Reinforcement Learning Based on the Bayesian Theorem for Electricity Markets Decision Support

This paper presents the applicability of a reinforcement learning algorithm based on the application of the Bayesian theorem of probability. The proposed reinforcement learning algorithm is an advantageous and indispensable tool for ALBidS (Adaptive Learning strategic Bidding System), a multi-agent system that has the purpose of providing decision support to electricity market negotiating players. ALBidS uses a set of different strategies for providing decision support to market players. These strategies are used accordingly to their probability of success for each different context. The approach proposed in this paper uses a Bayesian network for deciding the most probably successful action at each time, depending on past events. The performance of the proposed methodology is tested using electricity market simulations in MASCEM (Multi-Agent Simulator of Competitive Electricity Markets). MASCEM provides the means for simulating a real electricity market environment, based on real data from real electricity market operators.

Learning sustainability by developing a solar dryer for microalgae retrieval

The development of nations depends on energy consumption, which is generally based on fossil fuels. This dependency produces irreversible and dramatic effects on the environment, e.g. large greenhouse gas emissions, which in turn cause global warming and climate changes, responsible for the rise of the sea level, floods, and other extreme weather events. Transportation is one of the main uses of energy, and its excessive fossil fuel dependency is driving the search for alternative and sustainable sources of energy such as microalgae, from which biodiesel, among other useful compounds, can be obtained. The process includes harvesting and drying, two energy consuming steps, which are, therefore, expensive and unsustainable. The goal of this EPS@ISEP Spring 2013 project was to develop a solar microalgae dryer for the microalgae laboratory of ISEP. A multinational team of five students from distinct fields of study was responsible for designing and building the solar microalgae dryer prototype. The prototype includes a control system to ensure that the microalgae are not destroyed during the drying process. The solar microalgae dryer works as a distiller, extracting the excess water from the microalgae suspension. This paper details the design steps, the building technologies, the ethical and sustainable concerns and compares the prototype with existing solutions. The proposed sustainable microalgae drying process is competitive as far as energy usage is concerned. Finally, the project contributed to increase the deontological ethics, social compromise skills and sustainable development awareness of the students.

Context-aware personalization environment for mobile computing

Dissertação para obtenção do Grau de Mestre em Engenharia Informática

Adapter module for self-learning production systems

Dissertação para obtenção do Grau de Mestre em Engenharia Electrotécnica, Sistemas e Computadores

Learning new tricks: Corticostriatal dynamics during novel skill learning

Learning novel actions and skills is a prevalent ability across multiple species and a critical feature for survival and competence in a constantly changing world. Novel actions are generated and learned through a process of trial and error, where an animal explores the environment around itself, generates multiple patterns of behavior and selects the ones that increase the likelihood of positive outcomes. Proper adaptation and execution of the selected behavior requires the coordination of several biomechanical features by the animal. Cortico-basal ganglia circuits and loops are critically involved in the acquisition, learning and consolidation of motor skills.(...)

Peer interaction and learning opportunities in cohesive and less cohesive L2 classrooms

The present study investigates peer to peer oral interaction in two task based language teaching classrooms, one of which was a self-declared cohesive group, and the other a self- declared less cohesive group, both at B1 level. It studies how learners talk cohesion into being and considers how this talk leads to learning opportunities in these groups. The study was classroom-based and was carried out over the period of an academic year. Research was conducted in the classrooms and the tasks were part of regular class work. The research was framed within a sociocognitive perspective of second language learning and data came from a number of sources, namely questionnaires, interviews and audio recorded talk of dyads, triads and groups of four students completing a total of eight oral tasks. These audio recordings were transcribed and analysed qualitatively for interactions which encouraged a positive social dimension and behaviours which led to learning opportunities, using conversation analysis. In addition, recordings were analysed quantitatively for learning opportunities and quantity and quality of language produced. Results show that learners in both classes exhibited multiple behaviours in interaction which could promote a positive social dimension, although behaviours which could discourage positive affect amongst group members were also found. Analysis of interactions also revealed the many ways in which learners in both the cohesive and less cohesive class created learning opportunities. Further qualitative analysis of these interactions showed that a number of factors including how learners approach a task, the decisions they make at zones of interactional transition and the affective relationship between participants influence the amount of learning opportunities created, as well as the quality and quantity of language produced. The main conclusion of the study is that it is not the cohesive nature of the group as a whole but the nature of the relationship between the individual members of the small group completing the task which influences the effectiveness of oral interaction for learning.This study contributes to our understanding of the way in which learners individualise the learning space and highlights the situated nature of language learning. It shows how individuals interact with each other and the task, and how talk in interaction changes moment-by-moment as learners react to the ‘here and now’ of the classroom environment.

Introduction [to "Citizen voices: performing public participation in science and environment communication"]

"ECREA series, ISSN 1742-9420"

O desenvolvimento do pensamento abstrato através de game-based learning e do Scratch

Relatório de estágio de mestrado em Ensino de Informática

Olfactory traces and spatial learning in rats.

We conducted an experiment to assess the use of olfactory traces for spatial orientation in an open environment in rats, Rattus norvegicus. We trained rats to locate a food source at a fixed location from different starting points, in the presence or absence of visual information. A single food source was hidden in an array of 19 petri dishes regularly arranged in an open-field arena. Rats were trained to locate the food source either in white light (with full access to distant visuospatial information) or in darkness (without any visual information). In both cases, the goal was in a fixed location relative to the spatial frame of reference. The results of this experiment revealed that the presence of noncontrolled olfactory traces coherent with the spatial frame of reference enables rats to locate a unique position as accurately in darkness as with full access to visuospatial information. We hypothesize that the olfactory traces complement the use of other orientation mechanisms, such as path integration or the reliance on visuospatial information. This experiment demonstrates that rats can rely on olfactory traces for accurate orientation, and raises questions about the establishment of such traces in the absence of any other orientation mechanism. Copyright 1998 The Association for the Study of Animal Behaviour.

A behavioral study of alpha-1b adrenergic receptor knockout mice: increased reaction to novelty and selectively reduced learning capacities.

Knockout mice lacking the alpha-1b adrenergic receptor were tested in behavioral experiments. Reaction to novelty was first assessed in a simple test in which the time taken by the knockout mice and their littermate controls to enter a second compartment was compared. Then the mice were tested in an open field to which unknown objects were subsequently added. Special novelty was introduced by moving one of the familiar objects to another location in the open field. Spatial behavior and memory were further studied in a homing board test, and in the water maze. The alpha-1b knockout mice showed an enhanced reactivity to new situations. They were faster to enter the new environment, covered longer paths in the open field, and spent more time exploring the new objects. They reacted like controls to modification inducing spatial novelty. In the homing board test, both the knockout mice and the control mice seemed to use a combination of distant visual and proximal olfactory cues, showing place preference only if the two types of cues were redundant. In the water maze the alpha-1b knockout mice were unable to learn the task, which was confirmed in a probe trial without platform. They were perfectly able, however, to escape in a visible platform procedure. These results confirm previous findings showing that the noradrenergic pathway is important for the modulation of behaviors such as reaction to novelty and exploration, and suggest that this is mediated, at least partly, through the alpha-1b adrenergic receptors. The lack of alpha-1b adrenergic receptors in spatial orientation does not seem important in cue-rich tasks but may interfere with orientation in situations providing distant cues only.

The Role of model uncertainty and learning in the U.S. postwar policy response to oil prices

This paper studies optimal monetary policy in a framework that explicitly accounts for policymakers' uncertainty about the channels of transmission of oil prices into the economy. More specfically, I examine the robust response to the real price of oil that US monetary authorities would have been recommended to implement in the period 1970 2009; had they used the approach proposed by Cogley and Sargent (2005b) to incorporate model uncertainty and learning into policy decisions. In this context, I investigate the extent to which regulator' changing beliefs over different models of the economy play a role in the policy selection process. The main conclusion of this work is that, in the specific environment under analysis, one of the underlying models dominates the optimal interest rate response to oil prices. This result persists even when alternative assumptions on the model's priors change the pattern of the relative posterior probabilities, and can thus be attributed to the presence of model uncertainty itself.

Evolution of decision-making and learning under fluctuating social environments

The capacity to learn to associate sensory perceptions with appropriate motor actions underlies the success of many animal species, from insects to humans. The evolutionary significance of learning has long been a subject of interest for evolutionary biologists who emphasize the bene¬fit yielded by learning under changing environmental conditions, where it is required to flexibly switch from one behavior to another. However, two unsolved questions are particularly impor¬tant for improving our knowledge of the evolutionary advantages provided by learning, and are addressed in the present work. First, because it is possible to learn the wrong behavior when a task is too complex, the learning rules and their underlying psychological characteristics that generate truly adaptive behavior must be identified with greater precision, and must be linked to the specific ecological problems faced by each species. A framework for predicting behavior from the definition of a learning rule is developed here. Learning rules capture cognitive features such as the tendency to explore, or the ability to infer rewards associated to unchosen actions. It is shown that these features interact in a non-intuitive way to generate adaptive behavior in social interactions where individuals affect each other's fitness. Such behavioral predictions are used in an evolutionary model to demonstrate that, surprisingly, simple trial-and-error learn¬ing is not always outcompeted by more computationally demanding inference-based learning, when population members interact in pairwise social interactions. A second question in the evolution of learning is its link with and relative advantage compared to other simpler forms of phenotypic plasticity. After providing a conceptual clarification on the distinction between genetically determined vs. learned responses to environmental stimuli, a new factor in the evo¬lution of learning is proposed: environmental complexity. A simple mathematical model shows that a measure of environmental complexity, the number of possible stimuli in one's environ¬ment, is critical for the evolution of learning. In conclusion, this work opens roads for modeling interactions between evolving species and their environment in order to predict how natural se¬lection shapes animals' cognitive abilities. - La capacité d'apprendre à associer des sensations perceptives à des actions motrices appropriées est sous-jacente au succès évolutif de nombreuses espèces, depuis les insectes jusqu'aux êtres hu¬mains. L'importance évolutive de l'apprentissage est depuis longtemps un sujet d'intérêt pour les biologistes de l'évolution, et ces derniers mettent l'accent sur le bénéfice de l'apprentissage lorsque les conditions environnementales sont changeantes, car dans ce cas il est nécessaire de passer de manière flexible d'un comportement à l'autre. Cependant, deux questions non résolues sont importantes afin d'améliorer notre savoir quant aux avantages évolutifs procurés par l'apprentissage. Premièrement, puisqu'il est possible d'apprendre un comportement incorrect quand une tâche est trop complexe, les règles d'apprentissage qui permettent d'atteindre un com¬portement réellement adaptatif doivent être identifiées avec une plus grande précision, et doivent être mises en relation avec les problèmes écologiques spécifiques rencontrés par chaque espèce. Un cadre théorique ayant pour but de prédire le comportement à partir de la définition d'une règle d'apprentissage est développé ici. Il est démontré que les caractéristiques cognitives, telles que la tendance à explorer ou la capacité d'inférer les récompenses liées à des actions non ex¬périmentées, interagissent de manière non-intuitive dans les interactions sociales pour produire des comportements adaptatifs. Ces prédictions comportementales sont utilisées dans un modèle évolutif afin de démontrer que, de manière surprenante, l'apprentissage simple par essai-et-erreur n'est pas toujours battu par l'apprentissage basé sur l'inférence qui est pourtant plus exigeant en puissance de calcul, lorsque les membres d'une population interagissent socialement par pair. Une deuxième question quant à l'évolution de l'apprentissage concerne son lien et son avantage relatif vis-à-vis d'autres formes plus simples de plasticité phénotypique. Après avoir clarifié la distinction entre réponses aux stimuli génétiquement déterminées ou apprises, un nouveau fac¬teur favorisant l'évolution de l'apprentissage est proposé : la complexité environnementale. Un modèle mathématique permet de montrer qu'une mesure de la complexité environnementale - le nombre de stimuli rencontrés dans l'environnement - a un rôle fondamental pour l'évolution de l'apprentissage. En conclusion, ce travail ouvre de nombreuses perspectives quant à la mo¬délisation des interactions entre les espèces en évolution et leur environnement, dans le but de comprendre comment la sélection naturelle façonne les capacités cognitives des animaux.

Activity in inferior parietal and medial prefrontal cortex signals the accumulation of evidence in a probability learning task.

In an uncertain environment, probabilities are key to predicting future events and making adaptive choices. However, little is known about how humans learn such probabilities and where and how they are encoded in the brain, especially when they concern more than two outcomes. During functional magnetic resonance imaging (fMRI), young adults learned the probabilities of uncertain stimuli through repetitive sampling. Stimuli represented payoffs and participants had to predict their occurrence to maximize their earnings. Choices indicated loss and risk aversion but unbiased estimation of probabilities. BOLD response in medial prefrontal cortex and angular gyri increased linearly with the probability of the currently observed stimulus, untainted by its value. Connectivity analyses during rest and task revealed that these regions belonged to the default mode network. The activation of past outcomes in memory is evoked as a possible mechanism to explain the engagement of the default mode network in probability learning. A BOLD response relating to value was detected only at decision time, mainly in striatum. It is concluded that activity in inferior parietal and medial prefrontal cortex reflects the amount of evidence accumulated in favor of competing and uncertain outcomes.

Semi-online neural-Q_leaming for real-time robot learning

Reinforcement learning (RL) is a very suitable technique for robot learning, as it can learn in unknown environments and in real-time computation. The main difficulties in adapting classic RL algorithms to robotic systems are the generalization problem and the correct observation of the Markovian state. This paper attempts to solve the generalization problem by proposing the semi-online neural-Q_learning algorithm (SONQL). The algorithm uses the classic Q_learning technique with two modifications. First, a neural network (NN) approximates the Q_function allowing the use of continuous states and actions. Second, a database of the most representative learning samples accelerates and stabilizes the convergence. The term semi-online is referred to the fact that the algorithm uses the current but also past learning samples. However, the algorithm is able to learn in real-time while the robot is interacting with the environment. The paper shows simulated results with the "mountain-car" benchmark and, also, real results with an underwater robot in a target following behavior