Evolution of decision-making and learning under fluctuating social environments

The capacity to learn to associate sensory perceptions with appropriate motor actions underlies the success of many animal species, from insects to humans. The evolutionary significance of learning has long been a subject of interest for evolutionary biologists who emphasize the bene¬fit yielded by learning under changing environmental conditions, where it is required to flexibly switch from one behavior to another. However, two unsolved questions are particularly impor¬tant for improving our knowledge of the evolutionary advantages provided by learning, and are addressed in the present work. First, because it is possible to learn the wrong behavior when a task is too complex, the learning rules and their underlying psychological characteristics that generate truly adaptive behavior must be identified with greater precision, and must be linked to the specific ecological problems faced by each species. A framework for predicting behavior from the definition of a learning rule is developed here. Learning rules capture cognitive features such as the tendency to explore, or the ability to infer rewards associated to unchosen actions. It is shown that these features interact in a non-intuitive way to generate adaptive behavior in social interactions where individuals affect each other's fitness. Such behavioral predictions are used in an evolutionary model to demonstrate that, surprisingly, simple trial-and-error learn¬ing is not always outcompeted by more computationally demanding inference-based learning, when population members interact in pairwise social interactions. A second question in the evolution of learning is its link with and relative advantage compared to other simpler forms of phenotypic plasticity. After providing a conceptual clarification on the distinction between genetically determined vs. learned responses to environmental stimuli, a new factor in the evo¬lution of learning is proposed: environmental complexity. A simple mathematical model shows that a measure of environmental complexity, the number of possible stimuli in one's environ¬ment, is critical for the evolution of learning. In conclusion, this work opens roads for modeling interactions between evolving species and their environment in order to predict how natural se¬lection shapes animals' cognitive abilities. - La capacité d'apprendre à associer des sensations perceptives à des actions motrices appropriées est sous-jacente au succès évolutif de nombreuses espèces, depuis les insectes jusqu'aux êtres hu¬mains. L'importance évolutive de l'apprentissage est depuis longtemps un sujet d'intérêt pour les biologistes de l'évolution, et ces derniers mettent l'accent sur le bénéfice de l'apprentissage lorsque les conditions environnementales sont changeantes, car dans ce cas il est nécessaire de passer de manière flexible d'un comportement à l'autre. Cependant, deux questions non résolues sont importantes afin d'améliorer notre savoir quant aux avantages évolutifs procurés par l'apprentissage. Premièrement, puisqu'il est possible d'apprendre un comportement incorrect quand une tâche est trop complexe, les règles d'apprentissage qui permettent d'atteindre un com¬portement réellement adaptatif doivent être identifiées avec une plus grande précision, et doivent être mises en relation avec les problèmes écologiques spécifiques rencontrés par chaque espèce. Un cadre théorique ayant pour but de prédire le comportement à partir de la définition d'une règle d'apprentissage est développé ici. Il est démontré que les caractéristiques cognitives, telles que la tendance à explorer ou la capacité d'inférer les récompenses liées à des actions non ex¬périmentées, interagissent de manière non-intuitive dans les interactions sociales pour produire des comportements adaptatifs. Ces prédictions comportementales sont utilisées dans un modèle évolutif afin de démontrer que, de manière surprenante, l'apprentissage simple par essai-et-erreur n'est pas toujours battu par l'apprentissage basé sur l'inférence qui est pourtant plus exigeant en puissance de calcul, lorsque les membres d'une population interagissent socialement par pair. Une deuxième question quant à l'évolution de l'apprentissage concerne son lien et son avantage relatif vis-à-vis d'autres formes plus simples de plasticité phénotypique. Après avoir clarifié la distinction entre réponses aux stimuli génétiquement déterminées ou apprises, un nouveau fac¬teur favorisant l'évolution de l'apprentissage est proposé : la complexité environnementale. Un modèle mathématique permet de montrer qu'une mesure de la complexité environnementale - le nombre de stimuli rencontrés dans l'environnement - a un rôle fondamental pour l'évolution de l'apprentissage. En conclusion, ce travail ouvre de nombreuses perspectives quant à la mo¬délisation des interactions entre les espèces en évolution et leur environnement, dans le but de comprendre comment la sélection naturelle façonne les capacités cognitives des animaux.

Efficient learning of reactive robot behaviors with a Neural-Q_learning approach

The purpose of this paper is to propose a Neural-Q_learning approach designed for online learning of simple and reactive robot behaviors. In this approach, the Q_function is generalized by a multi-layer neural network allowing the use of continuous states and actions. The algorithm uses a database of the most recent learning samples to accelerate and guarantee the convergence. Each Neural-Q_learning function represents an independent, reactive and adaptive behavior which maps sensorial states to robot control actions. A group of these behaviors constitutes a reactive control scheme designed to fulfill simple missions. The paper centers on the description of the Neural-Q_learning based behaviors showing their performance with an underwater robot in a target following task. Real experiments demonstrate the convergence and stability of the learning system, pointing out its suitability for online robot learning. Advantages and limitations are discussed

Verbal labels selectively bias brain responses to high-energy foods.

The influence of external factors on food preferences and choices is poorly understood. Knowing which and how food-external cues impact the sensory processing and cognitive valuation of food would provide a strong benefit toward a more integrative understanding of food intake behavior and potential means of interfering with deviant eating patterns to avoid detrimental health consequences for individuals in the long run. We investigated whether written labels with positive and negative (as opposed to 'neutral') valence differentially modulate the spatio-temporal brain dynamics in response to the subsequent viewing of high- and low-energetic food images. Electrical neuroimaging analyses were applied to visual evoked potentials (VEPs) from 20 normal-weight participants. VEPs and source estimations in response to high- and low- energy foods were differentially affected by the valence of preceding word labels over the ~260-300 ms post-stimulus period. These effects were only observed when high-energy foods were preceded by labels with positive valence. Neural sources in occipital as well as posterior, frontal, insular and cingulate regions were down-regulated. These findings favor cognitive-affective influences especially on the visual responses to high-energetic food cues, potentially indicating decreases in cognitive control and goal-adaptive behavior. Inverse correlations between insular activity and effectiveness in food classification further indicate that this down-regulation directly impacts food-related behavior.

Brain dynamics of meal size selection in humans.

Although neuroimaging research has evidenced specific responses to visual food stimuli based on their nutritional quality (e.g., energy density, fat content), brain processes underlying portion size selection remain largely unexplored. We identified spatio-temporal brain dynamics in response to meal images varying in portion size during a task of ideal portion selection for prospective lunch intake and expected satiety. Brain responses to meal portions judged by the participants as 'too small', 'ideal' and 'too big' were measured by means of electro-encephalographic (EEG) recordings in 21 normal-weight women. During an early stage of meal viewing (105-145ms), data showed an incremental increase of the head-surface global electric field strength (quantified via global field power; GFP) as portion judgments ranged from 'too small' to 'too big'. Estimations of neural source activity revealed that brain regions underlying this effect were located in the insula, middle frontal gyrus and middle temporal gyrus, and are similar to those reported in previous studies investigating responses to changes in food nutritional content. In contrast, during a later stage (230-270ms), GFP was maximal for the 'ideal' relative to the 'non-ideal' portion sizes. Greater neural source activity to 'ideal' vs. 'non-ideal' portion sizes was observed in the inferior parietal lobule, superior temporal gyrus and mid-posterior cingulate gyrus. Collectively, our results provide evidence that several brain regions involved in attention and adaptive behavior track 'ideal' meal portion sizes as early as 230ms during visual encounter. That is, responses do not show an increase paralleling the amount of food viewed (and, in extension, the amount of reward), but are shaped by regulatory mechanisms.

An index for quantifying flocking behavior

One of the classic research topics in adaptive behavior is the collective displacement of groups of organisms such as flocks of birds, schools of fish, herds of mammals and crowds of people. However, most agent-based simulations of group behavior do not provide a quantitative index for determining the point at which the flock emerges. We have developed an index of the aggregation of moving individuals in a flock and have provided an example of how it can be used to quantify the degree to which a group of moving individuals actually forms a flock.

Attention in schizophrenia and in epileptic psychosis

The adaptive behavior of human beings is usually supported by rapid monitoring of outstanding events in the environment. Some investigators have suggested that a primary attention deficit might trigger symptoms of schizophrenia. In addition, researchers have long discussed the relationship between schizophrenia and the schizophrenia-like psychosis of epilepsy (SLPE). On the basis of these considerations, the objective of the present study was to investigate attention performance of patients with both disorders. Patient age was 18 to 60 years, and all patients had received formal schooling for at least four years. Patients were excluded if they had any systemic disease with neurologic or psychiatric comorbidity, or a history of brain surgery. The computer-assisted TAVIS-2R test was applied to all patients and to a control group to evaluate and discriminate between selective, alternating and sustained attention. The TAVIS-2R test is divided into three parts: one for selective attention (5 min), the second for alternating attention (5 min), and the third for the evaluation of vigilance or sustained attention (10 min). The same computer software was used for statistical analysis of reaction time, omission errors, and commission errors. The sample consisted of 36 patients with schizophrenia, 28 with interictal SLPE, and 47 healthy controls. The results of the selective attention tests for both patient groups were significantly lower than that for controls. The patients with schizophrenia and SLPE performed differently in the alternating and sustained attention tests: patients with SLPE had alternating attention deficits, whereas patients with schizophrenia showed deficits in sustained attention. These quantitative results confirmed the qualitative clinical observations for both patient groups, that is, that patients with schizophrenia had difficulties in focusing attention, whereas those with epilepsy showed perseveration in attention focus.

Codage de l’information visuelle par la plasticité et la synchronisation des réponses neuronales dans le cortex visuel primaire du chat

Les systèmes sensoriels encodent l’information sur notre environnement sous la forme d’impulsions électriques qui se propagent dans des réseaux de neurones. Élucider le code neuronal – les principes par lesquels l’information est représentée dans l’activité des neurones – est une question fondamentale des neurosciences. Cette thèse constituée de 3 études (E) s’intéresse à deux types de codes, la synchronisation et l’adaptation, dans les neurones du cortex visuel primaire (V1) du chat. Au niveau de V1, les neurones sont sélectifs pour des propriétés comme l’orientation des contours, la direction et la vitesse du mouvement. Chaque neurone ayant une combinaison de propriétés pour laquelle sa réponse est maximale, l’information se retrouve distribuée dans différents neurones situés dans diverses colonnes et aires corticales. Un mécanisme potentiel pour relier l’activité de neurones répondant à des items eux-mêmes reliés (e.g. deux contours appartenant au même objet) est la synchronisation de leur activité. Cependant, le type de relations potentiellement encodées par la synchronisation n’est pas entièrement clair (E1). Une autre stratégie de codage consiste en des changements transitoires des propriétés de réponse des neurones en fonction de l’environnement (adaptation). Cette plasticité est présente chez le chat adulte, les neurones de V1 changeant d’orientation préférée après exposition à une orientation non préférée. Cependant, on ignore si des neurones spatialement proches exhibent une plasticité comparable (E2). Finalement, nous avons étudié la dynamique de la relation entre synchronisation et plasticité des propriétés de réponse (E3). Résultats principaux — (E1) Nous avons montré que deux stimuli en mouvement soit convergent soit divergent élicitent plus de synchronisation entre les neurones de V1 que deux stimuli avec la même direction. La fréquence de décharge n’était en revanche pas différente en fonction du type de stimulus. Dans ce cas, la synchronisation semble coder pour la relation de cocircularité dont le mouvement convergent (centripète) et divergent (centrifuge) sont deux cas particuliers, et ainsi pourrait jouer un rôle dans l’intégration des contours. Cela indique que la synchronisation code pour une information qui n’est pas présente dans la fréquence de décharge des neurones. (E2) Après exposition à une orientation non préférée, les neurones changent d’orientation préférée dans la même direction que leurs voisins dans 75% des cas. Plusieurs propriétés de réponse des neurones de V1 dépendent de leur localisation dans la carte fonctionnelle corticale pour l’orientation. Les comportements plus diversifiés des 25% de neurones restants sont le fait de différences fonctionnelles que nous avons observé et qui suggèrent une localisation corticale particulière, les singularités, tandis que la majorité des neurones semblent situés dans les domaines d’iso-orientation. (E3) Après adaptation, les paires de neurones dont les propriétés de réponse deviennent plus similaires montrent une synchronisation accrue. Après récupération, la synchronisation retourne à son niveau initial. Par conséquent, la synchronisation semble refléter de façon dynamique la similarité des propriétés de réponse des neurones. Conclusions — Cette thèse contribue à notre connaissance des capacités d’adaptation de notre système visuel à un environnement changeant. Nous proposons également des données originales liées au rôle potentiel de la synchronisation. En particulier, la synchronisation semble capable de coder des relations entre objets similaires ou dissimilaires, suggérant l’existence d’assemblées neuronales superposées.

Reinforcement Learning by Policy Search

One objective of artificial intelligence is to model the behavior of an intelligent agent interacting with its environment. The environment's transformations can be modeled as a Markov chain, whose state is partially observable to the agent and affected by its actions; such processes are known as partially observable Markov decision processes (POMDPs). While the environment's dynamics are assumed to obey certain rules, the agent does not know them and must learn. In this dissertation we focus on the agent's adaptation as captured by the reinforcement learning framework. This means learning a policy---a mapping of observations into actions---based on feedback from the environment. The learning can be viewed as browsing a set of policies while evaluating them by trial through interaction with the environment. The set of policies is constrained by the architecture of the agent's controller. POMDPs require a controller to have a memory. We investigate controllers with memory, including controllers with external memory, finite state controllers and distributed controllers for multi-agent systems. For these various controllers we work out the details of the algorithms which learn by ascending the gradient of expected cumulative reinforcement. Building on statistical learning theory and experiment design theory, a policy evaluation algorithm is developed for the case of experience re-use. We address the question of sufficient experience for uniform convergence of policy evaluation and obtain sample complexity bounds for various estimators. Finally, we demonstrate the performance of the proposed algorithms on several domains, the most complex of which is simulated adaptive packet routing in a telecommunication network.

Efficient learning of reactive robot behaviors with a Neural-Q_learning approach

The purpose of this paper is to propose a Neural-Q_learning approach designed for online learning of simple and reactive robot behaviors. In this approach, the Q_function is generalized by a multi-layer neural network allowing the use of continuous states and actions. The algorithm uses a database of the most recent learning samples to accelerate and guarantee the convergence. Each Neural-Q_learning function represents an independent, reactive and adaptive behavior which maps sensorial states to robot control actions. A group of these behaviors constitutes a reactive control scheme designed to fulfill simple missions. The paper centers on the description of the Neural-Q_learning based behaviors showing their performance with an underwater robot in a target following task. Real experiments demonstrate the convergence and stability of the learning system, pointing out its suitability for online robot learning. Advantages and limitations are discussed

Equinoterapia en niños con autismo

Introducción: Autismo es un trastorno del desarrollo caracterizado por compromiso en interacción social, habilidades de lenguaje, presentando rituales con estereotipias. Sin tratamientos curativos, actualmente se buscan terapias alternativas. Un incremento de la literatura científica de terapias asistidas con animales se ha evidenciado, demostrando mejoría en pacientes autistas con la equinoterapia. Objetivo: Realizar una revisión sistemática de la literatura para evaluar efectividad de la equinoterapia en habilidades sociales y de lenguaje en niños autistas. Metodología: Revisión sistemática de la literatura de artículos obtenidos en bases de datos y Meta-buscadores que proporcionaron evidencia de equinoterapia en niños autistas. Tipo de artículos consultados: revisiones sistemáticas, meta análisis y ensayos clínicos. Trabajos publicados hasta 2013. En inglés y español. Se emplearon términos MeSH y EMTREE. Resultados: Cuatro artículos cumplieron criterios de inclusión y exclusión. Se analizaron los artículos individualmente, no se logró realizar un meta análisis por diferencias metodológicas entre los estudios. En total 85 sujetos fueron evaluados en dichos estudios. La equinoterapia en niños autistas evidenció mejoría en habilidades sociales y en las habilidades de lenguaje pre verbal. Discusión: La equinoterapia es prometedora en el manejo de niños autistas, los artículos evidencian consistentemente mejorías a nivel de habilidades sociales y de lenguaje. Debe ser considerado el tipo de paciente, el régimen de equinoterapia y la sostenibilidad de las mejoras. Conclusiones: Se necesitan nuevos estudio con un mayor rigor metodológico que permitan fortalecer la evidencia sobre la equinoterapia en niños con autismo y así poder realizar recomendaciones con un adecuado nivel de evidencia.

Neural processing of reward and punishment in young people at increased familial risk of depression

Background Abnormalities in the neural representation of rewarding and aversive stimuli have been well-described in patients with acute depression, and we previously found abnormal neural responses to rewarding and aversive sight and taste stimuli in recovered depressed patients. The aim of the present study was to determine whether similar abnormalities might be present in young people at increased familial risk of depression but with no personal history of mood disorder. Methods We therefore used functional magnetic resonance imaging to examine the neural responses to pleasant and aversive sights and tastes in 25 young people (16–21 years of age) with a biological parent with depression and 25 age- and gender-matched control subjects. Results We found that, relative to the control subjects, participants with a parental history of depression showed diminished responses in the orbitofrontal cortex to rewarding stimuli, whereas activations to aversive stimuli were increased in the lateral orbitofrontal cortex and insula. In anterior cingulate cortex the at-risk group showed blunted neural responses to both rewarding and aversive stimuli. Conclusions Our findings suggest that young people at increased familial risk of depression have altered neural representation of reward and punishment, particularly in cortical regions linked to the use of positive and negative feedback to guide adaptive behavior.

Investigação da relação entre conteúdo onírico e aprendizado de uma tarefa cognitiva complexa

Several lines of evidence indicate that sleep is beneficial for learning, but there is no experimental evidence yet that the content of dreams is adaptive, i.e., that dreams help the dreamer to cope with challenges of the following day. Our aim here is to investigate the role of dreams in the acquisition of a complex cognitive task. We investigated electroencephalographic recordings and dream reports of adult subjects exposed to a computer game comprising perceptual, motor, spatial, emotional and higher-level cognitive aspects (Doom). Subjects slept two nights in the sleep laboratory, a completely dark room with a comfortable bed and controlled temperature. Electroencephalographic recordings with 28 channels were continuously performed throughout the experiment to identify episodes of rapid-eye-movement (REM) sleep. Behaviors were continuously recorded in audio and video with an infrared camera. Dream reports were collected upon forced awakening from late REM sleep, and again in the morning after spontaneous awakening. On day 1, subjects were habituated to the sleep laboratory, no computer game was played, and negative controls for gamerelated dream reports were collected. On day 2, subjects played the computer game before and after sleep. Each game session lasted for an hour, and sleep for 7-9 hours. 9 different measures of performance indicated significant improve overnight. 81% of the subjects experienced intrusion of elements of the game into their dreams, including potentially adaptative strategies (insights). There was a linear correlation between performance and dream intrusion as well as for game improval and quantity of reported dreaming. In the electrophysiological analysis we mapped the subjects brain activities in different stages (SWS 1, REM 1, SWS 2, REM 2, Game 1 and Game 2), and found a modest reverberation in motor areas related to the joystick control during the sleep. When separated by gender, we found a significant difference on female subjects in the channels that indicate motor learning. Analysis of dream reports showed that the amount of gamerelated elements in dreams correlated with performance gains according to an inverted-U function analogous to the Yerkes-Dodson law that governs the relationship between arousal and learning. The results indicate that dreaming is an adaptive behavior

Uma abordagem baseada em aspectos e composição dinâmica para a construção de aplicações adaptativas cientes ao contexto

Ubiquitous computing systems operate in environments where the available resources significantly change during the system operation, thus requiring adaptive and context aware mechanisms to sense changes in the environment and adapt to new execution contexts. Motivated by this requirement, a framework for developing and executing adaptive context aware applications is proposed. The PACCA framework employs aspect-oriented techniques to modularize the adaptive behavior and to keep apart the application logic from this behavior. PACCA uses abstract aspect concept to provide flexibility by addition of new adaptive concerns that extend the abstract aspect. Furthermore, PACCA has a default aspect model that considers habitual adaptive concerns in ubiquitous applications. It exploits the synergy between aspect-orientation and dynamic composition to achieve context-aware adaptation, guided by predefined policies and aim to allow software modules on demand load making possible better use of mobile devices and yours limited resources. A Development Process for the ubiquitous applications conception is also proposed and presents a set of activities that guide adaptive context-aware developer. Finally, a quantitative study evaluates the approach based on aspects and dynamic composition for the construction of ubiquitous applications based in metrics

Optic Flow Contribution to Locomotion Adjustments in Obstacle Avoidance

Locomotion generates a visual movement pattern characterized as optic flow. To explore how the locomotor adjustments are affected by this pattern, an experimental paradigm was developed to eliminate optic flow during obstacle avoidance. The aim was to investigate the contribution of optic flow in obstacle avoidance by using a stroboscopic lamp. Ten young adults walked on an 8m pathway and stepped over obstacles at two heights. Visual sampling was determined by a stroboscopic lamp (static and dynamic visual sampling). Three-dimensional kinematics data showed that the visual information about self-motion provided by the optic flow was crucial for estimating the distance from and the height of the obstacle. Participants presented conservative behavior for obstacle avoidance under experimental visual sampling conditions, which suggests that optic flow favors the coupling of vision to adaptive behavior for obstacle avoidance.