Evolution of decision-making and learning under fluctuating social environments

The capacity to learn to associate sensory perceptions with appropriate motor actions underlies the success of many animal species, from insects to humans. The evolutionary significance of learning has long been a subject of interest for evolutionary biologists who emphasize the bene¬fit yielded by learning under changing environmental conditions, where it is required to flexibly switch from one behavior to another. However, two unsolved questions are particularly impor¬tant for improving our knowledge of the evolutionary advantages provided by learning, and are addressed in the present work. First, because it is possible to learn the wrong behavior when a task is too complex, the learning rules and their underlying psychological characteristics that generate truly adaptive behavior must be identified with greater precision, and must be linked to the specific ecological problems faced by each species. A framework for predicting behavior from the definition of a learning rule is developed here. Learning rules capture cognitive features such as the tendency to explore, or the ability to infer rewards associated to unchosen actions. It is shown that these features interact in a non-intuitive way to generate adaptive behavior in social interactions where individuals affect each other's fitness. Such behavioral predictions are used in an evolutionary model to demonstrate that, surprisingly, simple trial-and-error learn¬ing is not always outcompeted by more computationally demanding inference-based learning, when population members interact in pairwise social interactions. A second question in the evolution of learning is its link with and relative advantage compared to other simpler forms of phenotypic plasticity. After providing a conceptual clarification on the distinction between genetically determined vs. learned responses to environmental stimuli, a new factor in the evo¬lution of learning is proposed: environmental complexity. A simple mathematical model shows that a measure of environmental complexity, the number of possible stimuli in one's environ¬ment, is critical for the evolution of learning. In conclusion, this work opens roads for modeling interactions between evolving species and their environment in order to predict how natural se¬lection shapes animals' cognitive abilities. - La capacité d'apprendre à associer des sensations perceptives à des actions motrices appropriées est sous-jacente au succès évolutif de nombreuses espèces, depuis les insectes jusqu'aux êtres hu¬mains. L'importance évolutive de l'apprentissage est depuis longtemps un sujet d'intérêt pour les biologistes de l'évolution, et ces derniers mettent l'accent sur le bénéfice de l'apprentissage lorsque les conditions environnementales sont changeantes, car dans ce cas il est nécessaire de passer de manière flexible d'un comportement à l'autre. Cependant, deux questions non résolues sont importantes afin d'améliorer notre savoir quant aux avantages évolutifs procurés par l'apprentissage. Premièrement, puisqu'il est possible d'apprendre un comportement incorrect quand une tâche est trop complexe, les règles d'apprentissage qui permettent d'atteindre un com¬portement réellement adaptatif doivent être identifiées avec une plus grande précision, et doivent être mises en relation avec les problèmes écologiques spécifiques rencontrés par chaque espèce. Un cadre théorique ayant pour but de prédire le comportement à partir de la définition d'une règle d'apprentissage est développé ici. Il est démontré que les caractéristiques cognitives, telles que la tendance à explorer ou la capacité d'inférer les récompenses liées à des actions non ex¬périmentées, interagissent de manière non-intuitive dans les interactions sociales pour produire des comportements adaptatifs. Ces prédictions comportementales sont utilisées dans un modèle évolutif afin de démontrer que, de manière surprenante, l'apprentissage simple par essai-et-erreur n'est pas toujours battu par l'apprentissage basé sur l'inférence qui est pourtant plus exigeant en puissance de calcul, lorsque les membres d'une population interagissent socialement par pair. Une deuxième question quant à l'évolution de l'apprentissage concerne son lien et son avantage relatif vis-à-vis d'autres formes plus simples de plasticité phénotypique. Après avoir clarifié la distinction entre réponses aux stimuli génétiquement déterminées ou apprises, un nouveau fac¬teur favorisant l'évolution de l'apprentissage est proposé : la complexité environnementale. Un modèle mathématique permet de montrer qu'une mesure de la complexité environnementale - le nombre de stimuli rencontrés dans l'environnement - a un rôle fondamental pour l'évolution de l'apprentissage. En conclusion, ce travail ouvre de nombreuses perspectives quant à la mo¬délisation des interactions entre les espèces en évolution et leur environnement, dans le but de comprendre comment la sélection naturelle façonne les capacités cognitives des animaux.

Spatial learning by rats across visually disconnected environments

Experiments were designed to examine some properties of spatial representations in rats. Adult subjects were trained to escape through a hole at a fixed position in a large circular arena (see Schenk 1989). The experiments were conducted in the dark, with a limited number of controlled visual light cues in order to assess the minimal cue requirement for place learning. Three identical light cues (shape, height and distance from the table) were used. Depending on the condition, they were either permanently on, or alternatively on or off, depending on the position of the rat in the field. Two questions were asked: a) how many identical visual cues were necessary for spatial discrimination in the dark, and b) could rats integrate the relative positions of separate cues, under conditions in which the rat was never allowed to perceive all three cues simultaneously. The results suggest that rats are able to achieve a place discrimination task even if the three cues necessary for efficient orientation can never be seen simultaneously. A dissociation between the discrimination of the spatial position of the goal and the capacity to reach it by a direct path suggests that a reduced number of cues might require prolonged locomotion to allow an accurate orientation in the environment.

Spatial learning by rats across visually disconnected environments

Two spatial tasks were designed to test specific properties of spatial representation in rats. In the first task, rats were trained to locate an escape hole at a fixed position in a visually homogeneous arena. This arena was connected with a periphery where a full view of the room environment existed. Therefore, rats were dependent on their memory trace of the previous position in the periphery to discriminate a position within the central region. Under these experimental conditions, the test animals showed a significant discrimination of the training position without a specific local view. In the second task, rats were trained in a radial maze consisting of tunnels that were transparent at their distal ends only. Because the central part of the maze was non-transparent, rats had to plan and execute appropriate trajectories without specific visual feedback from the environment. This situation was intended to encourage the reliance on prospective memory of the non-visited arms in selecting the following move. Our results show that acquisition performance was only slightly decreased compared to that shown in a completely transparent maze and considerably higher than in a translucent maze or in darkness. These two series of experiments indicate (1) that rats can learn about the relative position of different places with no common visual panorama, and (2) that they are able to plan and execute a sequence of visits to several places without direct visual feed-back about their relative position.

Interactions between learning and immunity in Drosophila melanogaster

Learning and immunity are two adaptive traits with roles in central aspects of an organism's life: learning allows adjusting behaviours in changing environments, while immunity protects the body integrity against parasites and pathogens. While we know a lot about how these two traits interact in vertebrates, the interactions between learning and immunity remain poorly explored in insects. During my PhD, I studied three possible ways in which these two traits interact in the model system Drosophila melanogaster, a model organism in the study of learning and in the study of immunity. Learning can affect the behavioural defences against parasites and pathogens through the acquisition of new aversions for contaminated food for instance. This type of learning relies on the ability to associate a food-related cue with the visceral sickness following ingestion of contaminated food. Despite its potential implication in infection prevention, the existence of pathogen avoidance learning has been rarely explored in invertebrates. In a first part of my PhD, I tested whether D. melanogaster, which feed on food enriched in microorganisms, innately avoid the orally-acquired 'novel' virulent pathogen Pseudomonas entomophila, and whether it can learn to avoid it. Although flies did not innately avoid this pathogen, they decreased their preference for contaminated food over time, suggesting the existence of a form of learning based likely on infection-induced sickness. I further found that flies may be able to learn to avoid an odorant which was previously associated with the pathogen, but this requires confirmation with additional data. If this is confirmed, this would be the first time, to my knowledge, that pathogen avoidance learning is reported in an insect. The detrimental effect of infection on cognition and more specifically on learning ability is well documented in vertebrates and in social insects. While the underlying mechanisms are described in detail in vertebrates, experimental investigations are lacking in invertebrates. In a second part of my PhD, I tested the effect of an oral infection with natural pathogens on associative learning of D. melanogaster. By contrast with previous studies in insects, I found that flies orally infected with the virulent P. entomophila learned better the association of an odorant with mechanical shock than uninfected flies. The effect seems to be specific to a gut infection, and so far I have not been able to draw conclusions on the respective contributions of the pathogen's virulence and of the flies' immune activity in this effect. Interestingly, infected flies may display an increased sensitivity to physical pain. If the learning improvement observed in infected flies was due partially to the activity of the immune system, my results would suggest the existence of physiological connections between the immune system and the nervous system. The basis of these connections would then need to be addressed. Learning and immunity are linked at the physiological level in social insects. Physiological links between traits often result from the expression of genetic links between these traits. However, in social insects, there is no evidence that learning and immunity may be involved in an evolutionary trade-off. I previously reported a positive effect of infection on learning in D. melanogaster. This might suggest that a positive genetic link could exist between learning and immunity. We tested this hypothesis with two approaches: the diallel cross design with inbred lines, and the isofemale lines design. The two approaches provided consistent results: we found no additive genetic correlation between learning and resistance to infection with the diallel cross, and no genetic correlation in flies which are not yet adapted to laboratory conditions in isofemale lines. Consistently with the literature, the two studies suggested that the positive effect of infection on learning I observed might not be reflected by a positive evolutionary link between learning and immunity. Nevertheless, the existence of complex genetic relationships between the two traits cannot be excluded. - L'apprentissage et l'immunité sont deux caractères à valeur adaptative impliqués dans des aspects centraux de la vie d'un organisme : l'apprentissage permet d'ajuster les comportements pour faire face aux changements de l'environnement, tandis que l'immunité protège l'intégrité corporelle contre les attaques des parasites et des pathogènes. Alors que les interactions entre l'apprentissage et l'immunité sont bien documentées chez les vertébrés, ces interactions ont été très peu étudiées chez les insectes. Pendant ma thèse, je me suis intéressée à trois aspects des interactions possibles entre l'apprentissage et l'immunité chez la mouche du vinaigre Drosophila melanogaster, qui est un organisme modèle dans l'étude à la fois de l'apprentissage et de l'immunité. L'apprentissage peut affecter les défenses comportementales contre les parasites et les pathogènes par l'acquisition de nouvelles aversions pour la nourriture contaminée par exemple. Ce type d'apprentissage repose sur la capacité à associer une caractéristique de la nourriture avec la maladie qui suit l'ingestion de cette nourriture. Malgré les implications potentielles pour la prévention des infections, l'évitement appris des pathogènes a été rarement étudié chez les invertébrés. Dans une première partie de ma thèse, j'ai testé si les mouches, qui se nourrissent sur des milieux enrichis en micro-organismes, évitent de façon innée un 'nouveau' pathogène virulent Pseudomonas entomophila, et si elles ont la capacité d'apprendre à l'éviter. Bien que les mouches ne montrent pas d'évitement inné pour ce pathogène, elles diminuent leur préférence pour de la nourriture contaminée dans le temps, suggérant l'existence d'une forme d'apprentissage basée vraisemblablement sur la maladie générée par l'infection. J'ai ensuite observé que les mouches semblent être capables d'apprendre à éviter une odeur qui était au préalable associée avec ce pathogène, mais cela reste à confirmer par la collecte de données supplémentaires. Si cette observation est confirmée, cela sera la première fois, à ma connaissance, que l'évitement appris des pathogènes est décrit chez un insecte. L'effet détrimental des infections sur la cognition et plus particulièrement sur les capacités d'apprentissage est bien documenté chez les vertébrés et les insectes sociaux. Alors que les mécanismes sous-jacents sont détaillés chez les vertébrés, des études expérimentales font défaut chez les insectes. Dans une seconde partie de ma thèse, j'ai mesuré les effets d'une infection orale par des pathogènes naturels sur les capacités d'apprentissage associatif de la drosophile. Contrairement aux études précédentes chez les insectes, j'ai trouvé que les mouches infectées par le pathogène virulent P. entomophila apprennent mieux à associer une odeur avec des chocs mécaniques que des mouches non infectées. Cet effet semble spécifique à l'infection orale, et jusqu'à présent je n'ai pas pu conclure sur les contributions respectives de la virulence du pathogène et de l'activité immunitaire des mouches dans cet effet. De façon intéressante, les mouches infectées pourraient montrer une plus grande réactivité à la douleur physique. Si l'amélioration de l'apprentissage observée chez les mouches infectées était due en partie à l'activité du système immunitaire, mes résultats suggéreraient l'existence de connections physiologiques entre le système immunitaire et le système nerveux. Les mécanismes de ces connections seraient à explorer. L'apprentissage et l'immunité sont liés sur un plan physiologique chez les insectes sociaux. Les liens physiologiques entre les caractères résultent souvent de l'expression de liens entre ces caractères au niveau génétique. Cependant, chez les insectes sociaux, il n'y a pas de preuve que l'apprentissage et l'immunité soient liés par un compromis évolutif. J'ai précédemment rapporté un effet positif de l'infection sur l'apprentissage chez la drosophile. Cela pourrait suggérer qu'une relation génétique positive existerait entre l'apprentissage et l'immunité. Nous avons testé cette hypothèse par deux approches : le croisement diallèle avec des lignées consanguines, et les lignées isofemelles. Les deux approches ont fournies des résultats similaires : nous n'avons pas détecté de corrélation génétique additive entre l'apprentissage et la résistance à l'infection avec le croisement diallèle, et pas de corrélation génétique chez des mouches non adaptées aux conditions de laboratoire avec les lignées isofemelles. En ligne avec la littérature, ces deux études suggèrent que l'effet positif de l'infection sur l'apprentissage que j'ai précédemment observé ne refléterait pas un lien évolutif positif entre l'apprentissage et l'immunité. Néanmoins, l'existence de relations génétiques complexes n'est pas exclue.

Influence of learning on range expansion and adaptation to novel habitats.

Learning has been postulated to 'drive' evolution, but its influence on adaptive evolution in heterogeneous environments has not been formally examined. We used a spatially explicit individual-based model to study the effect of learning on the expansion and adaptation of a species to a novel habitat. Fitness was mediated by a behavioural trait (resource preference), which in turn was determined by both the genotype and learning. Our findings indicate that learning substantially increases the range of parameters under which the species expands and adapts to the novel habitat, particularly if the two habitats are separated by a sharp ecotone (rather than a gradient). However, for a broad range of parameters, learning reduces the degree of genetically-based local adaptation following the expansion and facilitates maintenance of genetic variation within local populations. Thus, in heterogeneous environments learning may facilitate evolutionary range expansions and maintenance of the potential of local populations to respond to subsequent environmental changes.

On learning dynamics underlying the evolution of learning rules.

In order to understand the development of non-genetically encoded actions during an animal's lifespan, it is necessary to analyze the dynamics and evolution of learning rules producing behavior. Owing to the intrinsic stochastic and frequency-dependent nature of learning dynamics, these rules are often studied in evolutionary biology via agent-based computer simulations. In this paper, we show that stochastic approximation theory can help to qualitatively understand learning dynamics and formulate analytical models for the evolution of learning rules. We consider a population of individuals repeatedly interacting during their lifespan, and where the stage game faced by the individuals fluctuates according to an environmental stochastic process. Individuals adjust their behavioral actions according to learning rules belonging to the class of experience-weighted attraction learning mechanisms, which includes standard reinforcement and Bayesian learning as special cases. We use stochastic approximation theory in order to derive differential equations governing action play probabilities, which turn out to have qualitative features of mutator-selection equations. We then perform agent-based simulations to find the conditions where the deterministic approximation is closest to the original stochastic learning process for standard 2-action 2-player fluctuating games, where interaction between learning rules and preference reversal may occur. Finally, we analyze a simplified model for the evolution of learning in a producer-scrounger game, which shows that the exploration rate can interact in a non-intuitive way with other features of co-evolving learning rules. Overall, our analyses illustrate the usefulness of applying stochastic approximation theory in the study of animal learning.

Drosophila rely on learning while foraging under semi-natural conditions

Learning is predicted to affect manifold ecological and evolutionary processes, but the extent to which animals rely on learning in nature remains poorly known, especially for short-lived non-social invertebrates. This is in particular the case for Drosophila, a favourite laboratory system to study molecular mechanisms of learning. Here we tested whether Drosophila melanogaster use learned information to choose food while free-flying in a large greenhouse emulating the natural environment. In a series of experiments flies were first given an opportunity to learn which of two food odours was associated with good versus unpalatable taste; subsequently, their preference for the two odours was assessed with olfactory traps set up in the greenhouse. Flies that had experienced palatable apple-flavoured food and unpalatable orange-flavoured food were more likely to be attracted to the odour of apple than flies with the opposite experience. This was true both when the flies first learned in the laboratory and were then released and recaptured in the greenhouse, and when the learning occurred under free-flying conditions in the greenhouse. Furthermore, flies retained the memory of their experience while exploring the greenhouse overnight in the absence of focal odours, pointing to the involvement of consolidated memory. These results support the notion that even small, short lived insects which are not central-place foragers make use of learned cues in their natural environments.

A model for the evolution of reinforcement learning in fluctuating games

Many species are able to learn to associate behaviours with rewards as this gives fitness advantages in changing environments. Social interactions between population members may, however, require more cognitive abilities than simple trial-and-error learning, in particular the capacity to make accurate hypotheses about the material payoff consequences of alternative action combinations. It is unclear in this context whether natural selection necessarily favours individuals to use information about payoffs associated with nontried actions (hypothetical payoffs), as opposed to simple reinforcement of realized payoff. Here, we develop an evolutionary model in which individuals are genetically determined to use either trial-and-error learning or learning based on hypothetical reinforcements, and ask what is the evolutionarily stable learning rule under pairwise symmetric two-action stochastic repeated games played over the individual's lifetime. We analyse through stochastic approximation theory and simulations the learning dynamics on the behavioural timescale, and derive conditions where trial-and-error learning outcompetes hypothetical reinforcement learning on the evolutionary timescale. This occurs in particular under repeated cooperative interactions with the same partner. By contrast, we find that hypothetical reinforcement learners tend to be favoured under random interactions, but stable polymorphisms can also obtain where trial-and-error learners are maintained at a low frequency. We conclude that specific game structures can select for trial-and-error learning even in the absence of costs of cognition, which illustrates that cost-free increased cognition can be counterselected under social interactions.