Evolution of decision-making and learning under fluctuating social environments

The capacity to learn to associate sensory perceptions with appropriate motor actions underlies the success of many animal species, from insects to humans. The evolutionary significance of learning has long been a subject of interest for evolutionary biologists who emphasize the bene¬fit yielded by learning under changing environmental conditions, where it is required to flexibly switch from one behavior to another. However, two unsolved questions are particularly impor¬tant for improving our knowledge of the evolutionary advantages provided by learning, and are addressed in the present work. First, because it is possible to learn the wrong behavior when a task is too complex, the learning rules and their underlying psychological characteristics that generate truly adaptive behavior must be identified with greater precision, and must be linked to the specific ecological problems faced by each species. A framework for predicting behavior from the definition of a learning rule is developed here. Learning rules capture cognitive features such as the tendency to explore, or the ability to infer rewards associated to unchosen actions. It is shown that these features interact in a non-intuitive way to generate adaptive behavior in social interactions where individuals affect each other's fitness. Such behavioral predictions are used in an evolutionary model to demonstrate that, surprisingly, simple trial-and-error learn¬ing is not always outcompeted by more computationally demanding inference-based learning, when population members interact in pairwise social interactions. A second question in the evolution of learning is its link with and relative advantage compared to other simpler forms of phenotypic plasticity. After providing a conceptual clarification on the distinction between genetically determined vs. learned responses to environmental stimuli, a new factor in the evo¬lution of learning is proposed: environmental complexity. A simple mathematical model shows that a measure of environmental complexity, the number of possible stimuli in one's environ¬ment, is critical for the evolution of learning. In conclusion, this work opens roads for modeling interactions between evolving species and their environment in order to predict how natural se¬lection shapes animals' cognitive abilities. - La capacité d'apprendre à associer des sensations perceptives à des actions motrices appropriées est sous-jacente au succès évolutif de nombreuses espèces, depuis les insectes jusqu'aux êtres hu¬mains. L'importance évolutive de l'apprentissage est depuis longtemps un sujet d'intérêt pour les biologistes de l'évolution, et ces derniers mettent l'accent sur le bénéfice de l'apprentissage lorsque les conditions environnementales sont changeantes, car dans ce cas il est nécessaire de passer de manière flexible d'un comportement à l'autre. Cependant, deux questions non résolues sont importantes afin d'améliorer notre savoir quant aux avantages évolutifs procurés par l'apprentissage. Premièrement, puisqu'il est possible d'apprendre un comportement incorrect quand une tâche est trop complexe, les règles d'apprentissage qui permettent d'atteindre un com¬portement réellement adaptatif doivent être identifiées avec une plus grande précision, et doivent être mises en relation avec les problèmes écologiques spécifiques rencontrés par chaque espèce. Un cadre théorique ayant pour but de prédire le comportement à partir de la définition d'une règle d'apprentissage est développé ici. Il est démontré que les caractéristiques cognitives, telles que la tendance à explorer ou la capacité d'inférer les récompenses liées à des actions non ex¬périmentées, interagissent de manière non-intuitive dans les interactions sociales pour produire des comportements adaptatifs. Ces prédictions comportementales sont utilisées dans un modèle évolutif afin de démontrer que, de manière surprenante, l'apprentissage simple par essai-et-erreur n'est pas toujours battu par l'apprentissage basé sur l'inférence qui est pourtant plus exigeant en puissance de calcul, lorsque les membres d'une population interagissent socialement par pair. Une deuxième question quant à l'évolution de l'apprentissage concerne son lien et son avantage relatif vis-à-vis d'autres formes plus simples de plasticité phénotypique. Après avoir clarifié la distinction entre réponses aux stimuli génétiquement déterminées ou apprises, un nouveau fac¬teur favorisant l'évolution de l'apprentissage est proposé : la complexité environnementale. Un modèle mathématique permet de montrer qu'une mesure de la complexité environnementale - le nombre de stimuli rencontrés dans l'environnement - a un rôle fondamental pour l'évolution de l'apprentissage. En conclusion, ce travail ouvre de nombreuses perspectives quant à la mo¬délisation des interactions entre les espèces en évolution et leur environnement, dans le but de comprendre comment la sélection naturelle façonne les capacités cognitives des animaux.

A model for the evolution of reinforcement learning in fluctuating games

Many species are able to learn to associate behaviours with rewards as this gives fitness advantages in changing environments. Social interactions between population members may, however, require more cognitive abilities than simple trial-and-error learning, in particular the capacity to make accurate hypotheses about the material payoff consequences of alternative action combinations. It is unclear in this context whether natural selection necessarily favours individuals to use information about payoffs associated with nontried actions (hypothetical payoffs), as opposed to simple reinforcement of realized payoff. Here, we develop an evolutionary model in which individuals are genetically determined to use either trial-and-error learning or learning based on hypothetical reinforcements, and ask what is the evolutionarily stable learning rule under pairwise symmetric two-action stochastic repeated games played over the individual's lifetime. We analyse through stochastic approximation theory and simulations the learning dynamics on the behavioural timescale, and derive conditions where trial-and-error learning outcompetes hypothetical reinforcement learning on the evolutionary timescale. This occurs in particular under repeated cooperative interactions with the same partner. By contrast, we find that hypothetical reinforcement learners tend to be favoured under random interactions, but stable polymorphisms can also obtain where trial-and-error learners are maintained at a low frequency. We conclude that specific game structures can select for trial-and-error learning even in the absence of costs of cognition, which illustrates that cost-free increased cognition can be counterselected under social interactions.

A pseudo-spectral method for simulating of poro-elastic seismic wave propagation in complex borehole environments

We present a novel numerical approach for the comprehensive, flexible, and accurate simulation of poro-elastic wave propagation in cylindrical coordinates. An important application of this method is the modeling of complex seismic wave phenomena in fluid-filled boreholes, which represents a major, and as of yet largely unresolved, computational problem in exploration geophysics. In view of this, we consider a numerical mesh consisting of three concentric domains representing the borehole fluid in the center, the borehole casing and the surrounding porous formation. The spatial discretization is based on a Chebyshev expansion in the radial direction, Fourier expansions in the other directions, and a Runge-Kutta integration scheme for the time evolution. A domain decomposition method based on the method of characteristics is used to match the boundary conditions at the fluid/porous-solid and porous-solid/porous-solid interfaces. The viability and accuracy of the proposed method has been tested and verified in 2D polar coordinates through comparisons with analytical solutions as well as with the results obtained with a corresponding, previously published, and independently benchmarked solution for 2D Cartesian coordinates. The proposed numerical solution also satisfies the reciprocity theorem, which indicates that the inherent singularity associated with the origin of the polar coordinate system is handled adequately.

Exposure to superfluous information reduces cooperation and increases antisocial punishment in reputation-based interactions

Human cooperation is often based on reputation gained from previous interactions with third parties. Such reputation can be built on generous or punitive actions, and both, one's own reputation and the reputation of others have been shown to influence decision making in experimental games that control for confounding variables. Here we test how reputation-based cooperation and punishment react to disruption of the cognitive processing in different kinds of helping games with observers. Saying a few superfluous words before each interaction was used to possibly interfere with working memory. In a first set of experiments, where reputation could only be based on generosity, the disruption reduced the frequency of cooperation and lowered mean final payoffs. In a second set of experiments where reputation could only be based on punishment, the disruption increased the frequency of antisocial punishment (i.e. of punishing those who helped) and reduced the frequency of punishing defectors. Our findings suggest that working memory can easily be constraining in reputation-based interactions within experimental games, even if these games are based on a few simple rules with a visual display that provides all the information the subjects need to play the strategies predicted from current theory. Our findings also highlight a weakness of experimental games, namely that they can be very sensitive to environmental variation and that quantitative conclusions about antisocial punishment or other behavioral strategies can easily be misleading.

The handaxe and the microscope: individual and social learning in a multidimensional model of adaptation

When individuals learn by trial-and-error, they perform randomly chosen actions and then reinforce those actions that led to a high payoff. However, individuals do not always have to physically perform an action in order to evaluate its consequences. Rather, they may be able to mentally simulate actions and their consequences without actually performing them. Such fictitious learners can select actions with high payoffs without making long chains of trial-and-error learning. Here, we analyze the evolution of an n-dimensional cultural trait (or artifact) by learning, in a payoff landscape with a single optimum. We derive the stochastic learning dynamics of the distance to the optimum in trait space when choice between alternative artifacts follows the standard logit choice rule. We show that for both trial-and-error and fictitious learners, the learning dynamics stabilize at an approximate distance of root n/(2 lambda(e)) away from the optimum, where lambda(e) is an effective learning performance parameter depending on the learning rule under scrutiny. Individual learners are thus unlikely to reach the optimum when traits are complex (n large), and so face a barrier to further improvement of the artifact. We show, however, that this barrier can be significantly reduced in a large population of learners performing payoff-biased social learning, in which case lambda(e) becomes proportional to population size. Overall, our results illustrate the effects of errors in learning, levels of cognition, and population size for the evolution of complex cultural traits. (C) 2013 Elsevier Inc. All rights reserved.

Literature report : a review of policy learning in five strands of political science research

This paper reviews the policy learning literature in political science. In recent years, the number of publications on learning in the political realm increased dramatically. Researchers have focused on how policymakers and administrators adapt policies based on learning processes or experiences. Thereby, learning has been discussed in very different ways. Authors have referred to learning in the context of ideas, understood as deeply held beliefs, and, as change and adaptation of policy instruments. Two other strands of literature have taken into consideration learning, namely the diffusion literature and research on transfer, which put forward learning to understand who learns from whom and what. Opposed to these views, political learning emphasizes the adaptation of new strategies by policymakers over the transfer of knowledge or broad ideas. In this approach, learning occurs due to the failure of existing policies, increasing problem pressure, scientific innovations, or a combination of these elements.

Policy learning in Europe : social policy and labour market reforms in 11 countries

This report compares policy learning processes in 11 European countries. Based on the country reports that were produced by the national teams of the INSPIRES project, this paper develops an argument that connects problem pressure and politicization to learning in different labor market innovations. In short, we argue that learning efforts are most likely to impact on policy change if there is a certain problem pressure that clearly necessitates political action. On the other hand, if problem pressure is very low, or so high that governments need to react immediately, chances are low that learning impacts on policy change. The second part of our argument contends that learning impacts on policy change especially if a problem is not very politicized, i.e. there are no main conflicts concerning a reform, because then, solutions are wound up in the search for a compromise. Our results confirm our first hypothesis regarding the connection between problem pressure and policy learning. Governments learn indeed up to a certain degree of problem pressure. However, once political action becomes really urgent, i.e. in anti-crisis policies, there is no time and room for learning. On the other hand, learning occurred independently from the politicization of problem. In fact, in countries that have a consensual political system, learning occurred before the decision on a reform, whereas in majoritarian systems, learning happened after the adoption of a policy during the process of implementation.

Age-related change in melanin-based coloration of Barn owls (Tyto alba): females that become more female-like and males that become more male-like perform better

Ornament expression fluctuates with age in many organisms. Whether these changes are adaptively plastic is poorly known. In order to understand the ultimate function of melanin-based ornaments, we studied their within-individual fluctuations and their covariation with fitness-related traits. In barn owls (Tyto alba), individuals vary from reddish-brown pheomelanic to white and from immaculate to marked with black eumelanic spots, males being less reddish and less spotted than females. During the first molt, both sexes became less pheomelanic, females displayed larger spots and males fewer spots, but the extent of these changes was not associated with reproduction. At subsequent molts, intra-individual changes in melanin-based traits covaried with simultaneous reproduction changes. Adult females bred earlier in the season and laid larger eggs when they became scattered with larger spots, while adults of both sexes produced larger broods when they became whiter. These results suggest that the production of melanin pigments and fitness-related life history traits are concomitantly regulated in a sex-specific way.

Olfactory traces and spatial learning in rats.

We conducted an experiment to assess the use of olfactory traces for spatial orientation in an open environment in rats, Rattus norvegicus. We trained rats to locate a food source at a fixed location from different starting points, in the presence or absence of visual information. A single food source was hidden in an array of 19 petri dishes regularly arranged in an open-field arena. Rats were trained to locate the food source either in white light (with full access to distant visuospatial information) or in darkness (without any visual information). In both cases, the goal was in a fixed location relative to the spatial frame of reference. The results of this experiment revealed that the presence of noncontrolled olfactory traces coherent with the spatial frame of reference enables rats to locate a unique position as accurately in darkness as with full access to visuospatial information. We hypothesize that the olfactory traces complement the use of other orientation mechanisms, such as path integration or the reliance on visuospatial information. This experiment demonstrates that rats can rely on olfactory traces for accurate orientation, and raises questions about the establishment of such traces in the absence of any other orientation mechanism. Copyright 1998 The Association for the Study of Animal Behaviour.

Influence of local environmental olfactory cues on place learning in rats

The aim of the present study was to assess the influence of local environmental olfactory cues on place learning in rats. We developed a new experimental design allowing the comparison of the use of local olfactory and visual cues in spatial and discrimination learning. We compared the effect of both types of cues on the discrimination of a single food source in an open-field arena. The goal was either in a fixed or in a variable location, and could be indicated by local olfactory and/or visual cues. The local cues enhanced the discrimination of the goal dish, whether it was in a fixed or in a variable location. However, we did not observe any overshadowing of the spatial information by the local olfactory or visual cue. Rats relied primarily on distant visuospatial information to locate the goal, neglecting local information when it was in conflict with the spatial information.

Stable hydrogen and oxygen isotope composition of waters from mine tailings in different climatic environments

The stable isotope composition of waters (delta H-2, delta O-18) can be used as a natural tracer of hydrologic processes in systems affected by acid mine drainage. We investigated the delta H-2 and delta O-18 values of pore waters from four oxidizing sulfidic mine tailings impoundments in different climatic regions of Chile (Piuquenes at La Andina with Alpine climate, Cauquenes and Caren at El Teniente with Mediterranean climate, and Talabre at the Chuquicamata deposit with hyperarid climate). No clear relationship was found between altitude and isotopic composition. The observed displacement of the tailings pore waters from the local meteoric water line toward higher delta O-18 values (by similar to +2% delta O-18 relative to delta H-2) is partly due to water-rock interaction processes, including hydration and O-isotope exchange with sulfates and Fe(III) oxyhydroxides produced by pyrite oxidation. In most tailings, from the saturated zone toward the surface, isotopically different zones can be distinguished. Zone I is characterized by an upward depletion of H-2 and O-18 in the pore waters from the saturated zone and the lowermost vadose zone, due to ascending diffused isotopically light water triggered by the constant loss of water vapor by evaporation at the surface. In zone II, the capillary flow of a mix of vapor and liquid water causes an evaporative isotopic enrichment in H-2 and O-18. At the top of the tailings in dry climate a zone III between the capillary zone and the surface contains isotopically light diffused and atmospheric water vapor. In temperate climates, the upper part of the profile is affected by recent rainfall and zone III may not differ isotopically from zone II.

Population Normalization with Ammonium in Wastewater-Based Epidemiology: Application to Illicit Drug Monitoring

Fluctuations in ammonium (NH4+), measured as NH4-N loads using an ion-selective electrode installed at the inlet of a sewage treatment plant, showed a distinctive pattern which was associated to weekly (i.e., commuters) and seasonal (i.e., holidays) fluctuations of the population. Moreover, population size estimates based on NH4-N loads were lower compared to census data. Diurnal profiles of benzoylecgonine (BE) and 11-nor-9-carboxy-Δ9-tetrahydrocannabinol (THC-COOH) were shown to be strongly correlated to NH4-N. Characteristic patterns, which reflect the prolonged nocturnal activity of people during the weekend, could be observed for BE, cocaine, and a major metabolite of MDMA (i.e., 4-hydroxy-3-methoxymethamphetamine). Additional 24 h composite samples were collected between February and September 2013. Per-capita loads (i.e., grams per day per 1000 inhabitants) were computed using census data and NH4-N measurements. Normalization with NH4-N did not modify the overall pattern, suggesting that the magnitude of fluctuations in the size of the population is negligible compared to those of illicit drug loads. Results show that fluctuations in the size of the population over longer periods of time or during major events can be monitored using NH4-N loads: either using raw NH4-N loads or population size estimates based on NH4-N loads, if information about site-specific NH4-N population equivalents is available.