A model for the evolution of reinforcement learning in fluctuating games

Many species are able to learn to associate behaviours with rewards as this gives fitness advantages in changing environments. Social interactions between population members may, however, require more cognitive abilities than simple trial-and-error learning, in particular the capacity to make accurate hypotheses about the material payoff consequences of alternative action combinations. It is unclear in this context whether natural selection necessarily favours individuals to use information about payoffs associated with nontried actions (hypothetical payoffs), as opposed to simple reinforcement of realized payoff. Here, we develop an evolutionary model in which individuals are genetically determined to use either trial-and-error learning or learning based on hypothetical reinforcements, and ask what is the evolutionarily stable learning rule under pairwise symmetric two-action stochastic repeated games played over the individual's lifetime. We analyse through stochastic approximation theory and simulations the learning dynamics on the behavioural timescale, and derive conditions where trial-and-error learning outcompetes hypothetical reinforcement learning on the evolutionary timescale. This occurs in particular under repeated cooperative interactions with the same partner. By contrast, we find that hypothetical reinforcement learners tend to be favoured under random interactions, but stable polymorphisms can also obtain where trial-and-error learners are maintained at a low frequency. We conclude that specific game structures can select for trial-and-error learning even in the absence of costs of cognition, which illustrates that cost-free increased cognition can be counterselected under social interactions.

Reversal of activity-mediated spine dynamics and learning impairment in a mouse model of Fragile X syndrome.

Fragile X syndrome (FXS) is characterized by intellectual disability and autistic traits, and results from the silencing of the FMR1 gene coding for a protein implicated in the regulation of protein synthesis at synapses. The lack of functional Fragile X mental retardation protein has been proposed to result in an excessive signaling of synaptic metabotropic glutamate receptors, leading to alterations of synapse maturation and plasticity. It remains, however, unclear how mechanisms of activity-dependent spine dynamics are affected in Fmr knockout (Fmr1-KO) mice and whether they can be reversed. Here we used a repetitive imaging approach in hippocampal slice cultures to investigate properties of structural plasticity and their modulation by signaling pathways. We found that basal spine turnover was significantly reduced in Fmr1-KO mice, but markedly enhanced by activity. Additionally, activity-mediated spine stabilization was lost in Fmr1-KO mice. Application of the metabotropic glutamate receptor antagonist α-Methyl-4-carboxyphenylglycine (MCPG) enhanced basal turnover, improved spine stability, but failed to reinstate activity-mediated spine stabilization. In contrast, enhancing phosphoinositide-3 kinase (PI3K) signaling, a pathway implicated in various aspects of synaptic plasticity, reversed both basal turnover and activity-mediated spine stabilization. It also restored defective long-term potentiation mechanisms in slices and improved reversal learning in Fmr1-KO mice. These results suggest that modulation of PI3K signaling could contribute to improve the cognitive deficits associated with FXS.

Spinal cord T-cell infiltration in the spared nerve injury model of neuropathic pain: a time course study

Background : Numerous studies have shown that immune cells infiltrate the spinal cord after peripheral nerve injury and that they play a major contribution to sensory hypersensitivity in rodents. In particular, the role of monocyte-derived cells and T lymphocytes seems to be prominent in this process. This exciting new perspective in research on neuropathic pain opens many different areas of work, including the understanding of the function of these cells and how they impact on neural function. However, no systematic description of the time course or cell types that characterize this infiltration has been published yet, although this seems to be the rational first step of an overall understanding of the phenomenon. Objective : Describe the time course and cell characteristics of T lymphocyte infiltration in the spinal cord in the Spared Nerve Injury (SNI) model of neuropathic pain in rats. Methods : Collect of lumbar spinal cords of rats at days 2, 7, 21 and 40 after SNI or sham operation (n=4). Immunofluorescence detecting different proteins of T cell subgroups (CD2+CD4+, CD2+CD8+, Th1 markers, Th2 markers, Th17 markers). Quantification of the infiltration rate of the different subgroups. Expected results : First, we expect to see an infiltration of T cells in the spinal cord ipsilateral to nerve injury, higher in SNI rats than in sham animals. Second, we anticipate that different subtypes of T cells penetrate at different time points. Finally, the number of T lymphocytes are expected to decrease at the latest time point, showing a resolution of the process underlying their infiltrating the spinal cord in the first place. Impact : A systematic description of the infiltration of T cells in the spinal cord after peripheral nerve injury is needed to have a better understanding of the role of immune cells in neuropathic pain. The time course that we want to establish will provide the scientific community with new perspectives. First, it will confirm that T cells do indeed infiltrate the spinal cord after SNI in rats. Second, the type of T cells infiltrating at different time points will give clues about their function, in particular their inflammatory or anti-inflammatory profile. From there on, other studies could be lead, investigating the functional side of the specific subtypes put to light by us. Ultimately, this could lead to the discovery of new drugs targeting T cells or their infiltration, in the hope of improving neuropathic pain.

The handaxe and the microscope: individual and social learning in a multidimensional model of adaptation

When individuals learn by trial-and-error, they perform randomly chosen actions and then reinforce those actions that led to a high payoff. However, individuals do not always have to physically perform an action in order to evaluate its consequences. Rather, they may be able to mentally simulate actions and their consequences without actually performing them. Such fictitious learners can select actions with high payoffs without making long chains of trial-and-error learning. Here, we analyze the evolution of an n-dimensional cultural trait (or artifact) by learning, in a payoff landscape with a single optimum. We derive the stochastic learning dynamics of the distance to the optimum in trait space when choice between alternative artifacts follows the standard logit choice rule. We show that for both trial-and-error and fictitious learners, the learning dynamics stabilize at an approximate distance of root n/(2 lambda(e)) away from the optimum, where lambda(e) is an effective learning performance parameter depending on the learning rule under scrutiny. Individual learners are thus unlikely to reach the optimum when traits are complex (n large), and so face a barrier to further improvement of the artifact. We show, however, that this barrier can be significantly reduced in a large population of learners performing payoff-biased social learning, in which case lambda(e) becomes proportional to population size. Overall, our results illustrate the effects of errors in learning, levels of cognition, and population size for the evolution of complex cultural traits. (C) 2013 Elsevier Inc. All rights reserved.

Learning through Computer Model Improvisations

It has been convincingly argued that computer simulation modeling differs from traditional science. If we understand simulation modeling as a new way of doing science, the manner in which scientists learn about the world through models must also be considered differently. This article examines how researchers learn about environmental processes through computer simulation modeling. Suggesting a conceptual framework anchored in a performative philosophical approach, we examine two modeling projects undertaken by research teams in England, both aiming to inform flood risk management. One of the modeling teams operated in the research wing of a consultancy firm, the other were university scientists taking part in an interdisciplinary project experimenting with public engagement. We found that in the first context the use of standardized software was critical to the process of improvisation, the obstacles emerging in the process concerned data and were resolved through exploiting affordances for generating, organizing, and combining scientific information in new ways. In the second context, an environmental competency group, obstacles were related to the computer program and affordances emerged in the combination of experience-based knowledge with the scientists' skill enabling a reconfiguration of the mathematical structure of the model, allowing the group to learn about local flooding.

The effect of learning on the evolution of new courtship behavior: a simulation model

The fact that individuals learn can change the relationship between genotype and phenotype in the population, and thus affect the evolutionary response to selection. Here we ask how male ability to learn from female response affects the evolution of a novel male behavioral courtship trait under pre-existing female preference (sensory drive). We assume a courtship trait which has both a genetic and a learned component, and a two-level female response to males. With individual-based simulations we show that, under this scenario, learning generally increases the strength of selection on the genetic component of the courtship trait, at least when the population genetic mean is still low. As a consequence, learning not only accelerates the evolution of the courtship trait, but also enables it when the trait is costly, which in the absence of learning results in an adaptive valley. Furthermore, learning can enable the evolution of the novel trait in the face of gene flow mediated by immigration of males that show superior attractiveness to females based on another, non-heritable trait. However, rather than increasing monotonically with the speed of learning, the effect of learning on evolution is maximized at intermediate learning rates. This model shows that, at least under some scenarios, the ability to learn can drive the evolution of mating behaviors through a process equivalent to Waddington's genetic assimilation.

Spinal Cord T-Cell Infiltration in the Rat Spared Nerve Injury Model: A Time Course Study.

The immune system is involved in the development of neuropathic pain. In particular, the infiltration of T-lymphocytes into the spinal cord following peripheral nerve injury has been described as a contributor to sensory hypersensitivity. We used the spared nerve injury (SNI) model of neuropathic pain in Sprague Dawley adult male rats to assess proliferation, and/or protein/gene expression levels for microglia (Iba1), T-lymphocytes (CD2) and cytotoxic T-lymphocytes (CD8). In the dorsal horn ipsilateral to SNI, Iba1 and BrdU stainings revealed microglial reactivity and proliferation, respectively, with different durations. Iba1 expression peaked at D4 and D7 at the mRNA and protein level, respectively, and was long-lasting. Proliferation occurred almost exclusively in Iba1 positive cells and peaked at D2. Gene expression observation by RT-qPCR array suggested that T-lymphocytes attracting chemokines were upregulated after SNI in rat spinal cord but only a few CD2/CD8 positive cells were found. A pronounced infiltration of CD2/CD8 positive T-cells was seen in the spinal cord injury (SCI) model used as a positive control for lymphocyte infiltration. Under these experimental conditions, we show early and long-lasting microglia reactivity in the spinal cord after SNI, but no lymphocyte infiltration was found.

Functional sphere profiling reveals the complexity of neuroblastoma tumor-initiating cell model.

Neuroblastoma (NB) is a neural crest-derived childhood tumor characterized by a remarkable phenotypic diversity, ranging from spontaneous regression to fatal metastatic disease. Although the cancer stem cell (CSC) model provides a trail to characterize the cells responsible for tumor onset, the NB tumor-initiating cell (TIC) has not been identified. In this study, the relevance of the CSC model in NB was investigated by taking advantage of typical functional stem cell characteristics. A predictive association was established between self-renewal, as assessed by serial sphere formation, and clinical aggressiveness in primary tumors. Moreover, cell subsets gradually selected during serial sphere culture harbored increased in vivo tumorigenicity, only highlighted in an orthotopic microenvironment. A microarray time course analysis of serial spheres passages from metastatic cells allowed us to specifically "profile" the NB stem cell-like phenotype and to identify CD133, ABC transporter, and WNT and NOTCH genes as spheres markers. On the basis of combined sphere markers expression, at least two distinct tumorigenic cell subpopulations were identified, also shown to preexist in primary NB. However, sphere markers-mediated cell sorting of parental tumor failed to recapitulate the TIC phenotype in the orthotopic model, highlighting the complexity of the CSC model. Our data support the NB stem-like cells as a dynamic and heterogeneous cell population strongly dependent on microenvironmental signals and add novel candidate genes as potential therapeutic targets in the control of high-risk NB.

Family relationships and learning experiences in anxious vs. non-anxious families: Are there differences ?

L'objectif principal de ce travail était d'explorer les relations parent-enfant et les processus d'apprentissage familiaux associés aux troubles anxieux. A cet effet, des familles ayant un membre anxieux (la mère ou l'enfant) ont été comparées avec des familles n'ayant aucun membre anxieux. Dans une première étude, l'observation de l'interaction mère-enfant, pendant une situation standardisée de jeu, a révélé que les mères présentant un trouble panique étaient plus susceptibles de se montrer verbalement contrôlantes, critiques et moins sensibles aux besoins de l'enfant, que les mères qui ne présentaient pas de trouble panique. Une deuxième étude a examiné les perceptions des différents membres de la famille quant aux relations au sein de la famille et a indiqué que, par comparaison aux adolescents non-anxieux, les adolescents anxieux étaient plus enclins à éprouver un sentiment d'autonomie individuelle diminué par rapport à leurs parents. Finalement, une troisième étude s'est intéressée à déterminer l'impact d'expériences d'apprentissage moins directes dans l'étiologie de l'anxiété. Les résultats ont indiqué que les mères présentant un trouble panique étaient plus enclines à s'engager dans des comportements qui maintiennent la panique et à impliquer leurs enfants dans ces comportements, que les mères ne présentant pas de trouble panique. En se basant sur des recherches antérieures qui ont établi une relation entre le contrôle parental, la perception de contrôle chez l'enfant et les troubles anxieux, le présent travail non seulement confirme ce lien mais propose également un modèle pour résumer l'état actuel des connaissances concernant les processus familiaux et le développement des troubles anxieux. Deux routes ont été suggérées par lesquelles l'anxiété pourrait être transmise de manière intergénérationnelle. Chacune de ces routes attribue un rôle important à la perception de contrôle chez l'enfant. L'idée est que lorsque les enfants présentent une prédisposition à interpréter le comportement de leurs parents comme hors de leur contrôle, ils seraient plus enclins à développer de l'anxiété. A ce titre, la perception du contrôle représenterait un tampon entre le comportement de contrôle/surprotection des parents et le trouble anxieux chez l'enfant. - The principal objective of the present work was to explore parent-child relationships and family learning processes associated with anxiety disorders. To this purpose, families with and without an anxious family member (mother or child) were compared. In a first study, observation of mother-child interaction, during a standard play situation, revealed that mothers with panic disorder were more likely to display verbal control and criticism, and less likely to display sensitivity toward their children than mothers without panic disorder. A second study examined family members' perceptions of family relationships and indicated that compared to non-anxious adolescents, anxious adolescents were more prone to experience a diminished sense of individual autonomy in relation to their parents. Finally a third study was interested in determining the effect of less direct learning experiences in the aetiology of anxiety. Results indicated that mothers with panic disorder were more likely to engage in panic-maintaining behaviour and to involve their children in this behaviour than mothers without panic disorder. Based on previous research showing a relationship between parental control, children's perception of control, and anxiety disorders, the present work not only further adds evidence to support this link but also proposes a model summarizing the current knowledge concerning family processes and the development of anxiety disorders. Two pathways have been suggested through which anxiety may be intergenerationally transmitted. Both pathways assign an important role to children's perception of control. The idea is that whenever children have a predisposition towards interpreting their parents' behaviour as beyond of their control, they may be more prone to develop anxiety. As such, perceived control may represent a buffer between parental overcontrolling/overprotective behaviours and childhood anxiety disorder.

Using active learning for monitoring networks design: The example of wind power plants sites evaluation

Locating new wind farms is of crucial importance for energy policies of the next decade. To select the new location, an accurate picture of the wind fields is necessary. However, characterizing wind fields is a difficult task, since the phenomenon is highly nonlinear and related to complex topographical features. In this paper, we propose both a nonparametric model to estimate wind speed at different time instants and a procedure to discover underrepresented topographic conditions, where new measuring stations could be added. Compared to space filling techniques, this last approach privileges optimization of the output space, thus locating new potential measuring sites through the uncertainty of the model itself.

Accurate performance of a rat model of schizophrenia in the water maze depends on visual cue availability and stability: a distortion in cognitive mapping abilities?

Rats were treated postnatally (PND 5-16) with BSO (l-buthionine-(S,R)-sulfoximine) in an animal model of schizophrenia based on transient glutathione deficit. The BSO treated rats were impaired in patrolling a maze or a homing table when adult, yet demonstrated preserved escape learning, place discrimination and reversal in a water maze task [37]. In the present work, BSO rats' performance in the water maze was assessed in conditions controlling for the available visual cues. First, in a completely curtained environment with two salient controlled cues, BSO rats showed little accuracy compared to control rats. Secondly, pre-trained BSO rats were impaired in reaching the familiar spatial position when curtains partially occluded different portions of the room environment in successive sessions. The apparently preserved place learning in a classical water maze task thus appears to require the stability and the richness of visual landmarks from the surrounding environment. In other words, the accuracy of BSO rats in place and reversal learning is impaired in a minimal cue condition or when the visual panorama changes between trials. However, if the panorama remains rich and stable between trials, BSO rats are equally efficient in reaching a familiar position or in learning a new one. This suggests that the BSO accurate performance in the water maze does not satisfy all the criteria for a cognitive map based navigation on the integration of polymodal cues. It supports the general hypothesis of a binding deficit in BSO rats.

Learning-induced plasticity in human audition: Objects, time, and space.

The human auditory system is comprised of specialized but interacting anatomic and functional pathways encoding object, spatial, and temporal information. We review how learning-induced plasticity manifests along these pathways and to what extent there are common mechanisms subserving such plasticity. A first series of experiments establishes a temporal hierarchy along which sounds of objects are discriminated along basic to fine-grained categorical boundaries and learned representations. A widespread network of temporal and (pre)frontal brain regions contributes to object discrimination via recursive processing. Learning-induced plasticity typically manifested as repetition suppression within a common set of brain regions. A second series considered how the temporal sequence of sound sources is represented. We show that lateralized responsiveness during the initial encoding phase of pairs of auditory spatial stimuli is critical for their accurate ordered perception. Finally, we consider how spatial representations are formed and modified through training-induced learning. A population-based model of spatial processing is supported wherein temporal and parietal structures interact in the encoding of relative and absolute spatial information over the initial ∼300ms post-stimulus onset. Collectively, these data provide insights into the functional organization of human audition and open directions for new developments in targeted diagnostic and neurorehabilitation strategies.

A Survey of Active Learning Algorithms for Supervised Remote Sensing Image Classification

Defining an efficient training set is one of the most delicate phases for the success of remote sensing image classification routines. The complexity of the problem, the limited temporal and financial resources, as well as the high intraclass variance can make an algorithm fail if it is trained with a suboptimal dataset. Active learning aims at building efficient training sets by iteratively improving the model performance through sampling. A user-defined heuristic ranks the unlabeled pixels according to a function of the uncertainty of their class membership and then the user is asked to provide labels for the most uncertain pixels. This paper reviews and tests the main families of active learning algorithms: committee, large margin, and posterior probability-based. For each of them, the most recent advances in the remote sensing community are discussed and some heuristics are detailed and tested. Several challenging remote sensing scenarios are considered, including very high spatial resolution and hyperspectral image classification. Finally, guidelines for choosing the good architecture are provided for new and/or unexperienced user.