Spatio-temporal credit assignment in population learning

Learning by reinforcement is important in shaping animal behavior, and in particular in behavioral decision making. Such decision making is likely to involve the integration of many synaptic events in space and time. However, using a single reinforcement signal to modulate synaptic plasticity, as suggested in classical reinforcement learning algorithms, a twofold problem arises. Different synapses will have contributed differently to the behavioral decision, and even for one and the same synapse, releases at different times may have had different effects. Here we present a plasticity rule which solves this spatio-temporal credit assignment problem in a population of spiking neurons. The learning rule is spike-time dependent and maximizes the expected reward by following its stochastic gradient. Synaptic plasticity is modulated not only by the reward, but also by a population feedback signal. While this additional signal solves the spatial component of the problem, the temporal one is solved by means of synaptic eligibility traces. In contrast to temporal difference (TD) based approaches to reinforcement learning, our rule is explicit with regard to the assumed biophysical mechanisms. Neurotransmitter concentrations determine plasticity and learning occurs fully online. Further, it works even if the task to be learned is non-Markovian, i.e. when reinforcement is not determined by the current state of the system but may also depend on past events. The performance of the model is assessed by studying three non-Markovian tasks. In the first task, the reward is delayed beyond the last action with non-related stimuli and actions appearing in between. The second task involves an action sequence which is itself extended in time and reward is only delivered at the last action, as it is the case in any type of board-game. The third task is the inspection game that has been studied in neuroeconomics, where an inspector tries to prevent a worker from shirking. Applying our algorithm to this game yields a learning behavior which is consistent with behavioral data from humans and monkeys, revealing themselves properties of a mixed Nash equilibrium. The examples show that our neuronal implementation of reward based learning copes with delayed and stochastic reward delivery, and also with the learning of mixed strategies in two-opponent games.

Spatio-temporal credit assignment in population learning

Learning by reinforcement is important in shaping animal behavior. But behavioral decision making is likely to involve the integration of many synaptic events in space and time. So in using a single reinforcement signal to modulate synaptic plasticity a twofold problem arises. Different synapses will have contributed differently to the behavioral decision and, even for one and the same synapse, releases at different times may have had different effects. Here we present a plasticity rule which solves this spatio-temporal credit assignment problem in a population of spiking neurons. The learning rule is spike time dependent and maximizes the expected reward by following its stochastic gradient. Synaptic plasticity is modulated not only by the reward but by a population feedback signal as well. While this additional signal solves the spatial component of the problem, the temporal one is solved by means of synaptic eligibility traces. In contrast to temporal difference based approaches to reinforcement learning, our rule is explicit with regard to the assumed biophysical mechanisms. Neurotransmitter concentrations determine plasticity and learning occurs fully online. Further, it works even if the task to be learned is non-Markovian, i.e. when reinforcement is not determined by the current state of the system but may also depend on past events. The performance of the model is assessed by studying three non-Markovian tasks. In the first task the reward is delayed beyond the last action with non-related stimuli and actions appearing in between. The second one involves an action sequence which is itself extended in time and reward is only delivered at the last action, as is the case in any type of board-game. The third is the inspection game that has been studied in neuroeconomics. It only has a mixed Nash equilibrium and exemplifies that the model also copes with stochastic reward delivery and the learning of mixed strategies.

Origins of the word "phenology"

Observing and documenting life cycle stages of plants and animals have been tradition and necessity for humans throughout history. Phenological observations—as called by their modern scientific name—were key to successful hunting and farming because the precise knowledge of animal behavior and plant growth, as well as their timing with changing seasons, was critical for survival. In today's context of environmental awareness and climate change research, phenological observations have become prime indicators of documenting altered life cycles due to environmental change in disciplines from biology to climatology, geography, and environmental history. Observations on the ground, from space, and from models of different complexity describe intra-annual and interannual changes of life cycles at individual, pixel, or grid box scale.

Dissection of floral pollination syndromes in petunia

Animal-mediated pollination is essential in the reproductive biology of many flowering plants and tends to be associated with pollination syndromes, sets of floral traits that are adapted to particular groups of pollinators. The complexity and functional convergence of various traits within pollination syndromes are outstanding examples of biological adaptation, raising questions about their mechanisms and origins. In the genus Petunia, complex pollination syndromes are found for nocturnal hawkmoths (P. axillaris) and diurnal bees (P. integrifolia), with characteristic differences in petal color, corolla shape, reproductive organ morphology, nectar quantity, nectar quality, and fragrance. We dissected the Petunia syndromes into their most important phenotypic and genetic components. They appear to include several distinct differences, such as cell-growth and cell-division patterns in the basal third of the petals, elongation of the ventral stamens, nectar secretion and nectar sugar metabolism, and enzymatic differentiation in the phenylpropanoid pathway. In backcross-inbred lines of species-derived chromosome segments in a transposon tagging strain of P. hybrida, one to five quantitative trait loci were identified for each syndrome component. Two loci for stamen elongation and nectar volume were confirmed in introgression lines and showed large allelic differences. The combined data provide a framework for a detailed understanding of floral syndromes from their developmental and molecular basis to their impact on animal behavior. With its molecular genetic tools, this Petunia system provides a novel venue for a pattern of adaptive radiation that is among the most characteristic of flowering plants.

Play in Rats: Association across Contexts and Types , and Analysis of Structure.

Play has been proposed as a promising indicator of positive animal welfare. We aimed to study play in rats across contexts (conspecific/heterospecific) and types (social: pinning, being pinned; solitary: scampering), and we investigated its structure using behavioral sequence analysis. Group-housed (three per cage) adolescent male Lister Hooded rats (n = 21) were subjected to a Play-In-Pairs test: after a 3 hour isolation period, a pair of cage-mates was returned to the home cage and both social and solitary play were scored for 20 min. This procedure was repeated for each pair combination across three consecutive days, and individual play scores were calculated. Heterospecific play was measured using a Tickling test: rats were individually tickled by the experimenter through bouts of gentle, rapid finger movements on their underside, and the number of positive 50 kHz frequency modulated vocalizations and experimenter-directed approach behaviors were recorded. Both of the above tests were compared with social play in the home cage. While conspecific play in both the Play-In-Pairs test and home cage were correlated, both seemed to be unrelated to heterospecific play in the Tickling test. During the Play-In-Pairs test, although both solitary and social play types occurred, they were unrelated, and solitary locomotor play of one rat did not predict the subsequent play behavior of its cage mate. Analysis of play structure revealed that social play occurred more often in bouts of repeated behaviors while solitary play sequences did not follow a specific pattern. If play is to be used as an indicator of positive welfare in rats, context, type and structure differences should be taken into account.

Effect of Population Heterogenization on the Reproducibility of Mouse Behavior: A Multi-Laboratory Study

In animal experiments, animals, husbandry and test procedures are traditionally standardized to maximize test sensitivity and minimize animal use, assuming that this will also guarantee reproducibility. However, by reducing within-experiment variation, standardization may limit inference to the specific experimental conditions. Indeed, we have recently shown in mice that standardization may generate spurious results in behavioral tests, accounting for poor reproducibility, and that this can be avoided by population heterogenization through systematic variation of experimental conditions. Here, we examined whether a simple form of heterogenization effectively improves reproducibility of test results in a multi-laboratory situation. Each of six laboratories independently ordered 64 female mice of two inbred strains (C57BL/6NCrl, DBA/2NCrl) and examined them for strain differences in five commonly used behavioral tests under two different experimental designs. In the standardized design, experimental conditions were standardized as much as possible in each laboratory, while they were systematically varied with respect to the animals' test age and cage enrichment in the heterogenized design. Although heterogenization tended to improve reproducibility by increasing within-experiment variation relative to between-experiment variation, the effect was too weak to account for the large variation between laboratories. However, our findings confirm the potential of systematic heterogenization for improving reproducibility of animal experiments and highlight the need for effective and practicable heterogenization strategies.

A dedicated animal model for mechanical thrombectomy in acute stroke

BACKGROUND: Recent studies have focused on mechanical thrombectomy as a means to reduce the time required for revascularization and increase the revascularization rate in acute stroke. To date no systematic evaluation has been made of the different mechanical devices in this novel and fast-developing field of endovascular interventions. To facilitate such evaluations, we developed a specific in vivo model for mechanical thrombectomy that allows visualization of dislocation or fragmentation of the thrombus during angiographic manipulation. METHODS: Angiography and embolization with a preformed thrombus was performed in 8 swine. The thrombus was generated by mixing 25 IU bovine thrombin and 10 mL autologous blood. For visualization during angiography, 1 g barium sulfate was added. RESULTS: The preformed thrombus exhibited mechanical stability, reproducibility, and high radiographic absorption, providing excellent visibility during angiography. The setting allowed selective embolization of targeted vessels without thrombus fragmentation. Despite the application of barium sulfate no local or systemic reaction occurred. Histologic evaluation revealed no intimal damage caused by the thrombus or contrast agent washout. CONCLUSION: The model presented here allows selective and reliable thromboembolization of vessels that reproduce the anatomic and hemodynamic situation in acute cerebrovascular stroke. It permits visualization of the thrombus during angiography and intervention, providing unique insight into the behavior of both thrombus and device, which is potentially useful in the development and evaluation of mechanical clot retrieval in acute cerebrovascular stroke.

Processing of temporal unpredictability in human and animal amygdala

The amygdala has been studied extensively for its critical role in associative fear conditioning in animals and humans. Noxious stimuli, such as those used for fear conditioning, are most effective in eliciting behavioral responses and amygdala activation when experienced in an unpredictable manner. Here, we show, using a translational approach in mice and humans, that unpredictability per se without interaction with motivational information is sufficient to induce sustained neural activity in the amygdala and to elicit anxiety-like behavior. Exposing mice to mere temporal unpredictability within a time series of neutral sound pulses in an otherwise neutral sensory environment increased expression of the immediate-early gene c-fos and prevented rapid habituation of single neuron activity in the basolateral amygdala. At the behavioral level, unpredictable, but not predictable, auditory stimulation induced avoidance and anxiety-like behavior. In humans, functional magnetic resonance imaging revealed that temporal unpredictably causes sustained neural activity in amygdala and anxiety-like behavior as quantified by enhanced attention toward emotional faces. Our findings show that unpredictability per se is an important feature of the sensory environment influencing habituation of neuronal activity in amygdala and emotional behavior and indicate that regulation of amygdala habituation represents an evolutionary-conserved mechanism for adapting behavior in anticipation of temporally unpredictable events.

Baclofen and gamma-hydroxybutyrate differentially altered behavior, EEG activity and sleep in rats.

OBJECTIVES Animal and human studies have shown that sleep may have an impact on functional recovery after brain damage. Baclofen (Bac) and gamma-hydroxybutyrate (GHB) have been shown to induce physiological sleep in humans, however, their effects in rodents are unclear. The aim of this study is to characterize sleep and electroencelphalogram (EEG) after Bac and GHB administration in rats. We hypothesized that both drugs would induce physiological sleep. METHODS Adult male Sprague-Dawley rats were implanted with EEG/electromyogram (EMG) electrodes for sleep recordings. Bac (10 or 20 mg/kg), GHB (150 or 300 mg/kg) or saline were injected 1 h after light and dark onset to evaluate time of day effect of the drugs. Vigilance states and EEG spectra were quantified. RESULTS Bac and GHB induced a non-physiological state characterized by atypical behavior and an abnormal EEG pattern. After termination of this state, Bac was found to increase the duration of non-rapid eye movement (NREM) and rapid eye movement (REM) sleep (∼90 and 10 min, respectively), reduce sleep fragmentation and affect NREM sleep episode frequency and duration (p<0.05). GHB had no major effect on vigilance states. Bac drastically increased EEG power density in NREM sleep in the frequencies 1.5-6.5 and 9.5-21.5 Hz compared to saline (p<0.05), while GHB enhanced power in the 1-5-Hz frequency band and reduced it in the 7-9-Hz band. Slow-wave activity in NREM sleep was enhanced 1.5-3-fold during the first 1-2 h following termination of the non-physiological state. The magnitude of drug effects was stronger during the dark phase. CONCLUSION While both Bac and GHB induced a non-physiological resting state, only Bac facilitated and consolidated sleep, and promoted EEG delta oscillations thereafter. Hence, Bac can be considered a sleep-promoting drug and its effects on functional recovery after stroke can be evaluated both in humans and rats.

The effects of long (C20/22) and short (C18) chain omega-3 fatty acids on keel bone fractures, bone biomechanics, behavior, and egg production in free-range laying hens.

Keel fractures in the laying hen are the most critical animal welfare issue facing the egg production industry, particularly with the increased use of extensive systems in response to the 2012 EU directive banning conventional battery cages. The current study is aimed at assessing the effects of 2 omega-3 (n3) enhanced diets on bone health, production endpoints, and behavior in free-range laying hens. Data was collected from 2 experiments over 2 laying cycles, each of which compared a (n3) supplemented diet with a control diet. Experiment 1 employed a diet supplemented with a 60:40 fish oil-linseed mixture (n3:n6 to 1.35) compared with a control diet (n3:n6 to 0.11), whereas the n3 diet in Experiment 2 was supplemented with a 40:60 fish oil-linseed (n3:n6 to 0.77) compared to the control diet (n3:n6 to 0.11). The n3 enhanced diet of Experiment 1 had a higher n3:n6 ratio, and a greater proportion of n3 in the long chain (C20/22) form (0.41 LC:SC) than that of Experiment 2 (0.12 LC:SC). Although dietary treatment was successful in reducing the frequency of fractures by approximately 27% in Experiment 2, data from Experiment 1 indicated the diet actually induced a greater likelihood of fracture (odds ratio: 1.2) and had substantial production detriment. Reduced keel breakage during Experiment 2 could be related to changes in bone health as n3-supplemented birds demonstrated greater load at failure of the keel, and tibiae and humeri that were more flexible. These results support previous findings that n3-supplemented diets can reduce fracture likely by increasing bone strength, and that this can be achieved without detriment to production. However, our findings suggest diets with excessive quantities of n3, or very high levels of C20/22, may experience health and production detriments. Further research is needed to optimize the quantity and type of n3 in terms of bone health and production variables and investigate the potential associated mechanisms.