Spatio-temporal credit assignment in population learning

Learning by reinforcement is important in shaping animal behavior, and in particular in behavioral decision making. Such decision making is likely to involve the integration of many synaptic events in space and time. However, using a single reinforcement signal to modulate synaptic plasticity, as suggested in classical reinforcement learning algorithms, a twofold problem arises. Different synapses will have contributed differently to the behavioral decision, and even for one and the same synapse, releases at different times may have had different effects. Here we present a plasticity rule which solves this spatio-temporal credit assignment problem in a population of spiking neurons. The learning rule is spike-time dependent and maximizes the expected reward by following its stochastic gradient. Synaptic plasticity is modulated not only by the reward, but also by a population feedback signal. While this additional signal solves the spatial component of the problem, the temporal one is solved by means of synaptic eligibility traces. In contrast to temporal difference (TD) based approaches to reinforcement learning, our rule is explicit with regard to the assumed biophysical mechanisms. Neurotransmitter concentrations determine plasticity and learning occurs fully online. Further, it works even if the task to be learned is non-Markovian, i.e. when reinforcement is not determined by the current state of the system but may also depend on past events. The performance of the model is assessed by studying three non-Markovian tasks. In the first task, the reward is delayed beyond the last action with non-related stimuli and actions appearing in between. The second task involves an action sequence which is itself extended in time and reward is only delivered at the last action, as it is the case in any type of board-game. The third task is the inspection game that has been studied in neuroeconomics, where an inspector tries to prevent a worker from shirking. Applying our algorithm to this game yields a learning behavior which is consistent with behavioral data from humans and monkeys, revealing themselves properties of a mixed Nash equilibrium. The examples show that our neuronal implementation of reward based learning copes with delayed and stochastic reward delivery, and also with the learning of mixed strategies in two-opponent games.

Photopic and scotopic spatiotemporal tuning of adult zebrafish vision.

Sensitivity to spatial and temporal patterns is a fundamental aspect of vision. Herein, we investigated this sensitivity in adult zebrafish for a wide range of spatial (0.014 to 0.511 cycles/degree [c/d]) and temporal frequencies (0.025 to 6 cycles/s) to better understand their visual system. Measurements were performed at photopic (1.8 log cd m(-2)) and scotopic (-4.5 log cd m(-2)) light levels to assess the optokinetic response (OKR). The resulting spatiotemporal contrast sensitivity (CS) functions revealed that the OKR of zebrafish is tuned to spatial frequency and speed but not to temporal frequencies. Thereby, optimal test parameters for CS measurements were identified. At photopic light levels, a spatial frequency of 0.116 ± 0.01 c/d (mean ± SD) and a grating speed of 8.42 ± 2.15 degrees/second (d/s) was ideal; at scotopic light levels, these values were 0.110 ± 0.02 c/d and 5.45 ± 1.31 d/s, respectively. This study allows to better characterize zebrafish mutants with altered vision and to distinguish between defects of rod and cone photoreceptors as measurements were performed under different light conditions.

Brain systems underlying encounter expectancy bias in spider phobia.

Spider-phobic individuals are characterized by exaggerated expectancies to be faced with spiders (so-called encounter expectancy bias). Whereas phobic responses have been linked to brain systems mediating fear, little is known about how the recruitment of these systems relates to exaggerated expectancies of threat. We used fMRI to examine spider-phobic and control participants while they imagined visiting different locations in a forest after having received background information about the likelihood of encountering different animals (spiders, snakes, and birds) at these locations. Critically, imagined encounter expectancies modulated brain responses differently in phobics as compared with controls. Phobics displayed stronger negative modulation of activity in the lateral prefrontal cortex, precuneus, and visual cortex by encounter expectancies for spiders, relative to snakes or birds (within-participants analysis); these effects were not seen in controls. Between-participants correlation analyses within the phobic group further corroborated the hypothesis that these phobia-specific modulations may underlie irrationality in encounter expectancies (deviations of encounter expectancies from objective background information) in spider phobia; the greater the negative modulation a phobic participant displayed in the lateral prefrontal cortex, precuneus, and visual cortex, the stronger was her bias in encounter expectancies for spiders. Interestingly, irrationality in expectancies reflected in frontal areas relied on right rather than left hemispheric deactivations. Our data accord with the idea that expectancy biases in spider phobia may reflect deficiencies in cognitive control and contextual integration that are mediated by right frontal and parietal areas.