Reinforcement learning using a continuous time actor-critic framework with spiking neurons.

Animals repeat rewarded behaviors, but the physiological basis of reward-based learning has only been partially elucidated. On one hand, experimental evidence shows that the neuromodulator dopamine carries information about rewards and affects synaptic plasticity. On the other hand, the theory of reinforcement learning provides a framework for reward-based learning. Recent models of reward-modulated spike-timing-dependent plasticity have made first steps towards bridging the gap between the two approaches, but faced two problems. First, reinforcement learning is typically formulated in a discrete framework, ill-adapted to the description of natural situations. Second, biologically plausible models of reward-modulated spike-timing-dependent plasticity require precise calculation of the reward prediction error, yet it remains to be shown how this can be computed by neurons. Here we propose a solution to these problems by extending the continuous temporal difference (TD) learning of Doya (2000) to the case of spiking neurons in an actor-critic network operating in continuous time, and with continuous state and action representations. In our model, the critic learns to predict expected future rewards in real time. Its activity, together with actual rewards, conditions the delivery of a neuromodulatory TD signal to itself and to the actor, which is responsible for action choice. In simulations, we show that such an architecture can solve a Morris water-maze-like navigation task, in a number of trials consistent with reported animal performance. We also use our model to solve the acrobot and the cartpole problems, two complex motor control tasks. Our model provides a plausible way of computing reward prediction error in the brain. Moreover, the analytically derived learning rule is consistent with experimental evidence for dopamine-modulated spike-timing-dependent plasticity.

Slowness: an objective for spike-timing-dependent plasticity?

Our nervous system can efficiently recognize objects in spite of changes in contextual variables such as perspective or lighting conditions. Several lines of research have proposed that this ability for invariant recognition is learned by exploiting the fact that object identities typically vary more slowly in time than contextual variables or noise. Here, we study the question of how this "temporal stability" or "slowness" approach can be implemented within the limits of biologically realistic spike-based learning rules. We first show that slow feature analysis, an algorithm that is based on slowness, can be implemented in linear continuous model neurons by means of a modified Hebbian learning rule. This approach provides a link to the trace rule, which is another implementation of slowness learning. Then, we show analytically that for linear Poisson neurons, slowness learning can be implemented by spike-timing-dependent plasticity (STDP) with a specific learning window. By studying the learning dynamics of STDP, we show that for functional interpretations of STDP, it is not the learning window alone that is relevant but rather the convolution of the learning window with the postsynaptic potential. We then derive STDP learning windows that implement slow feature analysis and the "trace rule." The resulting learning windows are compatible with physiological data both in shape and timescale. Moreover, our analysis shows that the learning window can be split into two functionally different components that are sensitive to reversible and irreversible aspects of the input statistics, respectively. The theory indicates that irreversible input statistics are not in favor of stable weight distributions but may generate oscillatory weight dynamics. Our analysis offers a novel interpretation for the functional role of STDP in physiological neurons.

Timing feedback control of a rhythmic system

This paper addresses the question relative to the role of sensory feedback in rhythmic tasks. We study the properties of a sinusoidally vibrating wedge-billiard as a model for 2-D bounce juggling. If this wedge is actuated with an harmonic sinusoidal input, it has been shown that some periodic orbits are exponentially stable. This paper explores an intuitive method to enlarge the parametric stability region of the simplest of these orbits. Accurate processing of timing is proven to be an important key to achieve frequency-locking in rhythmic tasks. © 2005 IEEE.

The temporal evolution of feedback gains rapidly update to task demands.

Recent theoretical frameworks such as optimal feedback control suggest that feedback gains should modulate throughout a movement and be tuned to task demands. Here we measured the visuomotor feedback gain throughout the course of movements made to "near" or "far" targets in human subjects. The visuomotor gain showed a systematic modulation over the time course of the reach, with the gain peaking at the middle of the movement and dropping rapidly as the target is approached. This modulation depends primarily on the proportion of the movement remaining, rather than hand position, suggesting that the modulation is sensitive to task demands. Model-predictive control suggests that the gains should be continuously recomputed throughout a movement. To test this, we investigated whether feedback gains update when the task goal is altered during a movement, that is when the target of the reach jumped. We measured the visuomotor gain either simultaneously with the jump or 100 ms after the jump. The visuomotor gain nonspecifically reduced for all target jumps when measured synchronously with the jump. However, the visuomotor gain 100 ms later showed an appropriate modulation for the revised task goal by increasing for jumps that increased the distance to the target and reducing for jumps that decreased the distance. We conclude that visuomotor feedback gain shows a temporal evolution related to task demands and that this evolution can be flexibly recomputed within 100 ms to accommodate online modifications to task goals.

Inhibitory Synaptic Plasticity - Spike timing dependence and putative network function.

While the plasticity of excitatory synaptic connections in the brain has been widely studied, the plasticity of inhibitory connections is much less understood. Here, we present recent experimental and theoretical □ndings concerning the rules of spike timing-dependent inhibitory plasticity and their putative network function. This is a summary of a workshop at the COSYNE conference 2012.

Functional magnetic resonance imaging of impaired sensory prediction in schizophrenia.

IMPORTANCE: Forward models predict the sensory consequences of planned actions and permit discrimination of self- and non-self-elicited sensation; their impairment in schizophrenia is implied by an abnormality in behavioral force-matching and the flawed agency judgments characteristic of positive symptoms, including auditory hallucinations and delusions of control. OBJECTIVE: To assess attenuation of sensory processing by self-action in individuals with schizophrenia and its relation to current symptom severity. DESIGN, SETTING, AND PARTICIPANTS: Functional magnetic resonance imaging data were acquired while medicated individuals with schizophrenia (n = 19) and matched controls (n = 19) performed a factorially designed sensorimotor task in which the occurrence and relative timing of action and sensation were manipulated. The study took place at the neuroimaging research unit at the Institute of Cognitive Neuroscience, University College London, and the Maudsley Hospital. RESULTS: In controls, a region of secondary somatosensory cortex exhibited attenuated activation when sensation and action were synchronous compared with when the former occurred after an unexpected delay or alone. By contrast, reduced attenuation was observed in the schizophrenia group, suggesting that these individuals were unable to predict the sensory consequences of their own actions. Furthermore, failure to attenuate secondary somatosensory cortex processing was predicted by current hallucinatory severity. CONCLUSIONS AND RELEVANCE: Although comparably reduced attenuation has been reported in the verbal domain, this work implies that a more general physiologic deficit underlies positive symptoms of schizophrenia.

Cognitive Tomography Reveals Complex, Task-Independent Mental Representations

Humans develop rich mental representations that guide their behavior in a variety of everyday tasks. However, it is unknown whether these representations, often formalized as priors in Bayesian inference, are specific for each task or subserve multiple tasks. Current approaches cannot distinguish between these two possibilities because they cannot extract comparable representations across different tasks [1-10]. Here, we develop a novel method, termed cognitive tomography, that can extract complex, multidimensional priors across tasks. We apply this method to human judgments in two qualitatively different tasks, "familiarity" and "odd one out," involving an ecologically relevant set of stimuli, human faces. We show that priors over faces are structurally complex and vary dramatically across subjects, but are invariant across the tasks within each subject. The priors we extract from each task allow us to predict with high precision the behavior of subjects for novel stimuli both in the same task as well as in the other task. Our results provide the first evidence for a single high-dimensional structured representation of a naturalistic stimulus set that guides behavior in multiple tasks. Moreover, the representations estimated by cognitive tomography can provide independent, behavior-based regressors for elucidating the neural correlates of complex naturalistic priors. © 2013 The Authors.

Daily timing of emersion and elevated atmospheric CO2 concentration affect photosynthetic performance of the intertidal macroalga Ulva lactuca (Chlorophyta) in sunlight

The lunar day differs in length from the solar day so that times of low tide vary from day to day. Thus, aerial exposure of intertidal seaweeds may be during the day or during the night. We measured photosynthetic CO, assimilation rates of the intertidal green macroalga Ulva lactuca during exposures of varied daily timings during sunny days of summer to establish how photosynthetic performance responds to emersion timing under varied CO2 levels [at ambient (360 ppmv) and 2x ambient (720 ppmv) atmospheric CO2 concentrations]. There was an increase in net photosynthetic rates following some duration of exposure when the initial timing of exposure occurred during early morning (06.30 h) and late afternoon (17.15 h). In contrast, net rates exhibited a sharp decline with exposure duration when the initial timing of exposure occurred at 09.30 h, 15.30 h and especially at noon (12.30 h), implying the occurrence of a severe photoinhibition resulting from mid-day insolation. Doubled atmospheric CO2 concentration significantly enhanced the emersed photosynthetic rates, indicating that the emersed photosynthesis is CO2-limited at ambient CO2 levels. However, increasing CO2 barely stimulates the emersed photosynthetic rates during mid-day insolation.

Study of high performance re-timing using a 40 Gb/s hybrid-integrated mach-zehnder all-optical regenerator

We experimentally show that a hybrid-integrated Mach-Zehnder switch with a high performance gate profile allows retiming of optical signals with an accuracy of 500-700fs even if the input timing jitter is increased to 3ps. © 2004 Optical Society of America.

Exploiting passive dynamics for robot throwing task

Throwing is a complex and highly dynamic task. Humans usually exploit passive dynamics of their limbs to optimize their movement and muscle activation. In order to approach human throwing, we developed a double pendulum robotic platform. To introduce passivity into the actuated joints, clutches were included in the drive train. In this paper, we demonstrate the advantage of exploiting passive dynamics in reducing the mechanical work. However, engaging and disengaging the clutches are done in discrete fashions. Therefore, we propose an optimization approach which can deal with such discontinuities. It is shown that properly engaging/disengaging the clutches can reduce the mechanical work of a throwing task. The result is compared to the solution of fully actuated double pendulum, both in simulation and experiment. © 2012 IEEE.