Reinforcement learning using a continuous time actor-critic framework with spiking neurons.

Animals repeat rewarded behaviors, but the physiological basis of reward-based learning has only been partially elucidated. On one hand, experimental evidence shows that the neuromodulator dopamine carries information about rewards and affects synaptic plasticity. On the other hand, the theory of reinforcement learning provides a framework for reward-based learning. Recent models of reward-modulated spike-timing-dependent plasticity have made first steps towards bridging the gap between the two approaches, but faced two problems. First, reinforcement learning is typically formulated in a discrete framework, ill-adapted to the description of natural situations. Second, biologically plausible models of reward-modulated spike-timing-dependent plasticity require precise calculation of the reward prediction error, yet it remains to be shown how this can be computed by neurons. Here we propose a solution to these problems by extending the continuous temporal difference (TD) learning of Doya (2000) to the case of spiking neurons in an actor-critic network operating in continuous time, and with continuous state and action representations. In our model, the critic learns to predict expected future rewards in real time. Its activity, together with actual rewards, conditions the delivery of a neuromodulatory TD signal to itself and to the actor, which is responsible for action choice. In simulations, we show that such an architecture can solve a Morris water-maze-like navigation task, in a number of trials consistent with reported animal performance. We also use our model to solve the acrobot and the cartpole problems, two complex motor control tasks. Our model provides a plausible way of computing reward prediction error in the brain. Moreover, the analytically derived learning rule is consistent with experimental evidence for dopamine-modulated spike-timing-dependent plasticity.

Motor systems: reaching out and grasping the molecular tools.

Two recent studies provide important insights into the organization of premotor circuitries, showing that control of highly-specific skilled forelimb movements, such as reaching and grasping, requires activation of specific subpopulations of neurons in the brainstem and spinal cord.

First human hNT neurons patterned on parylene-C/silicon dioxide substrates: Combining an accessible cell line and robust patterning technology for the study of the pathological adult human brain

In this communication, we describe a new method which has enabled the first patterning of human neurons (derived from the human teratocarcinoma cell line (hNT)) on parylene-C/silicon dioxide substrates. We reveal the details of the nanofabrication processes, cell differentiation and culturing protocols necessary to successfully pattern hNT neurons which are each key aspects of this new method. The benefits in patterning human neurons on silicon chip using an accessible cell line and robust patterning technology are of widespread value. Thus, using a combined technology such as this will facilitate the detailed study of the pathological human brain at both the single cell and network level. © 2010 Elsevier B.V.

Isolating single primary rat hippocampal neurons and astrocytes on ultra-thin patterned parylene-C/silicon dioxide substrates

We report here the patterning of primary rat neurons and astrocytes from the postnatal hippocampus on ultra-thin parylene-C deposited on a silicon dioxide substrate, following observations of neuronal, astrocytic and nuclear coverage on strips of different lengths, widths and thicknesses. Neuronal and glial growth was characterized 'on', 'adjacent to' and 'away from' the parylene strips. In addition, the article reports how the same material combination can be used to isolate single cells along thin tracks of parylene-C. This is demonstrated with a series of high magnification images of the experimental observations for varying parylene strip widths and thicknesses. Thus, the findings demonstrate the possibility to culture cells on ultra-thin layers of parylene-C and localize single cells on thin strips. Such work is of interest and significance to the Neuroengineering and Multi-Electrode Array (MEA) communities, as it provides an alternative insulating material in the fabrication of embedded micro-electrodes, which can be used to facilitate single cell stimulation and recording in capacitive coupling mode. © 2010 Elsevier Ltd.