Reinforcement learning in dendritic structures

The discovery of binary dendritic events such as local NMDA spikes in dendritic subbranches led to the suggestion that dendritic trees could be computationally equivalent to a 2-layer network of point neurons, with a single output unit represented by the soma, and input units represented by the dendritic branches. Although this interpretation endows a neuron with a high computational power, it is functionally not clear why nature would have preferred the dendritic solution with a single but complex neuron, as opposed to the network solution with many but simple units. We show that the dendritic solution has a distinguished advantage over the network solution when considering different learning tasks. Its key property is that the dendritic branches receive an immediate feedback from the somatic output spike, while in the corresponding network architecture the feedback would require additional backpropagating connections to the input units. Assuming a reinforcement learning scenario we formally derive a learning rule for the synaptic contacts on the individual dendritic trees which depends on the presynaptic activity, the local NMDA spikes, the somatic action potential, and a delayed reinforcement signal. We test the model for two scenarios: the learning of binary classifications and of precise spike timings. We show that the immediate feedback represented by the backpropagating action potential supplies the individual dendritic branches with enough information to efficiently adapt their synapses and to speed up the learning process.

Spatio-Temporal Credit Assignment in Neuronal Population Learning

n learning from trial and error, animals need to relate behavioral decisions to environmental reinforcement even though it may be difficult to assign credit to a particular decision when outcomes are uncertain or subject to delays. When considering the biophysical basis of learning, the credit-assignment problem is compounded because the behavioral decisions themselves result from the spatio-temporal aggregation of many synaptic releases. We present a model of plasticity induction for reinforcement learning in a population of leaky integrate and fire neurons which is based on a cascade of synaptic memory traces. Each synaptic cascade correlates presynaptic input first with postsynaptic events, next with the behavioral decisions and finally with external reinforcement. For operant conditioning, learning succeeds even when reinforcement is delivered with a delay so large that temporal contiguity between decision and pertinent reward is lost due to intervening decisions which are themselves subject to delayed reinforcement. This shows that the model provides a viable mechanism for temporal credit assignment. Further, learning speeds up with increasing population size, so the plasticity cascade simultaneously addresses the spatial problem of assigning credit to synapses in different population neurons. Simulations on other tasks, such as sequential decision making, serve to contrast the performance of the proposed scheme to that of temporal difference-based learning. We argue that, due to their comparative robustness, synaptic plasticity cascades are attractive basic models of reinforcement learning in the brain.

Responses to Central Oxotremorine and Scopolamine Support the Cholinergic Control of Male Mating Behavior in Hamsters

The responses of hamsters to intracranial injections of the cholinergic agonist oxotremorine (OXO) implicate cholinergic mechanisms in the medial preoptic area (MPOA) in the control of male mating behavior. To extend these observations, we ran three studies of responses to cholinergic drugs delivered singly or in combination to the vicinity of the MPOA. The first tested responses to OXO, confirming its ability to reduce the postejaculatory interval. The second complemented the first by examining responses to MPOA microinjections of the cholinergic antagonist scopolamine (SCO). These caused several changes revolving around intromission. These included increases in intromission frequency and ejaculation latency. They also included a change in the patterning of intromissions, marked by continuous strings without the usual separation by dismounts. The final study resembled the others in examining the effects of MPOA injections of OXO and SCO but focused on the ability of each drug to antagonize responses to the other. Most of the responses to OXO and SCO individually replicated earlier findings, though the measures examined here also permitted the description of effects on some noncopulatory sexual behaviors, specifically the male's inspection of the female. However, the most interesting results may be those suggesting asymmetry in the responses to the addition of the second drug: Whereas responses to OXO tended to be antagonized by SCO, OXO was less effective at counteracting responses to SCO. Though the explanation of this asymmetry is not completely clear, it is consistent with previous suggestions of differences in the affinities of these drugs for subtypes of muscarinic receptors. Therefore, it suggests that the cholinergic synapses and circuits controlling distinct elements of male behavior could differ in their dependence on these receptors. Copyright 2013 Elsevier Inc. All rights reserved.

LGI2 truncation causes a remitting focal epilepsy in dogs

One quadrillion synapses are laid in the first two years of postnatal construction of the human brain, which are then pruned until age 10 to 500 trillion synapses composing the final network. Genetic epilepsies are the most common neurological diseases with onset during pruning, affecting 0.5% of 2-10-year-old children, and these epilepsies are often characterized by spontaneous remission. We previously described a remitting epilepsy in the Lagotto romagnolo canine breed. Here, we identify the gene defect and affected neurochemical pathway. We reconstructed a large Lagotto pedigree of around 34 affected animals. Using genome-wide association in 11 discordant sib-pairs from this pedigree, we mapped the disease locus to a 1.7 Mb region of homozygosity in chromosome 3 where we identified a protein-truncating mutation in the Lgi2 gene, a homologue of the human epilepsy gene LGI1. We show that LGI2, like LGI1, is neuronally secreted and acts on metalloproteinase-lacking members of the ADAM family of neuronal receptors, which function in synapse remodeling, and that LGI2 truncation, like LGI1 truncations, prevents secretion and ADAM interaction. The resulting epilepsy onsets at around seven weeks (equivalent to human two years), and remits by four months (human eight years), versus onset after age eight in the majority of human patients with LGI1 mutations. Finally, we show that Lgi2 is expressed highly in the immediate post-natal period until halfway through pruning, unlike Lgi1, which is expressed in the latter part of pruning and beyond. LGI2 acts at least in part through the same ADAM receptors as LGI1, but earlier, ensuring electrical stability (absence of epilepsy) during pruning years, preceding this same function performed by LGI1 in later years. LGI2 should be considered a candidate gene for common remitting childhood epilepsies, and LGI2-to-LGI1 transition for mechanisms of childhood epilepsy remission.

Role of Acetylcholine in Control of Sexual Behavior of Male and Female Mammals

The results of studies using systemic or central applications of cholinergic drugs suggest that acetylcholine makes important contributions to the neurochemical control of male- and female-typical reproductive behaviors. In males, cholinergic control seems largely specific to some elements or aspects of copulatory behavior that can vary significantly across species. Synapses in or near the medial preoptic area represent part of this mechanism, but the entire system appears to extend more widely, perhaps especially to one or more structures flanking some part of the lateral ventricle. In females, the lordosis response that essentially defines sexual receptivity is clearly responsive to cholinergic drugs. The same seems likely to be true of other elements of female sexual behavior, but additional studies will be needed to confirm this. Changes in cholinergic activity may help to mediate estrogenic effects on female sexual behavior. However, estrogen exposure can increase or decrease cholinergic effects, suggesting a relationship that is complex and requires further analysis. Also presently unclear is the localization of the cholinergic effects on female sexual responses. Though periventricular sites again have been implicated, their identity is presently unknown. This review discusses these and other aspects of the central cholinergic systems affecting male and female sexual behaviors. Copyright 2014 Elsevier Inc. All rights reserved.

Eluding oblivion with smart stochastic selection of synaptic updates

The variables involved in the equations that describe realistic synaptic dynamics always vary in a limited range. Their boundedness makes the synapses forgetful, not for the mere passage of time, but because new experiences overwrite old memories. The forgetting rate depends on how many synapses are modified by each new experience: many changes means fast learning and fast forgetting, whereas few changes means slow learning and long memory retention. Reducing the average number of modified synapses can extend the memory span at the price of a reduced amount of information stored when a new experience is memorized. Every trick which allows to slow down the learning process in a smart way can improve the memory performance. We review some of the tricks that allow to elude fast forgetting (oblivion). They are based on the stochastic selection of the synapses whose modifications are actually consolidated following each new experience. In practice only a randomly selected, small fraction of the synapses eligible for an update are actually modified. This allows to acquire the amount of information necessary to retrieve the memory without compromising the retention of old experiences. The fraction of modified synapses can be further reduced in a smart way by changing synapses only when it is really necessary, i.e. when the post-synaptic neuron does not respond as desired. Finally we show that such a stochastic selection emerges naturally from spike driven synaptic dynamics which read noisy pre and post-synaptic neural activities. These activities can actually be generated by a chaotic system.

Vitamin E reduces antidepressant-related beta-adrenoceptor down-regulation in cultured cells. Comparable effects on St. John's wort and tricyclic antidepressant treatment

The mode of action of antidepressants is still a matter of debate. Acute inhibition of neurotransmitter reuptake in central neuronal synapses, followed by a down-regulation of central postsynaptic beta-adrenoceptor (beta-AR) numbers were consistently observed in vivo, while a reduction in surface beta-AR density was found in cell cultures. Effects of the tricyclic antidepressant desipramine (DMI) were abolished by vitamin E (alpha-TOC) in vitro as well as in vivo. Alpha-TOC interfered with antidepressant-induced changes of cellular plasma membrane properties and with recycling of beta-AR. St. John's wort (SJW) extract reduced beta-AR numbers in cultured cells to a similar extent as DMI or the selective serotonin re-uptake inhibitor fluoxetine. We chronically co-exposed cell cultures to SJW extract and to alpha-TOC. Receptor down-regulation following exposure to the plant extract was inhibited in the presence of alpha-TOC suggesting a mode of action of SJW extract comparable to that of synthetic antidepressants. Inhibition of cell proliferation by the plant extract was also significantly reduced by alpha-TOC.

Expression patterns of neurexin-1 and neuroligins in brain and retina of the chick embryo: Neuroligin-3 is absent in retina

Neuroligins (NLs) constitute a family of cell-surface proteins that interact with neurexins (beta-Nxs), another class of neuronal cell-surface proteins, one of each class functioning together in synapse formation. The localization of the various neurexins and neuroligins, however, has not yet been clarified in chicken. Therefore, we studied the expression patterns of neurexin-1 (Nx-1) and neuroligin-1 and -3 during embryonic development of the chick retina and brain by reverse-transcriptase polymerase chain reaction (RT-PCR) and in situ hybridization (ISH). While neurexin-1 increased continuously in both brain and retina, the expression of both neuroligins was more variable. As shown by ISH, Nx-1 is expressed in the inner half retina along with differentiation of ganglion and amacrine cells. Transcripts of NL-1 were detected as early as day 4 and increased with the maturation of the different brain regions. In different brain regions, NL-1 showed a different time regulation. Remarkably, neuroligin-3 was entirely absent in retina. This study indicates that synaptogenetic processes in brain and retina use different molecular machineries, whereby the neuroligins might represent the more distinctly regulated part of the neurexin-neuroligin complexes. Noticeably, NL-3 does not seem to be involved in the making of retinal synapses.

Learning real-world stimuli in a neural network with spike-driven synaptic dynamics

We present a model of spike-driven synaptic plasticity inspired by experimental observations and motivated by the desire to build an electronic hardware device that can learn to classify complex stimuli in a semisupervised fashion. During training, patterns of activity are sequentially imposed on the input neurons, and an additional instructor signal drives the output neurons toward the desired activity. The network is made of integrate-and-fire neurons with constant leak and a floor. The synapses are bistable, and they are modified by the arrival of presynaptic spikes. The sign of the change is determined by both the depolarization and the state of a variable that integrates the postsynaptic action potentials. Following the training phase, the instructor signal is removed, and the output neurons are driven purely by the activity of the input neurons weighted by the plastic synapses. In the absence of stimulation, the synapses preserve their internal state indefinitely. Memories are also very robust to the disruptive action of spontaneous activity. A network of 2000 input neurons is shown to be able to classify correctly a large number (thousands) of highly overlapping patterns (300 classes of preprocessed Latex characters, 30 patterns per class, and a subset of the NIST characters data set) and to generalize with performances that are better than or comparable to those of artificial neural networks. Finally we show that the synaptic dynamics is compatible with many of the experimental observations on the induction of long-term modifications (spike-timing-dependent plasticity and its dependence on both the postsynaptic depolarization and the frequency of pre- and postsynaptic neurons).

Properties of basal dendrites of layer 5 pyramidal neurons: a direct patch-clamp recording study

Basal dendrites receive the majority of synapses that contact neocortical pyramidal neurons, yet our knowledge of synaptic processing in these dendrites has been hampered by their inaccessibility for electrical recordings. A new approach to patch-clamp recordings enabled us to characterize the integrative properties of these cells. Despite the short physical length of rat basal dendrites, synaptic inputs were electrotonically remote from the soma (>30-fold excitatory postsynaptic potential (EPSP) attenuation) and back-propagating action potentials were significantly attenuated. Unitary EPSPs were location dependent, reaching large amplitudes distally (>8 mV), yet their somatic contribution was relatively location independent. Basal dendrites support sodium and NMDA spikes, but not calcium spikes, for 75% of their length. This suggests that basal dendrites, despite their proximity to the site of action potential initiation, do not form a single basal-somatic region but rather should be considered as a separate integrative compartment favoring two integration modes: subthreshold, location-independent summation versus local amplification of incoming spatiotemporally clustered information.

Angiotensinergic neurons in sympathetic coeliac ganglia innervating rat and human mesenteric resistance blood vessels

In contrast to the current belief that angiotensin II (Ang II) interacts with the sympathetic nervous system only as a circulating hormone, we document here the existence of endogenous Ang II in the neurons of rat and human sympathetic coeliac ganglia and their angiotensinergic innervation with mesenteric resistance blood vessels. Angiotensinogen - and angiotensin converting enzyme-mRNA were detected by using quantitative real time polymerase chain reaction in total RNA extracts of rat coeliac ganglia, while renin mRNA was untraceable. Cathepsin D, a protease responsible for cleavage beneath other substrates also angiotensinogen to angiotensin I, was successfully detected in rat coeliac ganglia indicating the possibility of existence of alternative pathways. Angiotensinogen mRNA was also detected by in situ hybridization in the cytoplasm of neurons of rat coeliac ganglia. Immunoreactivity for Ang II was demonstrated in rat and human coeliac ganglia as well as with mesenteric resistance blood vessels. By using confocal laser scanning microscopy we were able to demonstrate the presence of angiotensinergic synapses en passant along side of vascular smooth muscle cells. Our findings indicate that Ang II is synthesized inside the neurons of sympathetic coeliac ganglia and may act as an endogenous neurotransmitter locally with the mesenteric resistance blood vessels.

A 45nm CMOS DESIGN OF NEUROMORPHIC SYSTEM

Neuromorphic computing has become an emerging field in wide range of applications. Its challenge lies in developing a brain-inspired architecture that can emulate human brain and can work for real time applications. In this report a flexible neural architecture is presented which consists of 128 X 128 SRAM crossbar memory and 128 spiking neurons. For Neuron, digital integrate and fire model is used. All components are designed in 45nm technology node. The core can be configured for certain Neuron parameters, Axon types and synapses states and are fully digitally implemented. Learning for this architecture is done offline. To train this circuit a well-known algorithm Restricted Boltzmann Machine (RBM) is used and linear classifiers are trained at the output of RBM. Finally, circuit was tested for handwritten digit recognition application. Future prospects for this architecture are also discussed.

Simulating in vivo-like Synaptic Input Patterns in Multicompartmental Models

In the laboratory of Dr. Dieter Jaeger at Emory University, we use computer simulations to study how the biophysical properties of neurons—including their three-dimensional structure, passive membrane resistance and capacitance, and active membrane conductances generated by ion channels—affect the way that the neurons transfer synaptic inputs into the action potential streams that represent their output. Because our ultimate goal is to understand how neurons process and relay information in a living animal, we try to make our computer simulations as realistic as possible. As such, the computer models reflect the detailed morphology and all of the ion channels known to exist in the particular neuron types being simulated, and the model neurons are tested with synaptic input patterns that are intended to approximate the inputs that real neurons receive in vivo. The purpose of this workshop tutorial was to explain what we mean by ‘in vivo-like’ synaptic input patterns, and how we introduce these input patterns into our computer simulations using the freely available GENESIS software package (http://www.genesis-sim.org/GENESIS). The presentation was divided into four sections: first, an explanation of what we are talking about when we refer to in vivo-like synaptic input patterns

Learning by the Dendritic Prediction of Somatic Spiking

Recent modeling of spike-timing-dependent plasticity indicates that plasticity involves as a third factor a local dendritic potential, besides pre- and postsynaptic firing times. We present a simple compartmental neuron model together with a non-Hebbian, biologically plausible learning rule for dendritic synapses where plasticity is modulated by these three factors. In functional terms, the rule seeks to minimize discrepancies between somatic firings and a local dendritic potential. Such prediction errors can arise in our model from stochastic fluctuations as well as from synaptic input, which directly targets the soma. Depending on the nature of this direct input, our plasticity rule subserves supervised or unsupervised learning. When a reward signal modulates the learning rate, reinforcement learning results. Hence a single plasticity rule supports diverse learning paradigms.

Matching Recall and Storage in Sequence Learning with Spiking Neural Networks

Storing and recalling spiking sequences is a general problem the brain needs to solve. It is, however, unclear what type of biologically plausible learning rule is suited to learn a wide class of spatiotemporal activity patterns in a robust way. Here we consider a recurrent network of stochastic spiking neurons composed of both visible and hidden neurons. We derive a generic learning rule that is matched to the neural dynamics by minimizing an upper bound on the Kullback–Leibler divergence from the target distribution to the model distribution. The derived learning rule is consistent with spike-timing dependent plasticity in that a presynaptic spike preceding a postsynaptic spike elicits potentiation while otherwise depression emerges. Furthermore, the learning rule for synapses that target visible neurons can be matched to the recently proposed voltage-triplet rule. The learning rule for synapses that target hidden neurons is modulated by a global factor, which shares properties with astrocytes and gives rise to testable predictions.