Spatio-temporal credit assignment in population learning

Learning by reinforcement is important in shaping animal behavior, and in particular in behavioral decision making. Such decision making is likely to involve the integration of many synaptic events in space and time. However, using a single reinforcement signal to modulate synaptic plasticity, as suggested in classical reinforcement learning algorithms, a twofold problem arises. Different synapses will have contributed differently to the behavioral decision, and even for one and the same synapse, releases at different times may have had different effects. Here we present a plasticity rule which solves this spatio-temporal credit assignment problem in a population of spiking neurons. The learning rule is spike-time dependent and maximizes the expected reward by following its stochastic gradient. Synaptic plasticity is modulated not only by the reward, but also by a population feedback signal. While this additional signal solves the spatial component of the problem, the temporal one is solved by means of synaptic eligibility traces. In contrast to temporal difference (TD) based approaches to reinforcement learning, our rule is explicit with regard to the assumed biophysical mechanisms. Neurotransmitter concentrations determine plasticity and learning occurs fully online. Further, it works even if the task to be learned is non-Markovian, i.e. when reinforcement is not determined by the current state of the system but may also depend on past events. The performance of the model is assessed by studying three non-Markovian tasks. In the first task, the reward is delayed beyond the last action with non-related stimuli and actions appearing in between. The second task involves an action sequence which is itself extended in time and reward is only delivered at the last action, as it is the case in any type of board-game. The third task is the inspection game that has been studied in neuroeconomics, where an inspector tries to prevent a worker from shirking. Applying our algorithm to this game yields a learning behavior which is consistent with behavioral data from humans and monkeys, revealing themselves properties of a mixed Nash equilibrium. The examples show that our neuronal implementation of reward based learning copes with delayed and stochastic reward delivery, and also with the learning of mixed strategies in two-opponent games.

Spatio-temporal credit assignment in population learning

Learning by reinforcement is important in shaping animal behavior. But behavioral decision making is likely to involve the integration of many synaptic events in space and time. So in using a single reinforcement signal to modulate synaptic plasticity a twofold problem arises. Different synapses will have contributed differently to the behavioral decision and, even for one and the same synapse, releases at different times may have had different effects. Here we present a plasticity rule which solves this spatio-temporal credit assignment problem in a population of spiking neurons. The learning rule is spike time dependent and maximizes the expected reward by following its stochastic gradient. Synaptic plasticity is modulated not only by the reward but by a population feedback signal as well. While this additional signal solves the spatial component of the problem, the temporal one is solved by means of synaptic eligibility traces. In contrast to temporal difference based approaches to reinforcement learning, our rule is explicit with regard to the assumed biophysical mechanisms. Neurotransmitter concentrations determine plasticity and learning occurs fully online. Further, it works even if the task to be learned is non-Markovian, i.e. when reinforcement is not determined by the current state of the system but may also depend on past events. The performance of the model is assessed by studying three non-Markovian tasks. In the first task the reward is delayed beyond the last action with non-related stimuli and actions appearing in between. The second one involves an action sequence which is itself extended in time and reward is only delivered at the last action, as is the case in any type of board-game. The third is the inspection game that has been studied in neuroeconomics. It only has a mixed Nash equilibrium and exemplifies that the model also copes with stochastic reward delivery and the learning of mixed strategies.

In vivo evaluation in cynomolgus monkey brain and metabolism of [¹⁸F]fluorodeprenyl: a new MAO-B pet radioligand

In this study, we evaluated the in vivo characteristics of a new monoamine oxidase type B (MAO-B) radioligand, [¹⁸F]fluorodeprenyl, by positron emission tomography (PET) in two cynomolgus monkeys. The brain uptake of [¹⁸F]fluorodeprenyl was more than 7% (600% SUV) of the total injected radioactivity and similar to that of [¹¹C]deprenyl, an established MAO-B radioligand. The highest uptake was observed in the striatum, one of the MAO-B-rich regions, with a peak at approximately 2-3 min after injection, followed by lower uptake in the thalamus and the cortex and lowest uptake in the cerebellum. Brain uptake of [¹⁸F]fluorodeprenyl was largely inhibited by preadministration of the MAO-B inhibitor, L-deprenyl, whereas clorgyline, a MAO Type A blocker, had no significant inhibitory effect, thus demonstrating selectivity for MAO-B. [¹⁸F]Fluorodeprenyl showed relatively slow metabolism with the presence of two radiometabolite peaks with similar retention time as the labeled metabolites of [¹¹C]deprenyl. These results suggest that [¹⁸F]fluorodeprenyl is a potential PET radioligand for visualization of MAO-B activity.

Interactions between short-term and long-term plasticity: shooting for a moving target

Far from being static transmission units, synapses are highly dynamical elements that change over multiple time scales depending on the history of the neural activity of both the pre- and postsynaptic neuron. Moreover, synaptic changes on different time scales interact: long-term plasticity (LTP) can modify the properties of short-term plasticity (STP) in the same synapse. Most existing theories of synaptic plasticity focus on only one of these time scales (either STP or LTP or late-LTP) and the theoretical principles underlying their interactions are thus largely unknown. Here we develop a normative model of synaptic plasticity that combines both STP and LTP and predicts specific patterns for their interactions. Recently, it has been proposed that STP arranges for the local postsynaptic membrane potential at a synapse to behave as an optimal estimator of the presynaptic membrane potential based on the incoming spikes. Here we generalize this approach by considering an optimal estimator of a non-linear function of the membrane potential and the long-term synaptic efficacy—which itself may be subject to change on a slower time scale. We find that an increase in the long-term synaptic efficacy necessitates changes in the dynamics of STP. More precisely, for a realistic non-linear function to be estimated, our model predicts that after the induction of LTP, causing long-term synaptic efficacy to increase, a depressing synapse should become even more depressing. That is, in a protocol using trains of presynaptic stimuli, as the initial EPSP becomes stronger due to LTP, subsequent EPSPs should become weakened and this weakening should be more pronounced with LTP. This form of redistribution of synaptic efficacies agrees well with electrophysiological data on synapses connecting layer 5 pyramidal neurons.

LGI2 truncation causes a remitting focal epilepsy in dogs

One quadrillion synapses are laid in the first two years of postnatal construction of the human brain, which are then pruned until age 10 to 500 trillion synapses composing the final network. Genetic epilepsies are the most common neurological diseases with onset during pruning, affecting 0.5% of 2-10-year-old children, and these epilepsies are often characterized by spontaneous remission. We previously described a remitting epilepsy in the Lagotto romagnolo canine breed. Here, we identify the gene defect and affected neurochemical pathway. We reconstructed a large Lagotto pedigree of around 34 affected animals. Using genome-wide association in 11 discordant sib-pairs from this pedigree, we mapped the disease locus to a 1.7 Mb region of homozygosity in chromosome 3 where we identified a protein-truncating mutation in the Lgi2 gene, a homologue of the human epilepsy gene LGI1. We show that LGI2, like LGI1, is neuronally secreted and acts on metalloproteinase-lacking members of the ADAM family of neuronal receptors, which function in synapse remodeling, and that LGI2 truncation, like LGI1 truncations, prevents secretion and ADAM interaction. The resulting epilepsy onsets at around seven weeks (equivalent to human two years), and remits by four months (human eight years), versus onset after age eight in the majority of human patients with LGI1 mutations. Finally, we show that Lgi2 is expressed highly in the immediate post-natal period until halfway through pruning, unlike Lgi1, which is expressed in the latter part of pruning and beyond. LGI2 acts at least in part through the same ADAM receptors as LGI1, but earlier, ensuring electrical stability (absence of epilepsy) during pruning years, preceding this same function performed by LGI1 in later years. LGI2 should be considered a candidate gene for common remitting childhood epilepsies, and LGI2-to-LGI1 transition for mechanisms of childhood epilepsy remission.

Fine structure of hippocampal mossy fiber synapses following rapid high-pressure freezing

Synapses of hippocampal neurons play important roles in learning and memory processes and are involved in aberrant hippocampal function in temporal lobe epilepsy. Major neuronal types in the hippocampus as well as their input and output synapses are well known, but it has remained an open question to what extent conventional electron microscopy (EM) has provided us with the real appearance of synaptic fine structure under in vivo conditions. There is reason to assume that conventional aldehyde fixation and dehydration lead to protein denaturation and tissue shrinkage, likely associated with the occurrence of artifacts. However, realistic fine-structural data of synapses are required for our understanding of the transmission process and for its simulation. Here, we used high-pressure freezing and cryosubstitution of hippocampal tissue that was not subjected to aldehyde fixation and dehydration in ethanol to monitor the fine structure of an identified synapse in the hippocampal CA3 region, that is, the synapse between granule cell axons, the mossy fibers, and the proximal dendrites of CA3 pyramidal neurons. Our results showed that high-pressure freezing nicely preserved ultrastructural detail of this particular synapse and allowed us to study rapid structural changes associated with synaptic plasticity.

Gradient estimation in dendritic reinforcement learning

We study synaptic plasticity in a complex neuronal cell model where NMDA-spikes can arise in certain dendritic zones. In the context of reinforcement learning, two kinds of plasticity rules are derived, zone reinforcement (ZR) and cell reinforcement (CR), which both optimize the expected reward by stochastic gradient ascent. For ZR, the synaptic plasticity response to the external reward signal is modulated exclusively by quantities which are local to the NMDA-spike initiation zone in which the synapse is situated. CR, in addition, uses nonlocal feedback from the soma of the cell, provided by mechanisms such as the backpropagating action potential. Simulation results show that, compared to ZR, the use of nonlocal feedback in CR can drastically enhance learning performance. We suggest that the availability of nonlocal feedback for learning is a key advantage of complex neurons over networks of simple point neurons, which have previously been found to be largely equivalent with regard to computational capability.

Agrin is required for survival and function of monocytic cells

Agrin, an extracellular matrix protein belonging to the heterogeneous family of heparan sulfate proteoglycans (HSPGs), is expressed by cells of the hematopoietic system but its role in leukocyte biology is not yet clear. Here we demonstrate that agrin has a crucial, nonredundant role in myeloid cell development and functions. We have identified lineage-specific alterations that affect maturation, survival and properties of agrin-deficient monocytic cells, and occur at stages later than stem cell precursors. Our data indicate that the cell-autonomous signals delivered by agrin are sensed by macrophages through the α-DC (DG) receptor and lead to the activation of signaling pathways resulting in rearrangements of the actin cytoskeleton during the phagocytic synapse formation and phosphorylation of extracellular signal-regulated kinases (Erk 1/2). Altogether, these data identify agrin as a novel player of innate immunity.

Expression patterns of neurexin-1 and neuroligins in brain and retina of the chick embryo: Neuroligin-3 is absent in retina

Neuroligins (NLs) constitute a family of cell-surface proteins that interact with neurexins (beta-Nxs), another class of neuronal cell-surface proteins, one of each class functioning together in synapse formation. The localization of the various neurexins and neuroligins, however, has not yet been clarified in chicken. Therefore, we studied the expression patterns of neurexin-1 (Nx-1) and neuroligin-1 and -3 during embryonic development of the chick retina and brain by reverse-transcriptase polymerase chain reaction (RT-PCR) and in situ hybridization (ISH). While neurexin-1 increased continuously in both brain and retina, the expression of both neuroligins was more variable. As shown by ISH, Nx-1 is expressed in the inner half retina along with differentiation of ganglion and amacrine cells. Transcripts of NL-1 were detected as early as day 4 and increased with the maturation of the different brain regions. In different brain regions, NL-1 showed a different time regulation. Remarkably, neuroligin-3 was entirely absent in retina. This study indicates that synaptogenetic processes in brain and retina use different molecular machineries, whereby the neuroligins might represent the more distinctly regulated part of the neurexin-neuroligin complexes. Noticeably, NL-3 does not seem to be involved in the making of retinal synapses.

New ways of looking at synapses

Current concepts of synaptic fine-structure are derived from electron microscopic studies of tissue fixed by chemical fixation using aldehydes. However, chemical fixation with glutaraldehyde and paraformaldehyde and subsequent dehydration in ethanol result in uncontrolled tissue shrinkage. While electron microscopy allows for the unequivocal identification of synaptic contacts, it cannot be used for real-time analysis of structural changes at synapses. For the latter purpose advanced fluorescence microscopy techniques are to be applied which, however, do not allow for the identification of synaptic contacts. Here, two approaches are described that may overcome, at least in part, some of these drawbacks in the study of synapses. By focusing on a characteristic, easily identifiable synapse, the mossy fiber synapse in the hippocampus, we first describe high-pressure freezing of fresh tissue as a method that may be applied to study subtle changes in synaptic ultrastructure associated with functional synaptic plasticity. Next, we propose to label presynaptic mossy fiber terminals and postsynaptic complex spines on CA3 pyramidal neurons by different fluorescent dyes to allow for the real-time monitoring of these synapses in living tissue over extended periods of time. We expect these approaches to lead to new insights into the structure and function of central synapses.

Human immunodeficiency virus type 1 and influenza virus exit via different membrane microdomains

Directed release of human immunodeficiency virus type 1 (HIV-1) into the cleft of the virological synapse that can form between infected and uninfected T cells, for example, in lymph nodes, is thought to contribute to the systemic spread of this virus. In contrast, influenza virus, which causes local infections, is shed into the airways of the respiratory tract from free surfaces of epithelial cells. We now demonstrate that such differential release of HIV-1 and influenza virus is paralleled, at the subcellular level, by viral assembly at different microsegments of the plasma membrane of HeLa cells. HIV-1, but not influenza virus, buds through microdomains containing the tetraspanins CD9 and CD63. Consequently, the anti-CD9 antibody K41, which redistributes its antigen and also other tetraspanins to cell-cell adhesion sites, interferes with HIV-1 but not with influenza virus release. Altogether, these data strongly suggest that the bimodal egress of these two pathogenic viruses, like their entry into target cells, is guided by specific sets of host cell proteins.

Simulating in vivo-like Synaptic Input Patterns in Multicompartmental Models

In the laboratory of Dr. Dieter Jaeger at Emory University, we use computer simulations to study how the biophysical properties of neurons—including their three-dimensional structure, passive membrane resistance and capacitance, and active membrane conductances generated by ion channels—affect the way that the neurons transfer synaptic inputs into the action potential streams that represent their output. Because our ultimate goal is to understand how neurons process and relay information in a living animal, we try to make our computer simulations as realistic as possible. As such, the computer models reflect the detailed morphology and all of the ion channels known to exist in the particular neuron types being simulated, and the model neurons are tested with synaptic input patterns that are intended to approximate the inputs that real neurons receive in vivo. The purpose of this workshop tutorial was to explain what we mean by ‘in vivo-like’ synaptic input patterns, and how we introduce these input patterns into our computer simulations using the freely available GENESIS software package (http://www.genesis-sim.org/GENESIS). The presentation was divided into four sections: first, an explanation of what we are talking about when we refer to in vivo-like synaptic input patterns

Post-translational regulation of an Aplysia glutamate transporter during long-term facilitation.

Regulation of glutamate transporters accompanies plasticity of some glutamatergic synapses. The regulation of glutamate uptake at the Aplysia sensorimotor synapse during long-term facilitation (LTF) was investigated. Previously, increases in levels of ApGT1 (Aplysia glutamate transporter 1) in synaptic membranes were found to be related to long-term increases in glutamate uptake. In this study, we found that regulation of ApGT1 during LTF appears to occur post-translationally. Serotonin (5-HT) a transmitter that induces LTF did not increase synthesis of ApGT1. A pool of ApGT1 appears to exist in sensory neuron somata, which is transported to the terminals by axonal transport. Blocking the rough endoplasmic reticulum-Golgi-trans-Golgi network (TGN) pathway with Brefeldin A prevented the 5-HT-induced increase of ApGT1 in terminals. Also, 5-HT produced changes in post-translational modifications of ApGT1 as well as changes in the levels of an ApGT1-co-precipitating protein. These results suggest that regulation of trafficking of ApGT1 from the vesicular trafficking system (rough endoplasmic reticulum-Golgi-TGN) in the sensory neuron somata to the terminals by post-translational modifications and protein interactions appears to be the mechanism underlying the increase in ApGT1, and thus, glutamate uptake during memory formation.

Protein-Protein Interactions That Regulate Neurotransmitter Release from Retinal Ribbon Synapses

Protein-Protein Interactions That Regulate Neurotransmitter Release from Retinal Ribbon Synapses Photoreceptors and bipolar cells in the retina form specialized chemical synapses called ribbon synapses. This type of synapse differs physiologically from “conventional” chemical synapses. While “conventional” synapses exocytose neurotransmitter-filled vesicles in an all-or-none fashion in response to an action potential, a retinal ribbon synapse can release neurotransmitter tonically (sustained) in response to graded changes in membrane potential or phasically (transient) in response to a large change in membrane potential. Synaptic vesicle exocytosis is a tightly controlled process involving many protein-protein interactions. Therefore, it is likely that the dissimilarity in the release properties of retinal ribbon synapses and conventional synapses is the result of molecular differences between the two synapse types. Consistent with this idea, previous studies have demonstrated that ribbon synapses in the retina do not contain the t-SNARE (target-soluble N-ethylmaleimide-sensitive factor attachment protein receptor) syntaxin 1A that is found in conventional synapses of the nervous system. In contrast, ribbon synapses of the mammalian retina contain the related isoform, syntaxin 3B. Given that SNARE proteins play an important role in neurotransmitter release in conventional synapses, the purpose of this study was to characterize syntaxin 3B in order to elucidate what role this protein plays in neurotransmitter release from retinal ribbon synapses. Using molecular and biochemical techniques, it was demonstrated that syntaxin 3B is a binding partner of several presynaptic proteins that play a important role in synaptic vesicle exocytosis from retinal ribbon synapses and it is an evolutionarily conserved protein.