Spatio-temporal credit assignment in population learning

Learning by reinforcement is important in shaping animal behavior, and in particular in behavioral decision making. Such decision making is likely to involve the integration of many synaptic events in space and time. However, using a single reinforcement signal to modulate synaptic plasticity, as suggested in classical reinforcement learning algorithms, a twofold problem arises. Different synapses will have contributed differently to the behavioral decision, and even for one and the same synapse, releases at different times may have had different effects. Here we present a plasticity rule which solves this spatio-temporal credit assignment problem in a population of spiking neurons. The learning rule is spike-time dependent and maximizes the expected reward by following its stochastic gradient. Synaptic plasticity is modulated not only by the reward, but also by a population feedback signal. While this additional signal solves the spatial component of the problem, the temporal one is solved by means of synaptic eligibility traces. In contrast to temporal difference (TD) based approaches to reinforcement learning, our rule is explicit with regard to the assumed biophysical mechanisms. Neurotransmitter concentrations determine plasticity and learning occurs fully online. Further, it works even if the task to be learned is non-Markovian, i.e. when reinforcement is not determined by the current state of the system but may also depend on past events. The performance of the model is assessed by studying three non-Markovian tasks. In the first task, the reward is delayed beyond the last action with non-related stimuli and actions appearing in between. The second task involves an action sequence which is itself extended in time and reward is only delivered at the last action, as it is the case in any type of board-game. The third task is the inspection game that has been studied in neuroeconomics, where an inspector tries to prevent a worker from shirking. Applying our algorithm to this game yields a learning behavior which is consistent with behavioral data from humans and monkeys, revealing themselves properties of a mixed Nash equilibrium. The examples show that our neuronal implementation of reward based learning copes with delayed and stochastic reward delivery, and also with the learning of mixed strategies in two-opponent games.

Spatio-temporal credit assignment in population learning

Learning by reinforcement is important in shaping animal behavior. But behavioral decision making is likely to involve the integration of many synaptic events in space and time. So in using a single reinforcement signal to modulate synaptic plasticity a twofold problem arises. Different synapses will have contributed differently to the behavioral decision and, even for one and the same synapse, releases at different times may have had different effects. Here we present a plasticity rule which solves this spatio-temporal credit assignment problem in a population of spiking neurons. The learning rule is spike time dependent and maximizes the expected reward by following its stochastic gradient. Synaptic plasticity is modulated not only by the reward but by a population feedback signal as well. While this additional signal solves the spatial component of the problem, the temporal one is solved by means of synaptic eligibility traces. In contrast to temporal difference based approaches to reinforcement learning, our rule is explicit with regard to the assumed biophysical mechanisms. Neurotransmitter concentrations determine plasticity and learning occurs fully online. Further, it works even if the task to be learned is non-Markovian, i.e. when reinforcement is not determined by the current state of the system but may also depend on past events. The performance of the model is assessed by studying three non-Markovian tasks. In the first task the reward is delayed beyond the last action with non-related stimuli and actions appearing in between. The second one involves an action sequence which is itself extended in time and reward is only delivered at the last action, as is the case in any type of board-game. The third is the inspection game that has been studied in neuroeconomics. It only has a mixed Nash equilibrium and exemplifies that the model also copes with stochastic reward delivery and the learning of mixed strategies.

Fast kinase domain-containing protein 3 is a mitochondrial protein essential for cellular respiration

Fas-activated serine/threonine phosphoprotein (FAST) is the founding member of the FAST kinase domain-containing protein (FASTKD) family that includes FASTKD1-5. FAST is a sensor of mitochondrial stress that modulates protein translation to promote the survival of cells exposed to adverse conditions. Mutations in FASTKD2 have been linked to a mitochondrial encephalomyopathy that is associated with reduced cytochrome c oxidase activity, an essential component of the mitochondrial electron transport chain. We have confirmed the mitochondrial localization of FASTKD2 and shown that all FASTKD family members are found in mitochondria. Although human and mouse FASTKD1-5 genes are expressed ubiquitously, some of them are most abundantly expressed in mitochondria-enriched tissues. We have found that RNA interference-mediated knockdown of FASTKD3 severely blunts basal and stress-induced mitochondrial oxygen consumption without disrupting the assembly of respiratory chain complexes. Tandem affinity purification reveals that FASTKD3 interacts with components of mitochondrial respiratory and translation machineries. Our results introduce FASTKD3 as an essential component of mitochondrial respiration that may modulate energy balance in cells exposed to adverse conditions by functionally coupling mitochondrial protein synthesis to respiration.

Reduction in hepatic drug metabolizing CYP3A4 activities caused by P450 oxidoreductase mutations identified in patients with disordered steroid metabolism

Cytochrome P450 3A4 (CYP3A4), the major P450 present in human liver metabolizes approximately half the drugs in clinical use and requires electrons supplied from NADPH through NADPH-P450 reductase (POR, CPR). Mutations in human POR cause a rare form of congenital adrenal hyperplasia from diminished activities of steroid metabolizing P450s. In this study we examined the effect of mutations in POR on CYP3A4 activity. We used purified preparations of wild type and mutant human POR and in vitro reconstitution with purified CYP3A4 to perform kinetic studies. We are reporting that mutations in POR identified in patients with disordered steroidogenesis/Antley-Bixler syndrome (ABS) may reduce CYP3A4 activity, potentially affecting drug metabolism in individuals carrying mutant POR alleles. POR mutants Y181D, A457H, Y459H, V492E and R616X had more than 99% loss of CYP3A4 activity, while POR mutations A287P, C569Y and V608F lost 60-85% activity. Loss of CYP3A4 activity may result in increased risk of drug toxicities and adverse drug reactions in patients with POR mutations.

Altered heme catabolism by heme oxygenase-1 caused by mutations in human NADPH cytochrome P450 reductase

Human heme oxygenase-1 (HO-1) carries out heme catabolism supported by electrons supplied from the NADPH through NADPH P450 reductase (POR, CPR). Previously we have shown that mutations in human POR cause a rare form of congenital adrenal hyperplasia. In this study, we have evaluated the effects of mutations in POR on HO-1 activity. We used purified preparations of wild type and mutant human POR and in vitro reconstitution with purified HO-1 to measure heme degradation in a coupled assay using biliverdin reductase. Here we show that mutations in POR found in patients may reduce HO-1 activity, potentially influencing heme catabolism in individuals carrying mutant POR alleles. POR mutants Y181D, A457H, Y459H, V492E and R616X had total loss of HO-1 activity, while POR mutations A287P, C569Y and V608F lost 50-70% activity. The POR variants P228L, R316W and G413S, A503V and G504R identified as polymorphs had close to WT activity. Loss of HO-1 activity may result in increased oxidative neurotoxicity, anemia, growth retardation and iron deposition. Further examination of patients affected with POR deficiency will be required to assess the metabolic effects of reduced HO-1 activity in affected individuals.

Microneurography in rats: a minimally invasive method to record single C-fiber action potentials from peripheral nerves in vivo

Microneurography is a method suitable for recording intraneural single or multiunit action potentials in conscious subjects. Microneurography has rarely been applied to animal experiments, where more invasive methods, like the teased fiber recording technique, are widely used. We have tested the feasibility of microneurographic recordings from the peripheral nerves of rats. Tungsten microelectrodes were inserted into the sciatic nerve at mid-thigh level. Single or multiunit action potentials evoked by regular electrical stimulation were recorded, digitized and displayed as a raster plot of latencies. The method allows unambiguous recording and recognition of single C-fiber action potentials from an in vivo preparation, with minimal disruption of the nerve being recorded. Multiple C-fibers can be recorded simultaneously for several hours, and if the animal is allowed to recover, repeated recording sessions can be obtained from the same nerve at the same level over a period of weeks or months. Also, single C units can be functionally identified by their changes in latency to natural stimuli, and insensitive units can be recognized as 'silent' nociceptors or sympathetic efferents by their distinctive profiles of activity-dependent slowing during repetitive electrical stimulation, or by the effect on spontaneous efferent activity of a proximal anesthetic block. Moreover, information about the biophysical properties of C axons can be obtained from their latency recovery cycles. Finally, we show that this preparation is potentially suitable for the study of C-fiber behavior in models of neuropathies and nerve lesions, both under resting conditions and in response to drug administration.

Activation of sphingosine kinase 2 is an endogenous protective mechanism in cerebral ischemia

The two ubiquitously expressed sphingosine kinases (SphK) 1 and 2 are key regulators of the sphingolipid signaling pathway. Despite the formation of an identical messenger, i.e. sphingosine 1-phosphate (S1P), they exert strikingly different functions. Particularly, SphK2 is necessary for the phosphorylation of the sphingosine analog fingolimod (FTY720), which is protective in rodent stroke models. Using gene deficient mice lacking either SphK1 or SphK2, we investigated the role of the two lipid kinases in experimental stroke. We performed 2h transient middle cerebral artery occlusion (tMCAO) and analyzed lesion size and neurological function after 24h. Treatment groups received 1mg/kg FTY720. Neutrophil infiltration, microglia activation, mRNA and protein expression of SphK1, SphK2 and the S1P(1) receptor after tMCAO were studied. Genetic deletion of SphK2 but not SphK1 increased ischemic lesion size and worsened neurological function after tMCAO. The protective effect of FTY720 was conserved in SphK1(-/-) mice but not in SphK2(-/-) mice. This suggests that SphK2 activity is an important endogenous protective mechanism in cerebral ischemia and corroborates that the protective effect of FTY720 is mediated via phospho-FTY720.

DARPins recognizing the tumor-associated antigen EpCAM selected by phage and ribosome display and engineered for multivalency

Designed Ankyrin Repeat Proteins (DARPins) represent a novel class of binding molecules. Their favorable biophysical properties such as high affinity, stability and expression yields make them ideal candidates for tumor targeting. Here, we describe the selection of DARPins specific for the tumor-associated antigen epithelial cell adhesion molecule (EpCAM), an approved therapeutic target on solid tumors. We selected DARPins from combinatorial libraries by both phage display and ribosome display and compared their binding on tumor cells. By further rounds of random mutagenesis and ribosome display selection, binders with picomolar affinity were obtained that were entirely monomeric and could be expressed at high yields in the cytoplasm of Escherichia coli. One of the binders, denoted Ec1, bound to EpCAM with picomolar affinity (K(d)=68 pM), and another selected DARPin (Ac2) recognized a different epitope on EpCAM. Through the use of a variety of bivalent and tetravalent arrangements with these DARPins, the off-rate on cells was further improved by up to 47-fold. All EpCAM-specific DARPins were efficiently internalized by receptor-mediated endocytosis, which is essential for intracellular delivery of anticancer agents to tumor cells. Thus, using EpCAM as a target, we provide evidence that DARPins can be conveniently selected and rationally engineered to high-affinity binders of various formats for tumor targeting.

Farming Systems and Resilience Building in a Changing Climate On-Farm Comparisons of Organic and Conventional Cocoa Cultivation in Alto Beni, Bolivia

Resilience research has been applied to socioeconomic as well as for agroecological studies in the last 20 years. It provides a conceptual and methodological approach for a better understanding of interrelations between the performance of ecological and social systems. In the research area Alto Beni, Bolivia, the production of cocoa (Theobroma cacao L.), is one of the main sources of income. Farmers in the region have formed producers’ associations to enhance organic cocoa cultivation and obtain fair prices since the 1980s. In cooperation with the long-term system comparisons by the Research Institute of Organic Agriculture (FiBL) in Alto Beni, aspects of the field trial are applied for the use in on-farm research: a comparison of soil fertility, biomass and crop diversity is combined with qualitative interviews and participatory observation methods. Fieldwork is carried out together with Bolivian students through the Swiss KFPE-programme Echanges Universitaires. For the system comparisons, four different land-use types were classified according to their ecological complexity during a preliminary study in 2009: successional agroforestry systems, simple agroforestry systems (both organically managed and certified), traditional systems and conventional monocultures. The study focuses on interrelations between different ways of cocoa cultivation, livelihoods and the related socio-cultural rationales behind them. In particular this second aspect is innovative as it allows to broaden the biophysical perspective to a more comprehensive evaluation with socio-ecological aspects thereby increasing the relevance of the agronomic field studies for development policy and practice. Moreover, such a socio-ecological baseline allows to assess the potential of organic agriculture regarding resilience-building face to socio-environmental stress factors. Among others, the results of the pre-study illustrate local farmers’ perceptions of climate change and the consequences for the different crop-systems: all interviewees mentioned rising temperatures and/or an extended dry season as negative impacts more with regard to their own working conditions than to their crops. This was the case in particular for conventional monocultures and in plots where slash-and-burn cultivation was practised whereas for organic agroforestry systems the advantage of working in the shade was stressed indicating that their relevance rises in the context of climate change.

Spatio-Temporal Credit Assignment in Neuronal Population Learning

n learning from trial and error, animals need to relate behavioral decisions to environmental reinforcement even though it may be difficult to assign credit to a particular decision when outcomes are uncertain or subject to delays. When considering the biophysical basis of learning, the credit-assignment problem is compounded because the behavioral decisions themselves result from the spatio-temporal aggregation of many synaptic releases. We present a model of plasticity induction for reinforcement learning in a population of leaky integrate and fire neurons which is based on a cascade of synaptic memory traces. Each synaptic cascade correlates presynaptic input first with postsynaptic events, next with the behavioral decisions and finally with external reinforcement. For operant conditioning, learning succeeds even when reinforcement is delivered with a delay so large that temporal contiguity between decision and pertinent reward is lost due to intervening decisions which are themselves subject to delayed reinforcement. This shows that the model provides a viable mechanism for temporal credit assignment. Further, learning speeds up with increasing population size, so the plasticity cascade simultaneously addresses the spatial problem of assigning credit to synapses in different population neurons. Simulations on other tasks, such as sequential decision making, serve to contrast the performance of the proposed scheme to that of temporal difference-based learning. We argue that, due to their comparative robustness, synaptic plasticity cascades are attractive basic models of reinforcement learning in the brain.