Reinforcement learning in dendritic structures

The discovery of binary dendritic events such as local NMDA spikes in dendritic subbranches led to the suggestion that dendritic trees could be computationally equivalent to a 2-layer network of point neurons, with a single output unit represented by the soma, and input units represented by the dendritic branches. Although this interpretation endows a neuron with a high computational power, it is functionally not clear why nature would have preferred the dendritic solution with a single but complex neuron, as opposed to the network solution with many but simple units. We show that the dendritic solution has a distinguished advantage over the network solution when considering different learning tasks. Its key property is that the dendritic branches receive an immediate feedback from the somatic output spike, while in the corresponding network architecture the feedback would require additional backpropagating connections to the input units. Assuming a reinforcement learning scenario we formally derive a learning rule for the synaptic contacts on the individual dendritic trees which depends on the presynaptic activity, the local NMDA spikes, the somatic action potential, and a delayed reinforcement signal. We test the model for two scenarios: the learning of binary classifications and of precise spike timings. We show that the immediate feedback represented by the backpropagating action potential supplies the individual dendritic branches with enough information to efficiently adapt their synapses and to speed up the learning process.

Reinforcement learning in dendritic structures

The discovery of binary dendritic events such as local NMDA spikes in dendritic subbranches led to the suggestion that dendritic trees could be computationally equivalent to a 2-layer network of point neurons, with a single output unit represented by the soma, and input units represented by the dendritic branches. Although this interpretation endows a neuron with a high computational power, it is functionally not clear why nature would have preferred the dendritic solution with a single but complex neuron, as opposed to the network solution with many but simple units. We show that the dendritic solution has a distinguished advantage over the network solution when considering different learning tasks. Its key property is that the dendritic branches receive an immediate feedback from the somatic output spike, while in the corresponding network architecture the feedback would require additional backpropagating connections to the input units. Assuming a reinforcement learning scenario we formally derive a learning rule for the synaptic contacts on the individual dendritic trees which depends on the presynaptic activity, the local NMDA spikes, the somatic action potential, and a delayed reinforcement signal. We test the model for two scenarios: the learning of binary classifications and of precise spike timings. We show that the immediate feedback represented by the backpropagating action potential supplies the individual dendritic branches with enough information to efficiently adapt their synapses and to speed up the learning process.

Intermittent spike-wave dynamics in a heterogeneous, spatially extended neural mass model

Generalised epileptic seizures are frequently accompanied by sudden, reversible transitions from low amplitude, irregular background activity to high amplitude, regular spike-wave discharges (SWD) in the EEG. The underlying mechanisms responsible for SWD generation and for the apparently spontaneous transitions to SWD and back again are still not fully understood. Specifically, the role of spatial cortico-cortical interactions in ictogenesis is not well studied. We present a macroscopic, neural mass model of a cortical column which includes two distinct time scales of inhibition. This model can produce both an oscillatory background and a pathological SWD rhythm. We demonstrate that coupling two of these cortical columns can lead to a bistability between out-of-phase, low amplitude background dynamics and in-phase, high amplitude SWD activity. Stimuli can cause state-dependent transitions from background into SWD. In an extended local area of cortex, spatial heterogeneities in a model parameter can lead to spontaneous reversible transitions from a desynchronised background to synchronous SWD due to intermittency. The deterministic model is therefore capable of producing absence seizure-like events without any time dependent adjustment of model parameters. The emergence of such mechanisms due to spatial coupling demonstrates the importance of spatial interactions in modelling ictal dynamics, and in the study of ictogenesis.

Neuronal precursor cell-expressed developmentally down-regulated 4-1 (NEDD4-1) controls the sorting of newly synthesized Ca(V)1.2 calcium channels

Neuronal precursor cell-expressed developmentally down-regulated 4 (Nedd4) proteins are ubiquitin ligases, which attach ubiquitin moieties to their target proteins, a post-translational modification that is most commonly associated with protein degradation. Nedd4 ubiquitin ligases have been shown to down-regulate both potassium and sodium channels. In this study, we investigated whether Nedd4 ubiquitin ligases also regulate Ca(v) calcium channels. We expressed three Nedd4 family members, Nedd4-1, Nedd4-2, and WWP2, together with Ca(v)1.2 channels in tsA-201 cells. We found that Nedd4-1 dramatically decreased Ca(v) whole-cell currents, whereas Nedd4-2 and WWP2 failed to regulate the current. Surface biotinylation assays revealed that Nedd4-1 decreased the number of channels inserted at the plasma membrane. Western blots also showed a concomitant decrease in the total expression of the channels. Surprisingly, however, neither the Ca(v) pore-forming α1 subunit nor the associated Ca(v)β and Ca(v)α(2)δ subunits were ubiquitylated by Nedd4-1. The proteasome inhibitor MG132 prevented the degradation of Ca(v) channels, whereas monodansylcadaverine and chloroquine partially antagonized the Nedd4-1-induced regulation of Ca(v) currents. Remarkably, the effect of Nedd4-1 was fully prevented by brefeldin A. These data suggest that Nedd4-1 promotes the sorting of newly synthesized Ca(v) channels for degradation by both the proteasome and the lysosome. Most importantly, Nedd4-1-induced regulation required the co-expression of Ca(v)β subunits, known to antagonize the retention of the channels in the endoplasmic reticulum. Altogether, our results suggest that Nedd4-1 interferes with the chaperon role of Ca(v)β at the endoplasmic reticulum/Golgi level to prevent the delivery of Ca(v) channels at the plasma membrane.

A Decision-Tree-Based Approach to Smoke Spike Detection in a Heavy-Duty Diesel Engine

Smoke spikes occurring during transient engine operation have detrimental health effects and increase fuel consumption by requiring more frequent regeneration of the diesel particulate filter. This paper proposes a decision tree approach to real-time detection of smoke spikes for control and on-board diagnostics purposes. A contemporary, electronically controlled heavy-duty diesel engine was used to investigate the deficiencies of smoke control based on the fuel-to-oxygen-ratio limit. With the aid of transient and steady state data analysis and empirical as well as dimensional modeling, it was shown that the fuel-to-oxygen ratio was not estimated correctly during the turbocharger lag period. This inaccuracy was attributed to the large manifold pressure ratios and low exhaust gas recirculation flows recorded during the turbocharger lag period, which meant that engine control module correlations for the exhaust gas recirculation flow and the volumetric efficiency had to be extrapolated. The engine control module correlations were based on steady state data and it was shown that, unless the turbocharger efficiency is artificially reduced, the large manifold pressure ratios observed during the turbocharger lag period cannot be achieved at steady state. Additionally, the cylinder-to-cylinder variation during this period were shown to be sufficiently significant to make the average fuel-to-oxygen ratio a poor predictor of the transient smoke emissions. The steady state data also showed higher smoke emissions with higher exhaust gas recirculation fractions at constant fuel-to-oxygen-ratio levels. This suggests that, even if the fuel-to-oxygen ratios were to be estimated accurately for each cylinder, they would still be ineffective as smoke limiters. A decision tree trained on snap throttle data and pruned with engineering knowledge was able to use the inaccurate engine control module estimates of the fuel-to-oxygen ratio together with information on the engine control module estimate of the exhaust gas recirculation fraction, the engine speed, and the manifold pressure ratio to predict 94% of all spikes occurring over the Federal Test Procedure cycle. The advantages of this non-parametric approach over other commonly used parametric empirical methods such as regression were described. An application of accurate smoke spike detection in which the injection pressure is increased at points with a high opacity to reduce the cumulative particulate matter emissions substantially with a minimum increase in the cumulative nitrogrn oxide emissions was illustrated with dimensional and empirical modeling.

Spike-triggered reaction-time EEG as a possible assessment tool for driving ability

The impact of interictal epileptic activity (IEA) on driving is a rarely investigated issue. We analyzed the impact of IEA on reaction time in a pilot study. Reactions to simple visual stimuli (light flash) in the Flash test or complex visual stimuli (obstacle on a road) in a modified car driving computer game, the Steer Clear, were measured during IEA bursts and unremarkable electroencephalography (EEG) periods. Individual epilepsy patients showed slower reaction times (RTs) during generalized IEA compared to RTs during unremarkable EEG periods. RT differences were approximately 300 ms (p < 0.001) in the Flash test and approximately 200 ms (p < 0.001) in the Steer Clear. Prior work suggested that RT differences >100 ms may become clinically relevant. This occurred in 40% of patients in the Flash test and in up to 50% in the Steer Clear. When RT were pooled, mean RT differences were 157 ms in the Flash test (p < 0.0001) and 116 ms in the Steer Clear (p < 0.0001). Generalized IEA of short duration seems to impair brain function, that is, the ability to react. The reaction-time EEG could be used routinely to assess driving ability.

Spike-based decision learning of nash equilibria in two-player games

Humans and animals face decision tasks in an uncertain multi-agent environment where an agent's strategy may change in time due to the co-adaptation of others strategies. The neuronal substrate and the computational algorithms underlying such adaptive decision making, however, is largely unknown. We propose a population coding model of spiking neurons with a policy gradient procedure that successfully acquires optimal strategies for classical game-theoretical tasks. The suggested population reinforcement learning reproduces data from human behavioral experiments for the blackjack and the inspector game. It performs optimally according to a pure (deterministic) and mixed (stochastic) Nash equilibrium, respectively. In contrast, temporal-difference(TD)-learning, covariance-learning, and basic reinforcement learning fail to perform optimally for the stochastic strategy. Spike-based population reinforcement learning, shown to follow the stochastic reward gradient, is therefore a viable candidate to explain automated decision learning of a Nash equilibrium in two-player games.

The time window for generation of dendritic spikes by coincidence of action potentials and EPSPs is layer specific in somatosensory cortex

The precise timing of events in the brain has consequences for intracellular processes, synaptic plasticity, integration and network behaviour. Pyramidal neurons, the most widespread excitatory neuron of the neocortex have multiple spike initiation zones, which interact via dendritic and somatic spikes actively propagating in all directions within the dendritic tree. For these neurons, therefore, both the location and timing of synaptic inputs are critical. The time window for which the backpropagating action potential can influence dendritic spike generation has been extensively studied in layer 5 neocortical pyramidal neurons of rat somatosensory cortex. Here, we re-examine this coincidence detection window for pyramidal cell types across the rat somatosensory cortex in layers 2/3, 5 and 6. We find that the time-window for optimal interaction is widest and shifted in layer 5 pyramidal neurons relative to cells in layers 6 and 2/3. Inputs arriving at the same time and locations will therefore differentially affect spike-timing dependent processes in the different classes of pyramidal neurons.

THE RESONATE-AND-FIRE NEURON: TIME DEPENDENT AND FREQUENCY SELECTIVE NEURONS IN NEURAL NETWORKS

The means through which the nervous system perceives its environment is one of the most fascinating questions in contemporary science. Our endeavors to comprehend the principles of neural science provide an instance of how biological processes may inspire novel methods in mathematical modeling and engineering. The application ofmathematical models towards understanding neural signals and systems represents a vibrant field of research that has spanned over half a century. During this period, multiple approaches to neuronal modeling have been adopted, and each approach is adept at elucidating a specific aspect of nervous system function. Thus while bio-physical models have strived to comprehend the dynamics of actual physical processes occurring within a nerve cell, the phenomenological approach has conceived models that relate the ionic properties of nerve cells to transitions in neural activity. Further-more, the field of neural networks has endeavored to explore how distributed parallel processing systems may become capable of storing memory. Through this project, we strive to explore how some of the insights gained from biophysical neuronal modeling may be incorporated within the field of neural net-works. We specifically study the capabilities of a simple neural model, the Resonate-and-Fire (RAF) neuron, whose derivation is inspired by biophysical neural modeling. While reflecting further biological plausibility, the RAF neuron is also analytically tractable, and thus may be implemented within neural networks. In the following thesis, we provide a brief overview of the different approaches that have been adopted towards comprehending the properties of nerve cells, along with the framework under which our specific neuron model relates to the field of neuronal modeling. Subsequently, we explore some of the time-dependent neurocomputational capabilities of the RAF neuron, and we utilize the model to classify logic gates, and solve the classic XOR problem. Finally we explore how the resonate-and-fire neuron may be implemented within neural networks, and how such a network could be adapted through the temporal backpropagation algorithm.