Reinforcement learning in dendritic structures

The discovery of binary dendritic events such as local NMDA spikes in dendritic subbranches led to the suggestion that dendritic trees could be computationally equivalent to a 2-layer network of point neurons, with a single output unit represented by the soma, and input units represented by the dendritic branches. Although this interpretation endows a neuron with a high computational power, it is functionally not clear why nature would have preferred the dendritic solution with a single but complex neuron, as opposed to the network solution with many but simple units. We show that the dendritic solution has a distinguished advantage over the network solution when considering different learning tasks. Its key property is that the dendritic branches receive an immediate feedback from the somatic output spike, while in the corresponding network architecture the feedback would require additional backpropagating connections to the input units. Assuming a reinforcement learning scenario we formally derive a learning rule for the synaptic contacts on the individual dendritic trees which depends on the presynaptic activity, the local NMDA spikes, the somatic action potential, and a delayed reinforcement signal. We test the model for two scenarios: the learning of binary classifications and of precise spike timings. We show that the immediate feedback represented by the backpropagating action potential supplies the individual dendritic branches with enough information to efficiently adapt their synapses and to speed up the learning process.

Reinforcement learning in dendritic structures

The discovery of binary dendritic events such as local NMDA spikes in dendritic subbranches led to the suggestion that dendritic trees could be computationally equivalent to a 2-layer network of point neurons, with a single output unit represented by the soma, and input units represented by the dendritic branches. Although this interpretation endows a neuron with a high computational power, it is functionally not clear why nature would have preferred the dendritic solution with a single but complex neuron, as opposed to the network solution with many but simple units. We show that the dendritic solution has a distinguished advantage over the network solution when considering different learning tasks. Its key property is that the dendritic branches receive an immediate feedback from the somatic output spike, while in the corresponding network architecture the feedback would require additional backpropagating connections to the input units. Assuming a reinforcement learning scenario we formally derive a learning rule for the synaptic contacts on the individual dendritic trees which depends on the presynaptic activity, the local NMDA spikes, the somatic action potential, and a delayed reinforcement signal. We test the model for two scenarios: the learning of binary classifications and of precise spike timings. We show that the immediate feedback represented by the backpropagating action potential supplies the individual dendritic branches with enough information to efficiently adapt their synapses and to speed up the learning process.

Dual-energy multidetector CT: how does it work, what can it tell us, and when can we use it in abdominopelvic imaging?

Dual-energy CT provides information about how substances behave at different energies, the ability to generate virtual unenhanced datasets, and improved detection of iodine-containing substances on low-energy images. Knowing how a substance behaves at two different energies can provide information about tissue composition beyond that obtainable with single-energy techniques. The term K edge refers to the spike in attenuation that occurs at energy levels just greater than that of the K-shell binding because of the increased photoelectric absorption at these energy levels. K-edge values vary for each element, and they increase as the atomic number increases. The energy dependence of the photoelectric effect and the variability of K edges form the basis of dual-energy techniques, which may be used to detect substances such as iodine, calcium, and uric acid crystals. The closer the energy level used in imaging is to the K edge of a substance such as iodine, the more the substance attenuates. In the abdomen and pelvis, dual-energy CT may be used in the liver to increase conspicuity of hypervascular lesions; in the kidneys, to distinguish hyperattenuating cysts from enhancing renal masses and to characterize renal stone composition; in the adrenal glands, to characterize adrenal nodules; and in the pancreas, to differentiate between normal and abnormal parenchyma.

Reduction of Transient Particulate Matter Spikes with Decision Tree Based Control

Decision trees have been proposed as a basis for modifying table based injection to reduce transient particulate spikes during the turbocharger lag period. It has been shown that decision trees can detect particulate spikes in real time. In well calibrated electronically controlled diesel engines these spikes are narrow and are encompassed by a wider NOx spike. Decision trees have been shown to pinpoint the exact location of measured opacity spikes in real time thus enabling targeted PM reduction with near zero NOx penalty. A calibrated dimensional model has been used to demonstrate the possible reduction of particulate matter with targeted injection pressure pulses. Post injection strategy optimized for near stoichiometric combustion has been shown to provide additional benefits. Empirical models have been used to calculate emission tradeoffs over the entire FTP cycle. An empirical model based transient calibration has been used to demonstrate that such targeted transient modifiers are more beneficial at lower engine-out NOx levels.

The time window for generation of dendritic spikes by coincidence of action potentials and EPSPs is layer specific in somatosensory cortex

The precise timing of events in the brain has consequences for intracellular processes, synaptic plasticity, integration and network behaviour. Pyramidal neurons, the most widespread excitatory neuron of the neocortex have multiple spike initiation zones, which interact via dendritic and somatic spikes actively propagating in all directions within the dendritic tree. For these neurons, therefore, both the location and timing of synaptic inputs are critical. The time window for which the backpropagating action potential can influence dendritic spike generation has been extensively studied in layer 5 neocortical pyramidal neurons of rat somatosensory cortex. Here, we re-examine this coincidence detection window for pyramidal cell types across the rat somatosensory cortex in layers 2/3, 5 and 6. We find that the time-window for optimal interaction is widest and shifted in layer 5 pyramidal neurons relative to cells in layers 6 and 2/3. Inputs arriving at the same time and locations will therefore differentially affect spike-timing dependent processes in the different classes of pyramidal neurons.

Gradient estimation in dendritic reinforcement learning

We study synaptic plasticity in a complex neuronal cell model where NMDA-spikes can arise in certain dendritic zones. In the context of reinforcement learning, two kinds of plasticity rules are derived, zone reinforcement (ZR) and cell reinforcement (CR), which both optimize the expected reward by stochastic gradient ascent. For ZR, the synaptic plasticity response to the external reward signal is modulated exclusively by quantities which are local to the NMDA-spike initiation zone in which the synapse is situated. CR, in addition, uses nonlocal feedback from the soma of the cell, provided by mechanisms such as the backpropagating action potential. Simulation results show that, compared to ZR, the use of nonlocal feedback in CR can drastically enhance learning performance. We suggest that the availability of nonlocal feedback for learning is a key advantage of complex neurons over networks of simple point neurons, which have previously been found to be largely equivalent with regard to computational capability.

Full Cups, Full Coffers: Tax Strategies and Consumer Culture in the Early Modern German Cities

During the sixteenth and seventeenth centuries, the excise taxes (Ungeld) paid by town residents on the consumption of beer, wine, mead and brandy represented the single most important source of civic revenue for many German cities. In a crisis, these taxes could spike to 70-80% of civic income. This paper examines civic budgets and 'behind-the-scenes' deliberations in a sample of towns in southern Germany in order to illuminate how decisions affecting consumer taxes were made. Even during the sobriety movements of the Reformation and post-Reformation period, tax income from drinkers remained attractive to city leaders because the bulk of the excise tax burden could easily be shifted away from privileged members of society and placed on the population at large. At the same time, governments had to maintain a careful balance between what they needed in order to govern and what the consumer market could bear, for high taxes on drinks were also targeted in many popular revolts. This led to nimble politicking by those responsible for tax decisions. Drink taxes were introduced, raised, lowered and otherwise manipulated based not only on shifting fashions and tastes but also on the degree of economic stress faced by the community. Where civic rulers were successful in striking the right balance, the rewards were considerable. The income from drink sales was a major factor in how the cities of the Empire survived the wars and other crises of the early modern period without going into so much debt that they lost their independence.

Full Cups, Full Coffers: Tax Strategies and Consumer Culture in the Early Modern German Cities

During the sixteenth and seventeenth centuries, the excise taxes (Ungeld) paid by town residents on the consumption of beer, wine, mead and brandy represented the single most important source of civic revenue for many German cities. In a crisis, these taxes could spike to 70–80% of civic income. This paper examines civic budgets and ‘behind-the-scenes’ deliberations in a sample of towns in southern Germany in order to illuminate how decisions affecting consumer taxes were made. Even during the sobriety movements of the Reformation and post-Reformation period, tax income from drinkers remained attractive to city leaders because the bulk of the excise tax burden could easily be shifted away from privileged members of society and placed on the population at large. At the same time, governments had to maintain a careful balance between what they needed in order to govern and what the consumer market could bear, for high taxes on drinks were also targeted in many popular revolts. This led to nimble politicking by those responsible for tax decisions. Drink taxes were introduced, raised, lowered and otherwise manipulated based not only on shifting fashions and tastes but also on the degree of economic stress faced by the community. Where civic rulers were successful in striking the right balance, the rewards were considerable. The income from drink sales was a major factor in how the cities of the Empire survived the wars and other crises of the early modern period without going into so much debt that they lost their independence.

The Carrington event not observed in most ice core nitrate records

The Carrington Event of 1859 is considered to be among the largest space weather events of the last 150 years. We show that only one out of 14 well-resolved ice core records from Greenland and Antarctica has a nitrate spike dated to 1859. No sharp spikes are observed in the Antarctic cores studied here. In Greenland numerous spikes are observed in the 40 years surrounding 1859, but where other chemistry was measured, all large spikes have the unequivocal signal, including co-located spikes in ammonium, formate, black carbon and vanillic acid, of biomass burning plumes. It seems certain that most spikes in an earlier core, including that claimed for 1859, are also due to biomass burning plumes, and not to solar energetic particle (SEP) events. We conclude that an event as large as the Carrington Event did not leave an observable, widespread imprint in nitrate in polar ice. Nitrate spikes cannot be used to derive the statistics of SEPs.

The GABAB1b isoform mediates long-lasting inhibition of dendritic Ca2+ spikes in layer 5 somatosensory pyramidal neurons

The apical tuft of layer 5 pyramidal neurons is innervated by a large number of inhibitory inputs with unknown functions. Here, we studied the functional consequences and underlying molecular mechanisms of apical inhibition on dendritic spike activity. Extracellular stimulation of layer 1, during blockade of glutamatergic transmission, inhibited the dendritic Ca2+ spike for up to 400 ms. Activation of metabotropic GABAB receptors was responsible for a gradual and long-lasting inhibitory effect, whereas GABAA receptors mediated a short-lasting (approximately 150 ms) inhibition. Our results suggest that the mechanism underlying the GABAB inhibition of Ca2+ spikes involves direct blockade of dendritic Ca2+ channels. By using knockout mice for the two predominant GABAB1 isoforms, GABAB1a and GABAB1b, we showed that postsynaptic inhibition of Ca2+ spikes is mediated by GABAB1b, whereas presynaptic inhibition of GABA release is mediated by GABAB1a. We conclude that the molecular subtypes of GABAB receptors play strategically different physiological roles in neocortical neurons.

Multiple time scales of temporal response in pyramidal and fast spiking cortical neurons

Neural dynamic processes correlated over several time scales are found in vivo, in stimulus-evoked as well as spontaneous activity, and are thought to affect the way sensory stimulation is processed. Despite their potential computational consequences, a systematic description of the presence of multiple time scales in single cortical neurons is lacking. In this study, we injected fast spiking and pyramidal (PYR) neurons in vitro with long-lasting episodes of step-like and noisy, in-vivo-like current. Several processes shaped the time course of the instantaneous spike frequency, which could be reduced to a small number (1-4) of phenomenological mechanisms, either reducing (adapting) or increasing (facilitating) the neuron's firing rate over time. The different adaptation/facilitation processes cover a wide range of time scales, ranging from initial adaptation (<10 ms, PYR neurons only), to fast adaptation (<300 ms), early facilitation (0.5-1 s, PYR only), and slow (or late) adaptation (order of seconds). These processes are characterized by broad distributions of their magnitudes and time constants across cells, showing that multiple time scales are at play in cortical neurons, even in response to stationary stimuli and in the presence of input fluctuations. These processes might be part of a cascade of processes responsible for the power-law behavior of adaptation observed in several preparations, and may have far-reaching computational consequences that have been recently described.

Eluding oblivion with smart stochastic selection of synaptic updates

The variables involved in the equations that describe realistic synaptic dynamics always vary in a limited range. Their boundedness makes the synapses forgetful, not for the mere passage of time, but because new experiences overwrite old memories. The forgetting rate depends on how many synapses are modified by each new experience: many changes means fast learning and fast forgetting, whereas few changes means slow learning and long memory retention. Reducing the average number of modified synapses can extend the memory span at the price of a reduced amount of information stored when a new experience is memorized. Every trick which allows to slow down the learning process in a smart way can improve the memory performance. We review some of the tricks that allow to elude fast forgetting (oblivion). They are based on the stochastic selection of the synapses whose modifications are actually consolidated following each new experience. In practice only a randomly selected, small fraction of the synapses eligible for an update are actually modified. This allows to acquire the amount of information necessary to retrieve the memory without compromising the retention of old experiences. The fraction of modified synapses can be further reduced in a smart way by changing synapses only when it is really necessary, i.e. when the post-synaptic neuron does not respond as desired. Finally we show that such a stochastic selection emerges naturally from spike driven synaptic dynamics which read noisy pre and post-synaptic neural activities. These activities can actually be generated by a chaotic system.

A multivariate approach for processing magnetization effects in triggered event-related functional magnetic resonance imaging time series

Triggered event-related functional magnetic resonance imaging requires sparse intervals of temporally resolved functional data acquisitions, whose initiation corresponds to the occurrence of an event, typically an epileptic spike in the electroencephalographic trace. However, conventional fMRI time series are greatly affected by non-steady-state magnetization effects, which obscure initial blood oxygen level-dependent (BOLD) signals. Here, conventional echo-planar imaging and a post-processing solution based on principal component analysis were employed to remove the dominant eigenimages of the time series, to filter out the global signal changes induced by magnetization decay and to recover BOLD signals starting with the first functional volume. This approach was compared with a physical solution using radiofrequency preparation, which nullifies magnetization effects. As an application of the method, the detectability of the initial transient BOLD response in the auditory cortex, which is elicited by the onset of acoustic scanner noise, was used to demonstrate that post-processing-based removal of magnetization effects allows to detect brain activity patterns identical with those obtained using the radiofrequency preparation. Using the auditory responses as an ideal experimental model of triggered brain activity, our results suggest that reducing the initial magnetization effects by removing a few principal components from fMRI data may be potentially useful in the analysis of triggered event-related echo-planar time series. The implications of this study are discussed with special caution to remaining technical limitations and the additional neurophysiological issues of the triggered acquisition.

Riluzole-induced oscillations in spinal networks

We previously showed in dissociated cultures of fetal rat spinal cord that disinhibition-induced bursting is based on intrinsic spiking, network recruitment, and a network refractory period after the bursts. A persistent sodium current (I(NaP)) underlies intrinsic spiking, which, by recurrent excitation, generates the bursting activity. Although full blockade of I(NaP) with riluzole disrupts such bursting, the present study shows that partial blockade of I(NaP) with low doses of riluzole maintains bursting activity with unchanged burst rate and burst duration. More important, low doses of riluzole turned bursts composed of persistent activity into bursts composed of oscillatory activity at around 5 Hz. In a search for the mechanisms underlying the generation of such intraburst oscillations, we found that activity-dependent synaptic depression was not changed with low doses of riluzole. On the other hand, low doses of riluzole strongly increased spike-frequency adaptation and led to early depolarization block when bursts were simulated by injecting long current pulses into single neurons in the absence of fast synaptic transmission. Phenytoin is another I(NaP) blocker. When applied in doses that reduced intrinsic activity by 80-90%, as did low doses of riluzole, it had no effect either on spike-frequency adaptation or on depolarization block. Nor did phenytoin induce intraburst oscillations after disinhibition. A theoretical model incorporating a depolarization block mechanism could reproduce the generation of intraburst oscillations at the network level. From these findings we conclude that riluzole-induced intraburst oscillations are a network-driven phenomenon whose major accommodation mechanism is depolarization block arising from strong sodium channel inactivation.

BOLD correlates of continuously fluctuating epileptic activity isolated by independent component analysis

Combined EEG/fMRI recordings offer a promising opportunity to detect brain areas with altered BOLD signal during interictal epileptic discharges (IEDs). These areas are likely to represent the irritative zone, which is itself a reflection of the epileptogenic zone. This paper reports on the imaging findings using independent component analysis (ICA) to continuously quantify epileptiform activity in simultaneously acquired EEG and fMRI. Using ICA derived factors coding for the epileptic activity takes into account that epileptic activity is continuously fluctuating with each spike differing in amplitude, duration and maybe topography, including subthreshold epileptic activity besides clear IEDs and may thus increase the sensitivity and statistical power of combined EEG/fMRI in epilepsy. Twenty patients with different types of focal and generalized epilepsy syndromes were investigated. ICA separated epileptiform activity from normal physiological brain activity and artifacts. In 16/20 patients, BOLD correlates of epileptic activity matched the EEG sources, the clinical semiology, and, if present, the structural lesions. In clinically equivocal cases, the BOLD correlates aided to attribute proper diagnosis of the underlying epilepsy syndrome. Furthermore, in one patient with temporal lobe epilepsy, BOLD correlates of rhythmic delta activity could be employed to delineate the affected hippocampus. Compared to BOLD correlates of manually identified IEDs, the sensitivity was improved from 50% (10/20) to 80%. The ICA EEG/fMRI approach is a safe, non-invasive and easily applicable technique, which can be used to identify regions with altered hemodynamic effects related to IEDs as well as intermittent rhythmic discharges in different types of epilepsy.