Spatio-temporal credit assignment in population learning

Learning by reinforcement is important in shaping animal behavior, and in particular in behavioral decision making. Such decision making is likely to involve the integration of many synaptic events in space and time. However, using a single reinforcement signal to modulate synaptic plasticity, as suggested in classical reinforcement learning algorithms, a twofold problem arises. Different synapses will have contributed differently to the behavioral decision, and even for one and the same synapse, releases at different times may have had different effects. Here we present a plasticity rule which solves this spatio-temporal credit assignment problem in a population of spiking neurons. The learning rule is spike-time dependent and maximizes the expected reward by following its stochastic gradient. Synaptic plasticity is modulated not only by the reward, but also by a population feedback signal. While this additional signal solves the spatial component of the problem, the temporal one is solved by means of synaptic eligibility traces. In contrast to temporal difference (TD) based approaches to reinforcement learning, our rule is explicit with regard to the assumed biophysical mechanisms. Neurotransmitter concentrations determine plasticity and learning occurs fully online. Further, it works even if the task to be learned is non-Markovian, i.e. when reinforcement is not determined by the current state of the system but may also depend on past events. The performance of the model is assessed by studying three non-Markovian tasks. In the first task, the reward is delayed beyond the last action with non-related stimuli and actions appearing in between. The second task involves an action sequence which is itself extended in time and reward is only delivered at the last action, as it is the case in any type of board-game. The third task is the inspection game that has been studied in neuroeconomics, where an inspector tries to prevent a worker from shirking. Applying our algorithm to this game yields a learning behavior which is consistent with behavioral data from humans and monkeys, revealing themselves properties of a mixed Nash equilibrium. The examples show that our neuronal implementation of reward based learning copes with delayed and stochastic reward delivery, and also with the learning of mixed strategies in two-opponent games.

Spatio-temporal credit assignment in population learning

Learning by reinforcement is important in shaping animal behavior. But behavioral decision making is likely to involve the integration of many synaptic events in space and time. So in using a single reinforcement signal to modulate synaptic plasticity a twofold problem arises. Different synapses will have contributed differently to the behavioral decision and, even for one and the same synapse, releases at different times may have had different effects. Here we present a plasticity rule which solves this spatio-temporal credit assignment problem in a population of spiking neurons. The learning rule is spike time dependent and maximizes the expected reward by following its stochastic gradient. Synaptic plasticity is modulated not only by the reward but by a population feedback signal as well. While this additional signal solves the spatial component of the problem, the temporal one is solved by means of synaptic eligibility traces. In contrast to temporal difference based approaches to reinforcement learning, our rule is explicit with regard to the assumed biophysical mechanisms. Neurotransmitter concentrations determine plasticity and learning occurs fully online. Further, it works even if the task to be learned is non-Markovian, i.e. when reinforcement is not determined by the current state of the system but may also depend on past events. The performance of the model is assessed by studying three non-Markovian tasks. In the first task the reward is delayed beyond the last action with non-related stimuli and actions appearing in between. The second one involves an action sequence which is itself extended in time and reward is only delivered at the last action, as is the case in any type of board-game. The third is the inspection game that has been studied in neuroeconomics. It only has a mixed Nash equilibrium and exemplifies that the model also copes with stochastic reward delivery and the learning of mixed strategies.

Multiple time scales of temporal response in pyramidal and fast spiking cortical neurons

Neural dynamic processes correlated over several time scales are found in vivo, in stimulus-evoked as well as spontaneous activity, and are thought to affect the way sensory stimulation is processed. Despite their potential computational consequences, a systematic description of the presence of multiple time scales in single cortical neurons is lacking. In this study, we injected fast spiking and pyramidal (PYR) neurons in vitro with long-lasting episodes of step-like and noisy, in-vivo-like current. Several processes shaped the time course of the instantaneous spike frequency, which could be reduced to a small number (1-4) of phenomenological mechanisms, either reducing (adapting) or increasing (facilitating) the neuron's firing rate over time. The different adaptation/facilitation processes cover a wide range of time scales, ranging from initial adaptation (<10 ms, PYR neurons only), to fast adaptation (<300 ms), early facilitation (0.5-1 s, PYR only), and slow (or late) adaptation (order of seconds). These processes are characterized by broad distributions of their magnitudes and time constants across cells, showing that multiple time scales are at play in cortical neurons, even in response to stationary stimuli and in the presence of input fluctuations. These processes might be part of a cascade of processes responsible for the power-law behavior of adaptation observed in several preparations, and may have far-reaching computational consequences that have been recently described.

Brain areas involved in medial temporal lobe seizures: a principal component analysis of ictal SPECT data

The study describes brain areas involved in medial temporal lobe (mTL) seizures of 12 patients. All patients showed so-called oro-alimentary behavior within the first 20 s of clinical seizure manifestation characteristic of mTL seizures. Single photon emission computed tomography (SPECT) images of regional cerebral blood flow (rCBF) were acquired from the patients in ictal and interictal phases and from normal volunteers. Image analysis employed categorical comparisons with statistical parametric mapping and principal component analysis (PCA) to assess functional connectivity. PCA supplemented the findings of the categorical analysis by decomposing the covariance matrix containing images of patients and healthy subjects into distinct component images of independent variance, including areas not identified by the categorical analysis. Two principal components (PCs) discriminated the subject groups: patients with right or left mTL seizures and normal volunteers, indicating distinct neuronal networks implicated by the seizure. Both PCs were correlated with seizure duration, one positively and the other negatively, confirming their physiological significance. The independence of the two PCs yielded a clear clustering of subject groups. The local pattern within the temporal lobe describes critical relay nodes which are the counterpart of oro-alimentary behavior: (1) right mesial temporal zone and ipsilateral anterior insula in right mTL seizures, and (2) temporal poles on both sides that are densely interconnected by the anterior commissure. Regions remote from the temporal lobe may be related to seizure propagation and include positively and negatively loaded areas. These patterns, the covarying areas of the temporal pole and occipito-basal visual association cortices, for example, are related to known anatomic paths.

Processing of temporal unpredictability in human and animal amygdala

The amygdala has been studied extensively for its critical role in associative fear conditioning in animals and humans. Noxious stimuli, such as those used for fear conditioning, are most effective in eliciting behavioral responses and amygdala activation when experienced in an unpredictable manner. Here, we show, using a translational approach in mice and humans, that unpredictability per se without interaction with motivational information is sufficient to induce sustained neural activity in the amygdala and to elicit anxiety-like behavior. Exposing mice to mere temporal unpredictability within a time series of neutral sound pulses in an otherwise neutral sensory environment increased expression of the immediate-early gene c-fos and prevented rapid habituation of single neuron activity in the basolateral amygdala. At the behavioral level, unpredictable, but not predictable, auditory stimulation induced avoidance and anxiety-like behavior. In humans, functional magnetic resonance imaging revealed that temporal unpredictably causes sustained neural activity in amygdala and anxiety-like behavior as quantified by enhanced attention toward emotional faces. Our findings show that unpredictability per se is an important feature of the sensory environment influencing habituation of neuronal activity in amygdala and emotional behavior and indicate that regulation of amygdala habituation represents an evolutionary-conserved mechanism for adapting behavior in anticipation of temporally unpredictable events.

Differential effect of side of temporal lobe epilepsy on lateralization of hippocampal, temporolateral, and inferior frontal activation patterns during a verbal episodic memory task

The encoding of verbal stimuli elicits left-lateralized activation patterns within the medial temporal lobes in healthy adults. In our study, patients with left- and right-sided temporal lobe epilepsy (LTLE, RTLE) were investigated during the encoding and retrieval of word-pair associates using functional magnetic resonance imaging. Functional asymmetry of activation patterns in hippocampal, inferior frontal, and temporolateral neocortical areas associated with language functions was analyzed. Hippocampal activation patterns in patients with LTLE were more right-lateralized than those in patients with RTLE (P<0.05). There were no group differences with respect to lateralization in frontal or temporolateral regions of interest (ROIs). For both groups, frontal cortical activation patterns were significantly more left-lateralized than hippocampal patterns (P<0.05). For patients with LTLE, there was a strong trend toward a difference in functional asymmetry between the temporolateral and hippocampal ROIs (P=0.059). A graded effect of epileptic activity on laterality of the different regional activation patterns is discussed.

Disorganization and timing of motor behavior : insight from gesture impairments and movement patterns in schizophrenia

Background: Motor symptoms are frequent phenomena across the entire course of schizophrenia1. Some have argued that disorganized behavior was associated with aberrant motor behavior. We have studied the association of motor disturbances and disorganization in two projects focusing on the timing of movements. Method: In two studies, we assessed motor behavior and psychopathology. The first study applied a validated test of upper limb apraxia in 30 schizophrenia patients2,3. We used standardized video assessments of hand gestures by a blinded rater. The second study tested the stability of movement patterns using time series analysis in actigraphy data of 100 schizophrenia patients4. Both stability of movement patterns and the overall amount of movement were calculated from data of two hours with high degrees of social interaction comparable across the 100 subjects. Results: In total, 67% of the patients had gesture performance deficits3. Most frequently, they made spatial, temporal and body-part-as-object errors. Gesture performance relied on frontal lobe function2. Poor gesture performance was associated with increased disorganization scores. In the second study, we found disorganization to be predicted only by more irregular movement patterns irrespective of the overall amount of movement4. Conclusion : Both studies provide evidence for a link between aberrant timing of motor behavior and disorganization. Disturbed movement control seems critical for disorganized behavior in schizophrenia.

Calling and career preparation: Investigating developmental patterns and temporal precedence

The presence of a calling and career development are assumed to be closely related. However, the nature of and reason for this relationship has not been thoroughly investigated. We hypothesized the existence of reciprocal effects between calling and three dimensions of career preparation and assessed the change of the presence of a calling, career planning, decidedness, and self-efficacy with three waves of a diverse sample of German university students (N = 846) over one year. Latent growth analyses revealed that the intercepts of calling showed a siginficant positive correlation with the intercepts of all career preparation measures. The slope of calling was positively related to those of decidedness and self-efficacy but not to planning. Cross-lagged analyses showed that calling predicted a subsequent increase in planning and self-efficacy. Planning and decidedness predicted an incrrease in the presence of a calling. The results suggest that calling and career preparation are related due to mutual effects but that effects differ for different career preparation dimensions.

Repeated electrical stimulations as a tool to evoke temporal summation of nociceptive inputs in healthy, non-medicated experimental sheep.

The nociceptive withdrawal reflex (NWR) model is used in animal pain research to quantify nociception. The aim of this study was to evaluate the NWR evoked by repeated stimulations in healthy, non-medicated standing sheep. Repeated electrical stimulations were applied at 5Hz for 2s to the digital nerves of the right thoracic and the pelvic limbs of 25 standing sheep. The stimulation intensities applied were fractions (0.5, 0.6, 0.7, 0.8, 0.9 and 1) of the individual previously determined nociceptive threshold (It) after single stimulation. Surface-electromyographic activity (EMG) was recorded from the deltoid, the femoral biceps or the peroneus tertius muscles. The repeated stimulation threshold (RS It) was reached if at least one stimulus in the train was followed by a reflex with a minimal root-mean-square-amplitude (RMSA) of 20μV. The behavioural reaction following each series of stimulations was scored on a scale from 0 (no reaction) to 5 (vigorous whole-body reaction). For the deltoid muscle, RS It was 2.3mA (1.6-3mA) with a reaction score of 2 (1-2) and at a fraction of 0.6 (0.5-0.8)×It. For the biceps femoris muscle, RS It was 2.9mA (2.6-4mA) with a reaction score of 1 (1-2) at a fraction of and 0.55 (0.4-0.7)×It while for the peroneus tertius muscle RS It was 3mA (2.8-3.5mA) with a reaction score of 1 (1-2) and at a fraction of 0.8 (0.8-0.95)×It. Both, RMSA and reaction scores increased significantly with increasing stimulation intensities in all muscles (p<0.001). The repeated application of electrical stimuli led to temporal summation of nociceptive inputs and therefore a reduction of the stimulus intensity evoking a withdrawal reaction in healthy, standing sheep. Data achieved in this study can now serve as reference for further clinical or experimental applications of the model in this species.

Dynamic directed interictal connectivity in left and right temporal lobe epilepsy.

OBJECTIVE There is increasing evidence that epileptic activity involves widespread brain networks rather than single sources and that these networks contribute to interictal brain dysfunction. We investigated the fast-varying behavior of epileptic networks during interictal spikes in right and left temporal lobe epilepsy (RTLE and LTLE) at a whole-brain scale using directed connectivity. METHODS In 16 patients, 8 with LTLE and 8 with RTLE, we estimated the electrical source activity in 82 cortical regions of interest (ROIs) using high-density electroencephalography (EEG), individual head models, and a distributed linear inverse solution. A multivariate, time-varying, and frequency-resolved Granger-causal modeling (weighted Partial Directed Coherence) was applied to the source signal of all ROIs. A nonparametric statistical test assessed differences between spike and baseline epochs. Connectivity results between RTLE and LTLE were compared between RTLE and LTLE and with neuropsychological impairments. RESULTS Ipsilateral anterior temporal structures were identified as key drivers for both groups, concordant with the epileptogenic zone estimated invasively. We observed an increase in outflow from the key driver already before the spike. There were also important temporal and extratemporal ipsilateral drivers in both conditions, and contralateral only in RTLE. A different network pattern between LTLE and RTLE was found: in RTLE there was a much more prominent ipsilateral to contralateral pattern than in LTLE. Half of the RTLE patients but none of the LTLE patients had neuropsychological deficits consistent with contralateral temporal lobe dysfunction, suggesting a relationship between connectivity changes and cognitive deficits. SIGNIFICANCE The different patterns of time-varying connectivity in LTLE and RTLE suggest that they are not symmetrical entities, in line with our neuropsychological results. The highest outflow region was concordant with invasive validation of the epileptogenic zone. This enhanced characterization of dynamic connectivity patterns could better explain cognitive deficits and help the management of epilepsy surgery candidates.