Learning Spike-Based Population Codes by Reward and Population Feedback

We investigate a recently proposed model for decision learning in a population of spiking neurons where synaptic plasticity is modulated by a population signal in addition to reward feedback. For the basic model, binary population decision making based on spike/no-spike coding, a detailed computational analysis is given about how learning performance depends on population size and task complexity. Next, we extend the basic model to n-ary decision making and show that it can also be used in conjunction with other population codes such as rate or even latency coding.

Dopaminergic modulation of the human reward system: a placebo-controlled dopamine depletion fMRI study

Reward related behaviour is linked to dopaminergic neurotransmission. Our aim was to gain insight into dopaminergic involvement in the human reward system. Combining functional magnetic resonance imaging with dopaminergic depletion by α-methylparatyrosine we measured dopamine-related brain activity in 10 healthy volunteers. In addition to blood-oxygen-level-dependent (BOLD) contrast we assessed the effect of dopaminergic depletion on prolactin response, peripheral markers for dopamine and norepinephrine. In the placebo condition we found increased activation in the left caudate and left cingulate gyrus during anticipation of reward. In the α-methylparatyrosine condition there was no significant brain activation during anticipation of reward or loss. In α-methylparatyrosine, anticipation of reward vs. loss increased activation in the right insula, left frontal, right parietal cortices and right cingulate gyrus. Comparing placebo versus α-methylparatyrosine showed increased activation in the left cingulate gyrus during anticipation of reward and the left medial frontal gyrus during anticipation of loss. α-methylparatyrosine reduced levels of dopamine in urine and homovanillic acid in plasma and increased prolactin. No significant effect of α-methylparatyrosine was found on norepinephrine markers. Our findings implicate distinct patterns of BOLD underlying reward processing following dopamine depletion, suggesting a role of dopaminergic neurotransmission for anticipation of monetary reward.

Dopamine-related deficit in reward learning after catecholamine depletion in unmedicated, remitted subjects with bulimia nervosa

Disturbances in reward processing have been implicated in bulimia nervosa (BN). Abnormalities in processing reward-related stimuli might be linked to dysfunctions of the catecholaminergic neurotransmitter system, but findings have been inconclusive. A powerful way to investigate the relationship between catecholaminergic function and behavior is to examine behavioral changes in response to experimental catecholamine depletion (CD). The purpose of this study was to uncover putative catecholaminergic dysfunction in remitted subjects with BN who performed a reinforcement-learning task after CD. CD was achieved by oral alpha-methyl-para-tyrosine (AMPT) in 19 unmedicated female subjects with remitted BN (rBN) and 28 demographically matched healthy female controls (HC). Sham depletion administered identical capsules containing diphenhydramine. The study design consisted of a randomized, double-blind, placebo-controlled crossover, single-site experimental trial. The main outcome measures were reward learning in a probabilistic reward task analyzed using signal-detection theory. Secondary outcome measures included self-report assessments, including the Eating Disorder Examination-Questionnaire. Relative to healthy controls, rBN subjects were characterized by blunted reward learning in the AMPT-but not in placebo-condition. Highlighting the specificity of these findings, groups did not differ in their ability to perceptually distinguish between stimuli. Increased CD-induced anhedonic (but not eating disorder) symptoms were associated with a reduced response bias toward a more frequently rewarded stimulus. In conclusion, under CD, rBN subjects showed reduced reward learning compared with healthy control subjects. These deficits uncover disturbance of the central reward processing systems in rBN related to altered brain catecholamine levels, which might reflect a trait-like deficit increasing vulnerability to BN.

Overcommitment but not effort-reward imbalance relates to stress-induced coagulation changes in teachers

BACKGROUND: Stress-related hypercoagulability might link job stress with atherosclerosis. PURPOSE: This paper aims to study whether overcommitment, effort-reward imbalance, and the overcommitment by effort-reward imbalance interaction relate to an exaggerated procoagulant stress response. METHODS: We assessed job stress in 52 healthy teachers (49 +/- 8 years, 63% women) at study entry and, after a mean follow-up of 21 +/- 4 months, when they underwent an acute psychosocial stressor and had coagulation measures determined in plasma. In order to increase the reliability of job stress measures, entry and follow-up scores of overcommitment and of effort-reward imbalance were added up to total scores. RESULTS: During recovery from stress, elevated overcommitment correlated with D-dimer increase and with smaller fibrinogen decrease. In contrast, overcommitment was not associated with coagulation changes from pre-stress to immediately post-stress. Effort-reward imbalance and the interaction between overcommitment and effort-reward imbalance did not correlate with stress-induced changes in coagulation measures. CONCLUSIONS: Overcommitment predicted acute stress-induced hypercoagulability, particularly during the recovery period.

White matter pathway organization of the reward system is related to positive and negative symptoms in schizophrenia

The reward systemin schizophrenia has been linked to the emergence of delusions on the one hand and to negative symptoms such as affective flattening on the other hand. Previous Diffusion Tensor Imaging (DTI) studies reported white matter microstructure alterations of regions related to the reward system. The present study aimed at extending these findings by specifically investigating connection pathways of the reward system in schizophrenia. Therefore, 24 patients with schizophrenia and 22 healthy controls matched for age and gender underwent DTI-scans. Using a probabilistic fiber tracking approachwe bilaterally extracted pathways connecting the ventral tegmental area (VTA) with the nucleus accumbens (NAcc), themedial and lateral orbitofrontal cortices (mOFC, lOFC), the dorsolateral prefrontal cortex (dlPFC) and the amygdala; as well as pathways connecting NAcc with mOFC, lOFC, dlPFC and amygdala resulting in a total of 18 connections. Probability indices forming part of a bundle of interest (PIBI) were compared between groups using independent t-tests. In 6 connection pathways PIBI-valueswere increased in schizophrenia. In 3 of these pathways the spatial extension of connection pathways was decreased. In schizophrenia patients, there was a negative correlation of PIBI-values and PANSS negative scores in the left VTA–amygdala and in the left NAcc–mOFC connection. A sum score of delusions and hallucinations correlated positively with PIBI-values of the left amygdala–NAcc connection. Structural organization of specific segments ofwhite matter pathways of the reward systemin schizophrenia may contribute to the emergence of delusions and negative symptoms in schizophrenia.

Dopaminergic modulation of the reward system in schizophrenia: A placebo-controlled dopamine depletion fMRI study

Background The brain reward circuitry innervated by dopamine is critically disturbed in schizophrenia. This study aims to investigate the role of dopamine-related brain activity during prediction of monetary reward and loss in first episode schizophrenia patients. Methods We measured blood–oxygen-level dependent (BOLD) activity in 10 patients with schizophrenia (SCH) and 12 healthy controls during dopamine depletion with α-methylparatyrosine (AMPT) and during a placebo condition (PLA). Results AMPT reduced the activation of striatal and cortical brain regions in SCH. In SCH vs. controls reduced activation was found in the AMPT condition in several regions during anticipation of reward and loss, including areas of the striatum and frontal cortex. In SCH vs. controls reduced activation of the superior temporal gyrus and posterior cingulate was observed in PLA during anticipation of rewarding stimuli. PLA patients had reduced activation in the ventral striatum, frontal and cingulate cortex in anticipation of loss. The findings of reduced dopamine-related brain activity during AMPT were verified by reduced levels of dopamine in urine, homovanillic-acid in plasma and increased prolactin levels. Conclusions Our results indicate that dopamine depletion affects functioning of the cortico-striatal reward circuitry in SCH. The findings also suggest that neuronal functions associated with dopamine neurotransmission and attribution of salience to reward predicting stimuli are altered in schizophrenia.

The Role of BDNF, Leptin and Catecholamines in Reward Learning in Bulimia Nervosa

Background: A relationship between bulimia nervosa (BN) and reward-related behavior is supported by several lines of evidence. The dopaminergic dysfunctions in the processing of reward-related stimuli have been shown to be modulated by the neurotrophin brain derived neurotrophic factor (BDNF) and the hormone leptin. Methods: Using a randomized, double-blind, placebo-controlled, crossover design, a reward learning task was applied to study the behavior of 20 female subjects with remitted BN (rBN) and 27 female healthy controls under placebo and catecholamine depletion with alpha-methyl-para-tyrosine (AMPT). The plasma levels of BDNF and leptin were measured twice during the placebo and the AMPT condition, immediately before and 1 h after a standardized breakfast. Results: AMPT-induced differences in plasma BDNF levels were positively correlated with the AMPT-induced differences in reward learning in the whole sample (p = 0.05). Across conditions, plasma BDNF levels were higher in rBN subjects compared to controls (diagnosis effect; p = 0.001). Plasma BDNF and leptin levels were higher in the morning before compared to after a standardized breakfast across groups and conditions (time effect; p < 0.0001). The plasma leptin levels were higher under catecholamine depletion compared to placebo in the whole sample (treatment effect; p = 0.0004). Conclusions: This study reports on preliminary findings that suggest a catecholamine-dependent association of plasma BDNF and reward learning in subjects with rBN and controls. A role of leptin in reward learning is not supported by this study. However, leptin levels were sensitive to a depletion of catecholamine stores in both rBN and controls.

EEG marker of inhibitory brain activity correlates with resting-state cerebral blood flow in the reward system in major depression

Frontal alpha band asymmetry (FAA) is a marker of altered reward processing in major depressive disorder (MDD), associated with reduced approach behavior and withdrawal. However, its association with brain metabolism remains unclear. The aim of this study is to investigate FAA and its correlation with resting – state cerebral blood flow (rCBF). We hypothesized an association of FAA with regional rCBF in brain regions relevant for reward processing and motivated behavior, such as the striatum. We enrolled 20 patients and 19 healthy subjects. FAA scores and rCBF were quantified with the use of EEG and arterial spin labeling. Correlations of the two were evaluated, as well as the association with FAA and psychometric assessments of motivated behavior and anhedonia. Patients showed a left – lateralized pattern of frontal alpha activity and a correlation of FAA lateralization with subscores of Hamilton Depression Rating Scale linked to motivated behavior. An association of rCBF and FAA scores was found in clusters in the dorsolateral prefrontal cortex bilaterally (patients) and in the left medial frontal gyrus, in the right caudate head and in the right inferior parietal lobule (whole group). No correlations were found in healthy controls. Higher inhibitory right – lateralized alpha power was associated with lower rCBF values in prefrontal and striatal regions, predominantly in the right hemisphere, which are involved in the processing of motivated behavior and reward. Inhibitory brain activity in the reward system may contribute to some of the motivational problems observed in MDD.

Spatio-temporal credit assignment in population learning

Learning by reinforcement is important in shaping animal behavior, and in particular in behavioral decision making. Such decision making is likely to involve the integration of many synaptic events in space and time. However, using a single reinforcement signal to modulate synaptic plasticity, as suggested in classical reinforcement learning algorithms, a twofold problem arises. Different synapses will have contributed differently to the behavioral decision, and even for one and the same synapse, releases at different times may have had different effects. Here we present a plasticity rule which solves this spatio-temporal credit assignment problem in a population of spiking neurons. The learning rule is spike-time dependent and maximizes the expected reward by following its stochastic gradient. Synaptic plasticity is modulated not only by the reward, but also by a population feedback signal. While this additional signal solves the spatial component of the problem, the temporal one is solved by means of synaptic eligibility traces. In contrast to temporal difference (TD) based approaches to reinforcement learning, our rule is explicit with regard to the assumed biophysical mechanisms. Neurotransmitter concentrations determine plasticity and learning occurs fully online. Further, it works even if the task to be learned is non-Markovian, i.e. when reinforcement is not determined by the current state of the system but may also depend on past events. The performance of the model is assessed by studying three non-Markovian tasks. In the first task, the reward is delayed beyond the last action with non-related stimuli and actions appearing in between. The second task involves an action sequence which is itself extended in time and reward is only delivered at the last action, as it is the case in any type of board-game. The third task is the inspection game that has been studied in neuroeconomics, where an inspector tries to prevent a worker from shirking. Applying our algorithm to this game yields a learning behavior which is consistent with behavioral data from humans and monkeys, revealing themselves properties of a mixed Nash equilibrium. The examples show that our neuronal implementation of reward based learning copes with delayed and stochastic reward delivery, and also with the learning of mixed strategies in two-opponent games.

Spatio-temporal credit assignment in population learning

Learning by reinforcement is important in shaping animal behavior. But behavioral decision making is likely to involve the integration of many synaptic events in space and time. So in using a single reinforcement signal to modulate synaptic plasticity a twofold problem arises. Different synapses will have contributed differently to the behavioral decision and, even for one and the same synapse, releases at different times may have had different effects. Here we present a plasticity rule which solves this spatio-temporal credit assignment problem in a population of spiking neurons. The learning rule is spike time dependent and maximizes the expected reward by following its stochastic gradient. Synaptic plasticity is modulated not only by the reward but by a population feedback signal as well. While this additional signal solves the spatial component of the problem, the temporal one is solved by means of synaptic eligibility traces. In contrast to temporal difference based approaches to reinforcement learning, our rule is explicit with regard to the assumed biophysical mechanisms. Neurotransmitter concentrations determine plasticity and learning occurs fully online. Further, it works even if the task to be learned is non-Markovian, i.e. when reinforcement is not determined by the current state of the system but may also depend on past events. The performance of the model is assessed by studying three non-Markovian tasks. In the first task the reward is delayed beyond the last action with non-related stimuli and actions appearing in between. The second one involves an action sequence which is itself extended in time and reward is only delivered at the last action, as is the case in any type of board-game. The third is the inspection game that has been studied in neuroeconomics. It only has a mixed Nash equilibrium and exemplifies that the model also copes with stochastic reward delivery and the learning of mixed strategies.