Expected value, reward outcome, and temporal difference error representations in a probabilistic decision task

In probabilistic decision tasks, an expected value (EV) of a choice is calculated, and after the choice has been made, this can be updated based on a temporal difference (TD) prediction error between the EV and the reward magnitude (RM) obtained. The EV is measured as the probability of obtaining a reward x RM. To understand the contribution of different brain areas to these decision-making processes, functional magnetic resonance imaging activations related to EV versus RM (or outcome) were measured in a probabilistic decision task. Activations in the medial orbitofrontal cortex were correlated with both RM and with EV and confirmed in a conjunction analysis to extend toward the pregenual cingulate cortex. From these representations, TD reward prediction errors could be produced. Activations in areas that receive from the orbitofrontal cortex including the ventral striatum, midbrain, and inferior frontal gyrus were correlated with the TD error. Activations in the anterior insula were correlated negatively with EV, occurring when low reward outcomes were expected, and also with the uncertainty of the reward, implicating this region in basic and crucial decision-making parameters, low expected outcomes, and uncertainty.

Hierarchical Reinforcement Learning in Behavior and the Brain

Dissertation presented to obtain the Ph.D degree in Biology, Neuroscience

Dynamics of serotonergic neurons revealed by fiber photometry

This work was developed in the context of the MIT Portugal Program, area of Bioengineering Systems, in collaboration with the Champalimaud Research Programme, Champalimaud Center for the Unknown, Lisbon, Portugal. The project entitled Dynamics of serotonergic neurons revealed by fiber photometry was carried out at Instituto Gulbenkian de Ciência, Oeiras, Portugal and at the Champalimaud Research Programme, Champalimaud Center for the Unknown, Lisbon, Portugal

Stress induced risk-aversion is reverted by D2/D3 agonist in the rat

Stress exposure triggers cognitive and behavioral impairments that influence decision-making processes. Decisions under a context of uncertainty require complex reward-prediction processes that are known to be mediated by the mesocorticolimbic dopamine (DA) system in brain areas sensitive to the deleterious effects of chronic stress, in particular the orbitofrontal cortex (OFC). Using a decision-making task, we show that chronic stress biases risk-based decision-making to safer behaviors. This decision-making pattern is associated with an increased activation of the lateral part of the OFC and with morphological changes in pyramidal neurons specifically recruited by this task. Additionally, stress exposure induces a hypodopaminergic status accompanied by increased mRNA levels of the dopamine receptor type 2 (Drd2) in the OFC; importantly, treatment with a D2/D3 agonist quinpirole reverts the shift to safer behaviors induced by stress on risky decision-making. These results suggest that the brain mechanisms related to risk-based decision-making are altered after chronic stress, but can be modulated by manipulation of dopaminergic transmission.

Modèle informatique du coapprentissage des ganglions de la base et du cortex : l'apprentissage par renforcement et le développement de représentations

Tout au long de la vie, le cerveau développe des représentations de son environnement permettant à l’individu d’en tirer meilleur profit. Comment ces représentations se développent-elles pendant la quête de récompenses demeure un mystère. Il est raisonnable de penser que le cortex est le siège de ces représentations et que les ganglions de la base jouent un rôle important dans la maximisation des récompenses. En particulier, les neurones dopaminergiques semblent coder un signal d’erreur de prédiction de récompense. Cette thèse étudie le problème en construisant, à l’aide de l’apprentissage machine, un modèle informatique intégrant de nombreuses évidences neurologiques. Après une introduction au cadre mathématique et à quelques algorithmes de l’apprentissage machine, un survol de l’apprentissage en psychologie et en neuroscience et une revue des modèles de l’apprentissage dans les ganglions de la base, la thèse comporte trois articles. Le premier montre qu’il est possible d’apprendre à maximiser ses récompenses tout en développant de meilleures représentations des entrées. Le second article porte sur l'important problème toujours non résolu de la représentation du temps. Il démontre qu’une représentation du temps peut être acquise automatiquement dans un réseau de neurones artificiels faisant office de mémoire de travail. La représentation développée par le modèle ressemble beaucoup à l’activité de neurones corticaux dans des tâches similaires. De plus, le modèle montre que l’utilisation du signal d’erreur de récompense peut accélérer la construction de ces représentations temporelles. Finalement, il montre qu’une telle représentation acquise automatiquement dans le cortex peut fournir l’information nécessaire aux ganglions de la base pour expliquer le signal dopaminergique. Enfin, le troisième article évalue le pouvoir explicatif et prédictif du modèle sur différentes situations comme la présence ou l’absence d’un stimulus (conditionnement classique ou de trace) pendant l’attente de la récompense. En plus de faire des prédictions très intéressantes en lien avec la littérature sur les intervalles de temps, l’article révèle certaines lacunes du modèle qui devront être améliorées. Bref, cette thèse étend les modèles actuels de l’apprentissage des ganglions de la base et du système dopaminergique au développement concurrent de représentations temporelles dans le cortex et aux interactions de ces deux structures.

Induction of Psychosis by Delta 9-Tetrahydrocannabinol Reflects Modulation of Prefrontal and Striatal Function During Attentional Salience Processing

Context: The aberrant processing of salience is thought to be a fundamental factor underlying psychosis. Cannabis can induce acute psychotic symptoms, and its chronic use may increase the risk of schizophrenia. We investigated whether its psychotic effects are mediated through an influence on attentional salience processing. Objective: To examine the effects of Delta 9-tetrahydrocannabinol (Delta 9-THC) and cannabidiol (CBD) on regional brain function during salience processing. Design: Volunteers were studied using event-related functional magnetic resonance imaging on 3 occasions after administration of Delta 9-THC, CBD, or placebo while performing a visual oddball detection paradigm that involved allocation of attention to infrequent (oddball) stimuli within a string of frequent (standard) stimuli. Setting: University center. Participants: Fifteen healthy men with minimal previous cannabis use. Main Outcome Measures: Symptom ratings, task performance, and regional brain activation. Results: During the processing of oddball stimuli, relative to placebo, Delta 9-THC attenuated activation in the right caudate but augmented it in the right prefrontal cortex. Delta 9-Tetrahydrocannabinol also reduced the response latency to standard relative to oddball stimuli. The effect of Delta 9-THC in the right caudate was negatively correlated with the severity of the psychotic symptoms it induced and its effect on response latency. The effects of CBD on task-related activation were in the opposite direction of those of Delta 9-THC; relative to placebo, CBD augmented left caudate and hippocampal activation but attenuated right prefrontal activation. Conclusions: Delta 9-Tetrahydrocannabinol and CBD differentially modulate prefrontal, striatal, and hippocampal function during attentional salience processing. These effects may contribute to the effects of cannabis on psychotic symptoms and on the risk of psychotic disorders.

Electroencephalographic topography measures of experienced utility

Economic theory distinguishes two concepts of utility: decision utility, objectively quantifiable by choices, and experienced utility, referring to the satisfaction by an obtainment. To date, experienced utility is typically measured with subjective ratings. This study intended to quantify experienced utility by global levels of neuronal activity. Neuronal activity was measured by means of electroencephalographic (EEG) responses to gain and omission of graded monetary rewards at the level of the EEG topography in human subjects. A novel analysis approach allowed approximating psychophysiological value functions for the experienced utility of monetary rewards. In addition, we identified the time windows of the event-related potentials (ERP) and the respective intracortical sources, in which variations in neuronal activity were significantly related to the value or valence of outcomes. Results indicate that value functions of experienced utility and regret disproportionally increase with monetary value, and thus contradict the compressing value functions of decision utility. The temporal pattern of outcome evaluation suggests an initial (∼250 ms) coarse evaluation regarding the valence, concurrent with a finer-grained evaluation of the value of gained rewards, whereas the evaluation of the value of omitted rewards emerges later. We hypothesize that this temporal double dissociation is explained by reward prediction errors. Finally, a late, yet unreported, reward-sensitive ERP topography (∼500 ms) was identified. The sources of these topographical covariations are estimated in the ventromedial prefrontal cortex, the medial frontal gyrus, the anterior and posterior cingulate cortex and the hippocampus/amygdala. The results provide important new evidence regarding “how,” “when,” and “where” the brain evaluates outcomes with different hedonic impact.

Comparison between Eysenck's and Gray's models of personality in the prediction of motivational work criteria

Impulsivity based on Gray's [Gray, J. A. (1982) The neuropsychology of anxiety: an enquiry into the function of the septo-hippocampal system. New York: Oxford University Press: (1991). The neurophysiology of temperament. In J. Strelau & A. Angleitner. Explorations in temperament: international perspectives on theory and measurement. London. Plenum Press]. physiological model of personality was hypothesised to be more predictive of goal oriented criteria within the workplace than scales derived From Eysenck's [Eysenck. H.J. (1967). The biological basis of personality. Springfield, IL: Charles C. Thompson.] physiological model of personality. Results confirmed the hypothesis and also showed that Gray's scale of Impulsivity was generally a better predictor than attributional style and interest in money. Results were interpreted as providing support for Gray's Behavioural Activation System which moderates response to reward. (C) 2001 Elsevier Science Ltd. All rights reserved.

Negative anticipatory contrast: Does it involve anticipation of an impending reward?

Negative anticipatory contrast (NAC) corresponds to the suppression in consumption of a first rewarding substance (e.g., saccharin 0.15%) when it is followed daily by a second preferred substance (e.g., sucrose 32%). The NAC has been interpreted as resulting from anticipation of the impending preferred reward and its comparison with the currently available first reward [Flaherty, CF., Rowan, G.A., 1985. Anticipatory contrast: within-subjects analysis. Anim. Learn. Behav. 13, 2-5]. In this context, one should expect that devaluation of the preferred substance after the establishment of the NAC would either reduce or abolish the contrast effect. However, contrary to this prediction, the results of the present study show that the NAC is insensitive to devaluation of the second, preferred, substance. This allows one to question that interpretation. The results reported in this study support the view that the NAC effect is controlled by memory of the relative value of the first solution, which is updated daily by means of both a gustatory and/or post-ingestive comparison of the first and second solutions, and memory of past pairings. (C) 2010 Elsevier B.V. All rights reserved.

Learning by the Dendritic Prediction of Somatic Spiking

Recent modeling of spike-timing-dependent plasticity indicates that plasticity involves as a third factor a local dendritic potential, besides pre- and postsynaptic firing times. We present a simple compartmental neuron model together with a non-Hebbian, biologically plausible learning rule for dendritic synapses where plasticity is modulated by these three factors. In functional terms, the rule seeks to minimize discrepancies between somatic firings and a local dendritic potential. Such prediction errors can arise in our model from stochastic fluctuations as well as from synaptic input, which directly targets the soma. Depending on the nature of this direct input, our plasticity rule subserves supervised or unsupervised learning. When a reward signal modulates the learning rate, reinforcement learning results. Hence a single plasticity rule supports diverse learning paradigms.

Dopaminergic modulation of the reward system in schizophrenia: A placebo-controlled dopamine depletion fMRI study

Background The brain reward circuitry innervated by dopamine is critically disturbed in schizophrenia. This study aims to investigate the role of dopamine-related brain activity during prediction of monetary reward and loss in first episode schizophrenia patients. Methods We measured blood–oxygen-level dependent (BOLD) activity in 10 patients with schizophrenia (SCH) and 12 healthy controls during dopamine depletion with α-methylparatyrosine (AMPT) and during a placebo condition (PLA). Results AMPT reduced the activation of striatal and cortical brain regions in SCH. In SCH vs. controls reduced activation was found in the AMPT condition in several regions during anticipation of reward and loss, including areas of the striatum and frontal cortex. In SCH vs. controls reduced activation of the superior temporal gyrus and posterior cingulate was observed in PLA during anticipation of rewarding stimuli. PLA patients had reduced activation in the ventral striatum, frontal and cingulate cortex in anticipation of loss. The findings of reduced dopamine-related brain activity during AMPT were verified by reduced levels of dopamine in urine, homovanillic-acid in plasma and increased prolactin levels. Conclusions Our results indicate that dopamine depletion affects functioning of the cortico-striatal reward circuitry in SCH. The findings also suggest that neuronal functions associated with dopamine neurotransmission and attribution of salience to reward predicting stimuli are altered in schizophrenia.

The appetitive motivation scale and other BAS measures in the prediction of Approach and Active Avoidance

The Appetitive Motivation Scale (Jackson & Smillie, 2004) is a new trait conceptualisation of Gray's (I 970 199 1) Behavioural Activation System. In this experiment we explore relationships that the Appetitive Motivation Scale and other measures of Gray's model have with Approach and Active Avoidance responses. Using a sample of 144 undergraduate students, both Appetitive Motivation and Sensitivity to Reward (from the Sensitivity to Punishment and Sensitivity to Reward Questionnaire, SPSRQ; Torrubia, Avila, Molto, & Ceseras, 2001), were found to be significant predictors of Approach and Active Avoidance response latency. This confirms previous experimental validations of the SPSRQ (e.g., Avila, 2001) and provides the first experimental evidence for the validity of the Appetitive Motivation scale. Consistent with interactive views of Gray's model (e.g., Corr, 2001), high SPSRQ Sensitivity to Punishment diminished the relationship between Sensitivity to Reward and our BAS criteria. Measures of BIS did not however interact in this way with the appetitive motivation scale. A surprising result was the failure for any of Carver and White's (1994) BAS scales to correlate with RST criteria. Implications of these findings and potential future directions are discussed. (C) 2004 Elsevier Ltd. All rights reserved.

Exploring Iris Colour Prediction And Ancestry Inference In Admixed Populations Of South America.

New DNA-based predictive tests for physical characteristics and inference of ancestry are highly informative tools that are being increasingly used in forensic genetic analysis. Two eye colour prediction models: a Bayesian classifier - Snipper and a multinomial logistic regression (MLR) system for the Irisplex assay, have been described for the analysis of unadmixed European populations. Since multiple SNPs in combination contribute in varying degrees to eye colour predictability in Europeans, it is likely that these predictive tests will perform in different ways amongst admixed populations that have European co-ancestry, compared to unadmixed Europeans. In this study we examined 99 individuals from two admixed South American populations comparing eye colour versus ancestry in order to reveal a direct correlation of light eye colour phenotypes with European co-ancestry in admixed individuals. Additionally, eye colour prediction following six prediction models, using varying numbers of SNPs and based on Snipper and MLR, were applied to the study populations. Furthermore, patterns of eye colour prediction have been inferred for a set of publicly available admixed and globally distributed populations from the HGDP-CEPH panel and 1000 Genomes databases with a special emphasis on admixed American populations similar to those of the study samples.

Petroleomics By Electrospray Ionization Ft-icr Mass Spectrometry Coupled To Partial Least Squares With Variable Selection Methods: Prediction Of The Total Acid Number Of Crude Oils.

Negative-ion mode electrospray ionization, ESI(-), with Fourier transform ion cyclotron resonance mass spectrometry (FT-ICR MS) was coupled to a Partial Least Squares (PLS) regression and variable selection methods to estimate the total acid number (TAN) of Brazilian crude oil samples. Generally, ESI(-)-FT-ICR mass spectra present a power of resolution of ca. 500,000 and a mass accuracy less than 1 ppm, producing a data matrix containing over 5700 variables per sample. These variables correspond to heteroatom-containing species detected as deprotonated molecules, [M - H](-) ions, which are identified primarily as naphthenic acids, phenols and carbazole analog species. The TAN values for all samples ranged from 0.06 to 3.61 mg of KOH g(-1). To facilitate the spectral interpretation, three methods of variable selection were studied: variable importance in the projection (VIP), interval partial least squares (iPLS) and elimination of uninformative variables (UVE). The UVE method seems to be more appropriate for selecting important variables, reducing the dimension of the variables to 183 and producing a root mean square error of prediction of 0.32 mg of KOH g(-1). By reducing the size of the data, it was possible to relate the selected variables with their corresponding molecular formulas, thus identifying the main chemical species responsible for the TAN values.

Neck Circumference As A New Anthropometric Indicator For Prediction Of Insulin Resistance And Components Of Metabolic Syndrome In Adolescents: Brazilian Metabolic Syndrome Study.

To evaluate the correlation between neck circumference and insulin resistance and components of metabolic syndrome in adolescents with different adiposity levels and pubertal stages, as well as to determine the usefulness of neck circumference to predict insulin resistance in adolescents. Cross-sectional study with 388 adolescents of both genders from ten to 19 years old. The adolescents underwent anthropometric and body composition assessment, including neck and waist circumferences, and biochemical evaluation. The pubertal stage was obtained by self-assessment, and the blood pressure, by auscultation. Insulin resistance was evaluated by the Homeostasis Model Assessment-Insulin Resistance. The correlation between two variables was evaluated by partial correlation coefficient adjusted for the percentage of body fat and pubertal stage. The performance of neck circumference to identify insulin resistance was tested by Receiver Operating Characteristic Curve. After the adjustment for percentage body fat and pubertal stage, neck circumference correlated with waist circumference, blood pressure, triglycerides and markers of insulin resistance in both genders. The results showed that the neck circumference is a useful tool for the detection of insulin resistance and changes in the indicators of metabolic syndrome in adolescents. The easiness of application and low cost of this measure may allow its use in Public Health services.