Associative Pavlovian conditioning leads to an increase in spinophilin-immunoreactive dendritic spines in the lateral amygdala

Changes in dendritic spine number and shape are believed to reflect structural plasticity consequent to learning. Previous studies have strongly suggested that the dorsal subnucleus of the lateral amygdala is an important site of physiological plasticity in Pavlovian fear conditioning. In the present study, we examined the effect of auditory fear conditioning on dendritic spine numbers in the dorsal subnucleus of the lateral amygdala using an immunolabelling procedure to visualize the spine-associated protein spinophilin. Associatively conditioned rats that received paired tone and shock presentations had 35% more total spinophilin-immunoreactive spines than animals that had unpaired stimulation, consistent with the idea that changes in the number of dendritic spines occur during learning and account in part for memory.

Evaluative learning in human Pavlovian conditioning: Extinct, but still there?

Previous studies of human affective learning, the acquisition of likes and dislikes, provided evidence that extinction training does not affect changes in conditional stimulus (CS) valence as indexed by paper/pencil ratings. Experiment 1 (N = 32) investigated whether this is an artifact of the CS valence assessment, which is taken in test sessions before and after training. Pleasantness ratings were collected in pre/post training tests and, for half of the participants, on-line during training. Rated unpleasantness of the CS that preceded the aversive US (CS+) increased during acquisition and decreased during extinction back to neutral. However, as in previous studies, post extinction paper/pencil ratings revealed the maintenance of rated CS+ unpleasantness. Experiment 2 (N = 34) replicated this finding for two measures of CS valence, paper/pencil and the continuous measure used during training. The present results indicate that previous reports of failures to find extinction of affective learning may reflect renewal rather than maintenance of acquired CS valence across extinction training. (C) 2003 Elsevier Science (USA). All rights reserved.

The effects of unconditional stimulus valence and conditioning paradigm on verbal, skeleto-motor, and autonomic indices of human Pavlovian conditioning

The effects of unconditional stimulus (US) valence (aversive electro-tactile stimulus vs. nonaversive imperative stimulus of a RT task) and conditioning paradigm (delay vs. trace) on affective learning as indexed by verbal ratings of conditional stimulus (CS) pleasantness and blink startle modulation and on relational learning as indexed by electrodermal responses were investigated. Affective learning was not affected by the conditioning paradigm; however, electrodermal responses and blink latency shortening indicated delayed learning in the trace procedure. Changes in rated CS pleasantness were found with the aversive US, but not with the non-aversive US. Differential conditioning as indexed by electrodermal responses and startle modulation was found regardless of US valence. The finding of significant differential blink modulation and electrodermal responding in the absence of a change in rated CS pleasantness as a result of conditioning with a non-aversive US was replicated in a second experiment. These results seem to indicate that startle modulation during conditioning is mediated by the arousal level of the anticipated US, rather than by the valence of the CS. (C) 2002 Elsevier Science (USA). All rights reserved.

Dopaminergic and Non-Dopaminergic Value Systems in Conditioning and Outcome-Specific Revaluation

Animals are motivated to choose environmental options that can best satisfy current needs. To explain such choices, this paper introduces the MOTIVATOR (Matching Objects To Internal Values Triggers Option Revaluations) neural model. MOTIVATOR describes cognitiveemotional interactions between higher-order sensory cortices and an evaluative neuraxis composed of the hypothalamus, amygdala, and orbitofrontal cortex. Given a conditioned stimulus (CS), the model amygdala and lateral hypothalamus interact to calculate the expected current value of the subjective outcome that the CS predicts, constrained by the current state of deprivation or satiation. The amygdala relays the expected value information to orbitofrontal cells that receive inputs from anterior inferotemporal cells, and medial orbitofrontal cells that receive inputs from rhinal cortex. The activations of these orbitofrontal cells code the subjective values of objects. These values guide behavioral choices. The model basal ganglia detect errors in CS-specific predictions of the value and timing of rewards. Excitatory inputs from the pedunculopontine nucleus interact with timed inhibitory inputs from model striosomes in the ventral striatum to regulate dopamine burst and dip responses from cells in the substantia nigra pars compacta and ventral tegmental area. Learning in cortical and striatal regions is strongly modulated by dopamine. The model is used to address tasks that examine food-specific satiety, Pavlovian conditioning, reinforcer devaluation, and simultaneous visual discrimination. Model simulations successfully reproduce discharge dynamics of known cell types, including signals that predict saccadic reaction times and CS-dependent changes in systolic blood pressure.

Olfactory conditioning facilitates diet transition in human infants.

We evaluated whether Pavlovian conditioning methods could be used to increase the ingestion of non-preferred solutions by formula-fed human infants. In baseline measures, 5-7 month old infants sucked less frequently and consumed less water than regular formula. During a 3-day olfactory conditioning period, parents placed a small scented disk, the conditioned stimulus, on the rim of their infants' formula bottle at every feeding. Following this training, infants' responses to water were tested when their water bottles had a disk scented with the training odor, a novel odor, or no odor. Infants tested with the training odor sucked more frequently and consumed significantly more water than they had at baseline. Infants tested with no odor or a novel odor consumed water at or below baseline levels. These data demonstrate that olfactory conditioning can be used to enhance ingestion in infants and suggest that such methods may be useful for infants experiencing difficulty when making transitions from one diet to another.

Regional response differences across the human amygdaloid complex during social conditioning

The amygdala is consistently implicated in biologically relevant learning tasks such as Pavlovian conditioning. In humans, the ability to identify individual faces based on the social outcomes they have predicted in the past constitutes a critical form of associative learning that can be likened to “social conditioning.” To capture such learning in a laboratory setting, participants learned about faces that predicted negative, positive, or neutral social outcomes. Participants reported liking or disliking the faces in accordance with their learned social value. During acquisition, we observed differential functional magnetic resonance imaging activation across the human amygdaloid complex consistent with previous lesion, electrophysiological, and functional neuroimaging data. A region of the medial ventral amygdala and a region of the dorsal amygdala/substantia innominata showed signal increases to both Negative and Positive faces, whereas a lateral ventral region displayed a linear representation of the valence of faces such that Negative > Positive > Neutral. This lateral ventral locus also differed from the dorsal and medial loci in that the magnitude of these responses was more resistant to habituation. These findings document a role for the human amygdala in social learning and reveal coarse regional dissociations in amygdala activity that are consistent with previous human and nonhuman animal data.

Conditioning of stress in Nile tilapia

A Pavlovian conditioning paradigm was used to induce a connection between a conditioned stimulus, light (CS), associated with an unconditioned stimulus, confinement (US) in Nile tilapia Oreochromis niloticus, which resulted in a conditioned endocrine response (CR) to the CS alone manifested as an increase in plasma cortisol. Individual isolated Nile tilapia were submitted for 10 days to the conditioning treatment consisting of turning on a light (CS) for I min with subsequent 30 min confinement (US). on the 10th day of the experiment, plasma cortisol was not increased when fish were subjected to no handling at all, or only light, or even a daily stressor for the 9 days. on the other hand, at the 10th day cortisol was significantly increased only when light was presented either with or without pairing with the stressor. These results confirmed that the cue, light (CS), was not stressful in itself, but when given as the CS in the absence of the US post conditioning the hypothalamus-pituitary-interrenal axis was activated. Therefore, it was concluded that memory of a previous experience with a stressor can be recalled by a conditioned stimulus and induce stress, which is the first demonstration of a memory-induced stress in fishes. (C) 2004 the Fisheries Society of the British Isles.

Evaluating feeding as unconditioned stimulus for conditioning of an endocrine effect in Nile tilapia

This study tested the adequacy of feeding as an unconditioned stimulus (US) to condition an endocrine response (plasma cortisol increase) in the cichlid fish Nile tilapia (Oreochromis niloticus). In a first study, conditioning was confirmed in grouped fish in the only experiment using single-held Nile tilapia. In this test a conditioned stimulus (CS - aeration off) was associated with a stressor (air emersion for 2 min - US). We then assessed whether several events of paired CS-US resulted in a conditioned endocrine response (CR), in this case an increase in plasma cortisol after presentation of the CS only. Before testing feeding as US, the postprandial or social holding condition for feeding effects on cortisol levels was tested. Nile tilapia showed increased cortisol after feeding associated to social context (grouped fish), but not to food only (single-held fish). In a third study, feeding was tested as US in an experiment similar to the first study but an increase in feeding-induced cortisol could not be conditioned. The absence of CR suggests that the stressor affects acquisition of this response, which may be a consequence of stimulus intensity or biological relevance. This study expands the recently reported Pavlovian conditioning paradigm for endocrine response in fish. (C) 2007 Elsevier B.V. All rights reserved.

The effect of emotional and attentional load on attentional startle modulation

The interactive effects of emotion and attention on attentional startle modulation were investigated in two experiments. Participants performed a discrimination and counting task with two visual stimuli during which acoustic eyeblink startle-eliciting probes were presented at long lead intervals. In Experiment 1, this task was combined with aversive Pavlovian conditioning. In Group Attend CS+, the attended stimulus was followed by an aversive unconditional stimulus (US) and the ignored stimulus was presented alone whereas the ignored stimulus was paired with the US in Group Attend CS−. In Experiment 2, a non-aversive reaction time task US replaced the aversive US. Regardless of the conditioning manipulation and consistent with a modality non-specific account of attentional startle modulation, startle magnitude was larger during attended than ignored stimuli in both experiments. Blink latency shortening was differentially affected by the conditioning manipulations suggesting additive effects of conditioning and discrimination and counting task on blink startle.

Neural and computational representations of decision variables

These studies explore how, where, and when representations of variables critical to decision-making are represented in the brain. In order to produce a decision, humans must first determine the relevant stimuli, actions, and possible outcomes before applying an algorithm that will select an action from those available. When choosing amongst alternative stimuli, the framework of value-based decision-making proposes that values are assigned to the stimuli and that these values are then compared in an abstract “value space” in order to produce a decision. Despite much progress, in particular regarding the pinpointing of ventromedial prefrontal cortex (vmPFC) as a region that encodes the value, many basic questions remain. In Chapter 2, I show that distributed BOLD signaling in vmPFC represents the value of stimuli under consideration in a manner that is independent of the type of stimulus it is. Thus the open question of whether value is represented in abstraction, a key tenet of value-based decision-making, is confirmed. However, I also show that stimulus-dependent value representations are also present in the brain during decision-making and suggest a potential neural pathway for stimulus-to-value transformations that integrates these two results.

More broadly speaking, there is both neural and behavioral evidence that two distinct control systems are at work during action selection. These two systems compose the “goal-directed system”, which selects actions based on an internal model of the environment, and the “habitual” system, which generates responses based on antecedent stimuli only. Computational characterizations of these two systems imply that they have different informational requirements in terms of input stimuli, actions, and possible outcomes. Associative learning theory predicts that the habitual system should utilize stimulus and action information only, while goal-directed behavior requires that outcomes as well as stimuli and actions be processed. In Chapter 3, I test whether areas of the brain hypothesized to be involved in habitual versus goal-directed control represent the corresponding theorized variables.

The question of whether one or both of these neural systems drives Pavlovian conditioning is less well-studied. Chapter 4 describes an experiment in which subjects were scanned while engaged in a Pavlovian task with a simple non-trivial structure. After comparing a variety of model-based and model-free learning algorithms (thought to underpin goal-directed and habitual decision-making, respectively), it was found that subjects’ reaction times were better explained by a model-based system. In addition, neural signaling of precision, a variable based on a representation of a world model, was found in the amygdala. These data indicate that the influence of model-based representations of the environment can extend even to the most basic learning processes.

Knowledge of the state of hidden variables in an environment is required for optimal inference regarding the abstract decision structure of a given environment and therefore can be crucial to decision-making in a wide range of situations. Inferring the state of an abstract variable requires the generation and manipulation of an internal representation of beliefs over the values of the hidden variable. In Chapter 5, I describe behavioral and neural results regarding the learning strategies employed by human subjects in a hierarchical state-estimation task. In particular, a comprehensive model fit and comparison process pointed to the use of "belief thresholding". This implies that subjects tended to eliminate low-probability hypotheses regarding the state of the environment from their internal model and ceased to update the corresponding variables. Thus, in concert with incremental Bayesian learning, humans explicitly manipulate their internal model of the generative process during hierarchical inference consistent with a serial hypothesis testing strategy.

Differential encoding of losses and gains in the human striatum.

Studies on human monetary prediction and decision making emphasize the role of the striatum in encoding prediction errors for financial reward. However, less is known about how the brain encodes financial loss. Using Pavlovian conditioning of visual cues to outcomes that simultaneously incorporate the chance of financial reward and loss, we show that striatal activation reflects positively signed prediction errors for both. Furthermore, we show functional segregation within the striatum, with more anterior regions showing relative selectivity for rewards and more posterior regions for losses. These findings mirror the anteroposterior valence-specific gradient reported in rodents and endorse the role of the striatum in aversive motivational learning about financial losses, illustrating functional and anatomical consistencies with primary aversive outcomes such as pain.

The misbehavior of value and the discipline of the will.

Most reinforcement learning models of animal conditioning operate under the convenient, though fictive, assumption that Pavlovian conditioning concerns prediction learning whereas instrumental conditioning concerns action learning. However, it is only through Pavlovian responses that Pavlovian prediction learning is evident, and these responses can act against the instrumental interests of the subjects. This can be seen in both experimental and natural circumstances. In this paper we study the consequences of importing this competition into a reinforcement learning context, and demonstrate the resulting effects in an omission schedule and a maze navigation task. The misbehavior created by Pavlovian values can be quite debilitating; we discuss how it may be disciplined.

Opponent appetitive-aversive neural processes underlie predictive learning of pain relief.

Termination of a painful or unpleasant event can be rewarding. However, whether the brain treats relief in a similar way as it treats natural reward is unclear, and the neural processes that underlie its representation as a motivational goal remain poorly understood. We used fMRI (functional magnetic resonance imaging) to investigate how humans learn to generate expectations of pain relief. Using a pavlovian conditioning procedure, we show that subjects experiencing prolonged experimentally induced pain can be conditioned to predict pain relief. This proceeds in a manner consistent with contemporary reward-learning theory (average reward/loss reinforcement learning), reflected by neural activity in the amygdala and midbrain. Furthermore, these reward-like learning signals are mirrored by opposite aversion-like signals in lateral orbitofrontal cortex and anterior cingulate cortex. This dual coding has parallels to 'opponent process' theories in psychology and promotes a formal account of prediction and expectation during pain.

Temporal difference models describe higher-order learning in humans.

The ability to use environmental stimuli to predict impending harm is critical for survival. Such predictions should be available as early as they are reliable. In pavlovian conditioning, chains of successively earlier predictors are studied in terms of higher-order relationships, and have inspired computational theories such as temporal difference learning. However, there is at present no adequate neurobiological account of how this learning occurs. Here, in a functional magnetic resonance imaging (fMRI) study of higher-order aversive conditioning, we describe a key computational strategy that humans use to learn predictions about pain. We show that neural activity in the ventral striatum and the anterior insula displays a marked correspondence to the signals for sequential learning predicted by temporal difference models. This result reveals a flexible aversive learning process ideally suited to the changing and uncertain nature of real-world environments. Taken with existing data on reward learning, our results suggest a critical role for the ventral striatum in integrating complex appetitive and aversive predictions to coordinate behaviour.