The misbehavior of value and the discipline of the will.

Most reinforcement learning models of animal conditioning operate under the convenient, though fictive, assumption that Pavlovian conditioning concerns prediction learning whereas instrumental conditioning concerns action learning. However, it is only through Pavlovian responses that Pavlovian prediction learning is evident, and these responses can act against the instrumental interests of the subjects. This can be seen in both experimental and natural circumstances. In this paper we study the consequences of importing this competition into a reinforcement learning context, and demonstrate the resulting effects in an omission schedule and a maze navigation task. The misbehavior created by Pavlovian values can be quite debilitating; we discuss how it may be disciplined.

Soil conditioning under London

The CTRL Contract 220 covered 7.5km twin-bore tunnels excavated between late 2002 and early 2004 from Stratford Box to St Pancras station in Central London. To ensure efficient machine operation as well as the transport and disposal of soil, soil conditioning treatments were applied. Specifically, the foam injection ratio (FIR) and the polymer injection ratio (PIR) (injected volume of foam and polymer solution expressed as a percentage of the excavated soil volume) were employed. It was found that carefully selected soil conditioning allowed chamber pressures of 200kPa or more to be accurately controlled in the stiff London Clay and to an extent, in the very stiff clays of the Lambeth Group. Average FIRs of 50% and PIRs of 7 and 9% were used in the Thanet Sand and in the Lambeth Group Clays. In contrast, much lower quantities of foam were used in the London Clay.

Context-dependent human extinction memory is mediated by a ventromedial prefrontal and hippocampal network.

In fear extinction, an animal learns that a conditioned stimulus (CS) no longer predicts a noxious stimulus [unconditioned stimulus (UCS)] to which it had previously been associated, leading to inhibition of the conditioned response (CR). Extinction creates a new CS-noUCS memory trace, competing with the initial fear (CS-UCS) memory. Recall of extinction memory and, hence, CR inhibition at later CS encounters is facilitated by contextual stimuli present during extinction training. In line with theoretical predictions derived from animal studies, we show that, after extinction, a CS-evoked engagement of human ventromedial prefrontal cortex (VMPFC) and hippocampus is context dependent, being expressed in an extinction, but not a conditioning, context. Likewise, a positive correlation between VMPFC and hippocampal activity is extinction context dependent. Thus, a VMPFC-hippocampal network provides for context-dependent recall of human extinction memory, consistent with a view that hippocampus confers context dependence on VMPFC.

Contingency awareness in human aversive conditioning involves the middle frontal gyrus.

In contrast to the wealth of data describing the neural mechanisms underlying classical conditioning, we know remarkably little about the mechanisms involved in acquisition of explicit contingency awareness. Subjects variably acquire contingency awareness in classical conditioning paradigms, in which they are able to describe the temporal relationship between a conditioned cue and its outcome. Previous studies have implicated the hippocampus and prefrontal cortex in the acquisition of explicit knowledge, although their specific roles remain unclear. We used functional magnetic resonance imaging to track the trial-by-trial acquisition of explicit knowledge in a concurrent trace and delay conditioning paradigm. We show that activity in bilateral middle frontal gyrus and parahippocampal gyrus correlates with the accuracy of explicit contingency awareness on each trial. In contrast, amygdala activation correlates with conditioned responses indexed by skin conductance responses (SCRs). These results demonstrate that brain regions known to be involved in other aspects of learning and memory also play a specific role, reflecting on each trial the acquisition and representation of contingency awareness.