Encoding of marginal utility across time in the human brain.

Marginal utility theory prescribes the relationship between the objective property of the magnitude of rewards and their subjective value. Despite its pervasive influence, however, there is remarkably little direct empirical evidence for such a theory of value, let alone of its neurobiological basis. We show that human preferences in an intertemporal choice task are best described by a model that integrates marginally diminishing utility with temporal discounting. Using functional magnetic resonance imaging, we show that activity in the dorsal striatum encodes both the marginal utility of rewards, over and above that which can be described by their magnitude alone, and the discounting associated with increasing time. In addition, our data show that dorsal striatum may be involved in integrating subjective valuation systems inherent to time and magnitude, thereby providing an overall metric of value used to guide choice behavior. Furthermore, during choice, we show that anterior cingulate activity correlates with the degree of difficulty associated with dissonance between value and time. Our data support an integrative architecture for decision making, revealing the neural representation of distinct subcomponents of value that may contribute to impulsivity and decisiveness.

Anchors, scales and the relative coding of value in the brain.

People are alarmingly susceptible to manipulations that change both their expectations and experience of the value of goods. Recent studies in behavioral economics suggest such variability reflects more than mere caprice. People commonly judge options and prices in relative terms, rather than absolutely, and display strong sensitivity to exemplar and price anchors. We propose that these findings elucidate important principles about reward processing in the brain. In particular, relative valuation may be a natural consequence of adaptive coding of neuronal firing to optimise sensitivity across large ranges of value. Furthermore, the initial apparent arbitrariness of value may reflect the brains' attempts to optimally integrate diverse sources of value-relevant information in the face of perceived uncertainty. Recent findings in neuroscience support both accounts, and implicate regions in the orbitofrontal cortex, striatum, and ventromedial prefrontal cortex in the construction of value.

Frames, biases, and rational decision-making in the human brain.

Human choices are remarkably susceptible to the manner in which options are presented. This so-called "framing effect" represents a striking violation of standard economic accounts of human rationality, although its underlying neurobiology is not understood. We found that the framing effect was specifically associated with amygdala activity, suggesting a key role for an emotional system in mediating decision biases. Moreover, across individuals, orbital and medial prefrontal cortex activity predicted a reduced susceptibility to the framing effect. This finding highlights the importance of incorporating emotional processes within models of human choice and suggests how the brain may modulate the effect of these biasing influences to approximate rationality.

Reinforcement learning using a continuous time actor-critic framework with spiking neurons.

Animals repeat rewarded behaviors, but the physiological basis of reward-based learning has only been partially elucidated. On one hand, experimental evidence shows that the neuromodulator dopamine carries information about rewards and affects synaptic plasticity. On the other hand, the theory of reinforcement learning provides a framework for reward-based learning. Recent models of reward-modulated spike-timing-dependent plasticity have made first steps towards bridging the gap between the two approaches, but faced two problems. First, reinforcement learning is typically formulated in a discrete framework, ill-adapted to the description of natural situations. Second, biologically plausible models of reward-modulated spike-timing-dependent plasticity require precise calculation of the reward prediction error, yet it remains to be shown how this can be computed by neurons. Here we propose a solution to these problems by extending the continuous temporal difference (TD) learning of Doya (2000) to the case of spiking neurons in an actor-critic network operating in continuous time, and with continuous state and action representations. In our model, the critic learns to predict expected future rewards in real time. Its activity, together with actual rewards, conditions the delivery of a neuromodulatory TD signal to itself and to the actor, which is responsible for action choice. In simulations, we show that such an architecture can solve a Morris water-maze-like navigation task, in a number of trials consistent with reported animal performance. We also use our model to solve the acrobot and the cartpole problems, two complex motor control tasks. Our model provides a plausible way of computing reward prediction error in the brain. Moreover, the analytically derived learning rule is consistent with experimental evidence for dopamine-modulated spike-timing-dependent plasticity.

Inhibitory plasticity balances excitation and inhibition in sensory pathways and memory networks

Cortical neurons receive balanced excitatory and inhibitory synaptic currents. Such a balance could be established and maintained in an experience-dependent manner by synaptic plasticity at inhibitory synapses. We show that this mechanism provides an explanation for the sparse firing patterns observed in response to natural stimuli and fits well with a recently observed interaction of excitatory and inhibitory receptive field plasticity. The introduction of inhibitory plasticity in suitable recurrent networks provides a homeostatic mechanism that leads to asynchronous irregular network states. Further, it can accommodate synaptic memories with activity patterns that become indiscernible from the background state but can be reactivated by external stimuli. Our results suggest an essential role of inhibitory plasticity in the formation and maintenance of functional cortical circuitry.

Functional requirements for reward-modulated spike-timing-dependent plasticity.

Recent experiments have shown that spike-timing-dependent plasticity is influenced by neuromodulation. We derive theoretical conditions for successful learning of reward-related behavior for a large class of learning rules where Hebbian synaptic plasticity is conditioned on a global modulatory factor signaling reward. We show that all learning rules in this class can be separated into a term that captures the covariance of neuronal firing and reward and a second term that presents the influence of unsupervised learning. The unsupervised term, which is, in general, detrimental for reward-based learning, can be suppressed if the neuromodulatory signal encodes the difference between the reward and the expected reward-but only if the expected reward is calculated for each task and stimulus separately. If several tasks are to be learned simultaneously, the nervous system needs an internal critic that is able to predict the expected reward for arbitrary stimuli. We show that, with a critic, reward-modulated spike-timing-dependent plasticity is capable of learning motor trajectories with a temporal resolution of tens of milliseconds. The relation to temporal difference learning, the relevance of block-based learning paradigms, and the limitations of learning with a critic are discussed.

Slowness: an objective for spike-timing-dependent plasticity?

Our nervous system can efficiently recognize objects in spite of changes in contextual variables such as perspective or lighting conditions. Several lines of research have proposed that this ability for invariant recognition is learned by exploiting the fact that object identities typically vary more slowly in time than contextual variables or noise. Here, we study the question of how this "temporal stability" or "slowness" approach can be implemented within the limits of biologically realistic spike-based learning rules. We first show that slow feature analysis, an algorithm that is based on slowness, can be implemented in linear continuous model neurons by means of a modified Hebbian learning rule. This approach provides a link to the trace rule, which is another implementation of slowness learning. Then, we show analytically that for linear Poisson neurons, slowness learning can be implemented by spike-timing-dependent plasticity (STDP) with a specific learning window. By studying the learning dynamics of STDP, we show that for functional interpretations of STDP, it is not the learning window alone that is relevant but rather the convolution of the learning window with the postsynaptic potential. We then derive STDP learning windows that implement slow feature analysis and the "trace rule." The resulting learning windows are compatible with physiological data both in shape and timescale. Moreover, our analysis shows that the learning window can be split into two functionally different components that are sensitive to reversible and irreversible aspects of the input statistics, respectively. The theory indicates that irreversible input statistics are not in favor of stable weight distributions but may generate oscillatory weight dynamics. Our analysis offers a novel interpretation for the functional role of STDP in physiological neurons.

Plane-strain discrete dislocation plasticity with climb-assisted glide motion of dislocations

A small-strain two-dimensional discrete dislocation plasticity (DDP) framework is developed wherein dislocation motion is caused by climb-assisted glide. The climb motion of the dislocations is assumed to be governed by a drag-type relation similar to the glide-only motion of dislocations: such a relation is valid when vacancy kinetics is either diffusion limited or sink limited. The DDP framework is employed to predict the effect of dislocation climb on the uniaxial tensile and pure bending response of single crystals. Under uniaxial tensile loading conditions, the ability of dislocations to bypass obstacles by climb results in a reduced dislocation density over a wide range of specimen sizes in the climb-assisted glide case compared to when dislocation motion is only by glide. A consequence is that, at least in a single slip situation, size effects due to dislocation starvation are reduced. By contrast, under pure bending loading conditions, the dislocation density is unaffected by dislocation climb as geometrically necessary dislocations (GNDs) dominate. However, climb enables the dislocations to arrange themselves into lower energy configurations which significantly reduces the predicted bending size effect as well as the amount of reverse plasticity observed during unloading. The results indicate that the intrinsic plasticity material length scale associated with GNDs is strongly affected by thermally activated processes and will be a function of temperature. © 2013 IOP Publishing Ltd.

Characterisation of a high plasticity marine clay using a T-bar penetrometer

A series of laboratory-scale T-bar penetrometer tests have been conducted on a clay bed virgin consolidated from reconstituted high plasticity marine clay. This investigation was mainly concerned with the effects on the penetration resistance of rate of penetration and the presence of free water on the surface of the clay bed. The rate of penetration varied between 0.005mm/s and 50mm/s. The results showed that the nature of soil resistance was 'undrained' over the range of speeds studied, and the resistance showed a marked viscous rate effect. The virgin consolidated clay bed exhibited an increase in penetration resistance by up to 35% for a factor 10 increase in rate of penetration much larger than values previously reported for kaolin. The presence of water on the surface of clay bed had a profound impact on penetration resistance, particularly on the remoulded strength obtained by taking the T-bar through successive penetration and extraction cycles. This was true even when the remoulding cycles were conducted without the T-bar breaking through the clay surface.

Climb-enabled discrete dislocation plasticity

A small strain two-dimensional discrete dislocation plasticity framework coupled to vacancy diffusion is developed wherein the motion of edge dislocations is by a combination of glide and climb. The dislocations are modelled as line defects in a linear elastic medium and the mechanical boundary value problem is solved by the superposition of the infinite medium elastic fields of the dislocations and a complimentary non-singular solution that enforces the boundary conditions. Similarly, the climbing dislocations are modelled as line sources/sinks of vacancies and the vacancy diffusion boundary value problem is also solved by a superposition of the fields of the line sources/sinks in an infinite medium and a complementary non-singular solution that enforces the boundary conditions. The vacancy concentration field along with the stress field provides the climb rate of the dislocations. Other short-range interactions of the dislocations are incorporated via a set of constitutive rules. We first employ this formulation to investigate the climb of a single edge dislocation in an infinite medium and illustrate the existence of diffusion-limited and sink-limited climb regimes. Next, results are presented for the pure bending and uniaxial tension of single crystals oriented for single slip. These calculations show that plasticity size effects are reduced when dislocation climb is permitted. Finally, we contrast predictions of this coupled framework with an ad hoc model in which dislocation climb is modelled by a drag-type relation based on a quasi steady-state solution. © 2013 Elsevier Ltd. All rights reserved.

Grain boundary interface mechanics in strain gradient crystal plasticity

Interactions between dislocations and grain boundaries play an important role in the plastic deformation of polycrystalline metals. Capturing accurately the behaviour of these internal interfaces is particularly important for applications where the relative grain boundary fraction is significant, such as ultra fine-grained metals, thin films and microdevices. Incorporating these micro-scale interactions (which are sensitive to a number of dislocation, interface and crystallographic parameters) within a macro-scale crystal plasticity model poses a challenge. The innovative features in the present paper include (i) the formulation of a thermodynamically consistent grain boundary interface model within a microstructurally motivated strain gradient crystal plasticity framework, (ii) the presence of intra-grain slip system coupling through a microstructurally derived internal stress, (iii) the incorporation of inter-grain slip system coupling via an interface energy accounting for both the magnitude and direction of contributions to the residual defect from all slip systems in the two neighbouring grains, and (iv) the numerical implementation of the grain boundary model to directly investigate the influence of the interface constitutive parameters on plastic deformation. The model problem of a bicrystal deforming in plane strain is analysed. The influence of dissipative and energetic interface hardening, grain misorientation, asymmetry in the grain orientations and the grain size are systematically investigated. In each case, the crystal response is compared with reference calculations with grain boundaries that are either 'microhard' (impenetrable to dislocations) or 'microfree' (an infinite dislocation sink). © 2013 Elsevier Ltd. All rights reserved.