Encoding of marginal utility across time in the human brain.

Marginal utility theory prescribes the relationship between the objective property of the magnitude of rewards and their subjective value. Despite its pervasive influence, however, there is remarkably little direct empirical evidence for such a theory of value, let alone of its neurobiological basis. We show that human preferences in an intertemporal choice task are best described by a model that integrates marginally diminishing utility with temporal discounting. Using functional magnetic resonance imaging, we show that activity in the dorsal striatum encodes both the marginal utility of rewards, over and above that which can be described by their magnitude alone, and the discounting associated with increasing time. In addition, our data show that dorsal striatum may be involved in integrating subjective valuation systems inherent to time and magnitude, thereby providing an overall metric of value used to guide choice behavior. Furthermore, during choice, we show that anterior cingulate activity correlates with the degree of difficulty associated with dissonance between value and time. Our data support an integrative architecture for decision making, revealing the neural representation of distinct subcomponents of value that may contribute to impulsivity and decisiveness.

Cortical substrates for exploratory decisions in humans.

Decision making in an uncertain environment poses a conflict between the opposing demands of gathering and exploiting information. In a classic illustration of this 'exploration-exploitation' dilemma, a gambler choosing between multiple slot machines balances the desire to select what seems, on the basis of accumulated experience, the richest option, against the desire to choose a less familiar option that might turn out more advantageous (and thereby provide information for improving future decisions). Far from representing idle curiosity, such exploration is often critical for organisms to discover how best to harvest resources such as food and water. In appetitive choice, substantial experimental evidence, underpinned by computational reinforcement learning (RL) theory, indicates that a dopaminergic, striatal and medial prefrontal network mediates learning to exploit. In contrast, although exploration has been well studied from both theoretical and ethological perspectives, its neural substrates are much less clear. Here we show, in a gambling task, that human subjects' choices can be characterized by a computationally well-regarded strategy for addressing the explore/exploit dilemma. Furthermore, using this characterization to classify decisions as exploratory or exploitative, we employ functional magnetic resonance imaging to show that the frontopolar cortex and intraparietal sulcus are preferentially active during exploratory decisions. In contrast, regions of striatum and ventromedial prefrontal cortex exhibit activity characteristic of an involvement in value-based exploitative decision making. The results suggest a model of action selection under uncertainty that involves switching between exploratory and exploitative behavioural modes, and provide a computationally precise characterization of the contribution of key decision-related brain systems to each of these functions.

Modulation of pain processing in hyperalgesia by cognitive demand.

The relationship between pain and cognitive function is of theoretical and clinical interest, exemplified by observations that attention-demanding activities reduce pain in chronically afflicted patients. Previous studies have concentrated on phasic pain, which bears little correspondence to clinical pain conditions. Indeed, phasic pain is often associated with differential or opposing effects to tonic pain in behavioral, lesion, and pharmacological studies. To address how cognitive engagement interacts with tonic pain, we assessed the influence of an attention-demanding cognitive task on pain-evoked neural responses in an experimental model of chronic pain, the capsaicin-induced heat hyperalgesia model. Using functional magnetic resonance imaging (fMRI), we show that activity in the orbitofrontal and medial prefrontal cortices, insula, and cerebellum correlates with the intensity of tonic pain. This pain-related activity in medial prefrontal cortex and cerebellum was modulated by the demand level of the cognitive task. Our findings highlight a role for these structures in the integration of motivational and cognitive functions associated with a physiological state of injury. Within the limitations of an experimental model of pain, we suggest that the findings are relevant to understanding both the neurobiology and pathophysiology of chronic pain and its amelioration by cognitive strategies.

Dopamine and performance in a reinforcement learning task: Evidence from Parkinson's disease

The role dopamine plays in decision-making has important theoretical, empirical and clinical implications. Here, we examined its precise contribution by exploiting the lesion deficit model afforded by Parkinson's disease. We studied patients in a two-stage reinforcement learning task, while they were ON and OFF dopamine replacement medication. Contrary to expectation, we found that dopaminergic drug state (ON or OFF) did not impact learning. Instead, the critical factor was drug state during the performance phase, with patients ON medication choosing correctly significantly more frequently than those OFF medication. This effect was independent of drug state during initial learning and appears to reflect a facilitation of generalization for learnt information. This inference is bolstered by our observation that neural activity in nucleus accumbens and ventromedial prefrontal cortex, measured during simultaneously acquired functional magnetic resonance imaging, represented learnt stimulus values during performance. This effect was expressed solely during the ON state with activity in these regions correlating with better performance. Our data indicate that dopamine modulation of nucleus accumbens and ventromedial prefrontal cortex exerts a specific effect on choice behaviour distinct from pure learning. The findings are in keeping with the substantial other evidence that certain aspects of learning are unaffected by dopamine lesions or depletion, and that dopamine plays a key role in performance that may be distinct from its role in learning. © 2012 The Author.

Design and formulation of functional pluripotent stem cell-derived cardiac microtissues.

Access to robust and information-rich human cardiac tissue models would accelerate drug-based strategies for treating heart disease. Despite significant effort, the generation of high-fidelity adult-like human cardiac tissue analogs remains challenging. We used computational modeling of tissue contraction and assembly mechanics in conjunction with microfabricated constraints to guide the design of aligned and functional 3D human pluripotent stem cell (hPSC)-derived cardiac microtissues that we term cardiac microwires (CMWs). Miniaturization of the platform circumvented the need for tissue vascularization and enabled higher-throughput image-based analysis of CMW drug responsiveness. CMW tissue properties could be tuned using electromechanical stimuli and cell composition. Specifically, controlling self-assembly of 3D tissues in aligned collagen, and pacing with point stimulation electrodes, were found to promote cardiac maturation-associated gene expression and in vivo-like electrical signal propagation. Furthermore, screening a range of hPSC-derived cardiac cell ratios identified that 75% NKX2 Homeobox 5 (NKX2-5)+ cardiomyocytes and 25% Cluster of Differentiation 90 OR (CD90)+ nonmyocytes optimized tissue remodeling dynamics and yielded enhanced structural and functional properties. Finally, we demonstrate the utility of the optimized platform in a tachycardic model of arrhythmogenesis, an aspect of cardiac electrophysiology not previously recapitulated in 3D in vitro hPSC-derived cardiac microtissue models. The design criteria identified with our CMW platform should accelerate the development of predictive in vitro assays of human heart tissue function.

Gaussian process volatility model

The prediction of time-changing variances is an important task in the modeling of financial data. Standard econometric models are often limited as they assume rigid functional relationships for the evolution of the variance. Moreover, functional parameters are usually learned by maximum likelihood, which can lead to over-fitting. To address these problems we introduce GP-Vol, a novel non-parametric model for time-changing variances based on Gaussian Processes. This new model can capture highly flexible functional relationships for the variances. Furthermore, we introduce a new online algorithm for fast inference in GP-Vol. This method is much faster than current offline inference procedures and it avoids overfitting problems by following a fully Bayesian approach. Experiments with financial data show that GP-Vol performs significantly better than current standard alternatives.

Design and modeling of an integrated device for acoustic resonance spectroscopy

This paper investigates the design and modelling of an integrated device for acoustic resonance spectroscopy (ARS). Miniaturisation of such platforms can be achieved using MEMS technology thereby enabling scaling of device dimensions to investigate smaller specimens while simultaneously operating at higher frequencies. We propose an integrated device where the transducers are mounted in close proximity with the specimen to be analysed (e.g. by integrating ultrasound transducers within a microfluidic channel). A finite element (FE) model and a simplified analytical model have been constructed to predict the acoustic response of a sample embedded in such a device configuration. A FE simulation is performed in COMSOL by embedding the piezoelectric transducers in representative fluid media. Resonant frequencies associated with the measurement can be extracted from this data. The response of various media modelled through FEA matches with analytical predictions for a range of biological media. A variety of biological media may be identified by using the measured resonant frequencies as a signature of relevant physical characteristics. The paper establishes the modelling basis of an integrated acoustic resonant spectrometer that is then applied to examine the impact of geometrical scaling on system resolution. © 2013 IEEE.

Optimal recall from bounded metaplastic synapses: predicting functional adaptations in hippocampal area CA3.

A venerable history of classical work on autoassociative memory has significantly shaped our understanding of several features of the hippocampus, and most prominently of its CA3 area, in relation to memory storage and retrieval. However, existing theories of hippocampal memory processing ignore a key biological constraint affecting memory storage in neural circuits: the bounded dynamical range of synapses. Recent treatments based on the notion of metaplasticity provide a powerful model for individual bounded synapses; however, their implications for the ability of the hippocampus to retrieve memories well and the dynamics of neurons associated with that retrieval are both unknown. Here, we develop a theoretical framework for memory storage and recall with bounded synapses. We formulate the recall of a previously stored pattern from a noisy recall cue and limited-capacity (and therefore lossy) synapses as a probabilistic inference problem, and derive neural dynamics that implement approximate inference algorithms to solve this problem efficiently. In particular, for binary synapses with metaplastic states, we demonstrate for the first time that memories can be efficiently read out with biologically plausible network dynamics that are completely constrained by the synaptic plasticity rule, and the statistics of the stored patterns and of the recall cue. Our theory organises into a coherent framework a wide range of existing data about the regulation of excitability, feedback inhibition, and network oscillations in area CA3, and makes novel and directly testable predictions that can guide future experiments.

Novel fluid grid and voidage calculation techniques for a discrete element model of a 3D cylindrical fluidized bed

A discrete element model (DEM) combined with computational fluid dynamics (CFD) was developed to model particle and fluid behaviour in 3D cylindrical fluidized beds. Novel techniques were developed to (1) keep fluid cells, defined in cylindrical coordinates, at a constant volume in order to ensure the conditions for validity of the volume-averaged fluid equations were satisfied and (2) smoothly and accurately measure voidage in arbitrarily shaped fluid cells. The new technique for calculating voidage was more stable than traditional techniques, also examined in the paper, whilst remaining computationally-effective. The model was validated by quantitative comparison with experimental results from the magnetic resonance imaging of a fluidised bed analysed to give time-averaged particle velocities. Comparisons were also made between theoretical determinations of slug rise velocity in a tall bed. It was concluded that the DEM-CFD model is able to investigate aspects of the underlying physics of fluidisation not readily investigated by experiment. © 2014 The Authors.

Resonance based multi-gaited robot locomotion

In order to understand the underlying mechanisms of animals' agility, dexterity and efficiency in motor control, there has been an increasing interest in the study of gait patterns in biological and artificial legged systems. This paper presents a novel approach to the study of gait patterns which makes use of intrinsic mechanical dynamics of robotic systems. Each of these robots consists of a U-shape elastic beam and exploits free vibration to generate different gait patterns. We developed a conceptual model for these robots, and through simulation and real-world experiments, we show three distinct mechanisms for generating four different gait patterns in these robots. © 2012 IEEE.

Finding resonance: Adaptive frequency oscillators for dynamic legged locomotion

There is much to gain from providing walking machines with passive dynamics, e.g. by including compliant elements in the structure. These elements can offer interesting properties such as self-stabilization, energy efficiency and simplified control. However, there is still no general design strategy for such robots and their controllers. In particular, the calibration of control parameters is often complicated because of the highly nonlinear behavior of the interactions between passive components and the environment. In this article, we propose an approach in which the calibration of a key parameter of a walking controller, namely its intrinsic frequency, is done automatically. The approach uses adaptive frequency oscillators to automatically tune the intrinsic frequency of the oscillators to the resonant frequency of a compliant quadruped robot The tuning goes beyond simple synchronization and the learned frequency stays in the controller when the robot is put to halt. The controller is model free, robust and simple. Results are presented illustrating how the controller can robustly tune itself to the robot, as well as readapt when the mass of the robot is changed. We also provide an analysis of the convergence of the frequency adaptation for a linearized plant, and show how that analysis is useful for determining which type of sensory feedback must be used for stable convergence. This approach is expected to explain some aspects of developmental processes in biological and artificial adaptive systems that "develop" through the embodied system-environment interactions. © 2006 IEEE.