Dopamine and performance in a reinforcement learning task: evidence from Parkinson's disease.

The role dopamine plays in decision-making has important theoretical, empirical and clinical implications. Here, we examined its precise contribution by exploiting the lesion deficit model afforded by Parkinson's disease. We studied patients in a two-stage reinforcement learning task, while they were ON and OFF dopamine replacement medication. Contrary to expectation, we found that dopaminergic drug state (ON or OFF) did not impact learning. Instead, the critical factor was drug state during the performance phase, with patients ON medication choosing correctly significantly more frequently than those OFF medication. This effect was independent of drug state during initial learning and appears to reflect a facilitation of generalization for learnt information. This inference is bolstered by our observation that neural activity in nucleus accumbens and ventromedial prefrontal cortex, measured during simultaneously acquired functional magnetic resonance imaging, represented learnt stimulus values during performance. This effect was expressed solely during the ON state with activity in these regions correlating with better performance. Our data indicate that dopamine modulation of nucleus accumbens and ventromedial prefrontal cortex exerts a specific effect on choice behaviour distinct from pure learning. The findings are in keeping with the substantial other evidence that certain aspects of learning are unaffected by dopamine lesions or depletion, and that dopamine plays a key role in performance that may be distinct from its role in learning.

Dopamine and performance in a reinforcement learning task: Evidence from Parkinson's disease

The role dopamine plays in decision-making has important theoretical, empirical and clinical implications. Here, we examined its precise contribution by exploiting the lesion deficit model afforded by Parkinson's disease. We studied patients in a two-stage reinforcement learning task, while they were ON and OFF dopamine replacement medication. Contrary to expectation, we found that dopaminergic drug state (ON or OFF) did not impact learning. Instead, the critical factor was drug state during the performance phase, with patients ON medication choosing correctly significantly more frequently than those OFF medication. This effect was independent of drug state during initial learning and appears to reflect a facilitation of generalization for learnt information. This inference is bolstered by our observation that neural activity in nucleus accumbens and ventromedial prefrontal cortex, measured during simultaneously acquired functional magnetic resonance imaging, represented learnt stimulus values during performance. This effect was expressed solely during the ON state with activity in these regions correlating with better performance. Our data indicate that dopamine modulation of nucleus accumbens and ventromedial prefrontal cortex exerts a specific effect on choice behaviour distinct from pure learning. The findings are in keeping with the substantial other evidence that certain aspects of learning are unaffected by dopamine lesions or depletion, and that dopamine plays a key role in performance that may be distinct from its role in learning. © 2012 The Author.

Simultaneous reduction of particulate matter and NOx emissions using 4-way catalyzed filtration systems

The next generation of diesel emission control devices includes 4-way catalyzed filtration systems (4WCFS) consisting of both NOx and diesel particulate matter (DPM) control. A methodology was developed to simultaneously evaluate the NOx and DPM control performance of miniature 4WCFS made from acicular mullite, an advanced ceramic material (ACM), that were challenged with diesel exhaust. The impact of catalyst loading and substrate porosity on catalytic performance of the NOx trap was evaluated. Simultaneously with NOx measurements, the real-time solid particle filtration performance of catalyst-coated standard and high porosity filters was determined for steady-state and regenerative conditions. The use of high porosity ACM 4-way catalyzed filtration systems reduced NOx by 99% and solid and total particulate matter by 95% when averaged over 10 regeneration cycles. A "regeneration cycle" refers to an oxidizing ("lean") exhaust condition followed by a reducing ("rich") exhaust condition resulting in NOx storage and NOx reduction (i.e., trap "regeneration"), respectively. Standard porosity ACM 4-way catalyzed filtration systems reduced NOx by 60-75% and exhibited 99.9% filtration efficiency. The rich/lean cycling used to regenerate the filter had almost no impact on solid particle filtration efficiency but impacted NOx control. Cycling resulted in the formation of very low concentrations of semivolatile nucleation mode particles for some 4WCFS formulations. Overall, 4WCFS show promise for significantly reducing diesel emissions into the atmosphere in a single control device. © 2013 American Chemical Society.

Reinforcement learning using a continuous time actor-critic framework with spiking neurons.

Animals repeat rewarded behaviors, but the physiological basis of reward-based learning has only been partially elucidated. On one hand, experimental evidence shows that the neuromodulator dopamine carries information about rewards and affects synaptic plasticity. On the other hand, the theory of reinforcement learning provides a framework for reward-based learning. Recent models of reward-modulated spike-timing-dependent plasticity have made first steps towards bridging the gap between the two approaches, but faced two problems. First, reinforcement learning is typically formulated in a discrete framework, ill-adapted to the description of natural situations. Second, biologically plausible models of reward-modulated spike-timing-dependent plasticity require precise calculation of the reward prediction error, yet it remains to be shown how this can be computed by neurons. Here we propose a solution to these problems by extending the continuous temporal difference (TD) learning of Doya (2000) to the case of spiking neurons in an actor-critic network operating in continuous time, and with continuous state and action representations. In our model, the critic learns to predict expected future rewards in real time. Its activity, together with actual rewards, conditions the delivery of a neuromodulatory TD signal to itself and to the actor, which is responsible for action choice. In simulations, we show that such an architecture can solve a Morris water-maze-like navigation task, in a number of trials consistent with reported animal performance. We also use our model to solve the acrobot and the cartpole problems, two complex motor control tasks. Our model provides a plausible way of computing reward prediction error in the brain. Moreover, the analytically derived learning rule is consistent with experimental evidence for dopamine-modulated spike-timing-dependent plasticity.

Particulate matter emission during start-up and transient operation of a spark-ignition engine

In order to understand why emissions of Particulate Matter (PM) from Spark-Ignition (SI) automobiles peak during periods of transient operation such as rapid accelerations, a study of controlled, repeatable transients was performed. Time-resolved engine-out PM emissions from a modern four-cylinder engine during transient load and air/fuel ratio operation were examined, and the results could be fit in most cases to a first order time response. The time constants for the transient response are similar to those measured for changes in intake valve temperature, reflecting the strong dependence of PM emissions on the amount of liquid fuel in the combustion chamber. In only one unrepeatable case did the time response differ from a first order function: showing an overshoot in PM emissions during transition from the initial to the final steady state PM emission level. PM emissions during controlled, motored start-up experiments show a peak at start-up followed by a period during which emissions are either relatively constant or drift somewhat. When the fuel injection and ignition are shut off, PM emissions also peak briefly, but rapidly decay to low levels. Qualitative implications on the study and modeling of PM emissions during transient engine operation are discussed. Copyright © 1999 Society of Automotive Engineers, Inc.

Investigation of the dilution process for measurement of particulate matter from spark-ignition engines

Measurements of particulate matter (PM) from spark ignition (SI) engine exhaust using dilution tunnels will become more prevalent as emission standards are tightened. Hence, a study of the dilution process was undertaken in order to understand how various dilution related parameters affect the accuracy with which PM sizes and concentrations can be determined. A SI and a compression ignition (CI) engine were separately used to examine parameters of the dilution process; the present work discusses the results in the context of SI exhaust dilution. A Scanning Mobility Particle Sizer (SMPS) was used to measure the size distribution, number density, and volume fraction of PM. Temperature measurements in the exhaust pipe and dilution tunnel reveal the degree of mixing between exhaust and dilution air, the effect of flowrate on heat transfer from undiluted and diluted exhaust to the environment, and the minimum permissible dilution ratio for a maximum sample temperature of 52°C. Measurements of PM concentrations as a function of dilution ratio show the competing effects of temperature and particle/vapor concentrations on particle growth dynamics, which result in a range of dilution ratios-from 13 to 18-where the effect of dilution ratio, independent of flowrate, is kept to a minimum. This range of dilution ratios is therefore optimal in order to achieve repeatable PM concentration measurements. Particle dynamics during transit through the tunnel operating at the optimal dilution ratio was found statistically insignificant compared to data scatter. Such small differences in number concentration may be qualitatively representative of particle losses for SI exhaust, but small increases in PM volume fraction during transit through the tunnel may significantly underestimate accretion of mass due to unburned hydrocarbons (HCs) emitted by SI engines. The fraction of SI-derived PM mass due to adsorbed/absorbed vapor, estimated from these data, is consistent with previous chemical analyses of PM. © 1998 Society of Automotive Engineers, Inc.

An analysis of competing toughening mechanisms in layered and particulate solids

The relative potency of common toughening mechanisms is explored for layered solids and particulate solids, with an emphasis on crack multiplication and plasticity. First, the enhancement in toughness due to a parallel array of cracks in an elastic solid is explored, and the stability of co-operative cracking is quantified. Second, the degree of synergistic toughening is determined for combined crack penetration and crack kinking at the tip of a macroscopic, mode I crack; specifically, the asymptotic problem of self-similar crack advance (penetration mode) versus 90 ° symmetric kinking is considered for an isotropic, homogeneous solid with weak interfaces. Each interface is treated as a cohesive zone of finite strength and toughness. Third, the degree of toughening associated with crack multiplication is assessed for a particulate solid comprising isotropic elastic grains of hexagonal shape, bonded by cohesive zones of finite strength and toughness. The study concludes with the prediction of R-curves for a mode I crack in a multi-layer stack of elastic and elastic-plastic solids. A detailed comparison of the potency of the above mechanisms and their practical application are given. In broad terms, crack tip kinking can be highly potent, whereas multiple cracking is difficult to activate under quasi-static conditions. Plastic dissipation can give a significant toughening in multi-layers especially at the nanoscale. © 2013 Springer Science+Business Media Dordrecht.

Does temporal discounting explain unhealthy behavior? A systematic review and reinforcement learning perspective.

The tendency to make unhealthy choices is hypothesized to be related to an individual's temporal discount rate, the theoretical rate at which they devalue delayed rewards. Furthermore, a particular form of temporal discounting, hyperbolic discounting, has been proposed to explain why unhealthy behavior can occur despite healthy intentions. We examine these two hypotheses in turn. We first systematically review studies which investigate whether discount rates can predict unhealthy behavior. These studies reveal that high discount rates for money (and in some instances food or drug rewards) are associated with several unhealthy behaviors and markers of health status, establishing discounting as a promising predictive measure. We secondly examine whether intention-incongruent unhealthy actions are consistent with hyperbolic discounting. We conclude that intention-incongruent actions are often triggered by environmental cues or changes in motivational state, whose effects are not parameterized by hyperbolic discounting. We propose a framework for understanding these state-based effects in terms of the interplay of two distinct reinforcement learning mechanisms: a "model-based" (or goal-directed) system and a "model-free" (or habitual) system. Under this framework, while discounting of delayed health may contribute to the initiation of unhealthy behavior, with repetition, many unhealthy behaviors become habitual; if health goals then change, habitual behavior can still arise in response to environmental cues. We propose that the burgeoning development of computational models of these processes will permit further identification of health decision-making phenotypes.