Dopamine and performance in a reinforcement learning task: Evidence from Parkinson's disease

The role dopamine plays in decision-making has important theoretical, empirical and clinical implications. Here, we examined its precise contribution by exploiting the lesion deficit model afforded by Parkinson's disease. We studied patients in a two-stage reinforcement learning task, while they were ON and OFF dopamine replacement medication. Contrary to expectation, we found that dopaminergic drug state (ON or OFF) did not impact learning. Instead, the critical factor was drug state during the performance phase, with patients ON medication choosing correctly significantly more frequently than those OFF medication. This effect was independent of drug state during initial learning and appears to reflect a facilitation of generalization for learnt information. This inference is bolstered by our observation that neural activity in nucleus accumbens and ventromedial prefrontal cortex, measured during simultaneously acquired functional magnetic resonance imaging, represented learnt stimulus values during performance. This effect was expressed solely during the ON state with activity in these regions correlating with better performance. Our data indicate that dopamine modulation of nucleus accumbens and ventromedial prefrontal cortex exerts a specific effect on choice behaviour distinct from pure learning. The findings are in keeping with the substantial other evidence that certain aspects of learning are unaffected by dopamine lesions or depletion, and that dopamine plays a key role in performance that may be distinct from its role in learning. © 2012 The Author.

Reinforcement learning using a continuous time actor-critic framework with spiking neurons.

Animals repeat rewarded behaviors, but the physiological basis of reward-based learning has only been partially elucidated. On one hand, experimental evidence shows that the neuromodulator dopamine carries information about rewards and affects synaptic plasticity. On the other hand, the theory of reinforcement learning provides a framework for reward-based learning. Recent models of reward-modulated spike-timing-dependent plasticity have made first steps towards bridging the gap between the two approaches, but faced two problems. First, reinforcement learning is typically formulated in a discrete framework, ill-adapted to the description of natural situations. Second, biologically plausible models of reward-modulated spike-timing-dependent plasticity require precise calculation of the reward prediction error, yet it remains to be shown how this can be computed by neurons. Here we propose a solution to these problems by extending the continuous temporal difference (TD) learning of Doya (2000) to the case of spiking neurons in an actor-critic network operating in continuous time, and with continuous state and action representations. In our model, the critic learns to predict expected future rewards in real time. Its activity, together with actual rewards, conditions the delivery of a neuromodulatory TD signal to itself and to the actor, which is responsible for action choice. In simulations, we show that such an architecture can solve a Morris water-maze-like navigation task, in a number of trials consistent with reported animal performance. We also use our model to solve the acrobot and the cartpole problems, two complex motor control tasks. Our model provides a plausible way of computing reward prediction error in the brain. Moreover, the analytically derived learning rule is consistent with experimental evidence for dopamine-modulated spike-timing-dependent plasticity.

Optimizing the energy offset between dye and hole-transporting material in solid-state dye-sensitized solar cells

The power-conversion efficiency of solid-state dye-sensitized solar cells can be optimized by reducing the energy offset between the highest occupied molecular orbital (HOMO) levels of dye and hole-transporting material (HTM) to minimize the loss-in-potential. Here, we report a study of three novel HTMs with HOMO levels slightly above and below the one of the commonly used HTM 2,2′,7,7′- tetrakis(N,N-di-p-methoxyphenylamino)-9,9′- spirobifluorene (spiro-OMeTAD) to systematically explore this possibility. Using transient absorption spectroscopy and employing the ruthenium based dye Z907 as sensitizer, it is shown that, despite one new HTM showing a 100% hole-transfer yield, all devices based on the new HTMs performed worse than those incorporating spiro-OMeTAD. We further demonstrate that the design of the HTM has an additional impact on the electronic density of states present at the TiO2 electrode surface and hence influences not only hole- but also electron-transfer from the sensitizer. These results provide insight into the complex influence of the HTM on charge transfer and provide guidance for the molecular design of new materials. © 2013 American Chemical Society.

An integrated 2-D model of a lithium ion battery: The effect of material parameters and morphology on storage particle stress

An integrated 2-D model of a lithium ion battery is developed to study the mechanical stress in storage particles as a function of material properties. A previously developed coupled stress-diffusion model for storage particles is implemented in 2-D and integrated into a complete battery system. The effect of morphology on the stress and lithium concentration is studied for the case of extraction of lithium in terms of previously developed non-dimensional parameters. These non-dimensional parameters include the material properties of the storage particles in the system, among other variables. We examine particles functioning in isolation as well as in closely-packed systems. Our results show that the particle distance from the separator, in combination with the material properties of the particle, is critical in predicting the stress generated within the particle. © 2012 Springer-Verlag.

Behavior of RC beams retrofitted with CARDIFRC after thermal cycling

This study investigates the effect of thermal cycling on the performance of concrete beams retrofitted with CARDIFRC, a new class of high performance fiber-reinforced cement-based material that is compatible with concrete. Twenty four beams were subjected to 24 h thermal cycles between 25 and 90°C. One third of the beams were reinforced either in flexure only or in flexure and shear with conventional steel reinforcement and used as control specimens. The remaining sixteen beams were retrofitted with CARDIFRC strips to provide external flexural and/or shear strengthening. All beams were exposed to a varied number of 24 h thermal cycles ranging from 0 to 90 and were tested in four-point bending at room temperature. The tests indicated that the retrofitted members were stronger and stiffer than control beams, and more importantly, that their failure initiated in flexure without any signs of interfacial delamination cracking. The results of these tests are presented and compared to analytical predictions. The predictions show good correlation with the experimental results. © 2010 ASCE.

Material point method for coupled hydromechanical problems

This paper describes a new formulation of the material point method (MPM) for solving coupled hydromechanical problems of fluid-saturated soil subjected to large deformation. A soil-pore fluid coupled MPM algorithm based on Biot's mixture theory is proposed for solving hydromechanical interaction problems that include changes in water table location with time. The accuracy of the proposed method is examined by comparing the results of the simulation of a one-dimensional consolidation test with the corresponding analytical solution. A sensitivity analysis of the MPM parameters used in the proposed method is carried out for examining the effect of the number of particles per mesh and mesh size on solution accuracy. For demonstrating the capability of the proposed method, a physical model experiment of a large-scale levee failure by seepage is simulated. The behavior of the levee model with time-dependent changes in water table matches well to the experimental observations. The mechanisms of seepage-induced failure are discussed by examining the pore-water pressures, as well as the effective stresses computed from the simulations © 2013 American Society of Civil Engineers.

Control of material transport through pulse shape manipulation-a development toward designer pulses

The variety of laser systems available to industrial laser users is growing and the choice of the correct laser for a material target application is often based on an empirical assessment. Industrial master oscillator power amplifier systems with tuneable temporal pulse shapes have now entered the market, providing enormous pulse parameter flexibility in an already crowded parameter space. In this paper, an approach is developed to design interaction parameters based on observations of material responses. Energy and material transport mechanisms are studied using pulsed digital holography, post process analysis techniques and finite-difference modelling to understand the key response mechanisms for a variety of temporal pulse envelopes incident on a silicon (1/1/1) substrate. The temporal envelope is shown to be the primary control parameter of the source term that determines the subsequent material response and the resulting surface morphology. A double peak energy-bridged temporal pulse shape designed through direct application of holographic imaging data is shown to substantially improve surface quality. © 2014 IEEE.

Does temporal discounting explain unhealthy behavior? A systematic review and reinforcement learning perspective.

The tendency to make unhealthy choices is hypothesized to be related to an individual's temporal discount rate, the theoretical rate at which they devalue delayed rewards. Furthermore, a particular form of temporal discounting, hyperbolic discounting, has been proposed to explain why unhealthy behavior can occur despite healthy intentions. We examine these two hypotheses in turn. We first systematically review studies which investigate whether discount rates can predict unhealthy behavior. These studies reveal that high discount rates for money (and in some instances food or drug rewards) are associated with several unhealthy behaviors and markers of health status, establishing discounting as a promising predictive measure. We secondly examine whether intention-incongruent unhealthy actions are consistent with hyperbolic discounting. We conclude that intention-incongruent actions are often triggered by environmental cues or changes in motivational state, whose effects are not parameterized by hyperbolic discounting. We propose a framework for understanding these state-based effects in terms of the interplay of two distinct reinforcement learning mechanisms: a "model-based" (or goal-directed) system and a "model-free" (or habitual) system. Under this framework, while discounting of delayed health may contribute to the initiation of unhealthy behavior, with repetition, many unhealthy behaviors become habitual; if health goals then change, habitual behavior can still arise in response to environmental cues. We propose that the burgeoning development of computational models of these processes will permit further identification of health decision-making phenotypes.

Effect of natural dissolved humic material on bioavailability and acute toxicity of fenpropathrin to the grass carp, Ctenopharyngodon idellus

The effects of aquatic humic acids on the bioconcentration and acute toxicity of fenpropathrin were evaluated using grass carp, Ctenopharyngodan idellus, in laboratory freshwater systems. The results demonstrated that both bioavailability and acute toxicity decreased in the presence of aquatic humic acid 5 and 10 mg/liter. In addition, the extent of influence increased with increasing concentration of aquatic humic acid, (C) 1999 Academic Press.

The influence of growth temperature and input V/III ratio on the initial nucleation and material properties of InN on GaN by MOCVD

The effects of growth temperature and V/III ratio on the InN initial nucleation of islands on the GaN (0 0 0 1) surface were investigated. It is found that InN nuclei density increases with decreasing growth temperature between 375 and 525 degrees C. At lower growth temperatures, InN thin films take the form of small and closely packed islands with diameters of less than 100 nm, whereas at elevated temperatures the InN islands can grow larger and well separated, approaching an equilibrium hexagonal shape due to enhanced surface diffusion of adatoms. At a given growth temperature of 500 degrees C, a controllable density and size of separated InN islands can be achieved by adjusting the V/III ratio. The larger islands lead to fewer defects when they are coalesced. Comparatively, the electrical properties of the films grown under higher V/III ratio are improved.

Radiation hardness improvement of separation-by-implantation-of-oxygen/silicon-on-insulator material by nitrogen ion implantation

In our work, nitrogen ions were implanted into separation-by-implantation-of-oxygen (SIMOX) wafers to improve the radiation hardness of the SIMOX material. The experiments of secondary ion mass spectroscopy (SIMS) analysis showed that some nitrogen ions were distributed in the buried oxide layers and some others were collected at the Si/SiO2 interface after annealing. The results of electron paramagnetic resonance (EPR) suggested the density of the defects in the nitrided samples changed with different nitrogen ion implantation energies. Semiconductor-insulator-semiconductor (SIS) capacitors were made on the materials, and capacitance-voltage (C-V) measurements were carried out to confirm the results. The super total dose radiation tolerance of the materials was verified by the small increase of the drain leakage current of the metal-oxide-semiconductor field effect transistor with n-channel (NMOSFETs) fabricated on the materials before and after total dose irradiation. The optimum implantation energy was also determined.