Reinforcement learning using a continuous time actor-critic framework with spiking neurons.

Animals repeat rewarded behaviors, but the physiological basis of reward-based learning has only been partially elucidated. On one hand, experimental evidence shows that the neuromodulator dopamine carries information about rewards and affects synaptic plasticity. On the other hand, the theory of reinforcement learning provides a framework for reward-based learning. Recent models of reward-modulated spike-timing-dependent plasticity have made first steps towards bridging the gap between the two approaches, but faced two problems. First, reinforcement learning is typically formulated in a discrete framework, ill-adapted to the description of natural situations. Second, biologically plausible models of reward-modulated spike-timing-dependent plasticity require precise calculation of the reward prediction error, yet it remains to be shown how this can be computed by neurons. Here we propose a solution to these problems by extending the continuous temporal difference (TD) learning of Doya (2000) to the case of spiking neurons in an actor-critic network operating in continuous time, and with continuous state and action representations. In our model, the critic learns to predict expected future rewards in real time. Its activity, together with actual rewards, conditions the delivery of a neuromodulatory TD signal to itself and to the actor, which is responsible for action choice. In simulations, we show that such an architecture can solve a Morris water-maze-like navigation task, in a number of trials consistent with reported animal performance. We also use our model to solve the acrobot and the cartpole problems, two complex motor control tasks. Our model provides a plausible way of computing reward prediction error in the brain. Moreover, the analytically derived learning rule is consistent with experimental evidence for dopamine-modulated spike-timing-dependent plasticity.

Plastic limit solution for the response of plate anchors under combined translational and torsional undrained loading

Plate anchors are increasingly being used to moor large floating offshore structures in deep and ultradeep water. These facilities impart substantial vertical uplift loading to plate anchors. However, extreme operating conditions such as hurricane loading often result in partial system failures, with significant change in the orientation of the remaining intact mooring lines. The purpose of this study is to investigate the undrained pure translational (parallel to plate) and torsional bearing capacity of anchor plates idealized as square and rectangular shaped plates. Moreover, the interaction response of plate anchors under combined translational and torsional loading is studied using a modified plastic limit analysis (PLA) approach. The previous PLA formulation which did not account for shear-normal force interaction on the vertical end faces of the plate provides an exact solution to the idealized problem of an infinitely thin plate but only an approximate solution to the problem of a plate of finite thickness. This is also confirmed by the three-dimensional finite element (FE) results, since the PLA values exceed FE results as the thickness of the plate increases. By incorporating the shear-normal interaction relationship in the modified solution, the torsional bearing capacity factors, as well as the plate interaction responses are enhanced as they show satisfactory agreement with the FE results. The interaction relationship is then obtained for square and rectangular plates of different aspect ratios and thicknesses. The new interaction relationships could also be used as an associated plastic failure locus for combined shear and torsional loading to predict plastic displacements and rotations in translational and torsional loading modes as well. Copyright © 2011 by the International Society of Offshore and Polar Engineers (ISOPE).

Novel ZnO nanorod films by chemical solution deposition for planar device applications.

Smooth and continuous ZnO films consisting of densely packed ZnO nanorods (NRs), which can be used for electronic device fabrication, were synthesized using a hydro-thermo-chemical solution deposition method. Such devices would have the novelty of high performance, benefiting from the inherited unique properties of the nanomaterials, and can be fabricated on these smooth films using a conventional, low cost planar process. Photoluminescence measurements showed that the NR films have much stronger shallow donor to valence band emissions than those from discrete ZnO NRs, and hence have the potential for the development of ZnO light emission diodes and lasers, etc. The NR films have been used to fabricate large area surface acoustic wave devices by conventional photolithography. These demonstrated two well-defined resonant peaks and their potential for large area device applications. The chemical solution deposition method is simple, reproducible, scalable and economic. These NR films are suitable for large scale production on cost-effective substrates and are promising for various fields such as sensing systems, renewable energy and optoelectronic applications.

Structural and optical characterization of vertical GaAs/GaP core-shell nanowires grown on Si substrates

GaAs nanowires were grown on Si (111) substrates. By coating a thin GaAs buffer layer on Si surface and using a two-temperature growth, the morphology and crystal structure of GaAs nanowires were dramatically improved. The strained GaAs/GaP core-shell nanowires, based on the improved GaAs nanowires with a shell thickness of 25 nm, showed a significant shift in emission energy of 260 meV from the unstrained GaAs nanowires. © 2010 IEEE.

Spontaneous emission from radiative chiral nematic liquid crystals at the photonic band-gap edge: An investigation into the role of the density of photon states near resonance

In this article, we investigate the spontaneous emission properties of radiating molecules embedded in a chiral nematic liquid crystal, under the assumption that the electronic transition frequency is close to the photonic edge mode of the structure, i.e., at resonance. We take into account the transition broadening and the decay of electromagnetic field modes supported by the so-called "mirrorless"cavity. We employ the Jaynes-Cummings Hamiltonian to describe the electron interaction with the electromagnetic field, focusing on the mode with the diffracting polarization in the chiral nematic layer. As known in these structures, the density of photon states, calculated via the Wigner method, has distinct peaks on either side of the photonic band gap, which manifests itself as a considerable modification of the emission spectrum. We demonstrate that, near resonance, there are notable differences between the behavior of the density of states and the spontaneous emission profile of these structures. In addition, we examine in some detail the case of the logarithmic peak exhibited in the density of states in two-dimensional photonic structures and obtain analytic relations for the Lamb shift and the broadening of the atomic transition in the emission spectrum. The dynamical behavior of the atom-field system is described by a system of two first-order differential equations, solved using the Green's-function method and the Fourier transform. The emission spectra are then calculated and compared with experimental data. © 2013 American Physical Society.

Spontaneous emission from radiative chiral nematic liquid crystals at the photonic band-gap edge: an investigation into the role of the density of photon states near resonance.

In this article, we investigate the spontaneous emission properties of radiating molecules embedded in a chiral nematic liquid crystal, under the assumption that the electronic transition frequency is close to the photonic edge mode of the structure, i.e., at resonance. We take into account the transition broadening and the decay of electromagnetic field modes supported by the so-called "mirrorless"cavity. We employ the Jaynes-Cummings Hamiltonian to describe the electron interaction with the electromagnetic field, focusing on the mode with the diffracting polarization in the chiral nematic layer. As known in these structures, the density of photon states, calculated via the Wigner method, has distinct peaks on either side of the photonic band gap, which manifests itself as a considerable modification of the emission spectrum. We demonstrate that, near resonance, there are notable differences between the behavior of the density of states and the spontaneous emission profile of these structures. In addition, we examine in some detail the case of the logarithmic peak exhibited in the density of states in two-dimensional photonic structures and obtain analytic relations for the Lamb shift and the broadening of the atomic transition in the emission spectrum. The dynamical behavior of the atom-field system is described by a system of two first-order differential equations, solved using the Green's-function method and the Fourier transform. The emission spectra are then calculated and compared with experimental data.

Water, salt water, and alkaline solution uptake in epoxy thin films

As a means of characterizing the diffusion parameters of fiber reinforced polymer (FRP) composites within a relatively short time frame, the potential use of short term tests on epoxy films to predict the long-term behavior is investigated. Reference is made to the literature to assess the effectiveness of Fickian and anomalous diffusion models to describe solution uptake in epoxies. The influence of differing exposure conditions on the diffusion in epoxies, in particular the effect of solution type and temperature, are explored. Experimental results, where the solution uptake in desiccated (D) or undesiccated (U) thin films of a commercially available epoxy matrix subjected to water (W), salt water (SW), or alkali concrete pore solution (CPS) at either 20 or 60°C, are also presented. It was found that the type of solution did not significantly influence the diffusion behavior at 20°C and that the mass uptake profile was anomalous. Exposure to 60°C accelerated the initial diffusion behavior and appeared to raise the level of saturation. In spite of the accelerated approach, conclusive values of uptake at saturation remained elusive even at an exposure period of 5 years. This finding questions the viability of using short-term thin film results to predict the long-term mechanical performance of FRP materials. © 2013 Wiley Periodicals, Inc.

Tests of an adaptive QM/MM calculation on free energy profiles of chemical reactions in solution.

We present reaction free energy calculations using the adaptive buffered force mixing quantum mechanics/molecular mechanics (bf-QM/MM) method. The bf-QM/MM method combines nonadaptive electrostatic embedding QM/MM calculations with extended and reduced QM regions to calculate accurate forces on all atoms, which can be used in free energy calculation methods that require only the forces and not the energy. We calculate the free energy profiles of two reactions in aqueous solution: the nucleophilic substitution reaction of methyl chloride with a chloride anion and the deprotonation reaction of the tyrosine side chain. We validate the bf-QM/MM method against a full QM simulation, and show that it correctly reproduces both geometrical properties and free energy profiles of the QM model, while the electrostatic embedding QM/MM method using a static QM region comprising only the solute is unable to do so. The bf-QM/MM method is not explicitly dependent on the details of the QM and MM methods, so long as it is possible to compute QM forces in a small region and MM forces in the rest of the system, as in a conventional QM/MM calculation. It is simple, with only a few parameters needed to control the QM calculation sizes, and allows (but does not require) a varying and adapting QM region which is necessary for simulating solutions.