Reinforcement learning using a continuous time actor-critic framework with spiking neurons.

Animals repeat rewarded behaviors, but the physiological basis of reward-based learning has only been partially elucidated. On one hand, experimental evidence shows that the neuromodulator dopamine carries information about rewards and affects synaptic plasticity. On the other hand, the theory of reinforcement learning provides a framework for reward-based learning. Recent models of reward-modulated spike-timing-dependent plasticity have made first steps towards bridging the gap between the two approaches, but faced two problems. First, reinforcement learning is typically formulated in a discrete framework, ill-adapted to the description of natural situations. Second, biologically plausible models of reward-modulated spike-timing-dependent plasticity require precise calculation of the reward prediction error, yet it remains to be shown how this can be computed by neurons. Here we propose a solution to these problems by extending the continuous temporal difference (TD) learning of Doya (2000) to the case of spiking neurons in an actor-critic network operating in continuous time, and with continuous state and action representations. In our model, the critic learns to predict expected future rewards in real time. Its activity, together with actual rewards, conditions the delivery of a neuromodulatory TD signal to itself and to the actor, which is responsible for action choice. In simulations, we show that such an architecture can solve a Morris water-maze-like navigation task, in a number of trials consistent with reported animal performance. We also use our model to solve the acrobot and the cartpole problems, two complex motor control tasks. Our model provides a plausible way of computing reward prediction error in the brain. Moreover, the analytically derived learning rule is consistent with experimental evidence for dopamine-modulated spike-timing-dependent plasticity.

A PDE viewpoint on basic properties of coordination algorithms with symmetries

Several recent control applications consider the coordination of subsystems through local interaction. Often the interaction has a symmetry in state space, e.g. invariance with respect to a uniform translation of all subsystem values. The present paper shows that in presence of such symmetry, fundamental properties can be highlighted by viewing the distributed system as the discrete approximation of a partial differential equation. An important fact is that the symmetry on the state space differs from the popular spatial invariance property, which is not necessary for the present results. The relevance of the viewpoint is illustrated on two examples: (i) ill-conditioning of interaction matrices in coordination/consensus problems and (ii) the string instability issue. ©2009 IEEE.

Time-optimal control of a 3-level quantum system and its generalization

We solve the problem of steering a three-level quantum system from one eigen-state to another in minimum time and study its possible extension to the time-optimal control problem for a general n-level quantum system. For the three-level system we find all optimal controls by finding two types of symmetry in the problems: ℤ × S3 discrete symmetry and 51 continuous symmetry, and exploiting them to solve the problem through discrete reduction and symplectic reduction. We then study the geometry, in the same framework, which occurs in the time-optimal control of a general n-level quantum system. Copyright ©2007 Watam Press.

Continuous dynamical systems that realize discrete optimization on the hypercube

We study the problem of finding a local minimum of a multilinear function E over the discrete set {0,1}n. The search is achieved by a gradient-like system in [0,1]n with cost function E. Under mild restrictions on the metric, the stable attractors of the gradient-like system are shown to produce solutions of the problem, even when they are not in the vicinity of the discrete set {0,1}n. Moreover, the gradient-like system connects with interior point methods for linear programming and with the analog neural network studied by Vidyasagar (IEEE Trans. Automat. Control 40 (8) (1995) 1359), in the same context. © 2004 Elsevier B.V. All rights reserved.

Plane-strain discrete dislocation plasticity with climb-assisted glide motion of dislocations

A small-strain two-dimensional discrete dislocation plasticity (DDP) framework is developed wherein dislocation motion is caused by climb-assisted glide. The climb motion of the dislocations is assumed to be governed by a drag-type relation similar to the glide-only motion of dislocations: such a relation is valid when vacancy kinetics is either diffusion limited or sink limited. The DDP framework is employed to predict the effect of dislocation climb on the uniaxial tensile and pure bending response of single crystals. Under uniaxial tensile loading conditions, the ability of dislocations to bypass obstacles by climb results in a reduced dislocation density over a wide range of specimen sizes in the climb-assisted glide case compared to when dislocation motion is only by glide. A consequence is that, at least in a single slip situation, size effects due to dislocation starvation are reduced. By contrast, under pure bending loading conditions, the dislocation density is unaffected by dislocation climb as geometrically necessary dislocations (GNDs) dominate. However, climb enables the dislocations to arrange themselves into lower energy configurations which significantly reduces the predicted bending size effect as well as the amount of reverse plasticity observed during unloading. The results indicate that the intrinsic plasticity material length scale associated with GNDs is strongly affected by thermally activated processes and will be a function of temperature. © 2013 IOP Publishing Ltd.

The impact of sand slugs against beams and plates: Coupled discrete particle/finite element simulations

The impact of a slug of dry sand particles against a metallic sandwich beam or circular sandwich plate is analysed in order to aid the design of sandwich panels for shock mitigation. The sand particles interact via a combined linear-spring-and-dashpot law whereas the face sheets and compressible core of the sandwich beam and plate are treated as rate-sensitive, elastic-plastic solids. The majority of the calculations are performed in two dimensions and entail the transverse impact of end-clamped monolithic and sandwich beams, with plane strain conditions imposed. The sand slug is of rectangular shape and comprises a random loose packing of identical, circular cylindrical particles. These calculations reveal that loading due to the sand is primarily inertial in nature with negligible fluid-structure interaction: the momentum transmitted to the beam is approximately equal to that of the incoming sand slug. For a slug of given incoming momentum, the dynamic deflection of the beam increases with decreasing duration of sand-loading until the impulsive limit is attained. Sandwich beams with thick, strong cores significantly outperform monolithic beams of equal areal mass. This performance enhancement is traced to the "sandwich effect" whereby the sandwich beams have a higher bending strength than that of the monolithic beams. Three-dimensional (3D) calculations are also performed such that the sand slug has the shape of a circular cylindrical column of finite height, and contains spherical sand particles. The 3D slug impacts a circular monolithic plate or sandwich plate and we show that sandwich plates with thick strong cores again outperform monolithic plates of equal areal mass. Finally, we demonstrate that impact by sand particles is equivalent to impact by a crushable foam projectile. The calculations on the equivalent projectile are significantly less intensive computationally, yet give predictions to within 5% of the full discrete particle calculations for the monolithic and sandwich beams and plates. These foam projectile calculations suggest that metallic foam projectiles can be used to simulate the loading by sand particles within a laboratory setting. © 2013 Elsevier Ltd.

Study on small-strain behaviours of methane hydrate sandy sediments using discrete element method

Methane hydrate bearing soil has attracted increasing interest as a potential energy resource where methane gas can be extracted from dissociating hydrate-bearing sediments. Seismic testing techniques have been applied extensively and in various ways, to detect the presence of hydrates, due to the fact that hydrates increase the stiffness of hydrate-bearing sediments. With the recognition of the limitations of laboratory and field tests, wave propagation modelling using Discrete Element Method (DEM) was conducted in this study in order to provide some particle-scale insights on the hydrate-bearing sandy sediment models with pore-filling and cementation hydrate distributions. The relationship between shear wave velocity and hydrate saturation was established by both DEM simulations and analytical solutions. Obvious differences were observed in the dependence of wave velocity on hydrate saturation for these two cases. From the shear wave velocity measurement and particle-scale analysis, it was found that the small-strain mechanical properties of hydrate-bearing sandy sediments are governed by both the hydrate distribution patterns and hydrate saturation. © 2013 AIP Publishing LLC.

Coupling model uncertainty for coupled rainfall/runoff and surface water quality models in river problems

Coupled hydrology and water quality models are an important tool today, used in the understanding and management of surface water and watershed areas. Such problems are generally subject to substantial uncertainty in parameters, process understanding, and data. Component models, drawing on different data, concepts, and structures, are affected differently by each of these uncertain elements. This paper proposes a framework wherein the response of component models to their respective uncertain elements can be quantified and assessed, using a hydrological model and water quality model as two exemplars. The resulting assessments can be used to identify model coupling strategies that permit more appropriate use and calibration of individual models, and a better overall coupled model response. One key finding was that an approximate balance of water quality and hydrological model responses can be obtained using both the QUAL2E and Mike11 water quality models. The balance point, however, does not support a particularly narrow surface response (or stringent calibration criteria) with respect to the water quality calibration data, at least in the case examined here. Additionally, it is clear from the results presented that the structural source of uncertainty is at least as significant as parameter-based uncertainties in areal models. © 2012 John Wiley & Sons, Ltd.

Climb-enabled discrete dislocation plasticity

A small strain two-dimensional discrete dislocation plasticity framework coupled to vacancy diffusion is developed wherein the motion of edge dislocations is by a combination of glide and climb. The dislocations are modelled as line defects in a linear elastic medium and the mechanical boundary value problem is solved by the superposition of the infinite medium elastic fields of the dislocations and a complimentary non-singular solution that enforces the boundary conditions. Similarly, the climbing dislocations are modelled as line sources/sinks of vacancies and the vacancy diffusion boundary value problem is also solved by a superposition of the fields of the line sources/sinks in an infinite medium and a complementary non-singular solution that enforces the boundary conditions. The vacancy concentration field along with the stress field provides the climb rate of the dislocations. Other short-range interactions of the dislocations are incorporated via a set of constitutive rules. We first employ this formulation to investigate the climb of a single edge dislocation in an infinite medium and illustrate the existence of diffusion-limited and sink-limited climb regimes. Next, results are presented for the pure bending and uniaxial tension of single crystals oriented for single slip. These calculations show that plasticity size effects are reduced when dislocation climb is permitted. Finally, we contrast predictions of this coupled framework with an ad hoc model in which dislocation climb is modelled by a drag-type relation based on a quasi steady-state solution. © 2013 Elsevier Ltd. All rights reserved.

Adapting data processing to compare model and experiment accurately: A discrete element model and magnetic resonance measurements of a 3d cylindrical fluidized bed

Discrete element modeling is being used increasingly to simulate flow in fluidized beds. These models require complex measurement techniques to provide validation for the approximations inherent in the model. This paper introduces the idea of modeling the experiment to ensure that the validation is accurate. Specifically, a 3D, cylindrical gas-fluidized bed was simulated using a discrete element model (DEM) for particle motion coupled with computational fluid dynamics (CFD) to describe the flow of gas. The results for time-averaged, axial velocity during bubbling fluidization were compared with those from magnetic resonance (MR) experiments made on the bed. The DEM-CFD data were postprocessed with various methods to produce time-averaged velocity maps for comparison with the MR results, including a method which closely matched the pulse sequence and data processing procedure used in the MR experiments. The DEM-CFD results processed with the MR-type time-averaging closely matched experimental MR results, validating the DEM-CFD model. Analysis of different averaging procedures confirmed that MR time-averages of dynamic systems correspond to particle-weighted averaging, rather than frame-weighted averaging, and also demonstrated that the use of Gaussian slices in MR imaging of dynamic systems is valid. © 2013 American Chemical Society.

Material point method for coupled hydromechanical problems

This paper describes a new formulation of the material point method (MPM) for solving coupled hydromechanical problems of fluid-saturated soil subjected to large deformation. A soil-pore fluid coupled MPM algorithm based on Biot's mixture theory is proposed for solving hydromechanical interaction problems that include changes in water table location with time. The accuracy of the proposed method is examined by comparing the results of the simulation of a one-dimensional consolidation test with the corresponding analytical solution. A sensitivity analysis of the MPM parameters used in the proposed method is carried out for examining the effect of the number of particles per mesh and mesh size on solution accuracy. For demonstrating the capability of the proposed method, a physical model experiment of a large-scale levee failure by seepage is simulated. The behavior of the levee model with time-dependent changes in water table matches well to the experimental observations. The mechanisms of seepage-induced failure are discussed by examining the pore-water pressures, as well as the effective stresses computed from the simulations © 2013 American Society of Civil Engineers.

Novel fluid grid and voidage calculation techniques for a discrete element model of a 3D cylindrical fluidized bed

A discrete element model (DEM) combined with computational fluid dynamics (CFD) was developed to model particle and fluid behaviour in 3D cylindrical fluidized beds. Novel techniques were developed to (1) keep fluid cells, defined in cylindrical coordinates, at a constant volume in order to ensure the conditions for validity of the volume-averaged fluid equations were satisfied and (2) smoothly and accurately measure voidage in arbitrarily shaped fluid cells. The new technique for calculating voidage was more stable than traditional techniques, also examined in the paper, whilst remaining computationally-effective. The model was validated by quantitative comparison with experimental results from the magnetic resonance imaging of a fluidised bed analysed to give time-averaged particle velocities. Comparisons were also made between theoretical determinations of slug rise velocity in a tall bed. It was concluded that the DEM-CFD model is able to investigate aspects of the underlying physics of fluidisation not readily investigated by experiment. © 2014 The Authors.