Reinforcement learning using a continuous time actor-critic framework with spiking neurons.

Animals repeat rewarded behaviors, but the physiological basis of reward-based learning has only been partially elucidated. On one hand, experimental evidence shows that the neuromodulator dopamine carries information about rewards and affects synaptic plasticity. On the other hand, the theory of reinforcement learning provides a framework for reward-based learning. Recent models of reward-modulated spike-timing-dependent plasticity have made first steps towards bridging the gap between the two approaches, but faced two problems. First, reinforcement learning is typically formulated in a discrete framework, ill-adapted to the description of natural situations. Second, biologically plausible models of reward-modulated spike-timing-dependent plasticity require precise calculation of the reward prediction error, yet it remains to be shown how this can be computed by neurons. Here we propose a solution to these problems by extending the continuous temporal difference (TD) learning of Doya (2000) to the case of spiking neurons in an actor-critic network operating in continuous time, and with continuous state and action representations. In our model, the critic learns to predict expected future rewards in real time. Its activity, together with actual rewards, conditions the delivery of a neuromodulatory TD signal to itself and to the actor, which is responsible for action choice. In simulations, we show that such an architecture can solve a Morris water-maze-like navigation task, in a number of trials consistent with reported animal performance. We also use our model to solve the acrobot and the cartpole problems, two complex motor control tasks. Our model provides a plausible way of computing reward prediction error in the brain. Moreover, the analytically derived learning rule is consistent with experimental evidence for dopamine-modulated spike-timing-dependent plasticity.

A hybrid systems approach to trajectory tracking control for juggling systems

From a hybrid systems point of view, we provide a modeling framework and a trajectory tracking control design methodology for juggling systems. We present the main ideas and concepts in a one degree-of-freedom juggler, which consists of a ball bouncing on an actuated robot. We design a hybrid control strategy that, with only information of the ball's state at impacts, controls the ball to track a reference rhythmic pattern with arbitrary precision. We extend this hybrid control strategy to the case of juggling multiple balls with different rhythmic patterns. Simulation results for juggling of one and three balls with a single actuated robot are presented. © 2007 IEEE.

Time-optimal control of a 3-level quantum system and its generalization to an n-level system

We solve the problem of steering a three-level quantum system from one eigen-state to another in minimum time and study its possible extension to the time-optimal control problem for a general n-level quantum system. For the three-level system we find all optimal controls by finding two types of symmetry in the problem: ℤ2 × S3 discrete symmetry and S1 continuous symmetry, and exploiting them to solve the problem through discrete reduction and symplectic reduction. We then study the geometry, in the same framework, which occurs in the time-optimal control of a general n-level quantum system. © 2007 IEEE.

Time-optimal control of a 3-level quantum system and its generalization

We solve the problem of steering a three-level quantum system from one eigen-state to another in minimum time and study its possible extension to the time-optimal control problem for a general n-level quantum system. For the three-level system we find all optimal controls by finding two types of symmetry in the problems: ℤ × S3 discrete symmetry and 51 continuous symmetry, and exploiting them to solve the problem through discrete reduction and symplectic reduction. We then study the geometry, in the same framework, which occurs in the time-optimal control of a general n-level quantum system. Copyright ©2007 Watam Press.

Expressive RFID data access policies for the Pharmaceuticals supply chain

The Pharma(ceuticals) industry is at a cross-roads. There are growing concerns that illegitimate products are penetrating the supply chain. There are proposals in many countries to apply RFID and other traceability technologies to solve this problem. However there are several trade-offs and one of the most crucial is between data visibility and confidentiality. In this paper, we use the TrakChain assessment framework tools to study the US Pharma supply chain and to compare candidate solutions to achieve traceability data security: Point-of-Dispense Authentication, Network-based electronic Pedigree, and Document-based electronic Pedigree. We also propose extensions to a supply chain authorization language that is able to capture expressive data sharing conditions considered necessary by the industry's trading partners. © 2013 IEEE.

A novel theoretical approach to passive control of thermo-acoustic oscillations: Application to ducted heat sources

In this paper, we develop a linear technique that predicts how the stability of a thermo-acoustic system changes due to the action of a generic passive feedback device or a generic change in the base state. From this, one can calculate the passive device or base state change that most stabilizes the system. This theoretical framework, based on adjoint equations, is applied to two types of Rijke tube. The first contains an electrically-heated hot wire and the second contains a diffusion flame. Both heat sources are assumed to be compact so that the acoustic and heat release models can be decoupled. We find that the most effective passive control device is an adiabatic mesh placed at the downstream end of the Rijke tube. We also investigate the effects of a second hot wire and a local variation of the cross-sectional area but find that both affect the frequency more than the growth rate. This application of adjoint sensitivity analysis opens up new possibilities for the passive control of thermo-acoustic oscillations. For example, the influence of base state changes can be combined with other constraints, such as that the total heat release rate remains constant, in order to show how an unstable thermo-acoustic system should be changed in order to make it stable. Copyright © 2013 by ASME.

A framework for distributed intelligent automation systems developments

This chapter proposes a simple and pragmatic framework that has been developed for classifying and analyzing developments in distributed automation and information systems - especially those that have been labelled intelligent systems for different reasons. The framework dissects the different stages in the standard feedback process and assesses distribution in terms of the level of granularity of the organization that is being considered. The framework has been found to be useful in comparing and assessing different distributed industrial control paradigms and also for examining common features of different development projects - especially those that might be sourced from different sectors or domains. © Springer-Verlag Berlin Heidelberg 2013.

Control limitations from distributed sensing: Theory and Extremely Large Telescope application

We investigate performance bounds for feedback control of distributed plants where the controller can be centralized (i.e. it has access to measurements from the whole plant), but sensors only measure differences between neighboring subsystem outputs. Such "distributed sensing" can be a technological necessity in applications where system size exceeds accuracy requirements by many orders of magnitude. We formulate how distributed sensing generally limits feedback performance robust to measurement noise and to model uncertainty, without assuming any controller restrictions (among others, no "distributed control" restriction). A major practical consequence is the necessity to cut down integral action on some modes. We particularize the results to spatially invariant systems and finally illustrate implications of our developments for stabilizing the segmented primary mirror of the European Extremely Large Telescope. © 2013 Elsevier Ltd. All rights reserved.

A differential lyapunov framework for contraction analysis

Lyapunov's second theorem is an essential tool for stability analysis of differential equations. The paper provides an analog theorem for incremental stability analysis by lifting the Lyapunov function to the tangent bundle. The Lyapunov function endows the state-space with a Finsler structure. Incremental stability is inferred from infinitesimal contraction of the Finsler metrics through integration along solutions curves. © 2013 IEEE.

Second-order rate region of constant-composition codes for the multiple-access channel

This paper presents an achievable second-order rate region for the discrete memoryless multiple-access channel. The result is obtained using a random-coding ensemble in which each user's codebook contains codewords of a fixed composition. It is shown that this ensemble performs at least as well as i.i.d. random coding in terms of second-order asymptotics, and an example is given where a strict improvement is observed. © 2013 IEEE.

Trajectory control based on discrete full-range dynamics

There has been an increasing interest in the use of mechanical dynamics, (e.g., assive, Elastic, And viscous dynamics) for energy efficient and agile control of robotic systems. Despite the impressive demonstrations of behavioural performance, The mechanical dynamics of this class of robotic systems is still very limited as compared to those of biological systems. For example, Passive dynamic walkers are not capable of generating joint torques to compensate for disturbances from complex environments. In order to tackle such a discrepancy between biological and artificial systems, We present the concept and design of an adaptive clutch mechanism that discretely covers the full-range of dynamics. As a result, The system is capable of a large variety of joint operations, including dynamic switching among passive, actuated and rigid modes. The main innovation of this paper is the framework and algorithm developed for controlling the trajectory of such joint. We present different control strategies that exploit passive dynamics. Simulation results demonstrate a significant improvement in motion control with respect to the speed of motion and energy efficiency. The actuator is implemented in a simple pendulum platform to quantitatively evaluate this novel approach.