Reinforcement learning with reference tracking control in continuous state spaces

The contribution described in this paper is an algorithm for learning nonlinear, reference tracking, control policies given no prior knowledge of the dynamical system and limited interaction with the system through the learning process. Concepts from the field of reinforcement learning, Bayesian statistics and classical control have been brought together in the formulation of this algorithm which can be viewed as a form of indirect self tuning regulator. On the task of reference tracking using a simulated inverted pendulum it was shown to yield generally improved performance on the best controller derived from the standard linear quadratic method using only 30 s of total interaction with the system. Finally, the algorithm was shown to work on the simulated double pendulum proving its ability to solve nontrivial control tasks. © 2011 IEEE.

Partially Observable Markov Decision Processes with continuous observations for dialogue management

This work shows how a dialogue model can be represented as a Partially Observable Markov Decision Process (POMDP) with observations composed of a discrete and continuous component. The continuous component enables the model to directly incorporate a confidence score for automated planning. Using a testbed simulated dialogue management problem, we show how recent optimization techniques are able to find a policy for this continuous POMDP which outperforms a traditional MDP approach. Further, we present a method for automatically improving handcrafted dialogue managers by incorporating POMDP belief state monitoring, including confidence score information. Experiments on the testbed system show significant improvements for several example handcrafted dialogue managers across a range of operating conditions.

Continuous Diffraction Patterns from Circular Arrays of Carbon Nanotubes

We report the remarkable diffraction effects produced from circular patterned arrays of multiwalled carbon nanotubes (MWCNTs). Highly ordered circular arrays of multiwalled carbon nanotubes (with inter-nanotube spacings of 633 nm) display optical dispersion effects similar to compact discs. These arrays display remarkable diffraction patterns in the far field which are spatially continuous. High quality diffraction patterns were obtained experimentally which are in excellent agreement with the theoretical calculations. The achieved continuous diffraction patterns pave the way towards the utilization of engineered carbon nanotube arrays in applications like three dimensional holograms.

On the transition from finite-volume negatively buoyant releases to continuous fountains

An experimental investigation to identify the source conditions that distinguish finite-volume negatively buoyant fluid projectile behaviour from fountain behaviour in quiescent environments of uniform density is described. Finite-volume releases are governed by their source Froude number Fr D and the aspect ratio L/D of the release, where L denotes the length of the column of fluid dispensed vertically from the nozzle of diameter D. We establish the influence of L/D on the peak rise heights of a release formed by dispensing saline solution into fresh water for 0continuous flux. The critical aspect ratio (L/D) f, for a given Fr D, which when exceeded no longer influenced release behaviour, led to the determination of Fr D, (L/D) f paired source conditions that give rise to solely Froude-number-dependent, i.e. fountain-like, behaviour. As such, we make the link between finite-volume releases and continuous fountains. The Fr D(L/D) f pairs led us directly to the classification of a Fr D, L/D space from which source conditions giving rise to either negatively buoyant projectiles or fountains may be readily identified. The variation of (L/D) f with Fr D corresponds closely to established fountain regimes of very weak, weak and forced fountains. Moreover, our results indicate that the formation or otherwise of a primary vortex, as fluid is ejected, has a profound influence on the length of the dispensed fluid column that is necessary to achieve rise heights equal to fountain rise heights. © 2012 Cambridge University Press.

On the time delay in continuous in-cylinder sampling from IC engines

When gas sample is continuously drawn from the cylinder of an internal combustion engine, the sample that appears at the end of the sampling system corresponds to the in-cylinder content sometime ago because of the finite transit time which is a function of the cylinder pressure history. This variable delay causes a dispersion of the sample signal and makes the interpretation of the signal difficult An unsteady flow analysis of a typical sampling system was carried out for selected engine loads and speeds. For typical engine operation, a window in which the delay is approximately constant may be found. This window gets smaller with increase in engine speed, with decrease in load, and with the increase in exit pressure of the sampling system.

Reinforcement learning using a continuous time actor-critic framework with spiking neurons.

Animals repeat rewarded behaviors, but the physiological basis of reward-based learning has only been partially elucidated. On one hand, experimental evidence shows that the neuromodulator dopamine carries information about rewards and affects synaptic plasticity. On the other hand, the theory of reinforcement learning provides a framework for reward-based learning. Recent models of reward-modulated spike-timing-dependent plasticity have made first steps towards bridging the gap between the two approaches, but faced two problems. First, reinforcement learning is typically formulated in a discrete framework, ill-adapted to the description of natural situations. Second, biologically plausible models of reward-modulated spike-timing-dependent plasticity require precise calculation of the reward prediction error, yet it remains to be shown how this can be computed by neurons. Here we propose a solution to these problems by extending the continuous temporal difference (TD) learning of Doya (2000) to the case of spiking neurons in an actor-critic network operating in continuous time, and with continuous state and action representations. In our model, the critic learns to predict expected future rewards in real time. Its activity, together with actual rewards, conditions the delivery of a neuromodulatory TD signal to itself and to the actor, which is responsible for action choice. In simulations, we show that such an architecture can solve a Morris water-maze-like navigation task, in a number of trials consistent with reported animal performance. We also use our model to solve the acrobot and the cartpole problems, two complex motor control tasks. Our model provides a plausible way of computing reward prediction error in the brain. Moreover, the analytically derived learning rule is consistent with experimental evidence for dopamine-modulated spike-timing-dependent plasticity.

Continuous-time subspace flows related to the symmetric eigenproblem

The classes of continuous-time flows on Rn×p that induce the same flow on the set of p- dimensional subspaces of Rn×p are described. The power flow is briefly reviewed in this framework, and a subspace generalization of the Rayleigh quotient flow [Linear Algebra Appl. 368C, 2003, pp. 343-357] is proposed and analyzed. This new flow displays a property akin to deflation in finite time. © 2008 Yokohama Publishers.

Continuous dynamical systems that realize discrete optimization on the hypercube

We study the problem of finding a local minimum of a multilinear function E over the discrete set {0,1}n. The search is achieved by a gradient-like system in [0,1]n with cost function E. Under mild restrictions on the metric, the stable attractors of the gradient-like system are shown to produce solutions of the problem, even when they are not in the vicinity of the discrete set {0,1}n. Moreover, the gradient-like system connects with interior point methods for linear programming and with the analog neural network studied by Vidyasagar (IEEE Trans. Automat. Control 40 (8) (1995) 1359), in the same context. © 2004 Elsevier B.V. All rights reserved.

An experimental study of loss sources in a fan operating with continuous inlet stagnation pressure distortion

The viability of Boundary Layer Ingesting (BLI) engines for future aircraft propulsion is dependent on the ability to design robust, efficient engine fan systems for operation with continuously distorted inlet flow. A key step in this process is to develop an understanding of the specific mechanisms by which an inlet distortion affects the performance of a fan stage. In this paper, detailed full-annulus experimental measurements of the flow field within a low-speed fan stage operating with a continuous 60-degree inlet stagnation pressure distortion are presented. These results are used to describe the three-dimensional fluid mechanics governing the interaction between the fan and the distortion and to make a quantitative assessment of the impact on loss generation within the fan. A 5.3 percentage point reduction in stage total-to-total efficiency is observed as a result of the inlet distortion. The reduction in performance is shown to be dominated by increased loss generation in the rotor due to off-design incidence values at its leading edge, an effect which occurs throughout the annulus despite the localised nature of the inlet distortion. Increased loss generation in the stator row is also observed due to flow separations that are shown to be caused by whirl angle distortion at rotor exit. By addressing these losses, it should be possible to achieve improved efficiency in BLI fan systems. Copyright © 2012 by ASME.

Continuous weak measurement of a trapped electron using a percolation field effect transistor

We investigate the use of a percolation-field-effect-transistor for the continuous weak measurement of a spatially Rabi oscillating trapped electron through the change in percolation pathway of the transistor channel. In contrast to conventional devices, this detection mechanism in principle does not require a change in the stored energy of the gate capacitance to modify the drain current, so reducing the measurement back-action. The signal-to-noise ratio and measurement bandwidth are seen to be improved compared to conventional devices, allowing further aspects of the dynamic behaviour to be observed. © 2013 AIP Publishing LLC.