Reinforcement learning with reference tracking control in continuous state spaces

The contribution described in this paper is an algorithm for learning nonlinear, reference tracking, control policies given no prior knowledge of the dynamical system and limited interaction with the system through the learning process. Concepts from the field of reinforcement learning, Bayesian statistics and classical control have been brought together in the formulation of this algorithm which can be viewed as a form of indirect self tuning regulator. On the task of reference tracking using a simulated inverted pendulum it was shown to yield generally improved performance on the best controller derived from the standard linear quadratic method using only 30 s of total interaction with the system. Finally, the algorithm was shown to work on the simulated double pendulum proving its ability to solve nontrivial control tasks. © 2011 IEEE.

Thermal stratification produced by plumes and jets in enclosed spaces

The airflow and thermal stratification produced by a localised heat source located at floor level in a closed room is of considerable practical interest and is commonly referred to as a 'filling box'. In rooms with low aspect ratios H/R ≲ 1 (room height H to characteristic horizontal dimension R) the thermal plume spreads laterally on reaching the ceiling and a descending horizontal 'front' forms separating a stably stratified, warm upper region from cooler air below. The stratification is well predicted for H/R ≲ 1 by the original filling box model of Baines and Turner (J. Fluid. Mech. 37 (1968) 51). This model represents a somewhat idealised situation of a plume rising from a point source of buoyancy alone-in particular the momentum flux at the source is zero. In practical situations, real sources of heating and cooling in a ventilation system often include initial fluxes of both buoyancy and momentum, e.g. where a heating system vents warm air into a space. This paper describes laboratory experiments to determine the dependence of the 'front' formation and stratification on the source momentum and buoyancy fluxes of a single source, and on the location and relative strengths of two sources from which momentum and buoyancy fluxes were supplied separately. For a single source with a non-zero input of momentum, the rate of descent of the front is more rapid than for the case of zero source momentum flux and increases with increasing momentum input. Increasing the source momentum flux effectively increases the height of the enclosure, and leads to enhanced overturning motions and finally to complete mixing for highly momentum-driven flows. Stratified flows may be maintained by reducing the aspect ratio of the enclosure. At these low aspect ratios different long-time behaviour is observed depending on the nature of the heat input. A constant heat flux always produces a stratified interior at large times. On the other hand, a constant temperature supply ultimately produces a well-mixed space at the supply temperature. For separate sources of momentum and buoyancy, the developing stratification is shown to be strongly dependent on the separation of the sources and their relative strengths. Even at small separation distances the stratification initially exhibits horizontal inhomogeneity with localised regions of warm fluid (from the buoyancy source) and cool fluid. This inhomogeneity is less pronounced as the strength of one source is increased relative to the other. Regardless of the strengths of the sources, a constant buoyancy flux source dominates after sufficiently large times, although the strength of the momentum source determines whether the enclosure is initially well mixed (strong momentum source) or stably stratified (weak momentum source). © 2001 Elsevier Science Ltd. All rights reserved.

Consensus in non-commutative spaces

Convergence analysis of consensus algorithms is revisited in the light of the Hilbert distance. The Lyapunov function used in the early analysis by Tsitsiklis is shown to be the Hilbert distance to consensus in log coordinates. Birkhoff theorem, which proves contraction of the Hilbert metric for any positive homogeneous monotone map, provides an early yet general convergence result for consensus algorithms. Because Birkhoff theorem holds in arbitrary cones, we extend consensus algorithms to the cone of positive definite matrices. The proposed generalization finds applications in the convergence analysis of quantum stochastic maps, which are a generalization of stochastic maps to non-commutative probability spaces. ©2010 IEEE.

Efficient decoding with generative score-spaces using the expectation semiring

State-of-the-art speech recognisers are usually based on hidden Markov models (HMMs). They model a hidden symbol sequence with a Markov process, with the observations independent given that sequence. These assumptions yield efficient algorithms, but limit the power of the model. An alternative model that allows a wide range of features, including word- and phone-level features, is a log-linear model. To handle, for example, word-level variable-length features, the original feature vectors must be segmented into words. Thus, decoding must find the optimal combination of segmentation of the utterance into words and word sequence. Features must therefore be extracted for each possible segment of audio. For many types of features, this becomes slow. In this paper, long-span features are derived from the likelihoods of word HMMs. Derivatives of the log-likelihoods, which break the Markov assumption, are appended. Previously, decoding with this model took cubic time in the length of the sequence, and longer for higher-order derivatives. This paper shows how to decode in quadratic time. © 2013 IEEE.

Weighted least squares with expectation-maximization algorithm for burst detection in U.K. water distribution systems

Flow measurement data at the district meter area (DMA) level has the potential for burst detection in the water distribution systems. This work investigates using a polynomial function fitted to the historic flow measurements based on a weighted least-squares method for automatic burst detection in the U.K. water distribution networks. This approach, when used in conjunction with an expectationmaximization (EM) algorithm, can automatically select useful data from the historic flow measurements, which may contain normal and abnormal operating conditions in the distribution network, e.g., water burst. Thus, the model can estimate the normal water flow (nonburst condition), and hence the burst size on the water distribution system can be calculated from the difference between the measured flow and the estimated flow. The distinguishing feature of this method is that the burst detection is fully unsupervised, and the burst events that have occurred in the historic data do not affect the procedure and bias the burst detection algorithm. Experimental validation of the method has been carried out using a series of flushing events that simulate burst conditions to confirm that the simulated burst sizes are capable of being estimated correctly. This method was also applied to eight DMAs with known real burst events, and the results of burst detections are shown to relate to the water company's records of pipeline reparation work. © 2014 American Society of Civil Engineers.