Gaussian processes for fast policy optimisation of POMDP-based dialogue managers

Modelling dialogue as a Partially Observable Markov Decision Process (POMDP) enables a dialogue policy robust to speech understanding errors to be learnt. However, a major challenge in POMDP policy learning is to maintain tractability, so the use of approximation is inevitable. We propose applying Gaussian Processes in Reinforcement learning of optimal POMDP dialogue policies, in order (1) to make the learning process faster and (2) to obtain an estimate of the uncertainty of the approximation. We first demonstrate the idea on a simple voice mail dialogue task and then apply this method to a real-world tourist information dialogue task. © 2010 Association for Computational Linguistics.

On-line policy optimisation of spoken dialogue systems via live interaction with human subjects

Statistical dialogue models have required a large number of dialogues to optimise the dialogue policy, relying on the use of a simulated user. This results in a mismatch between training and live conditions, and significant development costs for the simulator thereby mitigating many of the claimed benefits of such models. Recent work on Gaussian process reinforcement learning, has shown that learning can be substantially accelerated. This paper reports on an experiment to learn a policy for a real-world task directly from human interaction using rewards provided by users. It shows that a usable policy can be learnt in just a few hundred dialogues without needing a user simulator and, using a learning strategy that reduces the risk of taking bad actions. The paper also investigates adaptation behaviour when the system continues learning for several thousand dialogues and highlights the need for robustness to noisy rewards. © 2011 IEEE.

Active control of high-frequency vibration: Optimisation using the hybrid modelling method

This work presents active control of high-frequency vibration using skyhook dampers. The choice of the damper gain and its optimal location is crucial for the effective implementation of active vibration control. In vibration control, certain sensor/actuator locations are preferable for reducing structural vibration while using minimum control effort. In order to perform optimisation on a general built-up structure to control vibration, it is necessary to have a good modelling technique to predict the performance of the controller. The present work exploits the hybrid modelling approach, which combines the finite element method (FEM) and statistical energy analysis (SEA) to provide efficient response predictions at medium to high frequencies. The hybrid method is implemented here for a general network of plates, coupled via springs, to allow study of a variety of generic control design problems. By combining the hybrid method with numerical optimisation using a genetic algorithm, optimal skyhook damper gains and locations are obtained. The optimal controller gain and location found from the hybrid method are compared with results from a deterministic modelling method. Good agreement between the results is observed, whereas results from the hybrid method are found in a significantly reduced amount of time. © 2012 Elsevier Ltd. All rights reserved.

Vehicle trajectory linearisation to enable efficient optimisation of the constant speed racing line

A driver model is presented capable of optimising the trajectory of a simple dynamic nonlinear vehicle, at constant forward speed, so that progression along a predefined track is maximised as a function of time. In doing so, the model is able to continually operate a vehicle at its lateral-handling limit, maximising vehicle performance. The technique used forms a part of the solution to the motor racing objective of minimising lap time. A new approach of formulating the minimum lap time problem is motivated by the need for a more computationally efficient and robust tool-set for understanding on-the-limit driving behaviour. This has been achieved through set point-dependent linearisation of the vehicle model and coupling the vehicle-track system using an intrinsic coordinate description. Through this, the geometric vehicle trajectory had been linearised relative to the track reference, leading to new path optimisation algorithm which can be formed as a computationally efficient convex quadratic programming problem. © 2012 Copyright Taylor and Francis Group, LLC.

Polarisation diversity of conformal arrays based on geometric algebra via convex optimisation

A simple and general design procedure is presented for the polarisation diversity of arbitrary conformal arrays; this procedure is based on the mathematical framework of geometric algebra and can be solved optimally using convex optimisation. Aside from being simpler and more direct than other derivations in the literature, this derivation is also entirely general in that it expresses the transformations in terms of rotors in geometric algebra which can easily be formulated for any arbitrary conformal array geometry. Convex optimisation has a number of advantages; solvers are widespread and freely available, the process generally requires a small number of iterations and a wide variety of constraints can be readily incorporated. The study outlines a two-step approach for addressing polarisation diversity in arbitrary conformal arrays: first, the authors obtain the array polarisation patterns using geometric algebra and secondly use a convex optimisation approach to find the optimal weights for the polarisation diversity problem. The versatility of this approach is illustrated via simulations of a 7×10 cylindrical conformal array. © 2012 The Institution of Engineering and Technology.

Uncertainty management for on-line optimisation of a POMDP-based large-scale spoken dialogue system

The optimization of dialogue policies using reinforcement learning (RL) is now an accepted part of the state of the art in spoken dialogue systems (SDS). Yet, it is still the case that the commonly used training algorithms for SDS require a large number of dialogues and hence most systems still rely on artificial data generated by a user simulator. Optimization is therefore performed off-line before releasing the system to real users. Gaussian Processes (GP) for RL have recently been applied to dialogue systems. One advantage of GP is that they compute an explicit measure of uncertainty in the value function estimates computed during learning. In this paper, a class of novel learning strategies is described which use uncertainty to control exploration on-line. Comparisons between several exploration schemes show that significant improvements to learning speed can be obtained and that rapid and safe online optimisation is possible, even on a complex task. Copyright © 2011 ISCA.

Making a shallow network deep: Conversion of a boosting classifier into a decision tree by boolean optimisation

This paper presents a novel way to speed up the evaluation time of a boosting classifier. We make a shallow (flat) network deep (hierarchical) by growing a tree from decision regions of a given boosting classifier. The tree provides many short paths for speeding up while preserving the reasonably smooth decision regions of the boosting classifier for good generalisation. For converting a boosting classifier into a decision tree, we formulate a Boolean optimization problem, which has been previously studied for circuit design but limited to a small number of binary variables. In this work, a novel optimisation method is proposed for, firstly, several tens of variables i.e. weak-learners of a boosting classifier, and then any larger number of weak-learners by using a two-stage cascade. Experiments on the synthetic and face image data sets show that the obtained tree achieves a significant speed up both over a standard boosting classifier and the Fast-exit-a previously described method for speeding-up boosting classification, at the same accuracy. The proposed method as a general meta-algorithm is also useful for a boosting cascade, where it speeds up individual stage classifiers by different gains. The proposed method is further demonstrated for fast-moving object tracking and segmentation problems. © 2011 Springer Science+Business Media, LLC.

Shape vs. volume: Invariant shape descriptors for 3D region of interest characterization in MRI

We propose a new approach for quantifying regions of interest (ROIs) in medical image data. Rotationally invariant shape descriptors (ISDs) were applied to 3D brain regions extracted from MRI scans of 5 Parkinson's patients and 10 control subjects. We concentrated on the thalamus and the caudate nucleus since prior studies have suggested they are affected in Parkinson's disease (PD). In the caudate, both the ISD and volumetric analyses found significant differences between control and PD subjects. The ISD analysis however revealed additional differences between the left and right caudate nuclei in both control and PD subjects. In the thalamus, the volumetric analysis showed significant differences between PD and control subjects, while ISD analysis found significant differences between the left and right thalami in control subjects but not in PD patients, implying disease-induced shape changes. These results suggest that employing ISDs for ROI characterization both complements and extends traditional volumetric analyses. © 2006 IEEE.

Coherent design methodology using modelling, simulation and optimisation

There is an increasing demand for optimising complete systems and the devices within that system, including capturing the interactions between the various multi-disciplinary (MD) components involved. Furthermore confidence in robust solutions is esential. As a consequence the computational cost rapidly increases and in many cases becomes infeasible to perform such conceptual designs. A coherent design methodology is proposed, where the aim is to improve the design process by effectively exploiting the potential of computational synthesis, search and optimisation and conventional simulation, with a reduction of the computational cost. This optimization framework consists of a hybrid optimization algorithm to handles multi-fidelity simulations. Simultaneously and in order to handles uncertainty without recasting the model and at affordable computational cost, a stochastic modelling method known as non-intrusive polynomial chaos is introduced. The effectiveness of the design methodology is demonstrated with the optimisation of a submarine propulsion system.

On the transition from finite-volume negatively buoyant releases to continuous fountains

An experimental investigation to identify the source conditions that distinguish finite-volume negatively buoyant fluid projectile behaviour from fountain behaviour in quiescent environments of uniform density is described. Finite-volume releases are governed by their source Froude number Fr D and the aspect ratio L/D of the release, where L denotes the length of the column of fluid dispensed vertically from the nozzle of diameter D. We establish the influence of L/D on the peak rise heights of a release formed by dispensing saline solution into fresh water for 0volume or continuous flux. The critical aspect ratio (L/D) f, for a given Fr D, which when exceeded no longer influenced release behaviour, led to the determination of Fr D, (L/D) f paired source conditions that give rise to solely Froude-number-dependent, i.e. fountain-like, behaviour. As such, we make the link between finite-volume releases and continuous fountains. The Fr D(L/D) f pairs led us directly to the classification of a Fr D, L/D space from which source conditions giving rise to either negatively buoyant projectiles or fountains may be readily identified. The variation of (L/D) f with Fr D corresponds closely to established fountain regimes of very weak, weak and forced fountains. Moreover, our results indicate that the formation or otherwise of a primary vortex, as fluid is ejected, has a profound influence on the length of the dispensed fluid column that is necessary to achieve rise heights equal to fountain rise heights. © 2012 Cambridge University Press.

Semi-implicit second order finite volume jet stream computations

Semi-implicit, second order temporal and spatial finite volume computations of the flow in a differentially heated rotating annulus are presented. For the regime considered, three cyclones and anticyclones separated by a relatively fast moving jet of fluid or "jet stream" are predicted. Two second order methods are compared with, first order spatial predictions, and experimental measurements. Velocity vector plots are used to illustrate the predicted flow structure. Computations made using second order central differences are shown to agree best with experimental measurements, and to be stable for integrations over long time periods (> 1000s). No periodic smoothing is required to prevent divergence.