An overview of Sequential Monte Carlo methods for parameter estimation in general state-space models

Nonlinear non-Gaussian state-space models arise in numerous applications in control and signal processing. Sequential Monte Carlo (SMC) methods, also known as Particle Filters, provide very good numerical approximations to the associated optimal state estimation problems. However, in many scenarios, the state-space model of interest also depends on unknown static parameters that need to be estimated from the data. In this context, standard SMC methods fail and it is necessary to rely on more sophisticated algorithms. The aim of this paper is to present a comprehensive overview of SMC methods that have been proposed to perform static parameter estimation in general state-space models. We discuss the advantages and limitations of these methods. © 2009 IFAC.

State-space inference and learning with Gaussian processes

State-space inference and learning with Gaussian processes (GPs) is an unsolved problem. We propose a new, general methodology for inference and learning in nonlinear state-space models that are described probabilistically by non-parametric GP models. We apply the expectation maximization algorithm to iterate between inference in the latent state-space and learning the parameters of the underlying GP dynamics model. Copyright 2010 by the authors.

Policy optimisation of POMDP-based dialogue systems without state space compression

The partially observable Markov decision process (POMDP) has been proposed as a dialogue model that enables automatic improvement of the dialogue policy and robustness to speech understanding errors. It requires, however, a large number of dialogues to train the dialogue policy. Gaussian processes (GP) have recently been applied to POMDP dialogue management optimisation showing an ability to substantially increase the speed of learning. Here, we investigate this further using the Bayesian Update of Dialogue State dialogue manager. We show that it is possible to apply Gaussian processes directly to the belief state, removing the need for a parametric policy representation. In addition, the resulting policy learns significantly faster while maintaining operational performance. © 2012 IEEE.

Rao-Blackwellized particle smoothers for mixed linear/nonlinear state-space models

We consider the smoothing problem for a class of conditionally linear Gaussian state-space (CLGSS) models, referred to as mixed linear/nonlinear models. In contrast to the better studied hierarchical CLGSS models, these allow for an intricate cross dependence between the linear and the nonlinear parts of the state vector. We derive a Rao-Blackwellized particle smoother (RBPS) for this model class by exploiting its tractable substructure. The smoother is of the forward filtering/backward simulation type. A key feature of the proposed method is that, unlike existing RBPS for this model class, the linear part of the state vector is marginalized out in both the forward direction and in the backward direction. © 2013 IEEE.

Bayesian inference and learning in Gaussian process state-space models with Particle MCMC

State-space models are successfully used in many areas of science, engineering and economics to model time series and dynamical systems. We present a fully Bayesian approach to inference and learning (i.e. state estimation and system identification) in nonlinear nonparametric state-space models. We place a Gaussian process prior over the state transition dynamics, resulting in a flexible model able to capture complex dynamical phenomena. To enable efficient inference, we marginalize over the transition dynamics function and, instead, infer directly the joint smoothing distribution using specially tailored Particle Markov Chain Monte Carlo samplers. Once a sample from the smoothing distribution is computed, the state transition predictive distribution can be formulated analytically. Our approach preserves the full nonparametric expressivity of the model and can make use of sparse Gaussian processes to greatly reduce computational complexity.

State estimation schemes for independent component coupled hidden Markov models

Conventional Hidden Markov models generally consist of a Markov chain observed through a linear map corrupted by additive noise. This general class of model has enjoyed a huge and diverse range of applications, for example, speech processing, biomedical signal processing and more recently quantitative finance. However, a lesser known extension of this general class of model is the so-called Factorial Hidden Markov Model (FHMM). FHMMs also have diverse applications, notably in machine learning, artificial intelligence and speech recognition [13, 17]. FHMMs extend the usual class of HMMs, by supposing the partially observed state process is a finite collection of distinct Markov chains, either statistically independent or dependent. There is also considerable current activity in applying collections of partially observed Markov chains to complex action recognition problems, see, for example, [6]. In this article we consider the Maximum Likelihood (ML) parameter estimation problem for FHMMs. Much of the extant literature concerning this problem presents parameter estimation schemes based on full data log-likelihood EM algorithms. This approach can be slow to converge and often imposes heavy demands on computer memory. The latter point is particularly relevant for the class of FHMMs where state space dimensions are relatively large. The contribution in this article is to develop new recursive formulae for a filter-based EM algorithm that can be implemented online. Our new formulae are equivalent ML estimators, however, these formulae are purely recursive and so, significantly reduce numerical complexity and memory requirements. A computer simulation is included to demonstrate the performance of our results. © Taylor & Francis Group, LLC.

A differential lyapunov framework for contraction analysis

Lyapunov's second theorem is an essential tool for stability analysis of differential equations. The paper provides an analog theorem for incremental stability analysis by lifting the Lyapunov function to the tangent bundle. The Lyapunov function endows the state-space with a Finsler structure. Incremental stability is inferred from infinitesimal contraction of the Finsler metrics through integration along solutions curves. © 2013 IEEE.

Sensitivity analysis of a time-delayed thermo-acoustic system via an adjoint-based approach

We apply adjoint-based sensitivity analysis to a time-delayed thermo-acoustic system: a Rijke tube containing a hot wire. We calculate how the growth rate and frequency of small oscillations about a base state are affected either by a generic passive control element in the system (the structural sensitivity analysis) or by a generic change to its base state (the base-state sensitivity analysis). We illustrate the structural sensitivity by calculating the effect of a second hot wire with a small heat-release parameter. In a single calculation, this shows how the second hot wire changes the growth rate and frequency of the small oscillations, as a function of its position in the tube. We then examine the components of the structural sensitivity in order to determine the passive control mechanism that has the strongest influence on the growth rate. We find that a force applied to the acoustic momentum equation in the opposite direction to the instantaneous velocity is the most stabilizing feedback mechanism. We also find that its effect is maximized when it is placed at the downstream end of the tube. This feedback mechanism could be supplied, for example, by an adiabatic mesh. We illustrate the base-state sensitivity by calculating the effects of small variations in the damping factor, the heat-release time-delay coefficient, the heat-release parameter, and the hot-wire location. The successful application of sensitivity analysis to thermo-acoustics opens up new possibilities for the passive control of thermo-acoustic oscillations by providing gradient information that can be combined with constrained optimization algorithms in order to reduce linear growth rates. © Cambridge University Press 2013.