Bayesian Learning of Noisy Markov Decision Processes

We consider the inverse reinforcement learning problem, that is, the problem of learning from, and then predicting or mimicking a controller based on state/action data. We propose a statistical model for such data, derived from the structure of a Markov decision process. Adopting a Bayesian approach to inference, we show how latent variables of the model can be estimated, and how predictions about actions can be made, in a unified framework. A new Markov chain Monte Carlo (MCMC) sampler is devised for simulation from the posterior distribution. This step includes a parameter expansion step, which is shown to be essential for good convergence properties of the MCMC sampler. As an illustration, the method is applied to learning a human controller.

Bayesian Structured Prediction Using Gaussian Processes

We introduce a conceptually novel structured prediction model, GPstruct, which is kernelized, non-parametric and Bayesian, by design. We motivate the model with respect to existing approaches, among others, conditional random fields (CRFs), maximum margin Markov networks (M3N), and structured support vector machines (SVMstruct), which embody only a subset of its properties. We present an inference procedure based on Markov Chain Monte Carlo. The framework can be instantiated for a wide range of structured objects such as linear chains, trees, grids, and other general graphs. As a proof of concept, the model is benchmarked on several natural language processing tasks and a video gesture segmentation task involving a linear chain structure. We show prediction accuracies for GPstruct which are comparable to or exceeding those of CRFs and SVMstruct.

Determinantal Clustering Processes - A Nonparametric Bayesian Approach to Kernel Based Semi-Supervised Clustering

Semi-supervised clustering is the task of clustering data points into clusters where only a fraction of the points are labelled. The true number of clusters in the data is often unknown and most models require this parameter as an input. Dirichlet process mixture models are appealing as they can infer the number of clusters from the data. However, these models do not deal with high dimensional data well and can encounter difficulties in inference. We present a novel nonparameteric Bayesian kernel based method to cluster data points without the need to prespecify the number of clusters or to model complicated densities from which data points are assumed to be generated from. The key insight is to use determinants of submatrices of a kernel matrix as a measure of how close together a set of points are. We explore some theoretical properties of the model and derive a natural Gibbs based algorithm with MCMC hyperparameter learning. The model is implemented on a variety of synthetic and real world data sets.

On-line policy optimisation of Bayesian spoken dialogue systems via human interaction

A partially observable Markov decision process has been proposed as a dialogue model that enables robustness to speech recognition errors and automatic policy optimisation using reinforcement learning (RL). However, conventional RL algorithms require a very large number of dialogues, necessitating a user simulator. Recently, Gaussian processes have been shown to substantially speed up the optimisation, making it possible to learn directly from interaction with human users. However, early studies have been limited to very low dimensional spaces and the learning has exhibited convergence problems. Here we investigate learning from human interaction using the Bayesian Update of Dialogue State system. This dynamic Bayesian network based system has an optimisation space covering more than one hundred features, allowing a wide range of behaviours to be learned. Using an improved policy model and a more robust reward function, we show that stable learning can be achieved that significantly outperforms a simulator trained policy. © 2013 IEEE.

Study on the hydrolysis/precipitation behavior of Keggin Al-13 and Al-30 polymers in polyaluminum solutions

The hydrolysis/precipitation behaviors of Al3+, Al-13 and Al-30 under conditions typical for flocculation in water treatment were investigated by studying the particulates' size development, charge characteristics, chemical species and speciation transformation of coagulant hydrolysis precipitates. The optimal pH conditions for hydrolysis precipitates formation for AlCl3, PAC(A113) and PAC(A130) were 6.5-7.5, 8.5-9.5, and 7.5-9.5, respectively. The precipitates' formation rate increased with the increase in dosage, and the relative rates were AlCl3 >> PAC(A130) > PACA113. The precipitates' size increased when the dosage increased from 50 mu M to 200 mu M, but it decreased when the dosage increased to 800 AM. The Zeta potential of coagulant hydrolysis precipitates decreased with the increase in pH for the three coagulants. The isoelectric points of the freshly formed precipitates for AlCl3, PAC(A113) and PAC(A130) were 7.3, 9.6 and 9.2, respectively. The Zeta potentials of AlCl3 hydrolysis precipitates were lower than those of PAC(A113) and PAC(A130) when pH > 5.0. The Zeta potential of PAC(A130) hydrolysis precipitates was higher than that of PACA113 at the acidic side, but lower at the alkaline side. The dosage had no obvious effect on the Zeta potential of hydrolysis precipitates under fixed pH conditions. The increase in Zeta potential with the increase in dosage under uncontrolled pH conditions was due to the pH depression caused by coagulant addition. Al-Ferron research indicated that the hydrolysis precipitates of AlCl3 were composed of amorphous AI(OH)3 precipitates, but those of PACA113 and PACA130 were composed of aggregates of Al-13 and Al-30, respectively. Al3+ was the most un-stable species in coagulants, and its hydrolysis was remarkably influenced by solution pH. Al-13 and Al-30 species were very stable, and solution pH and aging had little effect on the chemical species of their hydrolysis products. The research method involving coagulant hydrolysis precipitates based on Al-Ferron reaction kinetics was studied in detail. The Al species classification based on complex reaction kinetic of hydrolysis precipitates and Ferron reagent was different from that measured in a conventional coagulant assay using the Al--Ferron method. The chemical composition of Al-a, Al-b and Al-c depended on coagulant and solution pH. The Al-b measured in the current case was different from Keggin Al-13, and the high Alb content in the AlCl3 hydrolysis precipitates could not used as testimony that most of the Al3+ Was converted to highly charged Al-13 species during AlCl3 coagulation.

Precipitation at Lake Qinghai；NE Qinghai-Tibet Plateau；and its relation to Asian summer monsoons on decadal/interdecadal scales during the past 500 years

IEECAS SKLLQG

On the Dirichlet Prior and Bayesian Regularization

A common objective in learning a model from data is to recover its network structure, while the model parameters are of minor interest. For example, we may wish to recover regulatory networks from high-throughput data sources. In this paper we examine how Bayesian regularization using a Dirichlet prior over the model parameters affects the learned model structure in a domain with discrete variables. Surprisingly, a weak prior in the sense of smaller equivalent sample size leads to a strong regularization of the model structure (sparse graph) given a sufficiently large data set. In particular, the empty graph is obtained in the limit of a vanishing strength of prior belief. This is diametrically opposite to what one may expect in this limit, namely the complete graph from an (unregularized) maximum likelihood estimate. Since the prior affects the parameters as expected, the prior strength balances a "trade-off" between regularizing the parameters or the structure of the model. We demonstrate the benefits of optimizing this trade-off in the sense of predictive accuracy.