Bayesian learning of noisy Markov decision processes

This work addresses the problem of estimating the optimal value function in a Markov Decision Process from observed state-action pairs. We adopt a Bayesian approach to inference, which allows both the model to be estimated and predictions about actions to be made in a unified framework, providing a principled approach to mimicry of a controller on the basis of observed data. A new Markov chain Monte Carlo (MCMC) sampler is devised for simulation from theposterior distribution over the optimal value function. This step includes a parameter expansion step, which is shown to be essential for good convergence properties of the MCMC sampler. As an illustration, the method is applied to learning a human controller.

The unidirectional emptying box

A theoretical description of the turbulent mixing within and the draining of a dense fluid layer from a box connected to a uniform density, quiescent environment through openings in the top and the base of the box is presented in this paper. This is an extension of the draining model developed by Linden et al. (Annu. Rev. Fluid Mech. vol. 31, 1990, pp. 201-238) and includes terms that describe localized mixing within the emptying box at the density interface. Mixing is induced by a turbulent flow of replacement fluid into the box and as a consequence we predict, and observe in complementary experiments, the development of a three-layer stratification. Based on the data collated from previous researchers, three distinct formulations for entrainment fluxes across density interfaces are used to account for this localized mixing. The model was then solved numerically for the three mixing formulations. Analytical solutions were developed for one formulation directly and for a second on assuming that localized mixing is relatively weak though still significant in redistributing buoyancy on the timescale of the draining process. Comparisons between our theoretical predictions and the experimental data, which we have collected on the developing layer depths and their densities show good agreement. The differences in predictions between the three mixing formulations suggest that the normalized flux turbulently entrained across a density interface tends to a constant value for large values of a Froude number FrT, based on conditions of the inflow through the top of the box, and scales as the cube of FrT for small values of FrT. The upper limit on the rate of entrainment into the mixed layer results in a minimum time (tD) to remove the original dense layer. Using our analytical solutions, we bound this time and show that 0.2tE ≈tD tE, i.e. the original dense layer may be depleted up to five times more rapidly than when there is no internal mixing and the box empties in a time tE. © 2010 Cambridge University Press.

Manopt, a matlab toolbox for optimization on manifolds

Optimization on manifolds is a rapidly developing branch of nonlinear optimization. Its focus is on problems where the smooth geometry of the search space can be leveraged to design effcient numerical algorithms. In particular, optimization on manifolds is well-suited to deal with rank and orthogonality constraints. Such structured constraints appear pervasively in machine learning applications, including low-rank matrix completion, sensor network localization, camera network registration, independent component analysis, metric learning, dimensionality reduction and so on. The Manopt toolbox, available at www.manopt.org, is a user-friendly, documented piece of software dedicated to simplify experimenting with state of the art Riemannian optimization algorithms. By dealing internally with most of the differential geometry, the package aims particularly at lowering the entrance barrier. © 2014 Nicolas Boumal.