Partially Observable Markov Decision Processes with continuous observations for dialogue management

This work shows how a dialogue model can be represented as a Partially Observable Markov Decision Process (POMDP) with observations composed of a discrete and continuous component. The continuous component enables the model to directly incorporate a confidence score for automated planning. Using a testbed simulated dialogue management problem, we show how recent optimization techniques are able to find a policy for this continuous POMDP which outperforms a traditional MDP approach. Further, we present a method for automatically improving handcrafted dialogue managers by incorporating POMDP belief state monitoring, including confidence score information. Experiments on the testbed system show significant improvements for several example handcrafted dialogue managers across a range of operating conditions.

State estimation schemes for independent component coupled hidden Markov models

Conventional Hidden Markov models generally consist of a Markov chain observed through a linear map corrupted by additive noise. This general class of model has enjoyed a huge and diverse range of applications, for example, speech processing, biomedical signal processing and more recently quantitative finance. However, a lesser known extension of this general class of model is the so-called Factorial Hidden Markov Model (FHMM). FHMMs also have diverse applications, notably in machine learning, artificial intelligence and speech recognition [13, 17]. FHMMs extend the usual class of HMMs, by supposing the partially observed state process is a finite collection of distinct Markov chains, either statistically independent or dependent. There is also considerable current activity in applying collections of partially observed Markov chains to complex action recognition problems, see, for example, [6]. In this article we consider the Maximum Likelihood (ML) parameter estimation problem for FHMMs. Much of the extant literature concerning this problem presents parameter estimation schemes based on full data log-likelihood EM algorithms. This approach can be slow to converge and often imposes heavy demands on computer memory. The latter point is particularly relevant for the class of FHMMs where state space dimensions are relatively large. The contribution in this article is to develop new recursive formulae for a filter-based EM algorithm that can be implemented online. Our new formulae are equivalent ML estimators, however, these formulae are purely recursive and so, significantly reduce numerical complexity and memory requirements. A computer simulation is included to demonstrate the performance of our results. © Taylor & Francis Group, LLC.

Bayesian Learning of Noisy Markov Decision Processes

We consider the inverse reinforcement learning problem, that is, the problem of learning from, and then predicting or mimicking a controller based on state/action data. We propose a statistical model for such data, derived from the structure of a Markov decision process. Adopting a Bayesian approach to inference, we show how latent variables of the model can be estimated, and how predictions about actions can be made, in a unified framework. A new Markov chain Monte Carlo (MCMC) sampler is devised for simulation from the posterior distribution. This step includes a parameter expansion step, which is shown to be essential for good convergence properties of the MCMC sampler. As an illustration, the method is applied to learning a human controller.

Enzymatic activities in constructed wetlands and di-n-butyl phthalate (DBP) biodegradation

The bioaccumulation of phthalate acid esters (PAEs) from industrial products and their mutagenic action has been suggested to be a potential threat to human health. The effects of the most frequently identified PAE, Di-n-butyl phthalate (DBP), and its biodegradation, were examined by comparison of two small scale plots (SSP) of integrated vertical-flow constructed wetlands. The influent DBP concentration was 9.84 mg l(-1) in the treatment plot and the control plot received no DBP. Soil enzymatic activities of dehydrogenase, catalase, protease, phosphatase, urease, cellulase, beta-glucosidase, were measured in the two SSP after DBP application for 1 month and 2 months, and 1 month after the final application. Both treatment and control had significantly higher enzyme activity in the surface soil than in the subsurface soil (P < 0.001) and greater enzyme activity in the down-flow chamber than in the up-flow chamber (P < 0.05). In the constructed wetlands, DBP enhanced the activities of dehydrogenase, catalase, protease, phosphatase and inhibited the activities of urease, cellulase and beta-glucosidase. However, urease, cellulase, beta-glucosidase activities were restored 1 month following the final DBP addition. Degradation of DBP was greater in the surface soil and was reduced in sterile soil, indicating that this process may be mediated by aerobic microorgansims. DBP degradation fitted a first-order model, and the kinetic equation showed that the rate constant was 0.50 and 0.17 d(-1), the half-life was 1.39 and 4.02 d, and the r(2) was 0.99 and 0.98, in surface and subsurface soil, respectively. These results indicate that constructed wetlands are able to biodegrade organic PA-Es such as DBP. (c) 2005 Elsevier Ltd. All rights reserved.

Homogeneous Epitaxial Growth of N,N '-di(n-butyl)quinacridone Thin Films on Ag(110)

The structural evolution of the ordered N-N' dibutyl-substituted quinacridone (QA4C) multilayers (3 MLs) has been monitored in situ and in real time at various substrate temperatures using low energy electron diffraction (LEED) during organic molecular beam epitaxy (MBE). Experimental results of LEED patterns clearly reveal that the structure of the multilayer strongly depends on the substrate temperature. Multilayer growth can be achieved at the substrate temperatures below 300 K, while at the higher temperatures we can only get one ordered monolayer of QA4C. Two kinds of structures, the commensurate and incommensurate one, often coexist in the QA4C multilayer. With a method of the two-step substrate temperatures, the incommensurate one can be suppressed, and the commensurate, on the other hand, more similar to the (001) plane of the QA4C bulk crystal, prevails with the layer of QA4C increasing to 3 MLs. The two structures in the multilayers are compressed slightly in comparison to the original ones in the first monolayer.