Partially Observable Markov Decision Processes with continuous observations for dialogue management

This work shows how a dialogue model can be represented as a Partially Observable Markov Decision Process (POMDP) with observations composed of a discrete and continuous component. The continuous component enables the model to directly incorporate a confidence score for automated planning. Using a testbed simulated dialogue management problem, we show how recent optimization techniques are able to find a policy for this continuous POMDP which outperforms a traditional MDP approach. Further, we present a method for automatically improving handcrafted dialogue managers by incorporating POMDP belief state monitoring, including confidence score information. Experiments on the testbed system show significant improvements for several example handcrafted dialogue managers across a range of operating conditions.

State estimation schemes for independent component coupled hidden Markov models

Conventional Hidden Markov models generally consist of a Markov chain observed through a linear map corrupted by additive noise. This general class of model has enjoyed a huge and diverse range of applications, for example, speech processing, biomedical signal processing and more recently quantitative finance. However, a lesser known extension of this general class of model is the so-called Factorial Hidden Markov Model (FHMM). FHMMs also have diverse applications, notably in machine learning, artificial intelligence and speech recognition [13, 17]. FHMMs extend the usual class of HMMs, by supposing the partially observed state process is a finite collection of distinct Markov chains, either statistically independent or dependent. There is also considerable current activity in applying collections of partially observed Markov chains to complex action recognition problems, see, for example, [6]. In this article we consider the Maximum Likelihood (ML) parameter estimation problem for FHMMs. Much of the extant literature concerning this problem presents parameter estimation schemes based on full data log-likelihood EM algorithms. This approach can be slow to converge and often imposes heavy demands on computer memory. The latter point is particularly relevant for the class of FHMMs where state space dimensions are relatively large. The contribution in this article is to develop new recursive formulae for a filter-based EM algorithm that can be implemented online. Our new formulae are equivalent ML estimators, however, these formulae are purely recursive and so, significantly reduce numerical complexity and memory requirements. A computer simulation is included to demonstrate the performance of our results. © Taylor & Francis Group, LLC.

Bayesian Learning of Noisy Markov Decision Processes

We consider the inverse reinforcement learning problem, that is, the problem of learning from, and then predicting or mimicking a controller based on state/action data. We propose a statistical model for such data, derived from the structure of a Markov decision process. Adopting a Bayesian approach to inference, we show how latent variables of the model can be estimated, and how predictions about actions can be made, in a unified framework. A new Markov chain Monte Carlo (MCMC) sampler is devised for simulation from the posterior distribution. This step includes a parameter expansion step, which is shown to be essential for good convergence properties of the MCMC sampler. As an illustration, the method is applied to learning a human controller.

Markov���程在森林资源结构动态预测中的应用──以福建省南平地区的树种结构为例

根据马尔可夫决策过程理论和森林资源连续清查的固定样地调查资料，对南平地区的树种结构进行了预测与调整，结果表明，按现状发展，针阔比例将日趋严重，并且毛竹、经济林的占有率呈下降趋势，最终达到以杉木２８．０５％、马尾松１６．６３％、阔叶树１９．０１％、毛竹５．４３％、经济林２．２６％、其它类２８．７１％的树种结构．经调整后稳定状态的树种结构基本趋于合理，即各树种的占有率分别为杉木１８．７２％、马尾松１３．２４％、阔叶树２６．９８％、毛竹１０．８４％、经济林５．４５％、其它类２４．７７％．

Markov���灰色模型在土地利用预测中的应用

马尔柯夫和灰色模型都适用于土地利用变化预测,根据同一套土地利用数据分别用两种模型预测,将所得结果相互验证、对比分析,提高预测可信度。以江西省新建县为例,两种预测方法的预测结果都表明,若继续保持1996-2000年的变化速度,耕地和未利用地将持续减少,林地和建设用地呈增长趋势,而草地和水域相对较稳定,草地有下降趋势,水域呈缓慢上升趋向。预测结果可为土地利用规划管理及政策的制定提供科学依据,研究方法为土地利用变化预测研究提供一种思路。

景观动态的Markov���型研究——以长白山自然保护区为例

在 Markov���型假设的基础上 ,利用长白山自然保护区 1975年 MSS、1985年和 1997年 TM卫星遥感数据 ,在遥感图象处理软件和 GIS软件协助下 ,对遥感影像的计算机监督分类结果 (共分为 13类 )进行处理 ,对 Markov���型的可利用性进行分析与检验 ,得出长白山自然保护区景观变化无后效性 ,符合 Markov���型条件。根据 1985～ 1997年转移概率计算步长 10 a(1985～ 1995年 )的转移概率矩阵 ,从 1975年计算 1985年各景观类型的面积与 1985年各景观类型的实际面积值对比 ,计算得 χ2 >χ20 .0 5(12 ) ;再分别用 1975～ 1985年和 1985～ 1997年的转移矩阵计算 1995年和 2 0 4 7年各景观类型的面积 ,分析得χ2 >χ20 .0 5(12 ) ;对两阶段的转移概率矩阵分析得到 χ2 >χ20 .0 5(14 4 ) ;说明两阶段的 Markov���移过程不具同一性 ,属于两个不同的 Markov���程。不同景观类型转移方式对χ2 值的贡献率可以说明其对景观动态的重要性 ,分析结果表明有重要贡献的类型分别为 :阔叶红松林 5 2 .0 0 % >山杨白桦林 2 4 .6 6 % >云冷杉林 11.4 2 % >落叶松林 2 .4 3% ,说明这 4种景观类型的转移方式对长白山自然保护区的景观动态起重要作用 ,尤其以阔叶松林的作用最大 ;同时对 Markov���型在长白山自然保护区长期景观变化预测的可

Surtos da mosca-dos-estábulos, Stomoxys calcitrans, em Mato Grosso do Sul: novo problema para as cadeias produtivas da carne e sucroalcooleira?

No presente documento é registrada a ocorrência de infestações preocupantes pela mosca-dos-estábulos, Stomoxys calcitrans L. (Diptera: Muscidae), em propriedades pecuárias próximas de usinas sucroalcooleiras na região Sul de Mato Grosso do Sul, no ano de 2009. A biologia do inseto; sua importância epidemiológica, econômica e veterinária e; possíveis alternativas de controle são apresentadas e discutidas. Entre as alternativas de controle sugere-se, em especial, o revolvimento completo duas vezes por semana do material de compostagem nas usinas e a drenagem do local onde é executada, assim como o enterrio parcial da palha pós-colheita da cana-de-açúcar com uso de cultivador. Nas propriedades pecuárias recomenda-se a limpeza de locais de criação da mosca, representado por acúmulo de dejetos de animais domésticos e ou de resíduos alimentares, especialmente em confinamentos e leiterias, dentre outras recomendações. Somente com ações integradas será possível resolver o problema e contribuir para o desenvolvimento sustentável de duas importantes cadeias produtivas do agronegócio brasileiro.

Estimativas de parâmentros genéticos visando a seleção para resistência à queima-das-folhas em populações de cenoura do grupo Brasília.

2008

Learning Fine Motion by Markov Mixtures of Experts

Compliant control is a standard method for performing fine manipulation tasks, like grasping and assembly, but it requires estimation of the state of contact between the robot arm and the objects involved. Here we present a method to learn a model of the movement from measured data. The method requires little or no prior knowledge and the resulting model explicitly estimates the state of contact. The current state of contact is viewed as the hidden state variable of a discrete HMM. The control dependent transition probabilities between states are modeled as parametrized functions of the measurement We show that their parameters can be estimated from measurements concurrently with the estimation of the parameters of the movement in each state of contact. The learning algorithm is a variant of the EM procedure. The E step is computed exactly; solving the M step exactly would require solving a set of coupled nonlinear algebraic equations in the parameters. Instead, gradient ascent is used to produce an increase in likelihood.