Natural actor and belief critic: Reinforcement algorithm for learning parameters of dialogue systems modelled as POMDPs

This article presents a novel algorithm for learning parameters in statistical dialogue systems which are modeled as Partially Observable Markov Decision Processes (POMDPs). The three main components of a POMDP dialogue manager are a dialogue model representing dialogue state information; a policy that selects the system's responses based on the inferred state; and a reward function that specifies the desired behavior of the system. Ideally both the model parameters and the policy would be designed to maximize the cumulative reward. However, while there are many techniques available for learning the optimal policy, no good ways of learning the optimal model parameters that scale to real-world dialogue systems have been found yet. The presented algorithm, called the Natural Actor and Belief Critic (NABC), is a policy gradient method that offers a solution to this problem. Based on observed rewards, the algorithm estimates the natural gradient of the expected cumulative reward. The resulting gradient is then used to adapt both the prior distribution of the dialogue model parameters and the policy parameters. In addition, the article presents a variant of the NABC algorithm, called the Natural Belief Critic (NBC), which assumes that the policy is fixed and only the model parameters need to be estimated. The algorithms are evaluated on a spoken dialogue system in the tourist information domain. The experiments show that model parameters estimated to maximize the expected cumulative reward result in significantly improved performance compared to the baseline hand-crafted model parameters. The algorithms are also compared to optimization techniques using plain gradients and state-of-the-art random search algorithms. In all cases, the algorithms based on the natural gradient work significantly better. © 2011 ACM.

Effect of operating conditions and fuel type on crevice hc emissions: Model results and comparison with experiments

A one-dimensional model for crevice HC post-flame oxidation is used to calculate and understand the effect of operating parameters and fuel type (propane and isooctane) on the extent of crevice hydrocarbon and the product distribution in the post flame environment. The calculations show that the main parameters controlling oxidation are: bulk burned gas temperatures, wall temperatures, turbulent diffusivity, and fuel oxidation rates. Calculated extents of oxidation agree well with experimental values, and the sensitivities to operating conditions (wall temperatures, equivalence ratio, fuel type) are reasonably well captured. Whereas the bulk gas temperatures largely determine the extent of oxidation, the hydrocarbon product distribution is not very much affected by the burned gas temperatures, but mostly by diffusion rates. Uncertainties in both turbulent diffusion rates as well as in mechanisms are an important factor limiting the predictive capabilities of the model. However, it seems well suited to sensitivity calculations about a baseline. Copyright © 1999 Society of Automotive Engineers, Inc.

Bayesian updating of a unified soil compression model

Some amount of differential settlement occurs even in the most uniform soil deposit, but it is extremely difficult to estimate because of the natural heterogeneity of the soil. The compression response of the soil and its variability must be characterised in order to estimate the probability of the differential settlement exceeding a certain threshold value. The work presented in this paper introduces a probabilistic framework to address this issue in a rigorous manner, while preserving the format of a typical geotechnical settlement analysis. In order to avoid dealing with different approaches for each category of soil, a simplified unified compression model is used to characterise the nonlinear compression behavior of soils of varying gradation through a single constitutive law. The Bayesian updating rule is used to incorporate information from three different laboratory datasets in the computation of the statistics (estimates of the means and covariance matrix) of the compression model parameters, as well as of the uncertainty inherent in the model.