Natural actor and belief critic: Reinforcement algorithm for learning parameters of dialogue systems modelled as POMDPs

This article presents a novel algorithm for learning parameters in statistical dialogue systems which are modeled as Partially Observable Markov Decision Processes (POMDPs). The three main components of a POMDP dialogue manager are a dialogue model representing dialogue state information; a policy that selects the system's responses based on the inferred state; and a reward function that specifies the desired behavior of the system. Ideally both the model parameters and the policy would be designed to maximize the cumulative reward. However, while there are many techniques available for learning the optimal policy, no good ways of learning the optimal model parameters that scale to real-world dialogue systems have been found yet. The presented algorithm, called the Natural Actor and Belief Critic (NABC), is a policy gradient method that offers a solution to this problem. Based on observed rewards, the algorithm estimates the natural gradient of the expected cumulative reward. The resulting gradient is then used to adapt both the prior distribution of the dialogue model parameters and the policy parameters. In addition, the article presents a variant of the NABC algorithm, called the Natural Belief Critic (NBC), which assumes that the policy is fixed and only the model parameters need to be estimated. The algorithms are evaluated on a spoken dialogue system in the tourist information domain. The experiments show that model parameters estimated to maximize the expected cumulative reward result in significantly improved performance compared to the baseline hand-crafted model parameters. The algorithms are also compared to optimization techniques using plain gradients and state-of-the-art random search algorithms. In all cases, the algorithms based on the natural gradient work significantly better. © 2011 ACM.

Modular risk modeling and management of water infrastructure disaster incidents

Water supply and wastewater control are critical elements of society's infrastructure. The objective of this study will be to provide a generic risk assessment tool to provide municipalities and the nation as a whole with a quantifiable assessment of their vulnerability to water infrastructure threats. The approach will prioritize countermeasures and identify where research and development is required to further minimize risk. This paper outlines the current context, primary concerns and state-of-the art in critical infrastructure risk management for the water sector and proposes a novel approach to resolve existing questions in the field. The proposed approach is based on a modular framework that derives a quantitative risk index for varied domains of interest. The approach methodology is scaleable and based on formal definitions of event probability and severity. The framework is equally applicable to natural and human-induced hazard types and can be used for analysis of compound risk events.

Breathers and thermal relaxation as a temporal process: A possible way to detect breathers in experimental situations

Breather stability and longevity in thermally relaxing nonlinear arrays is investigated under the scrutiny of the analysis and tools employed for time series and state reconstruction of a dynamical system. We briefly review the methods used in the analysis and characterize a breather in terms of the results obtained with such methods. Our present work focuses on spontaneously appearing breathers in thermal Fermi-Pasta-Ulam arrays but we believe that the conclusions are general enough to describe many other related situations; the particular case described in detail is presented as another example of systems where three incommensurable frequencies dominate their chaotic dynamics (reminiscent of the Ruelle-Takens scenario for the appearance of chaotic behavior in nonlinear systems). This characterization may also be of great help for the discovery of breathers in experimental situations where the temporal evolution of a local variable (like the site energy) is the only available/measured data. © 2005 American Institute of Physics.