On-line policy optimisation of Bayesian spoken dialogue systems via human interaction

A partially observable Markov decision process has been proposed as a dialogue model that enables robustness to speech recognition errors and automatic policy optimisation using reinforcement learning (RL). However, conventional RL algorithms require a very large number of dialogues, necessitating a user simulator. Recently, Gaussian processes have been shown to substantially speed up the optimisation, making it possible to learn directly from interaction with human users. However, early studies have been limited to very low dimensional spaces and the learning has exhibited convergence problems. Here we investigate learning from human interaction using the Bayesian Update of Dialogue State system. This dynamic Bayesian network based system has an optimisation space covering more than one hundred features, allowing a wide range of behaviours to be learned. Using an improved policy model and a more robust reward function, we show that stable learning can be achieved that significantly outperforms a simulator trained policy. © 2013 IEEE.

On-line policy optimisation of spoken dialogue systems via live interaction with human subjects

Statistical dialogue models have required a large number of dialogues to optimise the dialogue policy, relying on the use of a simulated user. This results in a mismatch between training and live conditions, and significant development costs for the simulator thereby mitigating many of the claimed benefits of such models. Recent work on Gaussian process reinforcement learning, has shown that learning can be substantially accelerated. This paper reports on an experiment to learn a policy for a real-world task directly from human interaction using rewards provided by users. It shows that a usable policy can be learnt in just a few hundred dialogues without needing a user simulator and, using a learning strategy that reduces the risk of taking bad actions. The paper also investigates adaptation behaviour when the system continues learning for several thousand dialogues and highlights the need for robustness to noisy rewards. © 2011 IEEE.

Workspace comparisons of setup configurations for human-robot interaction

In virtual assembly verification or remote maintenance tasks, bimanual haptic interfaces play a crucial role in successful task completion. This paper proposes a method for objectively comparing how well a haptic interface covers the reachable workspace of human arms. Two system configurations are analyzed for a recently introduced haptic device that is based on two DLR-KUKA light weight robots: the standard configuration, where the device is opposite the human operator, and the ergonomic configuration, where the haptic device is mounted behind the human operator. The human operator directly controls the robotic arms using handles. The analysis is performed using a representation of the robot arm workspace. The merits of restricting the comparisons to the most significant regions of the human workspace are discussed. Using this method, a greater workspace correspondence for the ergonomic configuration was shown. ©2010 IEEE.