Reinforcement learning for parameter estimation in statistical spoken dialogue systems

Reinforcement techniques have been successfully used to maximise the expected cumulative reward of statistical dialogue systems. Typically, reinforcement learning is used to estimate the parameters of a dialogue policy which selects the system's responses based on the inferred dialogue state. However, the inference of the dialogue state itself depends on a dialogue model which describes the expected behaviour of a user when interacting with the system. Ideally the parameters of this dialogue model should be also optimised to maximise the expected cumulative reward. This article presents two novel reinforcement algorithms for learning the parameters of a dialogue model. First, the Natural Belief Critic algorithm is designed to optimise the model parameters while the policy is kept fixed. This algorithm is suitable, for example, in systems using a handcrafted policy, perhaps prescribed by other design considerations. Second, the Natural Actor and Belief Critic algorithm jointly optimises both the model and the policy parameters. The algorithms are evaluated on a statistical dialogue system modelled as a Partially Observable Markov Decision Process in a tourist information domain. The evaluation is performed with a user simulator and with real users. The experiments indicate that model parameters estimated to maximise the expected reward function provide improved performance compared to the baseline handcrafted parameters. © 2011 Elsevier Ltd. All rights reserved.

Statistical user simulation with a hidden agenda

Recent work in the area of probabilistic user simulation for training statistical dialogue managers has investigated a new agenda-based user model and presented preliminary experiments with a handcrafted model parameter set. Training the model on dialogue data is an important next step, but non-trivial since the user agenda states are not observable in data and the space of possible states and state transitions is intractably large. This paper presents a summary-space mapping which greatly reduces the number of state transitions and introduces a tree-based method for representing the space of possible agenda state sequences. Treating the user agenda as a hidden variable, the forward/backward algorithm can then be successfully applied to iteratively estimate the model parameters on dialogue data. © 2007 Association for Computational Linguistics.

Parameter estimation for agenda-based user simulation

This paper presents an agenda-based user simulator which has been extended to be trainable on real data with the aim of more closely modelling the complex rational behaviour exhibited by real users. The train-able part is formed by a set of random decision points that may be encountered during the process of receiving a system act and responding with a user act. A sample-based method is presented for using real user data to estimate the parameters that control these decisions. Evaluation results are given both in terms of statistics of generated user behaviour and the quality of policies trained with different simulators. Compared to a handcrafted simulator, the trained system provides a much better fit to corpus data and evaluations suggest that this better fit should result in improved dialogue performance. © 2010 Association for Computational Linguistics.

Phrase-based statistical language generation using graphical models and active learning

Most previous work on trainable language generation has focused on two paradigms: (a) using a statistical model to rank a set of generated utterances, or (b) using statistics to inform the generation decision process. Both approaches rely on the existence of a handcrafted generator, which limits their scalability to new domains. This paper presents BAGEL, a statistical language generator which uses dynamic Bayesian networks to learn from semantically-aligned data produced by 42 untrained annotators. A human evaluation shows that BAGEL can generate natural and informative utterances from unseen inputs in the information presentation domain. Additionally, generation performance on sparse datasets is improved significantly by using certainty-based active learning, yielding ratings close to the human gold standard with a fraction of the data. © 2010 Association for Computational Linguistics.

Partially Observable Markov Decision Processes with continuous observations for dialogue management

This work shows how a dialogue model can be represented as a Partially Observable Markov Decision Process (POMDP) with observations composed of a discrete and continuous component. The continuous component enables the model to directly incorporate a confidence score for automated planning. Using a testbed simulated dialogue management problem, we show how recent optimization techniques are able to find a policy for this continuous POMDP which outperforms a traditional MDP approach. Further, we present a method for automatically improving handcrafted dialogue managers by incorporating POMDP belief state monitoring, including confidence score information. Experiments on the testbed system show significant improvements for several example handcrafted dialogue managers across a range of operating conditions.

Satellite Constellation Design for Complex Coverage

Most traditional satellite constellation design methods are associated with a simple zonal or global, continuous or discontinuous coverage connected with a visibility of points on the Earth's surface. A new geometric approach for more complex coverage of a geographic region is proposed. Full and partial coverage of regions is considered. It implies that, at any time, the region is completely or partially within the instantaneous access area of a satellite of the constellation. The key idea of the method is a two-dimensional space application for maps of the satellite constellation and coverage requirements. The space dimensions are right ascension of ascending node and argument of latitude. Visibility requirements of each region can be presented as a polygon and satellite constellation as a uniform moving grid. At any time, at least one grid vertex must belong to the polygon. The optimal configuration of the satellite constellation corresponds to the maximum sparse grid. The method is suitable for continuous and discontinuous coverage. In the last case, a vertex belonging to the polygon should be examined with a revisit time. Examples of continuous coverage for a space communication network and of the United States are considered. Examples of discontinuous coverage are also presented.

Millstone Hill电离层F层等效风场的气候学研究

Neutral winds and electric fields in the ionospheric F layer play important roles in the variations of the ionosphere, and also affect the thermospheric circulation via the close coupling between the ionosphere and the thermosphere. By now, the neutral winds and electric drifts are generally observed with ground-based Fabry-Perot interferometers (FPI) and incoherent scatter radars (ISR), rockets, and satellite-borne instrument. Based on the servo theory, the ionospheric equivalent winds, which include the information of both the neutral winds and electric fields, can be derived from these characteristic parameters observed by ionosondes. This indirect derivation has potential values in climatological researches and space weather forecast. With the data set of the incoherent scatter radar observations at Millstone Hill, USA, from 1976 to 2006, we statistically analyzed the climatological variations of the vertical component of the equivalent winds (VEWs) over Millstone Hill, which are derived from the ionospheric key parameters (the peak electron number density and peak height of the F2 layer, NmF2 and hmF2) on the basis of the servo theory, Liu's method, and measurements from the ion line-of-sight velocity as well. The main results of this analysis are summarized as follows: (1) The values of VEWs over Millstone Hill during nighttime are stronger than in the daytime, and the upward drift dominates most of the day. In 1993, Hagan found that the component of the neutral winds in the magnetic meridion in daytime is weaker than during nighttime under both solar maximum and minimum conditions; he also found that the equatorward winds dominate most of the day. Both results suggest that the thermosphere in Millstone Hill is modulated by the aurorally driven high-latitude circulation cell; that is, during geomagnetic quiet periods, the average auroral activity is strong enough to drive thermospheric circulation equatorward for most of the day at Millstone Hill. Moreover, since ion drag is the strongest during daytime when F region densities are enhanced by photoionization, the wind speeds are smaller during the daytime than in the nighttime. (2) There is equinoctial symmetry in VEWs at Millstone Hill. The amplitudes and phases of VEWs in spring are quite similar to those in autumn. In contrast, the nighttime upward drift in winter is weaker than in summer and the difference becomes more significant with increasing solar activity. This solstice asymmetry indicates that, the aurorally driven circulation in the northern hemisphere at Millstone Hill latitude is weaker in winter due to arctic darkness, because the subsolar point is in the southern hemisphere. (3) The comparison of the VEWs derived from three methods, i.e., the servo theory, Liu's method, and the ISR ion line-of-sight velocity measurements, indicates that the amplitudes and main phase tendencies of these VEWs accord well with each other during nighttime hours. However, the case in the daytime is relatively worse. This daytime discrepancy can be explained in terms of the effects of photochemical processes and the choices of the servo constants. A larger servo constant gives a stronger plasma drift in daytime. Therefore, this study tells how important to choose a suitable constant for deriving VEWs at Millstone Hill.

Reuseable Silicon IP Cores for Discrete Wavelet Transform Applications

Architectures and methods for the rapid design of silicon cores for implementing discrete wavelet transforms over a wide range of specifications are described. These architectures are efficient, modular, scalable, and cover orthonormal and biorthogonal wavelet transform families. They offer efficient hardware utilization by exploiting a number of core wavelet filter properties and allow the creation of silicon designs that are highly parameterized, including in terms of wavelet type and wordlengths. Control circuitry is embedded within these systems allowing them to be cascaded for any desired level of decomposition without any interface glue logic. The time to produce chip designs for a specific wavelet application is typically less than a day and these are comparable in area and performance to handcrafted designs. They are also portable across a wide range of silicon foundries and suitable for field programmable gate array and programmable logic data implementation. The approach described has also been extended to wavelet packet transforms.

A Kernel Interleaved Scheduling Method for Streaming Applications on Soft-core Vector Processors

Massively parallel networks of highly efficient, high performance Single Instruction Multiple Data (SIMD) processors have been shown to enable FPGA-based implementation of real-time signal processing applications with performance and
cost comparable to dedicated hardware architectures. This is achieved by exploiting simple datapath units with deep processing pipelines. However, these architectures are highly susceptible to pipeline bubbles resulting from data and control hazards; the only way to mitigate against these is manual interleaving of
application tasks on each datapath, since no suitable automated interleaving approach exists. In this paper we describe a new automated integrated mapping/scheduling approach to map algorithm tasks to processors and a new low-complexity list scheduling technique to generate the interleaved schedules. When applied to a spatial Fixed-Complexity Sphere Decoding (FSD) detector
for next-generation Multiple-Input Multiple-Output (MIMO) systems, the resulting schedules achieve real-time performance for IEEE 802.11n systems on a network of 16-way SIMD processors on FPGA, enable better performance/complexity balance than current approaches and produce results comparable to handcrafted implementations.

Automated design of DSP array processor chips

Details are presented of the DAC (DSP ASIC Compiler) silicon compiler framework. DAC allows a non-specialist to automatically design DSP ASICs and DSP ASIC cores directly form a high level specification. Typical designs take only several minutes and the resulting layouts are comparable in area and performance to handcrafted designs.