Real user evaluation of spoken dialogue systems using Amazon Mechanical Turk

This paper describes a framework for evaluation of spoken dialogue systems. Typically, evaluation of dialogue systems is performed in a controlled test environment with carefully selected and instructed users. However, this approach is very demanding. An alternative is to recruit a large group of users who evaluate the dialogue systems in a remote setting under virtually no supervision. Crowdsourcing technology, for example Amazon Mechanical Turk (AMT), provides an efficient way of recruiting subjects. This paper describes an evaluation framework for spoken dialogue systems using AMT users and compares the obtained results with a recent trial in which the systems were tested by locally recruited users. The results suggest that the use of crowdsourcing technology is feasible and it can provide reliable results. Copyright © 2011 ISCA.

Applying systematic design: the flight refuelling probe project

This report is a product of close industry-academia collaboration between British Aerospace and the Cambridge Engineering Design Centre (EDC). British Aerospace designs and integrates some of the most complex systems in the world, and its expertise in this field has enabled the company to become the United Kingdom's largest exporter. However, to stay at the forefront of the highly competitive aerospace industry it is necessary to seek new ways to work more effectively and more efficiently. The Cambridge EDC has played a part in supporting these needs by providing access to the methods and tools that it has developed for improving the process of designing mechanical systems. The EDC has gained an international reputation for the quality of its work in this subject. Thus, the collaboration is between two organisations each of whom are leaders in their respective fields. The central aim of the project has been to demonstrate how a systematic design process can be applied to a real design task identified by industry. The task selected was the design of a flight refuelling probe which would enable a combat aircraft to refuel from a "flying tanker". However, the systematic approach, methods and tools described in this report are applicable to most engineering design tasks. The findings presented in this report provide a sound basis for comparing the recommended systematic design process with industrial practice. The results of this comparison would enable the company to define ways in which its existing design process can be improved. This research project has a high degree of industrial relevance. The value of the work may be judged in terms of the opportunities it opens up for positive changes to the company's engineering operations. Several members of the EDC have contributed to the project. These include Dr Lucienne Blessing, Dr Stuart Burgess, Dr Amaresh Chakrabarti, Major Mark Nowack, Aylmer Johnson and Dr Paul Weaver. At British Aerospace special thanks must go to Alan Dean and David Halliday for their interest and the support they have given. The project has been managed by Dr Nigel Upton of British Aerospace during a 3 year secondment to the EDC.

Robust design optimization of gas turbine compression systems

Gas turbine compression systems are required to perform adequately over a range of operating conditions. Complexity has encouraged the conventional design process for compressors to focus initially on one operating point, usually the most commonor arduous, to draw up an outline design. Generally, only as this initial design is refined is its offdesign performance assessed in detail. Not only does this necessarily introduce a potentially costly and timeconsuming extra loop in the design process, but it also may result in a design whose offdesign behavior is suboptimal. Aversion of nonintrusive polynomial chaos was previously developed in which a set of orthonormal polynomials was generated to facilitate a rapid analysis of robustness in the presence of generic uncertainties with good accuracy. In this paper, this analysis method is incorporated in real time into the design process for the compression system of a three-shaft gas turbine aeroengine. This approach to robust optimization is shown to lead to designs that exhibit consistently improved system performance with reduced sensitivity to offdesign operation.

Real time feature-based facial tracking using Lie algebras

We have developed a novel human facial tracking system that operates in real time at a video frame rate without needing any special hardware. The approach is based on the use of Lie algebra, and uses three-dimensional feature points on the targeted human face. It is assumed that the roughly estimated facial model (relative coordinates of the three-dimensional feature points) is known. First, the initial feature positions of the face are determined using a model fitting technique. Then, the tracking is operated by the following sequence: (1) capture the new video frame and render feature points to the image plane; (2) search for new positions of the feature points on the image plane; (3) get the Euclidean matrix from the moving vector and the three-dimensional information for the points; and (4) rotate and translate the feature points by using the Euclidean matrix, and render the new points on the image plane. The key algorithm of this tracker is to estimate the Euclidean matrix by using a least square technique based on Lie algebra. The resulting tracker performed very well on the task of tracking a human face.

Natural actor and belief critic: Reinforcement algorithm for learning parameters of dialogue systems modelled as POMDPs

This article presents a novel algorithm for learning parameters in statistical dialogue systems which are modeled as Partially Observable Markov Decision Processes (POMDPs). The three main components of a POMDP dialogue manager are a dialogue model representing dialogue state information; a policy that selects the system's responses based on the inferred state; and a reward function that specifies the desired behavior of the system. Ideally both the model parameters and the policy would be designed to maximize the cumulative reward. However, while there are many techniques available for learning the optimal policy, no good ways of learning the optimal model parameters that scale to real-world dialogue systems have been found yet. The presented algorithm, called the Natural Actor and Belief Critic (NABC), is a policy gradient method that offers a solution to this problem. Based on observed rewards, the algorithm estimates the natural gradient of the expected cumulative reward. The resulting gradient is then used to adapt both the prior distribution of the dialogue model parameters and the policy parameters. In addition, the article presents a variant of the NABC algorithm, called the Natural Belief Critic (NBC), which assumes that the policy is fixed and only the model parameters need to be estimated. The algorithms are evaluated on a spoken dialogue system in the tourist information domain. The experiments show that model parameters estimated to maximize the expected cumulative reward result in significantly improved performance compared to the baseline hand-crafted model parameters. The algorithms are also compared to optimization techniques using plain gradients and state-of-the-art random search algorithms. In all cases, the algorithms based on the natural gradient work significantly better. © 2011 ACM.

Performance plumbing: Installing performance management systems to deliver lasting value

Purpose - The purpose of this paper is to explore the concept of performance plumbing, arguing that too often performance management systems in organisations are not correctly installed. Without the appropriate plumbing, performance management systems do not drive organisational change and improvement. Design/methodology/approach - The paper draws on the consultancy experiences of two of the authors, as well as the research of the third. Specific case examples are provided throughout the paper to illustrate the points being made. Findings - The paper argues that the key elements of a plumbed-in performance management system are: performance architecture; performance insights; performance focus; and performance action. Taken together, these four elements provide the necessary plumbing to enable performance management systems to deliver real value. Research limitations/implications - The paper draws on the experience of the authors, rather than a formally designed piece of research. The ideas presented in the paper would therefore benefit from further investigation and testing. Originality/value - The paper will be valuable to scholars and practitioners interested in ensuring that performance management systems deliver lasting value. © Emerald Group Publishing Limited.

Adaptively modulated optical OFDM modems utilizing RSOAs as intensity modulators in IMDD SMF transmission systems.

Detailed investigations of the transmission performance of adaptively modulated optical orthogonal frequency division multiplexed (AMOOFDM) signals converted using reflective semiconductor optical amplifiers (RSOAs) are undertaken over intensity-modulation and direct-detection (IMDD) single-mode fiber (SMF) transmission systems for WDM-PONs. The theoretical RSOA model adopted for modulating the AMOOFDM signals is experimentally verified rigorously in the aforementioned transmission systems incorporating recently developed real-time end-to-end OOFDM transceivers. Extensive performance comparisons are also made between RSOA and SOA intensity modulators. Optimum RSOA operating conditions are identified, which are independent of RSOA rear-facet reflectivity and very similar to those corresponding to SOAs. Under the identified optimum operating conditions, the RSOA and SOA intensity modulators support the identical AMOOFDM transmission performance of 30Gb/s over 60km SMFs. Under low-cost optical component-enabled practical operating conditions, RSOA intensity modulators with rear-facet reflectivity values of >0.3 outperform considerably SOA intensity modulators in transmission performance, which decreases significantly with reducing RSOA rear-facet reflectivity and optical input power. In addition, results also show that use can be made of the RSOA/SOA intensity modulation-induced negative frequency chirp to improve the AMOOFDM transmission performance in IMDD SMF systems.

Experimental demonstration of colourless real-time end-to-end optical OFDM transmission at 7.5Gb/s over 25km SSMF using a 1GHz RSOA

7.5Gb/s real-time end-to-end optical OFDM (OOFDM) transceivers incorporating variable power loading on each individual subcarrier are demonstrated experimentally, for the first time, using a live-optimized RSOA intensity modulator having a modulation bandwidth as narrow as 1GHz. Colourless real-time 16-QAM-encoded OOFDM signal transmission at 7.5Gb/s over 25km SSMF is achieved across the C-band in simple IMDD systems without in-line optical amplification and dispersion compensation. Copyright © 2010 The authors.

First experimental demonstration of low-cost VCSEL-intensity modulated end-to-end real-time optical OFDM signal transmission at 11.25Gb/s over 25km SSMFs

Low-cost, narrow modulation bandwidth, un-cooled VCSELs can be utilized to directly modulate 64-QAM-encoded 11.25Gb/s signals for end-to-end real-time optical OFDM transmission over 25km SSMF IMDD systems with excellent performance robustness. © 2011 Optical Society of America.

Experimental demonstration of real-time optical OFDM Transmission at 7.5 Gb/s over 25-km SSMF using a 1-GHz RSOA

The 7.5-Gb/s real-time end-to-end optical orthogonal frequency-division- multiplexing (OOFDM) transceivers incorporating variable power loading on each individual subcarrier are demonstrated experimentally using a live-optimized reflective semiconductor optical amplifier intensity modulator having a modulation bandwidth as narrow as 1 GHz. Real-time OOFDM signal transmission at 7.5 Gb/s over 25-km standard single-mode fiber is achieved across the $C$-band in simple intensity modulation and direct detection systems without in-line optical amplification and dispersion compensation. © 2006 IEEE.

Building more sustainable solutions in production-consumption systems: the case of food for people with reduced access

In new product development, the ability to integrate different dimensions of sustainability at a value chain level is still a complex, problematic goal. As product-service approaches are increasingly enabling the introduction of more sustainable paths, this paper describes the authors' experience thus far when building insights into conditions for the implementation of integrated solutions in a process of co-development and testing in real life conditions, which are driven by a social need focusing on food for people with reduced access. Throughout this process, which brought together producers, consumers and other stakeholders to design and test industrialised, sustainable solutions, empirical evidence demonstrates feasibility and usefulness of the approach and insight into the conditions for implementing interactive, comprehensive multi-stakeholder processes in real life situations. In addition, results show that the delivery of innovative solutions enabled to offer social added value, economic profits and environmental improvements under specific experimental conditions. © 2006 Elsevier Ltd. All rights reserved.