Real user evaluation of spoken dialogue systems using Amazon Mechanical Turk

This paper describes a framework for evaluation of spoken dialogue systems. Typically, evaluation of dialogue systems is performed in a controlled test environment with carefully selected and instructed users. However, this approach is very demanding. An alternative is to recruit a large group of users who evaluate the dialogue systems in a remote setting under virtually no supervision. Crowdsourcing technology, for example Amazon Mechanical Turk (AMT), provides an efficient way of recruiting subjects. This paper describes an evaluation framework for spoken dialogue systems using AMT users and compares the obtained results with a recent trial in which the systems were tested by locally recruited users. The results suggest that the use of crowdsourcing technology is feasible and it can provide reliable results. Copyright © 2011 ISCA.

Design for all: designing for the motion-impaired user

Designing for all requires the adaptation and modification of current design best practices to encompass a broader range of user capabilities. This is particularly the case in the design of the human-product interface. Product interfaces exist everywhere and when designing them, there is a very strong temptation to jump to prescribing a solution with only a cursory attempt to understand the nature of the problem. This is particularly the case when attempting to adapt existing designs, optimised for able-bodied users, for use by disabled users. However, such approaches have led to numerous products that are neither usable nor commercially successful. In order to develop a successful design approach it is necessary consider the fundamental structure of the design process being applied. A three stage design process development strategy which includes problem definition, solution development and solution evaluation, should be adopted. This paper describes the development of a new design approach based on the application of usability heuristics to the design of interfaces. This is illustrated by reference to a particular case study of the re-design of a computer interface for controlling an assistive device.

Effects of the user model on simulation-based learning of dialogue strategies

Over the past decade, a variety of user models have been proposed for user simulation-based reinforcement-learning of dialogue strategies. However, the strategies learned with these models are rarely evaluated in actual user trials and it remains unclear how the choice of user model affects the quality of the learned strategy. In particular, the degree to which strategies learned with a user model generalise to real user populations has not be investigated. This paper presents a series of experiments that qualitatively and quantitatively examine the effect of the user model on the learned strategy. Our results show that the performance and characteristics of the strategy are in fact highly dependent on the user model. Furthermore, a policy trained with a poor user model may appear to perform well when tested with the same model, but fail when tested with a more sophisticated user model. This raises significant doubts about the current practice of learning and evaluating strategies with the same user model. The paper further investigates a new technique for testing and comparing strategies directly on real human-machine dialogues, thereby avoiding any evaluation bias introduced by the user model. © 2005 IEEE.

Preliminary evaluation of conceptual mechanical designs

Computational Design has traditionally required a great deal of geometrical and parametric data. This data can only be supplied at stages later than conceptual design, typically the detail stage, and design quality is given by some absolute fitness function. On the other hand, design evaluation offers a relative measure of design quality that requires only a sparse representation. Quality, in this case, is a measure of how well a design will complete its task.

The research intends to address the question: "Is it possible to evaluate a mechanical design at the conceptual design phase and be able to make some prediction of its quality?" Quality can be interpreted as success in the marketplace, success in performing the required task, or some other user requirement. This work aims to determine a minimum level of representation such that conceptual designs can be usefully evaluated without needing to capture detailed geometry. This representation will form the model for the conceptual designs that are being considered for evaluation. The method to be developed will be a case-based evaluation system, that uses a database of previous designs to support design exploration. The method will not be able to support novel design as case-based design implies the model topology must be fixed.

Parameter estimation for agenda-based user simulation

This paper presents an agenda-based user simulator which has been extended to be trainable on real data with the aim of more closely modelling the complex rational behaviour exhibited by real users. The train-able part is formed by a set of random decision points that may be encountered during the process of receiving a system act and responding with a user act. A sample-based method is presented for using real user data to estimate the parameters that control these decisions. Evaluation results are given both in terms of statistics of generated user behaviour and the quality of policies trained with different simulators. Compared to a handcrafted simulator, the trained system provides a much better fit to corpus data and evaluations suggest that this better fit should result in improved dialogue performance. © 2010 Association for Computational Linguistics.

Simulation in user-centred design: Helping designers to empathise with atypical users

Time and budget constraints frequently prevent designers from consulting with end-users while assessing the ease of use of the products they create. This has resulted in solutions that are difficult to use by a wide range of users, especially the growing older adult population and people with different types of impairments. To help designers with this problem, capability-loss simulators have been developed with the aim of temporarily representing users who are otherwise difficult to access. This paper questions the reliability of existing tools in providing designers with meaningful information about the users' capabilities. Consequently, a new capability-loss simulation toolkit is presented, followed by its empirical evaluation. The new toolkit proved to be significantly helpful for a group of designers identifying real usability problems with everyday devices. © 2012 Copyright Taylor and Francis Group, LLC.

Evaluation of multi-sensory feedback on the usability of a virtual assembly environment

Virtual assembly environment (VAE) technology has the great potential for benefiting the manufacturing applications in industry. Usability is an important aspect of the VAE. This paper presents the usability evaluation of a developed multi-sensory VAE. The evaluation is conducted by using its three attributes: (a) efficiency of use; (b) user satisfaction; and (c) reliability. These are addressed by using task completion times (TCTs), questionnaires, and human performance error rates (HPERs), respectively. A peg-in-a-hole and a Sener electronic box assembly task have been used to perform the experiments, using sixteen participants. The outcomes showed that the introduction of 3D auditory and/or visual feedback could improve the usability. They also indicated that the integrated feedback (visual plus auditory) offered better usability than either feedback used in isolation. Most participants preferred the integrated feedback to either feedback (visual or auditory) or no feedback. The participants' comments demonstrated that nonrealistic or inappropriate feedback had negative effects on the usability, and easily made them feel frustrated. The possible reasons behind the outcomes are also analysed. © 2007 ACADEMY PUBLISHER.

Evaluation of Setup Procedures on Mobile Devices Based on Users’ Initial Experience

Users’ initial perceptions of their competence are key motivational factors for further use. However, initial tasks on a mobile operating system (OS) require setup procedures, which are currently largely inconsistent, do not provide users with clear, visible and immediate feedback on their actions, and require significant adjustment time for first-time users. This paper reports on a study with ten users, carried out to better understand how both prior experience and initial interaction with two touchscreen mobile interfaces (Apple iOS and Google Android) affected setup task performance and motivation. The results show that the reactions to setup on mobile interfaces appear to be partially dependent on which device was experienced first. Initial experience with lower-complexity devices improves performance on higher-complexity devices, but not vice versa. Based on these results, the paper proposes six guidelines for designers to design more intuitive and motivating user interfaces (UI) for setup procedures. The preliminary results indicate that these guidelines can contribute to the design of more inclusive mobile platforms and further work to validate these findings is proposed.

A global model for effective use and evaluation of e-learning in health.

Healthcare systems worldwide face a wide range of challenges, including demographic change, rising drug and medical technology costs, and persistent and widening health inequalities both within and between countries. Simultaneously, issues such as professional silos, static medical curricula, and perceptions of "information overload" have made it difficult for medical training and continued professional development (CPD) to adapt to the changing needs of healthcare professionals in increasingly patient-centered, collaborative, and/or remote delivery contexts. In response to these challenges, increasing numbers of medical education and CPD programs have adopted e-learning approaches, which have been shown to provide flexible, low-cost, user-centered, and easily updated learning. The effectiveness of e-learning varies from context to context, however, and has also been shown to make considerable demands on users' motivation and "digital literacy" and on providing institutions. Consequently, there is a need to evaluate the effectiveness of e-learning in healthcare as part of ongoing quality improvement efforts. This article outlines the key issues for developing successful models for analyzing e-health learning.

A global model for effective use and evaluation of e-learning in health.

Healthcare systems worldwide face a wide range of challenges, including demographic change, rising drug and medical technology costs, and persistent and widening health inequalities both within and between countries. Simultaneously, issues such as professional silos, static medical curricula, and perceptions of "information overload" have made it difficult for medical training and continued professional development (CPD) to adapt to the changing needs of healthcare professionals in increasingly patient-centered, collaborative, and/or remote delivery contexts. In response to these challenges, increasing numbers of medical education and CPD programs have adopted e-learning approaches, which have been shown to provide flexible, low-cost, user-centered, and easily updated learning. The effectiveness of e-learning varies from context to context, however, and has also been shown to make considerable demands on users' motivation and "digital literacy" and on providing institutions. Consequently, there is a need to evaluate the effectiveness of e-learning in healthcare as part of ongoing quality improvement efforts. This article outlines the key issues for developing successful models for analyzing e-health learning.