Real user evaluation of spoken dialogue systems using Amazon Mechanical Turk

This paper describes a framework for evaluation of spoken dialogue systems. Typically, evaluation of dialogue systems is performed in a controlled test environment with carefully selected and instructed users. However, this approach is very demanding. An alternative is to recruit a large group of users who evaluate the dialogue systems in a remote setting under virtually no supervision. Crowdsourcing technology, for example Amazon Mechanical Turk (AMT), provides an efficient way of recruiting subjects. This paper describes an evaluation framework for spoken dialogue systems using AMT users and compares the obtained results with a recent trial in which the systems were tested by locally recruited users. The results suggest that the use of crowdsourcing technology is feasible and it can provide reliable results. Copyright © 2011 ISCA.

The user-value of rural electrification: An analysis and adoption of existing models and theories

User-value is a determining factor for product acceptance in product design. Research on rural electrification to date, however, does not draw sufficient attention to the importance of user-value with regard to the overall success of a project. This is evident from the analysis of project reports and applicable indicators from agencies active in the sector. Learning from the design, psychology and sociology literatures, it is important that rural electrification projects incorporate the value perception of the end-user and extend their success beyond the commonly used criteria of financial value, the appropriateness of the technology, capacity building and technology uptake. Creating value for the end-user is particularly important for project acceptance and the sustainability of a scheme once it has been handed over to the local community. In this research paper, existing theories and models of value-theory are transposed and applied to community operated rural electrification schemes and a user-value framework is developed. Furthermore, the importance of value to the end-user is clarified. Current literature on product design reveals that user-value has different properties, many of which are applicable to rural electrification. Five value pillars and their sub-categories important for the users of rural electrification projects are identified, namely: functional; social significance; epistemic; emotional; and cultural values. These pillars provide the main structure for the conceptual framework developed in this research paper. It is proposed that by targeting the values of the end-user, the key factors of user-value applicable to rural electrification projects will be identified and the sustainability of the project will be better ensured. © 2014 The Authors.