Real user evaluation of spoken dialogue systems using Amazon Mechanical Turk

This paper describes a framework for evaluation of spoken dialogue systems. Typically, evaluation of dialogue systems is performed in a controlled test environment with carefully selected and instructed users. However, this approach is very demanding. An alternative is to recruit a large group of users who evaluate the dialogue systems in a remote setting under virtually no supervision. Crowdsourcing technology, for example Amazon Mechanical Turk (AMT), provides an efficient way of recruiting subjects. This paper describes an evaluation framework for spoken dialogue systems using AMT users and compares the obtained results with a recent trial in which the systems were tested by locally recruited users. The results suggest that the use of crowdsourcing technology is feasible and it can provide reliable results. Copyright © 2011 ISCA.

An online daily diary study of alcohol use using Amazon's mechanical turk

Introduction and Aims: In recent years, unprecedented levels of Internet access and the widespread growth of emergent communication technologies have resulted in significantly greater population access for substance use researchers. Despite the research potential of such technologies, the use of the Internet to recruit individuals for participation in event-level research has been limited. The purpose of this paper is to provide a brief account of the methods and results from an online daily diary study of alcohol use. Design and Methods: Participants were recruited using Amazon's Mechanical Turk. Eligible participants completed a brief screener assessing demographics and health behaviours, with a subset of individuals subsequently recruited to participate in a 2 week daily diary study of alcohol use. Results: Multilevel models of the daily alcohol data derived from the Mechanical Turk sample (n=369) replicated several findings commonly reported in daily diary studies of alcohol use. Discussion and Conclusions: Results demonstrate that online participant recruitment and survey administration can be a fruitful method for conducting daily diary alcohol research. © 2014 Australasian Professional Society on Alcohol and other Drugs.

The rule of the turk a revised and enlarged edition of the Armenian crisis

UANL

Muṣayyab-nāmah-ʼi Turkī

Legendary account of Musayyab ibn Najabah's activities to avenge the death of Ḥusayn ibn ʻAlī.