Real user evaluation of spoken dialogue systems using Amazon Mechanical Turk

This paper describes a framework for evaluation of spoken dialogue systems. Typically, evaluation of dialogue systems is performed in a controlled test environment with carefully selected and instructed users. However, this approach is very demanding. An alternative is to recruit a large group of users who evaluate the dialogue systems in a remote setting under virtually no supervision. Crowdsourcing technology, for example Amazon Mechanical Turk (AMT), provides an efficient way of recruiting subjects. This paper describes an evaluation framework for spoken dialogue systems using AMT users and compares the obtained results with a recent trial in which the systems were tested by locally recruited users. The results suggest that the use of crowdsourcing technology is feasible and it can provide reliable results. Copyright © 2011 ISCA.

Efficient numerical models for investigating the statistics of random dynamic systems

The study of random dynamic systems usually requires the definition of an ensemble of structures and the solution of the eigenproblem for each member of the ensemble. If the process is carried out using a conventional numerical approach, the computational cost becomes prohibitive for complex systems. In this work, an alternative numerical method is proposed. The results for the response statistics are compared with values obtained from a detailed stochastic FE analysis of plates. The proposed method seems to capture the statistical behaviour of the response with a reduced computational cost.