Real user evaluation of spoken dialogue systems using Amazon Mechanical Turk

This paper describes a framework for evaluation of spoken dialogue systems. Typically, evaluation of dialogue systems is performed in a controlled test environment with carefully selected and instructed users. However, this approach is very demanding. An alternative is to recruit a large group of users who evaluate the dialogue systems in a remote setting under virtually no supervision. Crowdsourcing technology, for example Amazon Mechanical Turk (AMT), provides an efficient way of recruiting subjects. This paper describes an evaluation framework for spoken dialogue systems using AMT users and compares the obtained results with a recent trial in which the systems were tested by locally recruited users. The results suggest that the use of crowdsourcing technology is feasible and it can provide reliable results. Copyright © 2011 ISCA.

A Methodological Approach to User Evaluation and Assessment of a Virtual Environment Hangout

Innovation in virtual reality and motion sensing devices is pushing the development of virtual communication platforms towards completely immersive scenarios, which require full user interaction and create complex sensory experiences. This evolution influences user experiences and creates new paradigms for interaction, leading to an increased importance of user evaluation and assessment on new systems interfaces and usability, to validate platform design and development from the users’ point of view. The REVERIE research project aims to develop a virtual environment service for realistic inter-personal interaction. This paper describes the design challenges faced during the development process of user interfaces and the adopted methodological approach to user evaluation and assessment.

Supporting user evaluation of IT security certification schemes

IT Security Certification is an increasingly important qualification for information technology (IT) professionals seeking employment in IT security. Yet currently there is a lack of rigorously developed approaches to support the evaluation and selection by key stakeholders of the most appropriate IT security certification scheme from among hundreds of vendor-neutral and vendor-specific schemes. This paper develops a framework based on categories, characteristics and criteria to support user evaluation and selection of an (IT) Security Certification scheme that satisfies user priorities and requirements. The paper illustrates the use of the framework to support an experienced IT Professional’s evaluation. Theoretical and practical implications of the framework and trial evaluation are discussed.

Sensemaking tools for understanding research literatures:design, implementation and user evaluation

This paper describes the work undertaken in the Scholarly Ontologies Project. The aim of the project has been to develop a computational approach to support scholarly sensemaking, through interpretation and argumentation, enabling researchers to make claims: to describe and debate their view of a document's key contributions and relationships to the literature. The project has investigated the technicalities and practicalities of capturing conceptual relations, within and between conventional documents in terms of abstract ontological structures. In this way, we have developed a new kind of index to distributed digital library systems. This paper reports a case study undertaken to test the sensemaking tools developed by the Scholarly Ontologies project. The tools used were ClaiMapper, which allows the user to sketch argument maps of individual papers and their connections, ClaiMaker, a server on which such models can be stored and saved, which provides interpretative services to assist the querying of argument maps across multiple papers and ClaimFinder, a novice interface to the search services in ClaiMaker.

User interface design support for the development of knowledge-based systems

The present scarcity of operational knowledge-based systems (KBS) has been attributed, in part, to an inadequate consideration shown to user interface design during development. From a human factors perspective the problem has stemmed from an overall lack of user-centred design principles. Consequently the integration of human factors principles and techniques is seen as a necessary and important precursor to ensuring the implementation of KBS which are useful to, and usable by, the end-users for whom they are intended. Focussing upon KBS work taking place within commercial and industrial environments, this research set out to assess both the extent to which human factors support was presently being utilised within development, and the future path for human factors integration. The assessment consisted of interviews conducted with a number of commercial and industrial organisations involved in KBS development; and a set of three detailed case studies of individual KBS projects. Two of the studies were carried out within a collaborative Alvey project, involving the Interdisciplinary Higher Degrees Scheme (IHD) at the University of Aston in Birmingham, BIS Applied Systems Ltd (BIS), and the British Steel Corporation. This project, which had provided the initial basis and funding for the research, was concerned with the application of KBS to the design of commercial data processing (DP) systems. The third study stemmed from involvement on a KBS project being carried out by the Technology Division of the Trustees Saving Bank Group plc. The preliminary research highlighted poor human factors integration. In particular, there was a lack of early consideration of end-user requirements definition and user-centred evaluation. Instead concentration was given to the construction of the knowledge base and prototype evaluation with the expert(s). In response to this identified problem, a set of methods was developed that was aimed at encouraging developers to consider user interface requirements early on in a project. These methods were then applied in the two further projects, and their uptake within the overall development process was monitored. Experience from the two studies demonstrated that early consideration of user interface requirements was both feasible, and instructive for guiding future development work. In particular, it was shown a user interface prototype could be used as a basis for capturing requirements at the functional (task) level, and at the interface dialogue level. Extrapolating from this experience, a KBS life-cycle model is proposed which incorporates user interface design (and within that, user evaluation) as a largely parallel, rather than subsequent, activity to knowledge base construction. Further to this, there is a discussion of several key elements which can be seen as inhibiting the integration of human factors within KBS development. These elements stem from characteristics of present KBS development practice; from constraints within the commercial and industrial development environments; and from the state of existing human factors support.

Enabling effective user interactions in content-based image retrieval

This paper presents an interactive content-based image retrieval framework—uInteract, for delivering a novel four-factor user interaction model visually. The four-factor user interaction model is an interactive relevance feedback mechanism that we proposed, aiming to improve the interaction between users and the CBIR system and in turn users overall search experience. In this paper, we present how the framework is developed to deliver the four-factor user interaction model, and how the visual interface is designed to support user interaction activities. From our preliminary user evaluation result on the ease of use and usefulness of the proposed framework, we have learnt what the users like about the framework and the aspects we could improve in future studies. Whilst the framework is developed for our research purposes, we believe the functionalities could be adapted to any content-based image search framework.

Suivi postnatal à domicile après un congé précoce : Critères de sélection et Appréciation du délai

Aujourd’hui, la satisfaction des utilisateurs des services de santé est reconnue comme une mesure de la qualité des soins. Au Québec, le congé précoce en obstétrique constitue la norme pour les mères ayant donné naissance à un bébé en santé. Selon la littérature, cette pratique n’entraîne pas de répercussions négatives pour la santé des mères et de leur nouveau-né à condition qu’un suivi adéquat soit assuré. D’autre part, bien qu’il semble que la diminution de la durée du séjour hospitalier soit appréciée par les mères, peu de données sont disponibles relativement aux caractéristiques menant à l’appréciation du suivi postnatal. Objectifs : Cette étude s’intéresse principalement à la première visite à domicile effectuée par une infirmière suite au congé précoce en obstétrique. Dans un premier temps, elle vise à tracer un portrait des mères en fonction du délai de la première visite à domicile et, dans un second temps, à connaître les facteurs associés à l’appréciation, par les mères, du délai de cette visite. Méthode : Les données de cette étude ont été recueillies au Québec, entre janvier 2002 et janvier 2003, lors d’une enquête téléphonique effectuée auprès de mères de bébés nés en santé, un mois suivant leur accouchement vaginal sans complication (n=1548). Pour nos analyses, nous avons retranché les mères ayant eu une durée de séjour de plus de 60 heures, une grossesse de moins de 37semaines et un bébé pesant moins de 2500 g à la naissance. Notre échantillon se compose donc de 1351 mères. Résultats : 86,2 % des mères ont reçu une offre de visite à domicile. La majorité (80.2 %) des mères ont reçu la visite dans les trois premiers jours suivant leur retour à la maison, dont près du tiers (28,1 %), dans les 24 premières heures. Comparativement aux mères visitées au deuxième ou troisième jour suivant le congé, celles visitées dans les 24 premières heures ont jugé la durée de séjour hospitalier trop courte (p=0,018) et reçu un appel de l’infirmière qui a duré plus longtemps (p=0, 009). De plus, au moment du congé, elles perçoivent leur bébé en moins bonne santé (p=0,029). Elles ont aussi accouché d’un bébé plus petit (p=0,052) qui a tendance à avoir présenté des signes d’ictères pendant le séjour hospitalier (p=0,100). D’autre part, la majorité des mères (86,4 %) disent que le délai de la première visite à domicile est adéquat alors que 11,6 % le jugent trop court et 2,3 % trop long. Pour les mères visitées au premier jour, l’analyse multivariée révèle que certaines caractéristiques et certains besoins sont associés à la perception que le délai de la visite est trop court : une seule visite postnatale, un revenu familial de plus de 40 000 $, la perception que la durée de séjour est trop longue et le fait de ne pas allaiter. Pour les mères qui reçoivent la visite au deuxième et troisième jour, ce sont, seulement, le fait d’avoir été au rendez-vous médical et le fait d’avoir reçu une seule visite qui sont associés à la perception que le délai de la visite est trop court. Pour conclure, au Québec, le programme de suivi postnatal universel semble en mesure d’offrir une visite à domicile dans les délais prescrits à une majorité de mères. Les résultats de cette étude suggèrent que le délai de la première visite à domicile n’est pas optimal pour toutes les mères et permettent d’envisager que certaines mères auraient souhaité recevoir une seconde visite plus tardivement au cours de la période postnatale. D’autres recherches devront être effectuées afin de parfaire nos connaissances relativement au moment idéal pour réaliser les interventions postnatales.Mots clefs : Satisfaction, appréciation des utilisateurs, qualité des soins, programme universel, suivi postnatal, congé précoce en obstétrique, visite à domicile, délai de la visite, provision des services.

Att få ett jobb har betytt mycket : En kvalitativ brukarutvärdering av Krami

In this qualitative user evaluation, users from the Krami-programme in Borlänge have been interviewed on what they consider to have been positive changes in their social situation after participating in the programme. The respondents also point out factors in the programme that have supported these changes. The method consisted of interviews with five men who have undergone the Krami programme. The results show that Krami primarily helps the users to get an employment which results in the repeal of criminogenic factors. This plays an important part in the positive change of the users self-esteem, which tends to lead towards ”unlabeling” of the individual as “criminal”. The study also indicates other factors in the program that the users found helpful in their process of change highlighting the interaction between authorities, the approach of the Krami- staff and also the common rules that the participants has to agree upon. The users also poins out the importance of ones own motivation for changes to take place. 

A fuzzy rule-based decision support system for Duodopa treatment in Parkinson

A decision support system (DSS) was implemented based on a fuzzy logic inference system (FIS) to provide assistance in dose alteration of Duodopa infusion in patients with advanced Parkinson’s disease, using data from motor state assessments and dosage. Three-tier architecture with an object oriented approach was used. The DSS has a web enabled graphical user interface that presents alerts indicating non optimal dosage and states, new recommendations, namely typical advice with typical dose and statistical measurements. One data set was used for design and tuning of the FIS and another data set was used for evaluating performance compared with actual given dose. Overall goodness-of-fit for the new patients (design data) was 0.65 and for the ongoing patients (evaluation data) 0.98. User evaluation is now ongoing. The system could work as an assistant to clinical staff for Duodopa treatment in advanced Parkinson’s disease.

A web application for follow-up of results from a mobile device test battery for parkinson’s disease patients

Background: A test battery consisting of self-assessments and motor tests (tapping and spiral drawing) was developed for a hand computer with touch screen in a telemedicine setting. Objectives: To develop and evaluate a web-based system that delivers decision support information to the treating clinical staff for assessing PD symptoms in their patients based on the test battery data. Methods: The test battery is currently being used in a clinical trial (DAPHNE, EudraCT No. 2005-002654-21) by sixty five patients with advanced Parkinson’s disease (PD) on 9991 test occasions (four tests per day during in all 362 week-long test periods) at nine clinics around Sweden. Test results are sent continuously from the hand unit over a mobile net to a central computer and processed with statistical methods. They are summarized into scores for different dimensions of the symptom state and an ‘overall test score’ reflecting the overall condition of the patient during a test period. The information in the web application is organized and presented graphically in a way that the general overview of the patient performance per test period is emphasized. Focus is on the overall test score, symptom dimensions and daily summaries. In a recent preliminary user evaluation, the web application was demonstrated to the fifteen study nurses who had used the test battery in the clinical trial. At least one patient per clinic was shown. Results: In general, the responses from nurses were positive. They claimed that the test results shown in the system were consistent with their own clinical observations. They could follow complications, changes and trends within their patients. Discussion: In conclusion, the system is able to summarise the various time series of motor test results and self-assessments during test periods and present them in a useful manner. Its main contribution is a novel and reliable way to capture and easily access symptom information from patients’ home environment. The convenient access to current symptom profile as well as symptom history provides a basis for individualized evaluation and adjustment of treatments.

Flyktingguide: En väg till integration? : Brukarutvärdering av projektet Flyktingguide i Falun

In this qualitative user evaluation participants in the project refugee guide, Falun, was interviewed about their experience of the project and how the project contributed to their integration into society and establishment on the labor market. The collection of empirical data has been made by six interviews. The results of the evaluation indicate that participants feel that they through the project had the opportunity to meet new people, practice the language and learn about each other's culture. The results also show that participants are satisfied with the project. Some participants expressed a desire to meet with several people at once while others don´t have that need. Furthermore, the interviews shows that establishment on labor market occurred through job placement and not through participation in the project. Integration proved to be a difficult concept to define, but the interviews show that the participants have a perception that it is about learning the Swedish language and culture.

Objective and subjective measurements of office meeting rooms

This research is a preliminary investigation of the objective measurements ofthelmal comfort and C02 levels as well as subjective user evaluation surveys in six meeting rooms. The ongoing objective measurements were taken via thelmal comfort carts while the surveys evaluated parameters of temperature, lighting, air quality, meeting effectiveness etc. The outcomes are of particular interest since some of the objective measurements clearly to not support those of the users. In fact, the investigation has triggered off further interest in refining questionnaires to develop more precise indicators of 'discomfort'.
Furthelmore, meeting rooms present a challenge when it comes to conditioning spaces (HV AC design) which are sporadic in use and often with high occupancy for short periods. This paper investigates meeting rooms in four different buildings and their conditioning control concepts are discussed in relation to measured results.

Addressing online information needs in palliative care : an action research-inspired approach

Whereas user participation has been embraced worldwide as a means to provide better patient outcomes, the implementation of formative, action research approaches in online health information has remained under-explored. The purpose of this study is to present an action research-based methodology that allows the scoping of health information and design needs in complex, multi-user online environments. The project's four main stages were informed by an iterative, formative approach involving continuous expert and user evaluation. The study suggests that an action research-inspired formative approach can be successfully employed to generate user-participation. Moreover, sustained user-participation effectively addresses most quality issues regarding content, language, and accessibility raised in the recent literature. The paper concludes that an action research approach geared to develop online health resources deserves more attention.