Real user evaluation of spoken dialogue systems using Amazon Mechanical Turk

This paper describes a framework for evaluation of spoken dialogue systems. Typically, evaluation of dialogue systems is performed in a controlled test environment with carefully selected and instructed users. However, this approach is very demanding. An alternative is to recruit a large group of users who evaluate the dialogue systems in a remote setting under virtually no supervision. Crowdsourcing technology, for example Amazon Mechanical Turk (AMT), provides an efficient way of recruiting subjects. This paper describes an evaluation framework for spoken dialogue systems using AMT users and compares the obtained results with a recent trial in which the systems were tested by locally recruited users. The results suggest that the use of crowdsourcing technology is feasible and it can provide reliable results. Copyright © 2011 ISCA.

A Methodological Approach to User Evaluation and Assessment of a Virtual Environment Hangout

Innovation in virtual reality and motion sensing devices is pushing the development of virtual communication platforms towards completely immersive scenarios, which require full user interaction and create complex sensory experiences. This evolution influences user experiences and creates new paradigms for interaction, leading to an increased importance of user evaluation and assessment on new systems interfaces and usability, to validate platform design and development from the users’ point of view. The REVERIE research project aims to develop a virtual environment service for realistic inter-personal interaction. This paper describes the design challenges faced during the development process of user interfaces and the adopted methodological approach to user evaluation and assessment.

Sensemaking tools for understanding research literatures:design, implementation and user evaluation

This paper describes the work undertaken in the Scholarly Ontologies Project. The aim of the project has been to develop a computational approach to support scholarly sensemaking, through interpretation and argumentation, enabling researchers to make claims: to describe and debate their view of a document's key contributions and relationships to the literature. The project has investigated the technicalities and practicalities of capturing conceptual relations, within and between conventional documents in terms of abstract ontological structures. In this way, we have developed a new kind of index to distributed digital library systems. This paper reports a case study undertaken to test the sensemaking tools developed by the Scholarly Ontologies project. The tools used were ClaiMapper, which allows the user to sketch argument maps of individual papers and their connections, ClaiMaker, a server on which such models can be stored and saved, which provides interpretative services to assist the querying of argument maps across multiple papers and ClaimFinder, a novice interface to the search services in ClaiMaker.

User interface design support for the development of knowledge-based systems

The present scarcity of operational knowledge-based systems (KBS) has been attributed, in part, to an inadequate consideration shown to user interface design during development. From a human factors perspective the problem has stemmed from an overall lack of user-centred design principles. Consequently the integration of human factors principles and techniques is seen as a necessary and important precursor to ensuring the implementation of KBS which are useful to, and usable by, the end-users for whom they are intended. Focussing upon KBS work taking place within commercial and industrial environments, this research set out to assess both the extent to which human factors support was presently being utilised within development, and the future path for human factors integration. The assessment consisted of interviews conducted with a number of commercial and industrial organisations involved in KBS development; and a set of three detailed case studies of individual KBS projects. Two of the studies were carried out within a collaborative Alvey project, involving the Interdisciplinary Higher Degrees Scheme (IHD) at the University of Aston in Birmingham, BIS Applied Systems Ltd (BIS), and the British Steel Corporation. This project, which had provided the initial basis and funding for the research, was concerned with the application of KBS to the design of commercial data processing (DP) systems. The third study stemmed from involvement on a KBS project being carried out by the Technology Division of the Trustees Saving Bank Group plc. The preliminary research highlighted poor human factors integration. In particular, there was a lack of early consideration of end-user requirements definition and user-centred evaluation. Instead concentration was given to the construction of the knowledge base and prototype evaluation with the expert(s). In response to this identified problem, a set of methods was developed that was aimed at encouraging developers to consider user interface requirements early on in a project. These methods were then applied in the two further projects, and their uptake within the overall development process was monitored. Experience from the two studies demonstrated that early consideration of user interface requirements was both feasible, and instructive for guiding future development work. In particular, it was shown a user interface prototype could be used as a basis for capturing requirements at the functional (task) level, and at the interface dialogue level. Extrapolating from this experience, a KBS life-cycle model is proposed which incorporates user interface design (and within that, user evaluation) as a largely parallel, rather than subsequent, activity to knowledge base construction. Further to this, there is a discussion of several key elements which can be seen as inhibiting the integration of human factors within KBS development. These elements stem from characteristics of present KBS development practice; from constraints within the commercial and industrial development environments; and from the state of existing human factors support.

Enabling effective user interactions in content-based image retrieval

This paper presents an interactive content-based image retrieval framework—uInteract, for delivering a novel four-factor user interaction model visually. The four-factor user interaction model is an interactive relevance feedback mechanism that we proposed, aiming to improve the interaction between users and the CBIR system and in turn users overall search experience. In this paper, we present how the framework is developed to deliver the four-factor user interaction model, and how the visual interface is designed to support user interaction activities. From our preliminary user evaluation result on the ease of use and usefulness of the proposed framework, we have learnt what the users like about the framework and the aspects we could improve in future studies. Whilst the framework is developed for our research purposes, we believe the functionalities could be adapted to any content-based image search framework.

Suivi postnatal à domicile après un congé précoce : Critères de sélection et Appréciation du délai

Aujourd’hui, la satisfaction des utilisateurs des services de santé est reconnue comme une mesure de la qualité des soins. Au Québec, le congé précoce en obstétrique constitue la norme pour les mères ayant donné naissance à un bébé en santé. Selon la littérature, cette pratique n’entraîne pas de répercussions négatives pour la santé des mères et de leur nouveau-né à condition qu’un suivi adéquat soit assuré. D’autre part, bien qu’il semble que la diminution de la durée du séjour hospitalier soit appréciée par les mères, peu de données sont disponibles relativement aux caractéristiques menant à l’appréciation du suivi postnatal. Objectifs : Cette étude s’intéresse principalement à la première visite à domicile effectuée par une infirmière suite au congé précoce en obstétrique. Dans un premier temps, elle vise à tracer un portrait des mères en fonction du délai de la première visite à domicile et, dans un second temps, à connaître les facteurs associés à l’appréciation, par les mères, du délai de cette visite. Méthode : Les données de cette étude ont été recueillies au Québec, entre janvier 2002 et janvier 2003, lors d’une enquête téléphonique effectuée auprès de mères de bébés nés en santé, un mois suivant leur accouchement vaginal sans complication (n=1548). Pour nos analyses, nous avons retranché les mères ayant eu une durée de séjour de plus de 60 heures, une grossesse de moins de 37semaines et un bébé pesant moins de 2500 g à la naissance. Notre échantillon se compose donc de 1351 mères. Résultats : 86,2 % des mères ont reçu une offre de visite à domicile. La majorité (80.2 %) des mères ont reçu la visite dans les trois premiers jours suivant leur retour à la maison, dont près du tiers (28,1 %), dans les 24 premières heures. Comparativement aux mères visitées au deuxième ou troisième jour suivant le congé, celles visitées dans les 24 premières heures ont jugé la durée de séjour hospitalier trop courte (p=0,018) et reçu un appel de l’infirmière qui a duré plus longtemps (p=0, 009). De plus, au moment du congé, elles perçoivent leur bébé en moins bonne santé (p=0,029). Elles ont aussi accouché d’un bébé plus petit (p=0,052) qui a tendance à avoir présenté des signes d’ictères pendant le séjour hospitalier (p=0,100). D’autre part, la majorité des mères (86,4 %) disent que le délai de la première visite à domicile est adéquat alors que 11,6 % le jugent trop court et 2,3 % trop long. Pour les mères visitées au premier jour, l’analyse multivariée révèle que certaines caractéristiques et certains besoins sont associés à la perception que le délai de la visite est trop court : une seule visite postnatale, un revenu familial de plus de 40 000 $, la perception que la durée de séjour est trop longue et le fait de ne pas allaiter. Pour les mères qui reçoivent la visite au deuxième et troisième jour, ce sont, seulement, le fait d’avoir été au rendez-vous médical et le fait d’avoir reçu une seule visite qui sont associés à la perception que le délai de la visite est trop court. Pour conclure, au Québec, le programme de suivi postnatal universel semble en mesure d’offrir une visite à domicile dans les délais prescrits à une majorité de mères. Les résultats de cette étude suggèrent que le délai de la première visite à domicile n’est pas optimal pour toutes les mères et permettent d’envisager que certaines mères auraient souhaité recevoir une seconde visite plus tardivement au cours de la période postnatale. D’autres recherches devront être effectuées afin de parfaire nos connaissances relativement au moment idéal pour réaliser les interventions postnatales.Mots clefs : Satisfaction, appréciation des utilisateurs, qualité des soins, programme universel, suivi postnatal, congé précoce en obstétrique, visite à domicile, délai de la visite, provision des services.

Att få ett jobb har betytt mycket : En kvalitativ brukarutvärdering av Krami

In this qualitative user evaluation, users from the Krami-programme in Borlänge have been interviewed on what they consider to have been positive changes in their social situation after participating in the programme. The respondents also point out factors in the programme that have supported these changes. The method consisted of interviews with five men who have undergone the Krami programme. The results show that Krami primarily helps the users to get an employment which results in the repeal of criminogenic factors. This plays an important part in the positive change of the users self-esteem, which tends to lead towards ”unlabeling” of the individual as “criminal”. The study also indicates other factors in the program that the users found helpful in their process of change highlighting the interaction between authorities, the approach of the Krami- staff and also the common rules that the participants has to agree upon. The users also poins out the importance of ones own motivation for changes to take place. 

A fuzzy rule-based decision support system for Duodopa treatment in Parkinson

A decision support system (DSS) was implemented based on a fuzzy logic inference system (FIS) to provide assistance in dose alteration of Duodopa infusion in patients with advanced Parkinson’s disease, using data from motor state assessments and dosage. Three-tier architecture with an object oriented approach was used. The DSS has a web enabled graphical user interface that presents alerts indicating non optimal dosage and states, new recommendations, namely typical advice with typical dose and statistical measurements. One data set was used for design and tuning of the FIS and another data set was used for evaluating performance compared with actual given dose. Overall goodness-of-fit for the new patients (design data) was 0.65 and for the ongoing patients (evaluation data) 0.98. User evaluation is now ongoing. The system could work as an assistant to clinical staff for Duodopa treatment in advanced Parkinson’s disease.

A web application for follow-up of results from a mobile device test battery for parkinson’s disease patients

Background: A test battery consisting of self-assessments and motor tests (tapping and spiral drawing) was developed for a hand computer with touch screen in a telemedicine setting. Objectives: To develop and evaluate a web-based system that delivers decision support information to the treating clinical staff for assessing PD symptoms in their patients based on the test battery data. Methods: The test battery is currently being used in a clinical trial (DAPHNE, EudraCT No. 2005-002654-21) by sixty five patients with advanced Parkinson’s disease (PD) on 9991 test occasions (four tests per day during in all 362 week-long test periods) at nine clinics around Sweden. Test results are sent continuously from the hand unit over a mobile net to a central computer and processed with statistical methods. They are summarized into scores for different dimensions of the symptom state and an ‘overall test score’ reflecting the overall condition of the patient during a test period. The information in the web application is organized and presented graphically in a way that the general overview of the patient performance per test period is emphasized. Focus is on the overall test score, symptom dimensions and daily summaries. In a recent preliminary user evaluation, the web application was demonstrated to the fifteen study nurses who had used the test battery in the clinical trial. At least one patient per clinic was shown. Results: In general, the responses from nurses were positive. They claimed that the test results shown in the system were consistent with their own clinical observations. They could follow complications, changes and trends within their patients. Discussion: In conclusion, the system is able to summarise the various time series of motor test results and self-assessments during test periods and present them in a useful manner. Its main contribution is a novel and reliable way to capture and easily access symptom information from patients’ home environment. The convenient access to current symptom profile as well as symptom history provides a basis for individualized evaluation and adjustment of treatments.

Flyktingguide: En väg till integration? : Brukarutvärdering av projektet Flyktingguide i Falun

In this qualitative user evaluation participants in the project refugee guide, Falun, was interviewed about their experience of the project and how the project contributed to their integration into society and establishment on the labor market. The collection of empirical data has been made by six interviews. The results of the evaluation indicate that participants feel that they through the project had the opportunity to meet new people, practice the language and learn about each other's culture. The results also show that participants are satisfied with the project. Some participants expressed a desire to meet with several people at once while others don´t have that need. Furthermore, the interviews shows that establishment on labor market occurred through job placement and not through participation in the project. Integration proved to be a difficult concept to define, but the interviews show that the participants have a perception that it is about learning the Swedish language and culture.

Multidimensional Projections for Visual Analysis of Social Networks

Visual analysis of social networks is usually based on graph drawing algorithms and tools. However, social networks are a special kind of graph in the sense that interpretation of displayed relationships is heavily dependent on context. Context, in its turn, is given by attributes associated with graph elements, such as individual nodes, edges, and groups of edges, as well as by the nature of the connections between individuals. In most systems, attributes of individuals and communities are not taken into consideration during graph layout, except to derive weights for force-based placement strategies. This paper proposes a set of novel tools for displaying and exploring social networks based on attribute and connectivity mappings. These properties are employed to layout nodes on the plane via multidimensional projection techniques. For the attribute mapping, we show that node proximity in the layout corresponds to similarity in attribute, leading to easiness in locating similar groups of nodes. The projection based on connectivity yields an initial placement that forgoes force-based or graph analysis algorithm, reaching a meaningful layout in one pass. When a force algorithm is then applied to this initial mapping, the final layout presents better properties than conventional force-based approaches. Numerical evaluations show a number of advantages of pre-mapping points via projections. User evaluation demonstrates that these tools promote ease of manipulation as well as fast identification of concepts and associations which cannot be easily expressed by conventional graph visualization alone. In order to allow better space usage for complex networks, a graph mapping on the surface of a sphere is also implemented.

Aplicación móvil iOS de información a estudiantes universitarios

Reconociendo que podía ser útil para los alumnos una aplicación móvil para que pudieran acceder a información relacionada con sus estudios de forma rápida y sobre la marcha, se decide realizar una prueba de concepto cuyo resultado satisfactorio conduce a dar el siguiente paso en el desarrollo de la aplicación móvil. En este contexto es donde se enmarca el proyecto “Desarrollo de una aplicación móvil iOS de información a estudiantes universitarios” que tiene como finalidad aprovechar las ventajas que nos brindan las nuevas tecnologías. En el prototipo de aplicación móvil fueron encontrados problemas de mantenibilidad y la versión del sistema operativo había quedado obsoleta. Por lo tanto el primer paso fue refactorizar todos los paquetes del proyecto, después de esto fue necesario crear un estándar de codificación y una documentación del proyecto. El segundo paso fue adaptar el proyecto a la última versión del sistema operativo, iOS 7, siguiendo la guía de transición de la interfaz de usuario de Apple. Además de todo esto, había nuevas funcionalidades que incluir al prototipo, estas nuevas funcionalidades han sido probadas en una evaluación con usuarios para obtener comentarios y sugerencias de los alumnos universitarios para mejorar la aplicación en la medida de lo posible. ----ABSTRACT----Recognizing that could be useful a mobile app for students to enable them to access information related to their studies quickly and on the go, it was decided to perform a proof of concept whose satisfactory results leads to take the next step in the development of the mobile app. In this context is where the project “Desarrollo de una aplicación móvil iOS de información a estudiantes universitarios” takes part whose aim is to exploit the advantages offered by the new technologies. Maintenance problems were found in the mobile app prototype and the operating system version was outdated. So the first step was to refactor all the Project packages, after that it was necessary to create a coding standard and a Project documentation. The second step was to adapt the project to the latest versión of the operating system, iOS7, following the Apple UI transition guide. In addition to all of this, there were new features to include to the prototype, these new features have been tested in an user evaluation to obtain feedback and suggestions from college students to improve the app as far as possible.

Knowledge-based systems and engineering software

The work described was carried out as part of a collaborative Alvey software engineering project (project number SE057). The project collaborators were the Inter-Disciplinary Higher Degrees Scheme of the University of Aston in Birmingham, BIS Applied Systems Ltd. (BIS) and the British Steel Corporation. The aim of the project was to investigate the potential application of knowledge-based systems (KBSs) to the design of commercial data processing (DP) systems. The work was primarily concerned with BIS's Structured Systems Design (SSD) methodology for DP systems development and how users of this methodology could be supported using KBS tools. The problems encountered by users of SSD are discussed and potential forms of computer-based support for inexpert designers are identified. The architecture for a support environment for SSD is proposed based on the integration of KBS and non-KBS tools for individual design tasks within SSD - The Intellipse system. The Intellipse system has two modes of operation - Advisor and Designer. The design, implementation and user-evaluation of Advisor are discussed. The results of a Designer feasibility study, the aim of which was to analyse major design tasks in SSD to assess their suitability for KBS support, are reported. The potential role of KBS tools in the domain of database design is discussed. The project involved extensive knowledge engineering sessions with expert DP systems designers. Some practical lessons in relation to KBS development are derived from this experience. The nature of the expertise possessed by expert designers is discussed. The need for operational KBSs to be built to the same standards as other commercial and industrial software is identified. A comparison between current KBS and conventional DP systems development is made. On the basis of this analysis, a structured development method for KBSs in proposed - the POLITE model. Some initial results of applying this method to KBS development are discussed. Several areas for further research and development are identified.

The mosaic test:measuring the effectiveness of colour-based image retrieval

A variety of content-based image retrieval systems exist which enable users to perform image retrieval based on colour content - i.e., colour-based image retrieval. For the production of media for use in television and film, colour-based image retrieval is useful for retrieving specifically coloured animations, graphics or videos from large databases (by comparing user queries to the colour content of extracted key frames). It is also useful to graphic artists creating realistic computer-generated imagery (CGI). Unfortunately, current methods for evaluating colour-based image retrieval systems have 2 major drawbacks. Firstly, the relevance of images retrieved during the task cannot be measured reliably. Secondly, existing methods do not account for the creative design activity known as reflection-in-action. Consequently, the development and application of novel and potentially more effective colour-based image retrieval approaches, better supporting the large number of users creating media for use in television and film productions, is not possible as their efficacy cannot be reliably measured and compared to existing technologies. As a solution to the problem, this paper introduces the Mosaic Test. The Mosaic Test is a user-based evaluation approach in which participants complete an image mosaic of a predetermined target image, using the colour-based image retrieval system that is being evaluated. In this paper, we introduce the Mosaic Test and report on a user evaluation. The findings of the study reveal that the Mosaic Test overcomes the 2 major drawbacks associated with existing evaluation methods and does not require expert participants. © 2012 Springer Science+Business Media, LLC.