A User Centered Design Approach for Patient Interfaces to a Diabetes IT Platform

Improving patient self-management can have a greater impact than improving any clinical treatment (WHO). We propose here a systematic and comprehensive user centered design approach for delivering a technological platform for diabetes disease management. The system was developed under the METABO research project framework, involving patients from 3 different clinical centers in Parma, Modena and Madrid.

User experience design

Inmersos en un mundo digital en el que participan cada vez más personas con diferentes niveles de experiencia en tecnología, ha aparecido la necesidad de diseñar las aplicaciones pensando en el usuario. El principal problema que un desarrollador se encuentra hoy en día es la falta de tiempo o agilidad para conceder al usuario una experiencia a medida. Muchas empresas han conseguido destacar en un mundo donde la competencia es muy grande gracias al aporte de valor añadido de una buena experiencia de usuario. Empezando por el análisis psicológico y las reacciones que tenemos cuando estamos interactuando con un sistema digital se intenta conseguir un método para diseñar la experiencia de usuario de un modo satisfactorio para ambas partes. Con el objetivo de probar el método diseñado, en el cual se incorporan muchas técnicas de otros métodos, se diseña parte de la experiencia de usuario de “ESEM 2014”, una de las conferencias informáticas más importantes del norte de Italia. Los resultados obtenidos en este trabajo son muy favorables, ya que mejoran substancialmente el resultado final y permiten al desarrollador seguir mejorando la aplicación hasta el momento y objetivo deseado. Resumen Español---ABSTRACT---In a world where more people with different level of expertise is engaging with technology, designing for the user has become one of the main concerns for the developers. Lack of time or agility are the main problems from the developers’ point of view in order to provide a good user experience. One of most important differentiators for companies is the provided experience as the technology and knowledge is highly available for everyone. The main goal of this project is to compose a method to include experience design in current agile development where short iterations is the main characteristic. In order to test the designed method, were many parts of other techniques have been included, part of the “ESEM 2014” conference has been designed. The results achieved with this guidelines has been very positive from both sides of the development, users and developers. Having short iterations allows improvement until the desired experience is achieved.

Automated end user-centred adaptation of web components through automated description logic-based reasoning

Context: This paper addresses one of the major end-user development (EUD) challenges, namely, how to pack today?s EUD support tools with composable elements. This would give end users better access to more components which they can use to build a solution tailored to their own needs. The success of later end-user software engineering (EUSE) activities largely depends on how many components each tool has and how adaptable components are to multiple problem domains. Objective: A system for automatically adapting heterogeneous components to a common development environment would offer a sizeable saving of time and resources within the EUD support tool construction process. This paper presents an automated adaptation system for transforming EUD components to a standard format. Method: This system is based on the use of description logic. Based on a generic UML2 data model, this description logic is able to check whether an end-user component can be transformed to this modeling language through subsumption or as an instance of the UML2 model. Besides it automatically finds a consistent, non-ambiguous and finite set of XSLT mappings to automatically prepare data in order to leverage the component as part of a tool that conforms to the target UML2 component model. Results: The proposed system has been successfully applied to components from four prominent EUD tools. These components were automatically converted to a standard format. In order to validate the proposed system, rich internet applications (RIA) used as an operational support system for operators at a large services company were developed using automatically adapted standard format components. These RIAs would be impossible to develop using each EUD tool separately. Conclusion: The positive results of applying our system for automatically adapting components from current tool catalogues are indicative of the system?s effectiveness. Use of this system could foster the growth of web EUD component catalogues, leveraging a vast ecosystem of user-centred SaaS to further current EUSE trends.

Application of user-centered design in the development of an atomated/assisted testing system for self-service devices

Automated Teller Machines (ATMs) are sensitive self-service systems that require important investments in security and testing. ATM certifications are testing processes for machines that integrate software components from different vendors and are performed before their deployment for public use. This project was originated from the need of optimization of the certification process in an ATM manufacturing company. The process identifies compatibility problems between software components through testing. It is composed by a huge number of manual user tasks that makes the process very expensive and error-prone. Moreover, it is not possible to fully automate the process as it requires human intervention for manipulating ATM peripherals. This project presented important challenges for the development team. First, this is a critical process, as all the ATM operations rely on the software under test. Second, the context of use of ATMs applications is vastly different from ordinary software. Third, ATMs’ useful lifetime is beyond 15 years and both new and old models need to be supported. Fourth, the know-how for efficient testing depends on each specialist and it is not explicitly documented. Fifth, the huge number of tests and their importance implies the need for user efficiency and accuracy. All these factors led us conclude that besides the technical challenges, the usability of the intended software solution was critical for the project success. This business context is the motivation of this Master Thesis project. Our proposal focused in the development process applied. By combining user-centered design (UCD) with agile development we ensured both the high priority of usability and the early mitigation of software development risks caused by all the technology constraints. We performed 23 development iterations and finally we were able to provide a working solution on time according to users’ expectations. The evaluation of the project was carried out through usability tests, where 4 real users participated in different tests in the real context of use. The results were positive, according to different metrics: error rate, efficiency, effectiveness, and user satisfaction. We discuss the problems found, the benefits and the lessons learned in the process. Finally, we measured the expected project benefits by comparing the effort required by the current and the new process (once the new software tool is adopted). The savings corresponded to 40% less effort (man-hours) per certification. Future work includes additional evaluation of product usability in a real scenario (with customers) and the measuring of benefits in terms of quality improvement.

Usability Planner: A Tool to Support the Process of Selecting Usability Methods

There is increasing pressure on developers to produce usable systems, which requires the use of appropriate methods to support user centred design during development. There is currently no consistent advice on which methods are appropriate in which circumstances, so the selection of methods relies on individual experience and expertise. Considerable effort is required to collate information from various sources and to understand the applicability of each method in a particular situation. Usability Planner is a tool aimed to support the selection of the most appropriate methods depending on project and organizational constraints. Many of the rules employed are derived from ISO standards, complemented with rules from the authors’ experience.

Usability Planner: Herramienta para la planificación de técnicas de usabilidad en el ciclo de vida del software

A pesar del creciente interés de las organizaciones por la aplicación de métodos y técnicas de usabilidad en el proceso de desarrollo de software, seleccionar y poner en práctica aquellas técnicas que más se adecuan a las características de un determinado proyecto es una tarea compleja y con un soporte bajo. La diversidad de métodos de usabilidad, su desconocimiento en el mundo profesional y la falta de unas pautas claras para su aplicación en un determinado proyecto están causando que la introducción del Diseño Centrado en el Usuario (UCD, del inglés User Centred Design o DCU en español) en las compañías sea una tarea costosa y compleja. No obstante, las empresas que apuestan por la usabilidad de sus productos se están posicionando por encima del resto al crear sistemas que los usuarios valoran notablemente y por ver incrementadas sus ventas. Jefes de proyecto y consultores, que ven atractivo el desarrollo de sistemas usables, se hallan en un escenario en el que no encuentran un mecanismo sencillo e inmediato para seleccionar y aplicar las técnicas de usabilidad en sus proyectos de forma ágil y estructurada. El criterio para la selección de técnicas y métodos que manejan se basa únicamente en su experiencia profesional y en la bibliografía de numerosos autores. Usability Planner es una aplicación web dirigida a profesionales, estudiantes e investigadores involucrados en el desarrollo de software para ayudar a cambiar este escenario. Su objetivo es dar soporte a la selección de técnicas y métodos de usabilidad en el proceso de desarrollo de software minimizando riesgos y maximizando beneficios.

Methodology for developing an advanced communications system for the Deaf in a new domain

A methodology for developing an advanced communications system for the Deaf in a new domain is presented in this paper. This methodology is a user-centred design approach consisting of four main steps: requirement analysis, parallel corpus generation, technology adaptation to the new domain, and finally, system evaluation. During the requirement analysis, both the user and technical requirements are evaluated and defined. For generating the parallel corpus, it is necessary to collect Spanish sentences in the new domain and translate them into LSE (Lengua de Signos Española: Spanish Sign Language). LSE is represented by glosses and using video recordings. This corpus is used for training the two main modules of the advanced communications system to the new domain: the spoken Spanish into the LSE translation module and the Spanish generation from the LSE module. The main aspects to be generated are the vocabularies for both languages (Spanish words and signs), and the knowledge for translating in both directions. Finally, the field evaluation is carried out with deaf people using the advanced communications system to interact with hearing people in several scenarios. In this evaluation, the paper proposes several objective and subjective measurements for evaluating the performance. In this paper, the new considered domain is about dialogues in a hotel reception. Using this methodology, the system was developed in several months, obtaining very good performance: good translation rates (10% Sign Error Rate) with small processing times, allowing face-to-face dialogues.

An ICT-based closed-loop model to improve care for heart failure patients

Durante las últimas décadas se ha producido un fenómeno global de envejecimiento en la población. Esta tendencia se puede observar prácticamente en todos los países del mundo y se debe principalmente a los avances en la medicina, y a los descensos en las tasas de fertilidad y mortalidad. El envejecimiento de la población tiene un gran impacto en la salud de los ciudadanos, y a menudo es la causa de aparición de enfermedades crónicas. Este tipo de enfermedades supone una amenaza y una carga importantes para la sociedad, especialmente en aspectos como la mortalidad o los gastos en los sistemas sanitarios. Entre las enfermedades cardiovasculares, la insuficiencia cardíaca es probablemente la condición con mayor prevalencia y afecta a 23-26 millones de personas en todo el mundo. Normalmente, la insuficiencia cardíaca presenta un mal pronóstico y una tasa de supervivencia bajas, en algunos casos peores que algún tipo de cáncer. Además, suele ser la causa de hospitalizaciones frecuentes y es una de las enfermedades más costosas para los sistemas sanitarios. La tendencia al envejecimiento de la población y la creciente incidencia de las enfermedades crónicas están llevando a una situación en la que los sistemas de salud no son capaces de hacer frente a la demanda de la sociedad. Los servicios de salud existentes tendrán que adaptarse para ser efectivos y sostenibles en el futuro. Es necesario identificar nuevos paradigmas de cuidado de pacientes, así como mecanismos para la provisión de servicios que ayuden a transformar estos sistemas sanitarios. En este contexto, esta tesis se plantea la búsqueda de soluciones, basadas en las Tecnologías de la Información y la Comunicación (TIC), que contribuyan a realizar la transformación en los sistemas sanitarios. En concreto, la tesis se centra en abordar los problemas de una de las enfermedades con mayor impacto en estos sistemas: la insuficiencia cardíaca. Las siguientes hipótesis constituyen la base para la realización de este trabajo de investigación: 1. Es posible definir un modelo basado en el paradigma de lazo cerrado y herramientas TIC que formalice el diseño de mejores servicios para pacientes con insuficiencia cardíaca. 2. El modelo de lazo cerrado definido se puede utilizar para definir un servicio real que ayude a gestionar la insuficiencia cardíaca crónica. 3. La introducción, la adopción y el uso de un servicio basado en el modelo definido se traducirá en mejoras en el estado de salud de los pacientes que sufren insuficiencia cardíaca. a. La utilización de un sistema basado en el modelo de lazo cerrado definido mejorará la experiencia del usuario de los pacientes. La definición del modelo planteado se ha basado en el estándar ISO / EN 13940- Sistema de conceptos para dar soporte a la continuidad de la asistencia. Comprende un conjunto de conceptos, procesos, flujos de trabajo, y servicios como componentes principales, y representa una formalización de los servicios para los pacientes con insuficiencia cardíaca. Para evaluar el modelo definido se ha definido un servicio real basado en el mismo, además de la implementación de un sistema de apoyo a dicho servicio. El diseño e implementación de dicho sistema se realizó siguiendo la metodología de Diseño Orientado a Objetivos. El objetivo de la evaluación consistía en investigar el efecto que tiene un servicio basado en el modelo de lazo cerrado sobre el estado de salud de los pacientes con insuficiencia cardíaca. La evaluación se realizó en el marco de un estudio clínico observacional. El análisis de los resultados ha comprendido métodos de análisis cuantitativos y cualitativos. El análisis cuantitativo se ha centrado en determinar el estado de salud de los pacientes en base a datos objetivos (obtenidos en pruebas de laboratorio o exámenes médicos). Para realizar este análisis se definieron dos índices específicos: el índice de estabilidad y el índice de la evolución del estado de salud. El análisis cualitativo ha evaluado la autopercepción del estado de salud de los pacientes en términos de calidad de vida, auto-cuidado, el conocimiento, la ansiedad y la depresión, así como niveles de conocimiento. Se ha basado en los datos recogidos mediante varios cuestionarios o instrumentos estándar (i.e. EQ-5D, la Escala de Ansiedad y Depresión (HADS), el Cuestionario de Cardiomiopatía de Kansas City (KCCQ), la Escala Holandesa de Conocimiento de Insuficiencia Cardíaca (DHFKS), y la Escala Europea de Autocuidado en Insuficiencia Cardíaca (EHFScBS), así como cuestionarios dedicados no estandarizados de experiencia de usuario. Los resultados obtenidos en ambos análisis, cuantitativo y cualitativo, se compararon con el fin de evaluar la correlación entre el estado de salud objetivo y subjetivo de los pacientes. Los resultados de la validación demostraron que el modelo propuesto tiene efectos positivos en el cuidado de los pacientes con insuficiencia cardíaca y contribuye a mejorar su estado de salud. Asimismo, ratificaron al modelo como instrumento válido para la definición de servicios mejorados para la gestión de esta enfermedad. ABSTRACT During the last decades we have witnessed a global aging phenomenon in the population. This can be observed in practically every country in the world, and it is mainly caused by the advances in medicine, and the decrease of mortality and fertility rates. Population aging has an important impact on citizens’ health and it is often the cause for chronic diseases, which constitute global burden and threat to the society in terms of mortality and healthcare expenditure. Among chronic diseases, Chronic Heart Failure (CHF) or Heart Failure (HF) is probably the one with highest prevalence, affecting between 23 and 26 million people worldwide. Heart failure is a chronic, long-term and serious condition with very poor prognosis and worse survival rates than some type of cancers. Additionally, it is often the cause of frequent hospitalizations and one of the most expensive conditions for the healthcare systems. The aging trends in the population and the increasing incidence of chronic diseases are leading to a situation where healthcare systems are not able to cope with the society demand. Current healthcare services will have to be adapted and redefined in order to be effective and sustainable in the future. There is a need to find new paradigms for patients’ care, and to identify new mechanisms for services’ provision that help to transform the healthcare systems. In this context, this thesis aims to explore new solutions, based on ICT, that contribute to achieve the needed transformation within the healthcare systems. In particular, it focuses on addressing the problems of one of the diseases with higher impact within these systems: Heart Failure. The following hypotheses represent the basis to the elaboration of this research: 1. It is possible to define a model based on a closed-loop paradigm and ICT tools that formalises the design of enhanced healthcare services for chronic heart failure patients. 2. The described closed-loop model can be exemplified in a real service that supports the management of chronic heart failure disease. 3. The introduction, adoption and use of a service based on the outlined model will result in improvements in the health status of patients suffering heart failure. 4. The user experience of patients when utilizing a system based on the defined closed-loop model will be enhanced. The definition of the closed-loop model for health care support of heart failure patients have been based on the standard ISO/EN 13940 System of concepts to support continuity of care. It includes a set of concept, processes and workflows, and services as main components, and it represent a formalization of services for heart failure patients. In order to be validated, the proposed closed-loop model has been instantiated into a real service and a supporting IT system. The design and implementation of the system followed the user centred design methodology Goal Oriented Design. The validation, that included an observational clinical study, aimed to investigate the effect that a service based on the closed-loop model had on heart failure patients’ health status. The analysis of results comprised quantitative and qualitative analysis methods. The quantitative analysis was focused on determining the health status of patients based on objective data (obtained in lab tests or physical examinations). Two specific indexes where defined and considered in this analysis: the stability index and the health status evolution index. The qualitative analysis assessed the self-perception of patients’ health status in terms of quality of life, self-care, knowledge, anxiety and depression, as well as knowledge levels. It was based on the data gathered through several standard instruments (i.e. EQ-5D, the Hospital Anxiety and Depression Scale, the Kansas City Cardiomyopathy Questionnaire, the Dutch Heart Failure Knowledge Scale, and the European Heart Failure Self-care Behaviour Scale) as well as dedicated non-standardized user experience questionnaires. The results obtained in both analyses, quantitative and qualitative, were compared in order to assess the correlation between the objective and subjective health status of patients. The results of the validation showed that the proposed model contributed to improve the health status of the patients and had a positive effect on the patients’ care. It also proved that the model is a valid instrument for designing enhanced healthcare services for heart failure patients.

Intentions: a confident-based interaction design for smart spaces

The paradigm of ubiquitous computing has become a reference for the design of Smart Spaces. Current trends in Ambient Intelligence are increasingly related to the scope of Internet of Things. This paradigm has the potential to support cost-effective solutions in the fields of telecare, e-health and Ambient Assisted Living. Nevertheless, ubiquitous computing does not provide end users with a role for proactive interactions with the environment. Thus, the deployment of smart health care services at a private space like the home is still unsolved. This PhD dissertation aims to define a person-environment interaction model to foster acceptability and users confidence in private spaces by applying the concept of user-centred security and the human performance model of seven stages of action.

An overview of the ciao system

Ciao is a logic-based, multi-paradigm programming system. One of its most distinguishing features is that it supports a large number of semantic and syntactic language features which can be selectively activated or deactivated for each program module. As a result, a module can be written in, for example, ISO-Prolog plus constraints and higher order, while another can be a puré logic module with a different control rule such as iterative deepening and/or tabling, and perhaps using constructive negation. A powerful and modular extensión mechanism allows user-level design and implementation of such features and sub-languages. Another distinguishing feature of Ciao is its powerful assertion language, which allows expressing many kinds of program properties (ranging from, e.g., moded types to resource consumption), as well as tests and documentation. The compiler is capable of statically ñnding violations of these properties or verifying that programs comply with them, and issuing certiñcates of this compliance. The compiler also performs many types of optimizations, including automatic parallelization. It offers very competitive performance, while retaining the flexibility and interactive development of a dynamic language. We will present a hands-on overview of the system, through small examples which emphasize the novel aspects and the motivations which lie behind Ciao's design and implementation.

Developing front-end Web 2.0 technologies to access services, content and things in the future Internet

The future Internet is expected to be composed of a mesh of interoperable web services accessible from all over the web. This approach has not yet caught on since global user?service interaction is still an open issue. This paper states one vision with regard to next-generation front-end Web 2.0 technology that will enable integrated access to services, contents and things in the future Internet. In this paper, we illustrate how front-ends that wrap traditional services and resources can be tailored to the needs of end users, converting end users into prosumers (creators and consumers of service-based applications). To do this, we propose an architecture that end users without programming skills can use to create front-ends, consult catalogues of resources tailored to their needs, easily integrate and coordinate front-ends and create composite applications to orchestrate services in their back-end. The paper includes a case study illustrating that current user-centred web development tools are at a very early stage of evolution. We provide statistical data on how the proposed architecture improves these tools. This paper is based on research conducted by the Service Front End (SFE) Open Alliance initiative.

Adapting a speech into sign language translation system to a new domain

This paper presents a methodology for adapting an advanced communication system for deaf people in a new domain. This methodology is a user-centered design approach consisting of four main steps: requirement analysis, parallel corpus generation, technology adaptation to the new domain, and finally, system evaluation. In this paper, the new considered domain has been the dialogues in a hotel reception. With this methodology, it was possible to develop the system in a few months, obtaining very good performance: good speech recognition and translation rates (around 90%) with small processing times.

Validación de un marco abierto accesible de tercera generación para la integración nativa de productos de apoyo en tecnologías de la información y la comunicación

Las personas con discapacidad a menudo se encuentran con problemas de acceso a las Tecnologías de la Información y la Comunicación (TIC), debido a diseños y desarrollos que no tienen en cuenta sus diferencias funcionales, y en consecuencia se encuentran en riesgo de exclusión social. Cada vez es más común encontrar productos de apoyo que permitan utilizar diferentes tecnologías (ordenadores, Internet, dispositivos móviles), pero muchos de ellos no se encuentran integrados debido a que funcionan esencialmente modificando la plataforma donde están instalados, siendo soluciones de acceso de segunda generación. Más allá del desarrollo de los productos de apoyo, que sin duda ha evolucionado positivamente en los últimos años, cabe resaltar que existe una falta de herramientas y de aproximación holística que ayuden a los desarrolladores y diseñadores hacer las TIC accesibles. Esta tesis doctoral pretende validar la hipótesis de que una metodología holística de desarrollo de aplicaciones y productos de apoyo TIC, llamada Marco Abierto Accesible, facilita el desarrollo y la integración de modo nativo de la accesibilidad en las aplicaciones y productos apoyo, independientemente de la tecnología utilizada, dando lugar a soluciones de acceso de tercera generación que permitan mejorar la utilización de dichas aplicaciones por parte de personas con discapacidad. Este trabajo se ha desarrollado en el marco del proyecto AEGIS (del inglés, open Accessibility Everywhere: Groundwork, Infrastructure, Standards), que fue parcialmente financiado por la Comisión Europea (CE) bajo el VII Programa Marco y tuvo una duración de cuatro años. La metodología para el diseño, desarrollo y validación seguida en esta tesis es una adaptación de dos metodologías de diseño existentes (el Diseño Centrado en el Usuario y el Diseño Orientado a Metas), la implementación del Marco Abierto Accesible y el uso de diferentes técnicas de validación. Además se ha desarrollado un marco metodológico de entrenamiento para minimizar el efecto que tiene la curva de aprendizaje cuando los usuarios prueban por primera vez las soluciones desarrolladas. En esta tesis se presenta el Marco Abierto Accesible aplicado a las TIC en las tres áreas en las que se desarrolla este trabajo: ordenadores, Internet y dispositivos móviles, partiendo de las necesidades y problemas que tienen los usuarios con discapacidad en el uso de las TIC. Diferentes instanciaciones del Marco Abierto Accesible se definen en las tres áreas TIC mencionadas anteriormente y se describen varios ejemplos de sus implementaciones particulares. Los resultados de las evaluaciones de las implementaciones particulares con usuarios finales y expertos, una vez discutidos y contrastados con las hipótesis, sirven para probar la validez del Marco Abierto Accesible para la integración nativa de productos de apoyo en Tecnologías de la Información y la Comunicación. Finalmente, se presentan las líneas de investigación y trabajo futuro en el área de la accesibilidad en las TIC. ABSTRACT People with disabilities often encounter problems of access to Information and Communications Technology (ICT), due to design and developments that do not take into account their functional differences and therefore put them at risk of social exclusion. It is increasingly common to find assistive products that allow to use different technologies (computers, Internet, mobile devices), but many of them are not well integrated because they work essentially modifying the platform where they are installed, beeing the second-generation access solutions. Beyond the assistive product development, which has certainly evolved positively in recent years, it is notable that there is a lack of tools and holistic approach to help developers and designers make accessible ICT. This doctoral thesis aims to validate the hypothesis that a holistic approach to application development and assistive ICT products, called Open Accessibility Framework, facilitates the development and integration of native accessible applications and assistive products, regardless of the technology used, resulting in third-generation access solutions that improve the use of such applications by people with disabilities. This work was developed under the AEGIS project (open Accessibility Everywhere: Groundwork, Infrastructure, Standards), which was partially funded by the European Commission (EC) under the Seventh Framework Programme and lasted four years. The methodology for the design, development and validation followed in this thesis is an adaptation of two existing design methodologies (User Centered Design and Goal Oriented Design), the implementation of the Open Accessibility Framework and the usage of different validation techniques. A training methodological framework ha also been developed to minimize the effect of the learning curve when users first try the solutions developed. This thesis presents the Open Accessibility Framework applied to ICT in three areas in which this work is developed: computers, Internet and mobile devices, based on the needs and problems of the disabled users in the use of ICT. Different instantiations of the Open Accessibility Framework are defined in the three aforementioned ICT areas and various examples of its particular implementations are described. The results of the evaluations of the particular implementations which have been carried with end users and experts, once discussed and contrasted with the hypotheses, have been used to test the validity of the Open Accessibility Framework for the native integration of assistive products in Information and Communications Technology. Finally, the future research lines and future work in the area of ICT accessibility are presented.

Using the Journalistic Metaphor to Design User Interfaces That Explain Sensor Data

Facilitating general access to data from sensor networks (including traffic, hydrology and other domains) increases their utility. In this paper we argue that the journalistic metaphor can be effectively used to automatically generate multimedia presentations that help non-expert users analyze and understand sensor data. The journalistic layout and style are familiar to most users. Furthermore, the journalistic approach of ordering information from most general to most specific helps users obtain a high-level understanding while providing them the freedom to choose the depth of analysis to which they want to go. We describe the general characteristics and architectural requirements for an interactive intelligent user interface for exploring sensor data that uses the journalistic metaphor. We also describe our experience in developing this interface in real-world domains (e.g., hydrology).

User-centric need-driven affect modeling for spoken conversational agents: design and evaluation

It is easy to get frustrated at spoken conversational agents (SCAs), perhaps because they seem to be callous. By and large, the quality of human-computer interaction is affected due to the inability of the SCAs to recognise and adapt to user emotional state. Now with the mass appeal of artificially-mediated communication, there has been an increasing need for SCAs to be socially and emotionally intelligent, that is, to infer and adapt to their human interlocutors’ emotions on the fly, in order to ascertain an affective, empathetic and naturalistic interaction. An enhanced quality of interaction would reduce users’ frustrations and consequently increase their satisfactions. These reasons have motivated the development of SCAs towards including socio-emotional elements, turning them into affective and socially-sensitive interfaces. One barrier to the creation of such interfaces has been the lack of methods for modelling emotions in a task-independent environment. Most emotion models for spoken dialog systems are task-dependent and thus cannot be used “as-is” in different applications. This Thesis focuses on improving this, in which it concerns computational modeling of emotion, personality and their interrelationship for task-independent autonomous SCAs. The generation of emotion is driven by needs, inspired by human’s motivational systems. The work in this Thesis is organised in three stages, each one with its own contribution. The first stage involved defining, integrating and quantifying the psychological-based motivational and emotional models sourced from. Later these were transformed into a computational model by implementing them into software entities. The computational model was then incorporated and put to test with an existing SCA host, a HiFi-control agent. The second stage concerned automatic prediction of affect, which has been the main challenge towards the greater aim of infusing social intelligence into the HiFi agent. In recent years, studies on affect detection from voice have moved on to using realistic, non-acted data, which is subtler. However, it is more challenging to perceive subtler emotions and this is demonstrated in tasks such as labelling and machine prediction. In this stage, we attempted to address part of this challenge by considering the roles of user satisfaction ratings and conversational/dialog features as the respective target and predictors in discriminating contentment and frustration, two types of emotions that are known to be prevalent within spoken human-computer interaction. The final stage concerned the evaluation of the emotional model through the HiFi agent. A series of user studies with 70 subjects were conducted in a real-time environment, each in a different phase and with its own conditions. All the studies involved the comparisons between the baseline non-modified and the modified agent. The findings have gone some way towards enhancing our understanding of the utility of emotion in spoken dialog systems in several ways; first, an SCA should not express its emotions blindly, albeit positive. Rather, it should adapt its emotions to user states. Second, low performance in an SCA may be compensated by the exploitation of emotion. Third, the expression of emotion through the exploitation of prosody could better improve users’ perceptions of an SCA compared to exploiting emotions through just lexical contents. Taken together, these findings not only support the success of the emotional model, but also provide substantial evidences with respect to the benefits of adding emotion in an SCA, especially in mitigating users’ frustrations and ultimately improving their satisfactions. Resumen Es relativamente fácil experimentar cierta frustración al interaccionar con agentes conversacionales (Spoken Conversational Agents, SCA), a menudo porque parecen ser un poco insensibles. En general, la calidad de la interacción persona-agente se ve en cierto modo afectada por la incapacidad de los SCAs para identificar y adaptarse al estado emocional de sus usuarios. Actualmente, y debido al creciente atractivo e interés de dichos agentes, surge la necesidad de hacer de los SCAs unos seres cada vez más sociales y emocionalmente inteligentes, es decir, con capacidad para inferir y adaptarse a las emociones de sus interlocutores humanos sobre la marcha, de modo que la interacción resulte más afectiva, empática y, en definitiva, natural. Una interacción mejorada en este sentido permitiría reducir la posible frustración de los usuarios y, en consecuencia, mejorar el nivel de satisfacción alcanzado por los mismos. Estos argumentos justifican y motivan el desarrollo de nuevos SCAs con capacidades socio-emocionales, dotados de interfaces afectivas y socialmente sensibles. Una de las barreras para la creación de tales interfaces ha sido la falta de métodos de modelado de emociones en entornos independientes de tarea. La mayoría de los modelos emocionales empleados por los sistemas de diálogo hablado actuales son dependientes de tarea y, por tanto, no pueden utilizarse "tal cual" en diferentes dominios o aplicaciones. Esta tesis se centra precisamente en la mejora de este aspecto, la definición de modelos computacionales de las emociones, la personalidad y su interrelación para SCAs autónomos e independientes de tarea. Inspirada en los sistemas motivacionales humanos en el ámbito de la psicología, la tesis propone un modelo de generación/producción de la emoción basado en necesidades. El trabajo realizado en la presente tesis está organizado en tres etapas diferenciadas, cada una con su propia contribución. La primera etapa incluyó la definición, integración y cuantificación de los modelos motivacionales de partida y de los modelos emocionales derivados a partir de éstos. Posteriormente, dichos modelos emocionales fueron plasmados en un modelo computacional mediante su implementación software. Este modelo computacional fue incorporado y probado en un SCA anfitrión ya existente, un agente con capacidad para controlar un equipo HiFi, de alta fidelidad. La segunda etapa se orientó hacia el reconocimiento automático de la emoción, aspecto que ha constituido el principal desafío en relación al objetivo mayor de infundir inteligencia social en el agente HiFi. En los últimos años, los estudios sobre reconocimiento de emociones a partir de la voz han pasado de emplear datos actuados a usar datos reales en los que la presencia u observación de emociones se produce de una manera mucho más sutil. El reconocimiento de emociones bajo estas condiciones resulta mucho más complicado y esta dificultad se pone de manifiesto en tareas tales como el etiquetado y el aprendizaje automático. En esta etapa, se abordó el problema del reconocimiento de las emociones del usuario a partir de características o métricas derivadas del propio diálogo usuario-agente. Gracias a dichas métricas, empleadas como predictores o indicadores del grado o nivel de satisfacción alcanzado por el usuario, fue posible discriminar entre satisfacción y frustración, las dos emociones prevalentes durante la interacción usuario-agente. La etapa final corresponde fundamentalmente a la evaluación del modelo emocional por medio del agente Hifi. Con ese propósito se llevó a cabo una serie de estudios con usuarios reales, 70 sujetos, interaccionando con diferentes versiones del agente Hifi en tiempo real, cada uno en una fase diferente y con sus propias características o capacidades emocionales. En particular, todos los estudios realizados han profundizado en la comparación entre una versión de referencia del agente no dotada de ningún comportamiento o característica emocional, y una versión del agente modificada convenientemente con el modelo emocional propuesto. Los resultados obtenidos nos han permitido comprender y valorar mejor la utilidad de las emociones en los sistemas de diálogo hablado. Dicha utilidad depende de varios aspectos. En primer lugar, un SCA no debe expresar sus emociones a ciegas o arbitrariamente, incluso aunque éstas sean positivas. Más bien, debe adaptar sus emociones a los diferentes estados de los usuarios. En segundo lugar, un funcionamiento relativamente pobre por parte de un SCA podría compensarse, en cierto modo, dotando al SCA de comportamiento y capacidades emocionales. En tercer lugar, aprovechar la prosodia como vehículo para expresar las emociones, de manera complementaria al empleo de mensajes con un contenido emocional específico tanto desde el punto de vista léxico como semántico, ayuda a mejorar la percepción por parte de los usuarios de un SCA. Tomados en conjunto, los resultados alcanzados no sólo confirman el éxito del modelo emocional, sino xv que constituyen además una evidencia decisiva con respecto a los beneficios de incorporar emociones en un SCA, especialmente en cuanto a reducir el nivel de frustración de los usuarios y, en última instancia, mejorar su satisfacción.