A Satisfaction-based Model for Affect Recognition from Conversational Features in Spoken Dialog Systems

Detecting user affect automatically during real-time conversation is the main challenge towards our greater aim of infusing social intelligence into a natural-language mixed-initiative High-Fidelity (Hi-Fi) audio control spoken dialog agent. In recent years, studies on affect detection from voice have moved on to using realistic, non-acted data, which is subtler. However, it is more challenging to perceive subtler emotions and this is demonstrated in tasks such as labelling and machine prediction. This paper attempts to address part of this challenge by considering the role of user satisfaction ratings and also conversational/dialog features in discriminating contentment and frustration, two types of emotions that are known to be prevalent within spoken human-computer interaction. However, given the laboratory constraints, users might be positively biased when rating the system, indirectly making the reliability of the satisfaction data questionable. Machine learning experiments were conducted on two datasets, users and annotators, which were then compared in order to assess the reliability of these datasets. Our results indicated that standard classifiers were significantly more successful in discriminating the abovementioned emotions and their intensities (reflected by user satisfaction ratings) from annotator data than from user data. These results corroborated that: first, satisfaction data could be used directly as an alternative target variable to model affect, and that they could be predicted exclusively by dialog features. Second, these were only true when trying to predict the abovementioned emotions using annotator?s data, suggesting that user bias does exist in a laboratory-led evaluation.

User-centric need-driven affect modeling for spoken conversational agents: design and evaluation

It is easy to get frustrated at spoken conversational agents (SCAs), perhaps because they seem to be callous. By and large, the quality of human-computer interaction is affected due to the inability of the SCAs to recognise and adapt to user emotional state. Now with the mass appeal of artificially-mediated communication, there has been an increasing need for SCAs to be socially and emotionally intelligent, that is, to infer and adapt to their human interlocutors’ emotions on the fly, in order to ascertain an affective, empathetic and naturalistic interaction. An enhanced quality of interaction would reduce users’ frustrations and consequently increase their satisfactions. These reasons have motivated the development of SCAs towards including socio-emotional elements, turning them into affective and socially-sensitive interfaces. One barrier to the creation of such interfaces has been the lack of methods for modelling emotions in a task-independent environment. Most emotion models for spoken dialog systems are task-dependent and thus cannot be used “as-is” in different applications. This Thesis focuses on improving this, in which it concerns computational modeling of emotion, personality and their interrelationship for task-independent autonomous SCAs. The generation of emotion is driven by needs, inspired by human’s motivational systems. The work in this Thesis is organised in three stages, each one with its own contribution. The first stage involved defining, integrating and quantifying the psychological-based motivational and emotional models sourced from. Later these were transformed into a computational model by implementing them into software entities. The computational model was then incorporated and put to test with an existing SCA host, a HiFi-control agent. The second stage concerned automatic prediction of affect, which has been the main challenge towards the greater aim of infusing social intelligence into the HiFi agent. In recent years, studies on affect detection from voice have moved on to using realistic, non-acted data, which is subtler. However, it is more challenging to perceive subtler emotions and this is demonstrated in tasks such as labelling and machine prediction. In this stage, we attempted to address part of this challenge by considering the roles of user satisfaction ratings and conversational/dialog features as the respective target and predictors in discriminating contentment and frustration, two types of emotions that are known to be prevalent within spoken human-computer interaction. The final stage concerned the evaluation of the emotional model through the HiFi agent. A series of user studies with 70 subjects were conducted in a real-time environment, each in a different phase and with its own conditions. All the studies involved the comparisons between the baseline non-modified and the modified agent. The findings have gone some way towards enhancing our understanding of the utility of emotion in spoken dialog systems in several ways; first, an SCA should not express its emotions blindly, albeit positive. Rather, it should adapt its emotions to user states. Second, low performance in an SCA may be compensated by the exploitation of emotion. Third, the expression of emotion through the exploitation of prosody could better improve users’ perceptions of an SCA compared to exploiting emotions through just lexical contents. Taken together, these findings not only support the success of the emotional model, but also provide substantial evidences with respect to the benefits of adding emotion in an SCA, especially in mitigating users’ frustrations and ultimately improving their satisfactions. Resumen Es relativamente fácil experimentar cierta frustración al interaccionar con agentes conversacionales (Spoken Conversational Agents, SCA), a menudo porque parecen ser un poco insensibles. En general, la calidad de la interacción persona-agente se ve en cierto modo afectada por la incapacidad de los SCAs para identificar y adaptarse al estado emocional de sus usuarios. Actualmente, y debido al creciente atractivo e interés de dichos agentes, surge la necesidad de hacer de los SCAs unos seres cada vez más sociales y emocionalmente inteligentes, es decir, con capacidad para inferir y adaptarse a las emociones de sus interlocutores humanos sobre la marcha, de modo que la interacción resulte más afectiva, empática y, en definitiva, natural. Una interacción mejorada en este sentido permitiría reducir la posible frustración de los usuarios y, en consecuencia, mejorar el nivel de satisfacción alcanzado por los mismos. Estos argumentos justifican y motivan el desarrollo de nuevos SCAs con capacidades socio-emocionales, dotados de interfaces afectivas y socialmente sensibles. Una de las barreras para la creación de tales interfaces ha sido la falta de métodos de modelado de emociones en entornos independientes de tarea. La mayoría de los modelos emocionales empleados por los sistemas de diálogo hablado actuales son dependientes de tarea y, por tanto, no pueden utilizarse "tal cual" en diferentes dominios o aplicaciones. Esta tesis se centra precisamente en la mejora de este aspecto, la definición de modelos computacionales de las emociones, la personalidad y su interrelación para SCAs autónomos e independientes de tarea. Inspirada en los sistemas motivacionales humanos en el ámbito de la psicología, la tesis propone un modelo de generación/producción de la emoción basado en necesidades. El trabajo realizado en la presente tesis está organizado en tres etapas diferenciadas, cada una con su propia contribución. La primera etapa incluyó la definición, integración y cuantificación de los modelos motivacionales de partida y de los modelos emocionales derivados a partir de éstos. Posteriormente, dichos modelos emocionales fueron plasmados en un modelo computacional mediante su implementación software. Este modelo computacional fue incorporado y probado en un SCA anfitrión ya existente, un agente con capacidad para controlar un equipo HiFi, de alta fidelidad. La segunda etapa se orientó hacia el reconocimiento automático de la emoción, aspecto que ha constituido el principal desafío en relación al objetivo mayor de infundir inteligencia social en el agente HiFi. En los últimos años, los estudios sobre reconocimiento de emociones a partir de la voz han pasado de emplear datos actuados a usar datos reales en los que la presencia u observación de emociones se produce de una manera mucho más sutil. El reconocimiento de emociones bajo estas condiciones resulta mucho más complicado y esta dificultad se pone de manifiesto en tareas tales como el etiquetado y el aprendizaje automático. En esta etapa, se abordó el problema del reconocimiento de las emociones del usuario a partir de características o métricas derivadas del propio diálogo usuario-agente. Gracias a dichas métricas, empleadas como predictores o indicadores del grado o nivel de satisfacción alcanzado por el usuario, fue posible discriminar entre satisfacción y frustración, las dos emociones prevalentes durante la interacción usuario-agente. La etapa final corresponde fundamentalmente a la evaluación del modelo emocional por medio del agente Hifi. Con ese propósito se llevó a cabo una serie de estudios con usuarios reales, 70 sujetos, interaccionando con diferentes versiones del agente Hifi en tiempo real, cada uno en una fase diferente y con sus propias características o capacidades emocionales. En particular, todos los estudios realizados han profundizado en la comparación entre una versión de referencia del agente no dotada de ningún comportamiento o característica emocional, y una versión del agente modificada convenientemente con el modelo emocional propuesto. Los resultados obtenidos nos han permitido comprender y valorar mejor la utilidad de las emociones en los sistemas de diálogo hablado. Dicha utilidad depende de varios aspectos. En primer lugar, un SCA no debe expresar sus emociones a ciegas o arbitrariamente, incluso aunque éstas sean positivas. Más bien, debe adaptar sus emociones a los diferentes estados de los usuarios. En segundo lugar, un funcionamiento relativamente pobre por parte de un SCA podría compensarse, en cierto modo, dotando al SCA de comportamiento y capacidades emocionales. En tercer lugar, aprovechar la prosodia como vehículo para expresar las emociones, de manera complementaria al empleo de mensajes con un contenido emocional específico tanto desde el punto de vista léxico como semántico, ayuda a mejorar la percepción por parte de los usuarios de un SCA. Tomados en conjunto, los resultados alcanzados no sólo confirman el éxito del modelo emocional, sino xv que constituyen además una evidencia decisiva con respecto a los beneficios de incorporar emociones en un SCA, especialmente en cuanto a reducir el nivel de frustración de los usuarios y, en última instancia, mejorar su satisfacción.

Assessing user bias in affect detection within context-based spoken dialog systems

This paper presents an empirical evidence of user bias within a laboratory-oriented evaluation of a Spoken Dialog System. Specifically, we addressed user bias in their satisfaction judgements. We question the reliability of this data for modeling user emotion, focusing on contentment and frustration in a spoken dialog system. This bias is detected through machine learning experiments that were conducted on two datasets, users and annotators, which were then compared in order to assess the reliability of these datasets. The target used was the satisfaction rating and the predictors were conversational/dialog features. Our results indicated that standard classifiers were significantly more successful in discriminating frustration and contentment and the intensities of these emotions (reflected by user satisfaction ratings) from annotator data than from user data. Indirectly, the results showed that conversational features are reliable predictors of the two abovementioned emotions.

I feel you: Towards affect-sensitive domotic Spoken Conversational Agents

We describe the work on infusion of emotion into limitedtask autonomous spoken conversational agents (SCAs) situated in the domestic environment, using a Need-inspired task-independentEmotion model (NEMO). In order to demonstrate the generation of a?ect through the use of the model, we describe the work of integrating it with a naturallanguage mixed-initiative HiFi-control SCA. NEMO and the host system communicates externally, removing the need for the Dialog Manager to be modi?ed as done in most existing dialog systems, in order to be adaptive. We also summarize the work on automatic a?ect prediction, namely frustration and contentment from dialog features, a non-conventional source, in the attempt of moving towards a more user-centric approach.

Study about the Problem of Dropping of Subjects by Students During an Academinc Year at the University.

In university studies, it is not unusual for students to drop some of the subjects they have enrolled in for the academic year. They start by not attending lectures, sometimes due to neglect or carelessness, or because they find the subject too difficult, this means that they lose the continuity in the topics that the professor follows. If they try to attend again they discover that they hardly understand anything and become discouraged and so decide to give up attending lectures and study on their own. However some fail to turn up to do their final exams and the failure rate of those who actually do the exams is high. The problem is that this is not only the case with one specific subject, but it is often the same with many subjects. The result is that students arent’s productive enough, wasting time and also prolonging their years of study which entails a great cost for families. Degree courses structured to be conducted and completed in three academic courses, it may in fact take up to an average of six or more academic courses. In this paper, we have studied this problem, which apart from the waste of money and time, produces frustration in the student, who finds that he has not been able to achieve what he had proposed at the beginning of the course. It is quite common, to find students who do not even pass nor 50% of the subjects they had enrolled in for the academic year. If this happens repeatedly to a student, it can be the point when he considers dropping out altogether. This is also a concern for the universities, especially in the early courses. In our experience as professors, we have found that students, who attend lectures regularly and follow the explanations, approach the final exams with confidence and rarely fail the subject. In this proposal we present some techniques and methods carried out to solve in possible, the problem of lack of attendance to lectures. This involves "rewarding students for their assistance and participation in lectures". Rewarding assistance with a "prize" that counts for the final mark on the subject and involving more participation in the development of lectures. We believe that we have to teach students to use the lectures as part of their learning in a non-passive way. We consider the professor's work as fundamental in terms of how to convey the usefulness of these topics explained and the applications that they will have for their professional life in the future. In this way the student see for himself the use and importance of what he is learning. When his participation is required, he will feel more involved and confident participating in the educational system. Finally we present statistical results of studies carried out on different degrees and on different subjects over two consecutive years. In the first year we assessed only the final exams without considering the students attendance, or participation. In the second year, we have applied the techniques and methods proposed here. In addition we have compared the two ways of assessing subjects.

I feel you: the design and evaluation of a domotic affect-sensitive spoken conversational agent

We describe the work on infusion of emotion into a limited-task autonomous spoken conversational agent situated in the domestic environment, using a need-inspired task-independent emotion model (NEMO). In order to demonstrate the generation of affect through the use of the model, we describe the work of integrating it with a natural-language mixed-initiative HiFi-control spoken conversational agent (SCA). NEMO and the host system communicate externally, removing the need for the Dialog Manager to be modified, as is done in most existing dialog systems, in order to be adaptive. The first part of the paper concerns the integration between NEMO and the host agent. The second part summarizes the work on automatic affect prediction, namely, frustration and contentment, from dialog features, a non-conventional source, in the attempt of moving towards a more user-centric approach. The final part reports the evaluation results obtained from a user study, in which both versions of the agent (non-adaptive and emotionally-adaptive) were compared. The results provide substantial evidences with respect to the benefits of adding emotion in a spoken conversational agent, especially in mitigating users' frustrations and, ultimately, improving their satisfaction.

The educational learning lifecycle and the cloud computing

Currently, student dropout rates are a matter of concern among universities. Many research studies, aimed at discovering the causes, have been carried out. However, few solutions, that could serve all students and related problems, have been proposed so far. One such problem is caused by the lack of the "knowledge chain educational links" that occurs when students move onto higher studies without mastering their basic studies. Most regulated studies imparted at universities are designed so that some basic subjects serve as support for other, more complicated, subjects, thus forming a complicated knowledge network. When a link in this chain fails, student frustration occurs as it prevents him from fully understanding the following educational links. In this proposal we try to mitigate these disasters that stem, for the most part, the student?s frustration beyond his college stay. On one hand, we make a dissertation on the student?s learning process, which we divide into a series of phases that amount to what we call the "learning lifecycle." Also, we analyze at what stage the action by the stakeholders involved in this scenario: teachers and students; is most important. On the other hand, we consider that Information and Communication Technologies ICT, such as Cloud Computing, can help develop new ways, allowing for the teaching of higher education, while easing and facilitating the student?s learning process. But, methods and processes need to be defined as to direct the use of such technologies; in the teaching process in general, and within higher education in particular; in order to achieve optimum results. Our methodology integrates, as another actor, the ICT into the "Learning Lifecycle". We stimulate students to stop being merely spectators of their own education, and encourage them to take an active part in their training process. To do this, we offer a set of useful tools to determine not only academic failure causes, (for self assessment), but also to remedy these failures (with corrective actions); "discovered the causes it is easier to determine solutions?. We believe this study will be useful for both students and teachers. Students learn from their own experience and improve their learning process, while obtaining all of the "knowledge chain educational links? required in their studies. We stand by the motto "Studying to learn instead of studying to pass". Teachers will also be benefited by detecting where and how to strengthen their teaching proposals. All of this will also result in decreasing dropout rates.

Usability study for the design of a communicator for ICU patients with communication difficulties

Most of the patients that reside in the intensive care unit experience fear, frustration and high levels of anxiety as they are not able to communicate properly. In this sense, the use of communication tools can be helpful to reduce the frustration levels and also, to improve the efficiency and the speed of the communication. The objective of this work, is to design a tool that allows solving the communication problems that patients suffer when they are admitted in the intensive care unit. In order to achieve the objective of this work, a qualitative study that involved interviews with former patients, hospital staff members and family relatives was performed. Afterwards, the design of a prototype was developed to later conduct and analyze usability evaluations with former patients, hospital staff members and patients relatives. The results expose that participants of the usability evaluations were able to perform most of the tasks effectively.

Transiciones irreversibles en redes complejas

En esta tesis se aborda la emergencia de sincronización en sistemas de osciladores acoplados. En particular, nos centraremos en la emergencia de un tipo de transición discontinua entre el estado incoherente y el estado síncrono, llamada transición explosiva. Este fenómeno es análogo al de las transiciones de fase de primer orden asociadas a los cambios de agregación de la materia, cuya importancia abarca diversos campos, desde la sincronización espontánea de redes neuronales al riesgo de desincronización súbita entre los osciladores que componen la red de suministro de potencia eléctrica. Para analizar el problema, se introducen varios métodos de creciente generalidad cuyo efecto es inducir una transición explosiva al imponer una serie condiciones sobre la topología y las frecuencias naturales de cada oscilador. Así mismo, se aborda el estudio de un modelo algo más complejo con características similares para entender en mayor profundidad las características asociadas a este tipo de transiciones, siendo la histéresis una de las más destacadas. Finalmente, se propone un método cuantitativo para describir la importancia de cada nodo en el proceso de sincronización con el objetivo de estudiar y caracterizar el efecto sobre los nodos del sistema de los diversos métodos que inducen una transición explosiva. Este nuevo enfoque permite descubrir un proceso de frustración de la sincronización local en redes de osciladores acoplados, siendo el responsable de la emergencia de la sincronización explosiva. ABSTRACT In this thesis we address the emergence of synchronization in systems of coupled oscillators in complex networks. We focus our attention on a particular kind of discontinuous transitions, named explosive synchronization, where the system changes abruptly from an incoherent state to a synchronous state. This emergent phenomena is analogous to those first order transitions typically associated with changes among the aggregate states of matter, and it is important in many different fields, such as spontaneous synchronization of neurons or spontaneous desynchronization in power grids. To analyze it, we introduce some methods of increasing generality in order to induce such a discontinuous transition by acting over the topology and the natural frequencies in several different ways. Likewise, we address the study of a more complex model in order to acquire deeper knowledge on the properties of this kind of transitions, where a hysteretic behavior is specially relevant. Finally, we propose a new quantitative approach in order to find the importance of each node in the route to synchronization, aiming to provide a characterization of the effects over the network’s units of the different methods able to induce an explosive transition. This approach allows us to show the inner mechanisms behind such explosive behavior in networks of coupled oscillators, being rooted by a frustration of the local synchronization process previous to the emergence of global coherence.