A blueprint for affective computing: A sourcebook

A Blueprint for Affective Computing: A sourcebook and manual is the very first attempt to ground affective computing within the disciplines of psychology, affective neuroscience, and philosophy. This book illustrates the contributions of each of these disciplines to the development of the ever-growing field of affective computing. In addition, it demonstrates practical examples of cross-fertilization between disciplines in order to highlight the need for integration of computer science, engineering and the affective sciences.

Towards a lightweight adaptative multicomponential affective recognition system

As part of the Affective Computing research field, the development of automatic affective recognition systems can enhance human-computer interactions by allowing the creation of interfaces that react to the user's emotional state. To that end, this Master Thesis brings affect recognition to nowadays most used human computer interface, mobile devices, by developing a facial expression recognition system able to perform detection under the difficult conditions of viewing angle and illumination that entails the interaction with a mobile device. Moreover, this Master Thesis proposes to combine emotional features detected from expression with contextual information of the current situation, to infer a complex and extensive emotional state of the user. Thus, a cognitive computational model of emotion is defined that provides a multicomponential affective state of the user through the integration of the detected emotional features into appraisal processes. In order to account for individual differences in the emotional experience, these processes can be adapted to the culture and personality of the user.

A Digital Signal Processing Approach for Affective Sensing of a Computer User through Pupil Diameter Monitoring

Recent research has indicated that the pupil diameter (PD) in humans varies with their affective states. However, this signal has not been fully investigated for affective sensing purposes in human-computer interaction systems. This may be due to the dominant separate effect of the pupillary light reflex (PLR), which shrinks the pupil when light intensity increases. In this dissertation, an adaptive interference canceller (AIC) system using the H∞ time-varying (HITV) adaptive algorithm was developed to minimize the impact of the PLR on the measured pupil diameter signal. The modified pupil diameter (MPD) signal, obtained from the AIC was expected to reflect primarily the pupillary affective responses (PAR) of the subject. Additional manipulations of the AIC output resulted in a processed MPD (PMPD) signal, from which a classification feature, PMPDmean, was extracted. This feature was used to train and test a support vector machine (SVM), for the identification of stress states in the subject from whom the pupil diameter signal was recorded, achieving an accuracy rate of 77.78%. The advantages of affective recognition through the PD signal were verified by comparatively investigating the classification of stress and relaxation states through features derived from the simultaneously recorded galvanic skin response (GSR) and blood volume pulse (BVP) signals, with and without the PD feature. The discriminating potential of each individual feature extracted from GSR, BVP and PD was studied by analysis of its receiver operating characteristic (ROC) curve. The ROC curve found for the PMPDmean feature encompassed the largest area (0.8546) of all the single-feature ROCs investigated. The encouraging results seen in affective sensing based on pupil diameter monitoring were obtained in spite of intermittent illumination increases purposely introduced during the experiments. Therefore, these results confirmed the benefits of using the AIC implementation with the HITV adaptive algorithm to isolate the PAR and the potential of using PD monitoring to sense the evolving affective states of a computer user.

Off-line and On-line Affective Recognition of a Computer User through A Biosignal Processing Approach

Physiological signals, which are controlled by the autonomic nervous system (ANS), could be used to detect the affective state of computer users and therefore find applications in medicine and engineering. The Pupil Diameter (PD) seems to provide a strong indication of the affective state, as found by previous research, but it has not been investigated fully yet. In this study, new approaches based on monitoring and processing the PD signal for off-line and on-line affective assessment (“relaxation” vs. “stress”) are proposed. Wavelet denoising and Kalman filtering methods are first used to remove abrupt changes in the raw Pupil Diameter (PD) signal. Then three features (PDmean, PDmax and PDWalsh) are extracted from the preprocessed PD signal for the affective state classification. In order to select more relevant and reliable physiological data for further analysis, two types of data selection methods are applied, which are based on the paired t-test and subject self-evaluation, respectively. In addition, five different kinds of the classifiers are implemented on the selected data, which achieve average accuracies up to 86.43% and 87.20%, respectively. Finally, the receiver operating characteristic (ROC) curve is utilized to investigate the discriminating potential of each individual feature by evaluation of the area under the ROC curve, which reaches values above 0.90. For the on-line affective assessment, a hard threshold is implemented first in order to remove the eye blinks from the PD signal and then a moving average window is utilized to obtain the representative value PDr for every one-second time interval of PD. There are three main steps for the on-line affective assessment algorithm, which are preparation, feature-based decision voting and affective determination. The final results show that the accuracies are 72.30% and 73.55% for the data subsets, which were respectively chosen using two types of data selection methods (paired t-test and subject self-evaluation). In order to further analyze the efficiency of affective recognition through the PD signal, the Galvanic Skin Response (GSR) was also monitored and processed. The highest affective assessment classification rate obtained from GSR processing is only 63.57% (based on the off-line processing algorithm). The overall results confirm that the PD signal should be considered as one of the most powerful physiological signals to involve in future automated real-time affective recognition systems, especially for detecting the “relaxation” vs. “stress” states.

Towards Emotional Interaction: Using Movies to Automatically Learn Users' Emotional States

The HCI community is actively seeking novel methodologies to gain insight into the user’s experience during interaction with both the application and the content. We propose an emotional recognition engine capable of automatically recognizing a set of human emotional states using psychophysiological measures of the autonomous nervous system, including galvanic skin response, respiration, and heart rate. A novel pattern recognition system, based on discriminant analysis and support vector machine classifiers is trained using movies’ scenes selected to induce emotions ranging from the positive to the negative valence dimension, including happiness, anger, disgust, sadness, and fear. In this paper we introduce an emotion recognition system and evaluate its accuracy by presenting the results of an experiment conducted with three physiologic sensors.

Visualização personalizada sensível ao contexto e às emoções

Mestrado em Engenharia Informática

Satisfação de agentes afectivos no processo de tomada de decisão em grupo

Um dos factores mais determinantes para o sucesso de uma organização é a qualidade das decisões tomadas. Para que as decisões tomadas sejam melhores e potenciem a competitividade das organizações, sistemas como os Sistemas de Apoio à Tomada de Decisão em Grupo (SADG) têm sido fortemente desenvolvidos e estudados nas últimas décadas. Cada vez mais, estes sistemas são populados com um maior número de dados, com o objectivo de serem mais assertivos. Considera-se que com determinados dados seja possível que o sistema possa aferir a satisfação dos participantes com as decisões tomadas, tendencialmente de forma automática. Hoje em dia, as análises de satisfação com as decisões não contemplam na sua maioria factores imprescindíveis, como os emocionais e de personalidade, sendo que os modelos existentes tendem a ser incompletos. Nesta dissertação propõe-se uma metodologia que permite a um SADG aferir a satisfação do participante com a decisão, considerando aspectos como a personalidade, as emoções e as expectativas. A metodologia desenvolvida foi implementada num SADG (ArgEmotionsAgents) com uma arquitectura multiagente, composto por agentes que reflectem participantes reais e que estão modelados com a sua personalidade. De acordo com a sua personalidade, esses agentes trocam argumentos persuasivos de forma a obterem uma decisão consensual. No processo de troca de argumentos os agentes geram emoções que afectam o seu humor. A implementação da metodologia no ArgEmotionsAgents permite que, no final de uma reunião, seja possível aferir a satisfação dos agentes participantes com a decisão final e com o processo que levou à tomada de decisão. De forma a validar a metodologia proposta bem como a implementação que foi desenvolvida, foram realizadas quatro experiências em diferentes cenários. Numa primeira experiência, foi aferida a satisfação dos quatro agentes participantes. Nas experiências seguintes, um dos agentes participantes foi usado como referência e foram alteradas configurações (expectativas, personalidade e reavaliação das alternativas) para perceber de que forma os vários factores afectam a satisfação. Com o estudo concluiu-se que todos os factores considerados no modelo afectam a satisfação. A forma como a satisfação é afectada por cada um dos factores vai ao encontro da lógica apresentada no estado da arte. Os resultados de satisfação aferidos pelo modelo são congruentes.

Emotional and user-specific cues for improved analysis of naturalistic interactions

... ist es nicht möglich, die in dieser Arbeit identifizierten offenen Fragen restlos zu klären. Die Erweiterung der puren akustischen Emotionserkennung durch Berücksichtigung von anderen Modalitäten, Sprechercharakteristiken, Feedbacksignalen und Persönlichkeitsmerkmalen erlaubt es jedoch, länger andauernde natürliche Interaktionen zu untersuchen und dialogkritische Situationen zu erkennen. Technische Systeme, die diese erweiterte Emotionserkennung nutzen, passen sich an ihren Nutzer an und werden so zu seinem Begleiter und letztendlich zu seinem Companion.

Configuration et exploitation d'une machine émotionnelle

Dans ce travail, nous explorons la faisabilité de doter les machines de la capacité de prédire, dans un contexte d'interaction homme-machine (IHM), l'émotion d'un utilisateur, ainsi que son intensité, de manière instantanée pour une grande variété de situations. Plus spécifiquement, une application a été développée, appelée machine émotionnelle, capable de «comprendre» la signification d'une situation en se basant sur le modèle théorique d'évaluation de l'émotion Ortony, Clore et Collins (OCC). Cette machine est apte, également, à prédire les réactions émotionnelles des utilisateurs, en combinant des versions améliorées des k plus proches voisins et des réseaux de neurones. Une procédure empirique a été réalisée pour l'acquisition des données. Ces dernières ont fourni une connaissance consistante aux algorithmes d'apprentissage choisis et ont permis de tester la performance de la machine. Les résultats obtenus montrent que la machine émotionnelle proposée est capable de produire de bonnes prédictions. Une telle réalisation pourrait encourager son utilisation future dans des domaines exploitant la reconnaissance automatique de l'émotion.

Using an animated pedagogical agent to interact affectively with the student

This work proposes an animated pedagogical agent that has the role of providing emotional support to the student: motivating and encouraging him, making him believe in his self-ability, and promoting a positive mood in him, which fosters learning. This careful support of the agent, its affective tactics, is expressed through emotional behaviour and encouragement messages of the lifelike character. Due to human social tendency of anthropomorphising software, we believe that a software agent can accomplish this affective role. In order to choose the adequate affective tactics, the agent should also know the student’s emotions. The proposed agent recognises the student’s emotions: joy/distress, satisfaction/disappointment, anger/gratitude, and shame, from the student’s observable behaviour, i. e. his actions in the interface of the educational system. The inference of emotions is psychologically grounded on the cognitive theory of emotions. More specifically, we use the OCC model which is based on the cognitive approach of emotion and can be computationally implemented. Due to the dynamic nature of the student’s affective information, we adopted a BDI approach to implement the affective user model and the affective diagnosis. Besides, in our work we profit from the reasoning capacity of the BDI approach in order for the agent to deduce the student’s appraisal, which allows it to infer the student’s emotions. As a case study, the proposed agent is implemented as the Mediating Agent of MACES: an educational collaborative environment modelled as a multi-agent system and pedagogically based on the sociocultural theory of Vygotsky.

Avaliação da influência de emoções na tomada de decisão de sistemas computacionais.

Este trabalho avalia a influência das emoções humanas expressas pela mímica da face na tomada de decisão de sistemas computacionais, com o objetivo de melhorar a experiência do usuário. Para isso, foram desenvolvidos três módulos: o primeiro trata-se de um sistema de computação assistiva - uma prancha de comunicação alternativa e ampliada em versão digital. O segundo módulo, aqui denominado Módulo Afetivo, trata-se de um sistema de computação afetiva que, por meio de Visão Computacional, capta a mímica da face do usuário e classifica seu estado emocional. Este segundo módulo foi implementado em duas etapas, as duas inspiradas no Sistema de Codificação de Ações Faciais (FACS), que identifica expressões faciais com base no sistema cognitivo humano. Na primeira etapa, o Módulo Afetivo realiza a inferência dos estados emocionais básicos: felicidade, surpresa, raiva, medo, tristeza, aversão e, ainda, o estado neutro. Segundo a maioria dos pesquisadores da área, as emoções básicas são inatas e universais, o que torna o módulo afetivo generalizável a qualquer população. Os testes realizados com o modelo proposto apresentaram resultados 10,9% acima dos resultados que usam metodologias semelhantes. Também foram realizadas análises de emoções espontâneas, e os resultados computacionais aproximam-se da taxa de acerto dos seres humanos. Na segunda etapa do desenvolvimento do Módulo Afetivo, o objetivo foi identificar expressões faciais que refletem a insatisfação ou a dificuldade de uma pessoa durante o uso de sistemas computacionais. Assim, o primeiro modelo do Módulo Afetivo foi ajustado para este fim. Por fim, foi desenvolvido um Módulo de Tomada de Decisão que recebe informações do Módulo Afetivo e faz intervenções no Sistema Computacional. Parâmetros como tamanho do ícone, arraste convertido em clique e velocidade de varredura são alterados em tempo real pelo Módulo de Tomada de Decisão no sistema computacional assistivo, de acordo com as informações geradas pelo Módulo Afetivo. Como o Módulo Afetivo não possui uma etapa de treinamento para inferência do estado emocional, foi proposto um algoritmo de face neutra para resolver o problema da inicialização com faces contendo emoções. Também foi proposto, neste trabalho, a divisão dos sinais faciais rápidos entre sinais de linha base (tique e outros ruídos na movimentação da face que não se tratam de sinais emocionais) e sinais emocionais. Os resultados dos Estudos de Caso realizados com os alunos da APAE de Presidente Prudente demonstraram que é possível melhorar a experiência do usuário, configurando um sistema computacional com informações emocionais expressas pela mímica da face.

Influencing the perceived emotions of music with intent

Music is an immensely powerful affective medium that pervades our everyday life. With ever advancing technology, the reproduction and application of music for emotive and information transfer purposes has never been more prevalent. In this paper we introduce a rule-based engine for influencing the perceived emotions of music. Based on empirical music psychology, we attempt to formalise the relationship between musical elements and their perceived emotion. We examine the modification to structural aspects of music to allow for a graduated transition between perceived emotive states. This engine is intended to provide music reproduction systems with a finer grained control over this affective medium; where perceived musical emotion can be influenced with intent. This intent comes from both an external application and the audience. Using a series of affective computing technologies, an audience’s response metrics and attitudes can be incorporated to model this intent. A generative feedback loop is set up between the external application, the influencing process and the audience’s response to this, which together shape the modification of musical structure. The effectiveness of our rule system for influencing perceived musical emotion was examined in earlier work, with a small test study providing generally encouraging results.