Impact Study of Nonverbal Facial Cues on Spontaneous Chatting with Virtual Humans

Non-verbal communication (NVC) is considered to represent more than 90 percent of everyday communication. In virtual world, this important aspect of interaction between virtual humans (VH) is strongly neglected. This paper presents a user-test study to demonstrate the impact of automatically generated graphics-based NVC expression on the dialog quality: first, we wanted to compare impassive and emotion facial expression simulation for impact on the chatting. Second, we wanted to see whether people like chatting within a 3D graphical environment. Our model only proposes facial expressions and head movements induced from spontaneous chatting between VHs. Only subtle facial expressions are being used as nonverbal cues - i.e. related to the emotional model. Motion capture animations related to hand gestures, such as cleaning glasses, were randomly used to make the virtual human lively. After briefly introducing the technical architecture of the 3D-chatting system, we focus on two aspects of chatting through VHs. First, what is the influence of facial expressions that are induced from text dialog? For this purpose, we exploited an emotion engine extracting an emotional content from a text and depicting it into a virtual character developed previously [GAS11]. Second, as our goal was not addressing automatic generation of text, we compared the impact of nonverbal cues in conversation with a chatbot or with a human operator with a wizard of oz approach. Among main results, the within group study -involving 40 subjects- suggests that subtle facial expressions impact significantly not only on the quality of experience but also on dialog understanding.

Cortical control of facial expression.

The present topical review deals with the motor control of facial expressions in humans. Facial expressions are a central part of human communication. Emotional face expressions have a crucial role in human non-verbal behavior, allowing a rapid transfer of information between individuals. Facial expressions can be both voluntarily or emotionally controlled. Recent studies in non-human primates and humans revealed that the motor control of facial expressions has a distributed neural representation. At least 5 cortical regions on the medial and lateral aspects of each hemisphere are involved: the primary motor cortex, the ventral lateral premotor cortex, the supplementary motor area on the medial wall, and, finally, the rostral and caudal cingulate cortex. The results of studies in humans and non-human primates suggest that the innervation of the face is bilaterally controlled for the upper part, and mainly contralaterally controlled for the lower part. Furthermore, the primary motor cortex, the ventral lateral premotor cortex, and the supplementary motor area are essential for the voluntary control of facial expressions. In contrast, the cingulate cortical areas are important for emotional expression, since they receive input from different structures of the limbic system. This article is protected by copyright. All rights reserved.

A facial expression for anxiety

Anxiety and fear are often confounded in discussions of human emotions. However, studies of rodent defensive reactions under naturalistic conditions suggest anxiety is functionally distinct from fear. Unambiguous threats, such as predators, elicit flight from rodents (if an escape-route is available), whereas ambiguous threats (e.g., the odor of a predator) elicit risk assessment behavior, which is associated with anxiety as it is preferentially modulated by anti-anxiety drugs. However, without human evidence, it would be premature to assume that rodent-based psychological models are valid for humans. We tested the human validity of the risk assessment explanation for anxiety by presenting 8 volunteers with emotive scenarios and asking them to pose facial expressions. Photographs and videos of these expressions were shown to 40 participants who matched them to the scenarios and labeled each expression. Scenarios describing ambiguous threats were preferentially matched to the facial expression posed in response to the same scenario type. This expression consisted of two plausible environmental-scanning behaviors (eye darts and head swivels) and was labeled as anxiety, not fear. The facial expression elicited by unambiguous threat scenarios was labeled as fear. The emotion labels generated were then presented to another 18 participants who matched them back to photographs of the facial expressions. This back-matching of labels to faces also linked anxiety to the environmental-scanning face rather than fear face. Results therefore suggest that anxiety produces a distinct facial expression and that it has adaptive value in situations that are ambiguously threatening, supporting a functional, risk-assessing explanation for human anxiety.

Investigation of neural correlates of faces and emotional expressions using magnetoencephalography

The recognition of faces and of facial expressions in an important evolutionary skill, and an integral part of social communication. It has been argued that the processing of faces is distinct from the processing of non-face stimuli and functional neuroimaging investigations have even found evidence of a distinction between the perception of faces and of emotional expressions. Structural and temporal correlates of face perception and facial affect have only been separately identified. Investigation neural dynamics of face perception per se as well as facial affect would allow the mapping of these in space, time and frequency specific domains. Participants were asked to perform face categorisation and emotional discrimination tasks and Magnetoencephalography (MEG) was used to measure the neurophysiology of face and facial emotion processing. SAM analysis techniques enable the investigation of spectral changes within specific time-windows and frequency bands, thus allowing the identification of stimulus specific regions of cortical power changes. Furthermore, MEG’s excellent temporal resolution allows for the detection of subtle changes associated with the processing of face and non-face stimuli and different emotional expressions. The data presented reveal that face perception is associated with spectral power changes within a distributed cortical network comprising occipito-temporal as well as parietal and frontal areas. For the perception of facial affect, spectral power changes were also observed within frontal and limbic areas including the parahippocampal gyrus and the amygdala. Analyses of temporal correlates also reveal a distinction between the processing of faces and facial affect. Face perception per se occurred at earlier latencies whereas the discrimination of facial expression occurred within a longer time-window. In addition, the processing of faces and facial affect was differentially associated with changes in cortical oscillatory power for alpha, beta and gamma frequencies. The perception of faces and facial affect is associated with distinct changes in cortical oscillatory activity that can be mapped to specific neural structures, specific time-windows and latencies as well as specific frequency bands. Therefore, the work presented in this thesis provides further insight into the sequential processing of faces and facial affect.

Facial affect processing inbipolar disorder:a magnetoencephalography study

Background: Identifying biological markers to aid diagnosis of bipolar disorder (BD) is critically important. To be considered a possible biological marker, neural patterns in BD should be discriminant from those in healthy individuals (HI). We examined patterns of neuromagnetic responses revealed by magnetoencephalography (MEG) during implicit emotion-processing using emotional (happy, fearful, sad) and neutral facial expressions, in sixteen BD and sixteen age- and gender-matched healthy individuals. Methods: Neuromagnetic data were recorded using a 306-channel whole-head MEG ELEKTA Neuromag System, and preprocessed using Signal Space Separation as implemented in MaxFilter (ELEKTA). Custom Matlab programs removed EOG and ECG signals from filtered MEG data, and computed means of epoched data (0-250ms, 250-500ms, 500-750ms). A generalized linear model with three factors (individual, emotion intensity and time) compared BD and HI. A principal component analysis of normalized mean channel data in selected brain regions identified principal components that explained 95% of data variation. These components were used in a quadratic support vector machine (SVM) pattern classifier. SVM classifier performance was assessed using the leave-one-out approach. Results: BD and HI showed significantly different patterns of activation for 0-250ms within both left occipital and temporal regions, specifically for neutral facial expressions. PCA analysis revealed significant differences between BD and HI for mild fearful, happy, and sad facial expressions within 250-500ms. SVM quadratic classifier showed greatest accuracy (84%) and sensitivity (92%) for neutral faces, in left occipital regions within 500-750ms. Conclusions: MEG responses may be used in the search for disease specific neural markers.

Traitement des émotions évoquées par les expressions faciales, vocales et musicales à la suite d’un traumatisme craniocérébral

Le traitement des émotions joue un rôle essentiel dans les relations interpersonnelles. Des déficits dans la reconnaissance des émotions évoquées par les expressions faciales et vocales ont été démontrés à la suite d’un traumatisme craniocérébral (TCC). Toutefois, la majorité des études n’ont pas différencié les participants selon le niveau de gravité du TCC et n’ont pas évalué certains préalables essentiels au traitement émotionnel, tels que la capacité à percevoir les caractéristiques faciales et vocales, et par le fait même, la capacité à y porter attention. Aucune étude ne s’est intéressée au traitement des émotions évoquées par les expressions musicales, alors que la musique est utilisée comme méthode d’intervention afin de répondre à des besoins de prise en charge comportementale, cognitive ou affective chez des personnes présentant des atteintes neurologiques. Ainsi, on ignore si les effets positifs de l’intervention musicale sont basés sur la préservation de la reconnaissance de certaines catégories d’émotions évoquées par les expressions musicales à la suite d’un TCC. La première étude de cette thèse a évalué la reconnaissance des émotions de base (joie, tristesse, peur) évoquées par les expressions faciales, vocales et musicales chez quarante et un adultes (10 TCC modéré-sévère, 9 TCC léger complexe, 11 TCC léger simple et 11 témoins), à partir de tâches expérimentales et de tâches perceptuelles contrôles. Les résultats suggèrent un déficit de la reconnaissance de la peur évoquée par les expressions faciales à la suite d’un TCC modéré-sévère et d’un TCC léger complexe, comparativement aux personnes avec un TCC léger simple et sans TCC. Le déficit n’est pas expliqué par un trouble perceptuel sous-jacent. Les résultats montrent de plus une préservation de la reconnaissance des émotions évoquées par les expressions vocales et musicales à la suite d’un TCC, indépendamment du niveau de gravité. Enfin, malgré une dissociation observée entre les performances aux tâches de reconnaissance des émotions évoquées par les modalités visuelle et auditive, aucune corrélation n’a été trouvée entre les expressions vocales et musicales. La deuxième étude a mesuré les ondes cérébrales précoces (N1, N170) et plus tardives (N2) de vingt-cinq adultes (10 TCC léger simple, 1 TCC léger complexe, 3 TCC modéré-sévère et 11 témoins), pendant la présentation d’expressions faciales évoquant la peur, la neutralité et la joie. Les résultats suggèrent des altérations dans le traitement attentionnel précoce à la suite d’un TCC, qui amenuisent le traitement ultérieur de la peur évoquée par les expressions faciales. En somme, les conclusions de cette thèse affinent notre compréhension du traitement des émotions évoquées par les expressions faciales, vocales et musicales à la suite d’un TCC selon le niveau de gravité. Les résultats permettent également de mieux saisir les origines des déficits du traitement des émotions évoquées par les expressions faciales à la suite d’un TCC, lesquels semblent secondaires à des altérations attentionnelles précoces. Cette thèse pourrait contribuer au développement éventuel d’interventions axées sur les émotions à la suite d’un TCC.

Traitement des émotions évoquées par les expressions faciales, vocales et musicales à la suite d’un traumatisme craniocérébral

Le traitement des émotions joue un rôle essentiel dans les relations interpersonnelles. Des déficits dans la reconnaissance des émotions évoquées par les expressions faciales et vocales ont été démontrés à la suite d’un traumatisme craniocérébral (TCC). Toutefois, la majorité des études n’ont pas différencié les participants selon le niveau de gravité du TCC et n’ont pas évalué certains préalables essentiels au traitement émotionnel, tels que la capacité à percevoir les caractéristiques faciales et vocales, et par le fait même, la capacité à y porter attention. Aucune étude ne s’est intéressée au traitement des émotions évoquées par les expressions musicales, alors que la musique est utilisée comme méthode d’intervention afin de répondre à des besoins de prise en charge comportementale, cognitive ou affective chez des personnes présentant des atteintes neurologiques. Ainsi, on ignore si les effets positifs de l’intervention musicale sont basés sur la préservation de la reconnaissance de certaines catégories d’émotions évoquées par les expressions musicales à la suite d’un TCC. La première étude de cette thèse a évalué la reconnaissance des émotions de base (joie, tristesse, peur) évoquées par les expressions faciales, vocales et musicales chez quarante et un adultes (10 TCC modéré-sévère, 9 TCC léger complexe, 11 TCC léger simple et 11 témoins), à partir de tâches expérimentales et de tâches perceptuelles contrôles. Les résultats suggèrent un déficit de la reconnaissance de la peur évoquée par les expressions faciales à la suite d’un TCC modéré-sévère et d’un TCC léger complexe, comparativement aux personnes avec un TCC léger simple et sans TCC. Le déficit n’est pas expliqué par un trouble perceptuel sous-jacent. Les résultats montrent de plus une préservation de la reconnaissance des émotions évoquées par les expressions vocales et musicales à la suite d’un TCC, indépendamment du niveau de gravité. Enfin, malgré une dissociation observée entre les performances aux tâches de reconnaissance des émotions évoquées par les modalités visuelle et auditive, aucune corrélation n’a été trouvée entre les expressions vocales et musicales. La deuxième étude a mesuré les ondes cérébrales précoces (N1, N170) et plus tardives (N2) de vingt-cinq adultes (10 TCC léger simple, 1 TCC léger complexe, 3 TCC modéré-sévère et 11 témoins), pendant la présentation d’expressions faciales évoquant la peur, la neutralité et la joie. Les résultats suggèrent des altérations dans le traitement attentionnel précoce à la suite d’un TCC, qui amenuisent le traitement ultérieur de la peur évoquée par les expressions faciales. En somme, les conclusions de cette thèse affinent notre compréhension du traitement des émotions évoquées par les expressions faciales, vocales et musicales à la suite d’un TCC selon le niveau de gravité. Les résultats permettent également de mieux saisir les origines des déficits du traitement des émotions évoquées par les expressions faciales à la suite d’un TCC, lesquels semblent secondaires à des altérations attentionnelles précoces. Cette thèse pourrait contribuer au développement éventuel d’interventions axées sur les émotions à la suite d’un TCC.

La retroalimentación facial y su efecto en la evaluación de publicidad de humor

La hipótesis de retroalimentación facial planteada por Tomkins en 1962 sustenta que la activación de algunos músculos faciales envía información sensorial al cerebro y se induce entonces una experiencia emocional en el sujeto. Partiendo de dicha teoría y de investigaciones que la sustentan, el presente estudio se propuso confirmar el efecto de la emoción inducida a través de la retroalimentación facial sobre la evaluación de cinco tipos de humor en publicidad. Para ello se realizó un experimento con 60 hombres y 60 mujeres, que fueron asignados aleatoriamente a una de dos condiciones: estimulación de sonrisa –músculos hacia arriba- o inhibición de sonrisa –músculos hacía abajo-, mientras evaluaban 16 imágenes de publicidad de humor. A partir del análisis de los resultados se encontraron diferencias significativas entre las condiciones; en línea con la hipótesis formulada, los participantes expuestos a la condición estimulación de sonrisa –músculos hacía arriba- evaluaron más positivamente los comerciales. También se encontraron diferencias significativas en función del sexo y los tipos de humor evaluados. El estudio ofrece evidencia empírica de la teoría propuesta hace más de medio siglo y su efecto en el ámbito de la publicidad actual.

Geometry vs appearance for discriminating between posed and spontaneous emotions

Spontaneous facial expressions differ from posed ones in appearance, timing and accompanying head movements. Still images cannot provide timing or head movement information directly. However, indirectly the distances between key points on a face extracted from a still image using active shape models can capture some movement and pose changes. This information is superposed on information about non-rigid facial movement that is also part of the expression. Does geometric information improve the discrimination between spontaneous and posed facial expressions arising from discrete emotions? We investigate the performance of a machine vision system for discrimination between posed and spontaneous versions of six basic emotions that uses SIFT appearance based features and FAP geometric features. Experimental results on the NVIE database demonstrate that fusion of geometric information leads only to marginal improvement over appearance features. Using fusion features, surprise is the easiest emotion (83.4% accuracy) to be distinguished, while disgust is the most difficult (76.1%). Our results find different important facial regions between discriminating posed versus spontaneous version of one emotion and classifying the same emotion versus other emotions. The distribution of the selected SIFT features shows that mouth is more important for sadness, while nose is more important for surprise, however, both the nose and mouth are important for disgust, fear, and happiness. Eyebrows, eyes, nose and mouth are important for anger.

Face recognition using fractal codes

In this paper we propose a new method for face recognition using fractal codes. Fractal codes represent local contractive, affine transformations which when iteratively applied to range-domain pairs in an arbitrary initial image result in a fixed point close to a given image. The transformation parameters such as brightness offset, contrast factor, orientation and the address of the corresponding domain for each range are used directly as features in our method. Features of an unknown face image are compared with those pre-computed for images in a database. There is no need to iterate, use fractal neighbor distances or fractal dimensions for comparison in the proposed method. This method is robust to scale change, frame size change and rotations as well as to some noise, facial expressions and blur distortion in the image

Social signal processing for pain monitoring using a hidden conditional random field

utomatic pain monitoring has the potential to greatly improve patient diagnosis and outcomes by providing a continuous objective measure. One of the most promising methods is to do this via automatically detecting facial expressions. However, current approaches have failed due to their inability to: 1) integrate the rigid and non-rigid head motion into a single feature representation, and 2) incorporate the salient temporal patterns into the classification stage. In this paper, we tackle the first problem by developing a “histogram of facial action units” representation using Active Appearance Model (AAM) face features, and then utilize a Hidden Conditional Random Field (HCRF) to overcome the second issue. We show that both of these methods improve the performance on the task of pain detection in sequence level compared to current state-of-the-art-methods on the UNBC-McMaster Shoulder Pain Archive.

Exploring emotional climate in preservice science teacher education

Classroom emotional climates are interrelated with students’ engagement with university courses. Despite growing interest in emotions and emotional climate research, little is known about the ways in which social interactions and different subject matter mediate emotional climates in preservice science teacher education classes. In this study we investigated the emotional climate and associated classroom interactions in a preservice science teacher education class. We were interested in the ways in which salient classroom interactions were related to the emotional climate during lessons centered on debates about science-based issues (e.g., nuclear energy alternatives). Participants used audience response technology to indicate their perceptions of the emotional climate. Analysis of conversation for salient video clips and analysis of non-verbal conduct (acoustic parameters, body movements, and facial expressions) supplemented emotional climate data. One key contribution that this study makes to preservice science teacher education is to identify the micro-processes of successful and unsuccessful class interactions that were associated with positive and neutral emotional climate. The structure of these interactions can inform the practice of other science educators who wish to produce positive emotional climates in their classes. The study also extends and explicates the construct of intensity of emotional climate.

Emotional climate and high quality learning experiences in science teacher education

The role of emotion during learning encounters in science teacher education is under-researched and under-theorized. In this case study we explore the emotional climates, that is, the collective states of emotional arousal, of a preservice secondary science education class to illuminate practice for producing and reproducing high quality learning experiences for preservice science teachers. Theories related to the sociology of emotions informed our analyses from data sources such as preservice teachers’ perceptions of the emotional climate of their class, emotional facial expressions, classroom conversations, and cogenerative dialogue. The major outcome from our analyses was that even though preservice teachers reported high positive emotional climate during the professor’s science demonstrations, they also valued the professor’s in the moment reflections on her teaching that were associated with low emotional climate ratings. We co-relate emotional climate data and preservice teachers’ comments during cogenerative dialogue to expand our understanding of high quality experiences and emotional climate in science teacher education. Our study also contributes refinements to research perspectives on emotional climate.

Linguistic and perceptual processing of communicative cues in Asperger Syndrome

Asperger Syndrome (AS) belongs to autism spectrum disorders where both verbal and non-verbal communication difficulties are at the core of the impairment. Social communication requires a complex use of affective, linguistic-cognitive and perceptual processes. In the four studies included in the current thesis, some of the linguistic and perceptual factors that are important for face-to-face communication were studied using behavioural methods. In all four studies the results obtained from individuals with AS were compared with typically developed age, gender and IQ matched controls. First, the language skills of school-aged children were characterized in detail with standardized tests that measured different aspects of receptive and expressive language (Study I). The children with AS were found to be worse than the controls in following complex verbal instructions. Next, the visual perception of facial expressions of emotion with varying degrees of visual detail was examined (Study II). Adults with AS were found to have impaired recognition of facial expressions on the basis of very low spatial frequencies which are important for processing global information. Following that, multisensory perception was investigated by looking at audiovisual speech perception (Studies III and IV). Adults with AS were found to perceive audiovisual speech qualitatively differently from typically developed adults, although both groups were equally accurate in recognizing auditory and visual speech presented alone. Finally, the effect of attention on audiovisual speech perception was studied by registering eye gaze behaviour (Study III) and by studying the voluntary control of visual attention (Study IV). The groups did not differ in eye gaze behaviour or in the voluntary control of visual attention. The results of the study series demonstrate that many factors underpinning face-to-face social communication are atypical in AS. In contrast with previous assumptions about intact language abilities, the current results show that children with AS have difficulties in understanding complex verbal instructions. Furthermore, the study makes clear that deviations in the perception of global features in faces expressing emotions as well as in the multisensory perception of speech are likely to harm face-to-face social communication.

Electronic dictionary entries with animated pictures: Lookup preferences and word retention

http://ijl.oxfordjournals.org/cgi/reprint/ecp022?ijkey=FWAwWPvILuZDT1S&keytype=ref