FACSGen: a tool to synthesize emotional facial expressions through systematic manipulation of facial action units

To investigate the perception of emotional facial expressions, researchers rely on shared sets of photos or videos, most often generated by actor portrayals. The drawback of such standardized material is a lack of flexibility and controllability, as it does not allow the systematic parametric manipulation of specific features of facial expressions on the one hand, and of more general properties of the facial identity (age, ethnicity, gender) on the other. To remedy this problem, we developed FACSGen: a novel tool that allows the creation of realistic synthetic 3D facial stimuli, both static and dynamic, based on the Facial Action Coding System. FACSGen provides researchers with total control over facial action units, and corresponding informational cues in 3D synthetic faces. We present four studies validating both the software and the general methodology of systematically generating controlled facial expression patterns for stimulus presentation.

FACSGen 2.0 animation software: generating three-dimensional FACS-valid facial expressions for emotion research

In this article, we present FACSGen 2.0, new animation software for creating static and dynamic threedimensional facial expressions on the basis of the Facial Action Coding System (FACS). FACSGen permits total control over the action units (AUs), which can be animated at all levels of intensity and applied alone or in combination to an infinite number of faces. In two studies, we tested the validity of the software for the AU appearance defined in the FACS manual and the conveyed emotionality of FACSGen expressions. In Experiment 1, four FACS-certified coders evaluated the complete set of 35 single AUs and 54 AU combinations for AU presence or absence, appearance quality, intensity, and asymmetry. In Experiment 2, lay participants performed a recognition task on emotional expressions created with FACSGen software and rated the similarity of expressions displayed by human and FACSGen faces. Results showed good to excellent classification levels for all AUs by the four FACS coders, suggesting that the AUs are valid exemplars of FACS specifications. Lay participants’ recognition rates for nine emotions were high, and comparisons of human and FACSGen expressions were very similar. The findings demonstrate the effectiveness of the software in producing reliable and emotionally valid expressions, and suggest its application in numerous scientific areas, including perception, emotion, and clinical and euroscience research.

I see happy people: attention bias towards happy but not angry facial expressions in Williams syndrome

Introduction: Observations of behaviour and research using eye-tracking technology have shown that individuals with Williams syndrome (WS) pay an unusual amount of attention to other people’s faces. The present research examines whether this attention to faces is moderated by the valence of emotional expression. Method: Sixteen participants with WS aged between 13 and 29 years (Mean=19 years 9 months) completed a dot-probe task in which pairs of faces displaying happy, angry and neutral expressions were presented. The performance of the WS group was compared to two groups of typically developing control participants, individually matched to the participants in the WS group on either chronological age or mental age. General mental age was assessed in the WS group using the Woodcock Johnson Test of Cognitive Ability Revised (WJ-COG-R; Woodcock & Johnson, 1989; 1990). Results: Compared to both control groups, the WS group exhibited a greater attention bias for happy faces. In contrast, no between-group differences in bias for angry faces were obtained. Conclusions: The results are discussed in relation to recent neuroimaging findings and the hypersocial behaviour that is characteristic of the WS population.

Interpretation of infant facial expression in the context of maternal postnatal depression

Postnatal maternal depression is associated with difficulties in maternal responsiveness. As most signals arising from the infant come from facial expressions one possible explanation for these difficulties is that mothers with postnatal depression are differentially affected by particular infant facial expressions. Thus, this study investigates the effects of postnatal depression on mothers’ perceptions of infant facial expressions. Participants (15 controls, 15 depressed and 15 anxious mothers) were asked to rate a number of infant facial expressions, ranging from very positive to very negative. Each face was shown twice, for a short and for a longer period of time in random order. Results revealed that mothers used more extreme ratings when shown the infant faces (i.e. more negative or more positive) for a longer period of time. Mothers suffering from postnatal depression were more likely to rate negative infant faces shown for a longer period more negatively than controls. The differences were specific to depression rather than an effect of general postnatal psychopathology—as no differences were observed between anxious mothers and controls. There were no other significant differences in maternal ratings of infant faces showed for short periods or for positive or neutral valence faces of either length. The findings that mothers with postnatal depression rate negative infant faces more negatively indicate that appraisal bias might underlie some of the difficulties that these mothers have in responding to their own infants signals.

Beta event-related desynchronization as an index of individual differences in processing human facial expression: further investigations of autistic traits in typically developing adults

The human mirror neuron system (hMNS) has been associated with various forms of social cognition and affective processing including vicarious experience. It has also been proposed that a faulty hMNS may underlie some of the deficits seen in the autism spectrum disorders (ASDs). In the present study we set out to investigate whether emotional facial expressions could modulate a putative EEG index of hMNS activation (mu suppression) and if so, would this differ according to the individual level of autistic traits [high versus low Autism Spectrum Quotient (AQ) score]. Participants were presented with 3 s films of actors opening and closing their hands (classic hMNS mu-suppression protocol) while simultaneously wearing happy, angry, or neutral expressions. Mu-suppression was measured in the alpha and low beta bands. The low AQ group displayed greater low beta event-related desynchronization (ERD) to both angry and neutral expressions. The high AQ group displayed greater low beta ERD to angry than to happy expressions. There was also significantly more low beta ERD to happy faces for the low than for the high AQ group. In conclusion, an interesting interaction between AQ group and emotional expression revealed that hMNS activation can be modulated by emotional facial expressions and that this is differentiated according to individual differences in the level of autistic traits. The EEG index of hMNS activation (mu suppression) seems to be a sensitive measure of the variability in facial processing in typically developing individuals with high and low self-reported traits of autism.

Visual search for basic emotional expressions in autism; impaired processing of anger, fear and sadness, but a typical happy face advantage

Facial expression recognition was investigated in 20 males with high functioning autism (HFA) or Asperger syndrome (AS), compared to typically developing individuals matched for chronological age (TD CA group) and verbal and non-verbal ability (TD V/NV group). This was the first study to employ a visual search, “face in the crowd” paradigm with a HFA/AS group, which explored responses to numerous facial expressions using real-face stimuli. Results showed slower response times for processing fear, anger and sad expressions in the HFA/AS group, relative to the TD CA group, but not the TD V/NV group. Reponses to happy, disgust and surprise expressions showed no group differences. Results are discussed with reference to the amygdala theory of autism.

An optimized ERP brain–computer interface based on facial expression changes

Objective. Interferences from spatially adjacent non-target stimuli are known to evoke event-related potentials (ERPs) during non-target flashes and, therefore, lead to false positives. This phenomenon was commonly seen in visual attention-based brain–computer interfaces (BCIs) using conspicuous stimuli and is known to adversely affect the performance of BCI systems. Although users try to focus on the target stimulus, they cannot help but be affected by conspicuous changes of the stimuli (such as flashes or presenting images) which were adjacent to the target stimulus. Furthermore, subjects have reported that conspicuous stimuli made them tired and annoyed. In view of this, the aim of this study was to reduce adjacent interference, annoyance and fatigue using a new stimulus presentation pattern based upon facial expression changes. Our goal was not to design a new pattern which could evoke larger ERPs than the face pattern, but to design a new pattern which could reduce adjacent interference, annoyance and fatigue, and evoke ERPs as good as those observed during the face pattern. Approach. Positive facial expressions could be changed to negative facial expressions by minor changes to the original facial image. Although the changes are minor, the contrast is big enough to evoke strong ERPs. In this paper, a facial expression change pattern between positive and negative facial expressions was used to attempt to minimize interference effects. This was compared against two different conditions, a shuffled pattern containing the same shapes and colours as the facial expression change pattern, but without the semantic content associated with a change in expression, and a face versus no face pattern. Comparisons were made in terms of classification accuracy and information transfer rate as well as user supplied subjective measures. Main results. The results showed that interferences from adjacent stimuli, annoyance and the fatigue experienced by the subjects could be reduced significantly (p < 0.05) by using the facial expression change patterns in comparison with the face pattern. The offline results show that the classification accuracy of the facial expression change pattern was significantly better than that of the shuffled pattern (p < 0.05) and the face pattern (p < 0.05). Significance. The facial expression change pattern presented in this paper reduced interference from adjacent stimuli and decreased the fatigue and annoyance experienced by BCI users significantly (p < 0.05) compared to the face pattern.

Decreasing the interference of visual-based P300 BCI using facial expression changes

Interferences from the spatially adjacent non-target stimuli evoke ERPs during non-target sub-trials and lead to false positives. This phenomenon is commonly seen in visual attention based BCIs and affects the performance of BCI system. Although, users or subjects tried to focus on the target stimulus, they still could not help being affected by conspicuous changes of the stimuli (flashes or presenting images) which were adjacent to the target stimulus. In view of this case, the aim of this study is to reduce the adjacent interference using new stimulus presentation pattern based on facial expression changes. Positive facial expressions can be changed to negative facial expressions by minor changes to the original facial image. Although the changes are minor, the contrast will be big enough to evoke strong ERPs. In this paper, two different conditions (Pattern_1, Pattern_2) were used to compare across objective measures such as classification accuracy and information transfer rate as well as subjective measures. Pattern_1 was a “flash-only” pattern and Pattern_2 was a facial expression change of a dummy face. In the facial expression change patterns, the background is a positive facial expression and the stimulus is a negative facial expression. The results showed that the interferences from adjacent stimuli could be reduced significantly (P<;0.05) by using the facial expression change patterns. The online performance of the BCI system using the facial expression change patterns was significantly better than that using the “flash-only” patterns in terms of classification accuracy (p<;0.01), bit rate (p<;0.01), and practical bit rate (p<;0.01). Subjects reported that the annoyance and fatigue could be significantly decreased (p<;0.05) using the new stimulus presentation pattern presented in this paper.

An optimized ERP Brain-computer interface based on facial expression changes

OBJECTIVE: Interferences from spatially adjacent non-target stimuli are known to evoke event-related potentials (ERPs) during non-target flashes and, therefore, lead to false positives. This phenomenon was commonly seen in visual attention-based brain-computer interfaces (BCIs) using conspicuous stimuli and is known to adversely affect the performance of BCI systems. Although users try to focus on the target stimulus, they cannot help but be affected by conspicuous changes of the stimuli (such as flashes or presenting images) which were adjacent to the target stimulus. Furthermore, subjects have reported that conspicuous stimuli made them tired and annoyed. In view of this, the aim of this study was to reduce adjacent interference, annoyance and fatigue using a new stimulus presentation pattern based upon facial expression changes. Our goal was not to design a new pattern which could evoke larger ERPs than the face pattern, but to design a new pattern which could reduce adjacent interference, annoyance and fatigue, and evoke ERPs as good as those observed during the face pattern. APPROACH: Positive facial expressions could be changed to negative facial expressions by minor changes to the original facial image. Although the changes are minor, the contrast is big enough to evoke strong ERPs. In this paper, a facial expression change pattern between positive and negative facial expressions was used to attempt to minimize interference effects. This was compared against two different conditions, a shuffled pattern containing the same shapes and colours as the facial expression change pattern, but without the semantic content associated with a change in expression, and a face versus no face pattern. Comparisons were made in terms of classification accuracy and information transfer rate as well as user supplied subjective measures. MAIN RESULTS: The results showed that interferences from adjacent stimuli, annoyance and the fatigue experienced by the subjects could be reduced significantly (p < 0.05) by using the facial expression change patterns in comparison with the face pattern. The offline results show that the classification accuracy of the facial expression change pattern was significantly better than that of the shuffled pattern (p < 0.05) and the face pattern (p < 0.05). SIGNIFICANCE: The facial expression change pattern presented in this paper reduced interference from adjacent stimuli and decreased the fatigue and annoyance experienced by BCI users significantly (p < 0.05) compared to the face pattern.

Exploring combinations of different color and facial expression stimuli for gaze-independent BCIs

Background: Some studies have proven that a conventional visual brain computer interface (BCI) based on overt attention cannot be used effectively when eye movement control is not possible. To solve this problem, a novel visual-based BCI system based on covert attention and feature attention has been proposed and was called the gaze-independent BCI. Color and shape difference between stimuli and backgrounds have generally been used in examples of gaze-independent BCIs. Recently, a new paradigm based on facial expression changes has been presented, and obtained high performance. However, some facial expressions were so similar that users couldn't tell them apart, especially when they were presented at the same position in a rapid serial visual presentation (RSVP) paradigm. Consequently, the performance of the BCI is reduced. New Method: In this paper, we combined facial expressions and colors to optimize the stimuli presentation in the gaze-independent BCI. This optimized paradigm was called the colored dummy face pattern. It is suggested that different colors and facial expressions could help users to locate the target and evoke larger event-related potentials (ERPs). In order to evaluate the performance of this new paradigm, two other paradigms were presented, called the gray dummy face pattern and the colored ball pattern. Comparison with Existing Method(s): The key point that determined the value of the colored dummy faces stimuli in BCI systems was whether the dummy face stimuli could obtain higher performance than gray faces or colored balls stimuli. Ten healthy participants (seven male, aged 21–26 years, mean 24.5 ± 1.25) participated in our experiment. Online and offline results of four different paradigms were obtained and comparatively analyzed. Results: The results showed that the colored dummy face pattern could evoke higher P300 and N400 ERP amplitudes, compared with the gray dummy face pattern and the colored ball pattern. Online results showed that the colored dummy face pattern had a significant advantage in terms of classification accuracy (p < 0.05) and information transfer rate (p < 0.05) compared to the other two patterns. Conclusions: The stimuli used in the colored dummy face paradigm combined color and facial expressions. This had a significant advantage in terms of the evoked P300 and N400 amplitudes and resulted in high classification accuracies and information transfer rates. It was compared with colored ball and gray dummy face stimuli.

Inverse amygdala and medial prefrontal cortex responses to surprised faces

Here we show inverse fMRI activation patterns in amygdala and medial prefrontal cortex (mPFC) depending upon whether subjects interpreted surprised facial expressions positively or negatively. More negative interpretations of surprised faces were associated with greater signal changes in the right ventral amygdala, while more positive interpretations were associated with greater signal changes in the ventral mPFC. Accordingly, signal change within these two areas was inversely correlated. Thus, individual differences in the judgment of surprised faces are related to a systematic inverse relationship between amygdala and mPFC activity, a circuitry that the animal literature suggests is critical to the assessment of stimuli that predict potential positive vs negative outcomes.

Contextual modulation of amygdala responsivity to surprised faces

We recently demonstrated a functional relationship between fMRI responses within the amygdala and the medial prefrontal cortex based upon whether subjects interpreted surprised facial expressions positively or negatively. In the present fMRI study, we sought to assess amygdala-medial prefrontal cortex responsivity when the interpretations of surprised faces were determined by contextual experimental stimuli, rather than subjective judgment. Subjects passively viewed individual presentations of surprised faces preceded by either a negatively or positively valenced contextual sentence (e. g., She just found $500 vs. She just lost $500). Negative and positive sentences were carefully matched in terms of length, situations described, and arousal level. Negatively cued surprised faces produced greater ventral amygdala activation compared to positively cued surprised faces. Responses to negative versus positive sentences were greater within the ventrolateral prefrontal cortex, whereas responses to positive versus negative sentences were greater within the ventromedial prefrontal cortex. The present study demonstrates that amygdala response to surprised facial expressions can be modulated by negatively versus positively valenced verbal contextual information. Connectivity analyses identified candidate cortical-subcortical systems subserving this modulation.

Human amygdala responses during presentation of happy and neutral faces: correlations with state anxiety

BACKGROUND: Previous functional imaging studies demonstrating amygdala response to happy facial expressions have all included the presentation of negatively valenced primary comparison expressions within the experimental context. This study assessed amygdala response to happy and neutral facial expressions in an experimental paradigm devoid of primary negatively valenced comparison expressions. METHODS: Sixteen human subjects (eight female) viewed 16-sec blocks of alternating happy and neutral faces interleaved with a baseline fixation condition during two functional magnetic resonance imaging scans. RESULTS: Within the ventral amygdala, a negative correlation between happy versus neutral signal changes and state anxiety was observed. The majority of the variability associated with this effect was explained by a positive relationship between state anxiety and signal change to neutral faces. CONCLUSIONS: Interpretation of amygdala responses to facial expressions of emotion will be influenced by considering the contribution of each constituent condition within a greater subtractive finding, as well as 1) their spatial location within the amygdaloid complex; and 2) the experimental context in which they were observed. Here, an observed relationship between state anxiety and ventral amygdala response to happy versus neutral faces was explained by response to neutral faces.

A Functional Magnetic Resonance Imaging Predictor of Treatment Response to Venlafaxine in Generalized Anxiety Disorder

Background: Functional magnetic resonance imaging (fMRI) holds promise as a noninvasive means of identifying neural responses that can be used to predict treatment response before beginning a drug trial. Imaging paradigms employing facial expressions as presented stimuli have been shown to activate the amygdala and anterior cingulate cortex (ACC). Here, we sought to determine whether pretreatment amygdala and rostral ACC (rACC) reactivity to facial expressions could predict treatment outcomes in patients with generalized anxiety disorder (GAD).Methods: Fifteen subjects (12 female subjects) with GAD participated in an open-label venlafaxine treatment trial. Functional magnetic resonance imaging responses to facial expressions of emotion collected before subjects began treatment were compared with changes in anxiety following 8 weeks of venlafaxine administration. In addition, the magnitude of fMRI responses of subjects with GAD were compared with that of 15 control subjects (12 female subjects) who did not have GAD and did not receive venlafaxine treatment.Results The magnitude of treatment response was predicted by greater pretreatment reactivity to fearful faces in rACC and lesser reactivity in the amygdala. These individual differences in pretreatment rACC and amygdala reactivity within the GAD group were observed despite the fact that 1) the overall magnitude of pretreatment rACC and amygdala reactivity did not differ between subjects with GAD and control subjects and 2) there was no main effect of treatment on rACC-amygdala reactivity in the GAD group.Conclusions: These findings show that this pattern of rACC-amygdala responsivity could prove useful as a predictor of venlafaxine treatment response in patients with GAD.

Variations in the human cannabinoid receptor (CNR1) gene modulate striatal responses to happy faces.

Happy facial expressions are innate social rewards and evoke a response in the striatum, a region known for its role in reward processing in rats, primates and humans. The cannabinoid receptor 1 (CNR1) is the best-characterized molecule of the endocannabinoid system, involved in processing rewards. We hypothesized that genetic variation in human CNR1 gene would predict differences in the striatal response to happy faces. In a 3T functional magnetic resonance imaging (fMRI) scanning study on 19 Caucasian volunteers, we report that four single nucleotide polymorphisms (SNPs) in the CNR1 locus modulate differential striatal response to happy but not to disgust faces. This suggests a role for the variations of the CNR1 gene in underlying social reward responsivity. Future studies should aim to replicate this finding with a balanced design in a larger sample, but these preliminary results suggest neural responsivity to emotional and socially rewarding stimuli varies as a function of CNR1 genotype. This has implications for medical conditions involving hypo-responsivity to emotional and social stimuli, such as autism.