Towards an emotion-driven interactive music system: ridging the gaps between affect, physiology and sound generation

Report for the scientific sojourn at the Stanford University from January until June 2007. Music is well known for affecting human emotional states, yet the relationship between specific musical parameters and emotional responses is still not clear. With the advent of new human-computer interaction (HCI) technologies, it is now possible to derive emotion-related information from physiological data and use it as an input to interactive music systems. Providing such implicit musical HCI will be highly relevant for a number of applications including music therapy, diagnosis, nteractive gaming, and physiologically-based musical instruments. A key question in such physiology-based compositions is how sound synthesis parameters can be mapped to emotional states of valence and arousal. We used both verbal and heart rate responses to evaluate the affective power of five musical parameters. Our results show that a significant correlation exists between heart rate and the subjective evaluation of well-defined musical parameters. Brightness and loudness showed to be arousing parameters on subjective scale while harmonicity and even partial attenuation factor resulted in heart rate changes typically associated to valence. This demonstrates that a rational approach to designing emotion-driven music systems for our public installations and music therapy applications is possible.

Influència de la música sobre l'atenció humana: especial a la metodologia d'anàlisi, tot combinant metodologia quantitavia amb qualitativa (mix method)

Projecte de recerca elaborat a partir d’una estada a la University of Calgary, Canadà, entre desembre del 2007 i febrer del 2008. El projecte ha consistit en l'anàlisi de les dades d'una recerca en el camp de la psicologia de la música, concretament en com influeix la música en l'atenció a través de les vies dels estats emocionals i enèrgics de la persona. Per a la recerca es feu ús de videu en les sessions, obtenint dades visuals i auditives per a complementar les dades de tipus quantitatiu provinents dels resultats d'uns tests d'atenció subministrats. L'anàlisi es realitzà segons mètodes i tècniques de caràcter qualitatiu, apresos durant l'estada. Així mateix també s'ha aprofundit en la comprensió del paradigma qualitatiu com a paradigma vàlid i realment complementari del paradigma qualitatiu. S'ha focalitzat especialment en l'anàlisi de la conversa des d'un punt de vista interpretatiu així com l'anàlisi de llenguatge corporal i facial a partir de l'observació de videu, tot formulant descriptors i subdescriptors de la conducta que està relacionada amb la hipòtesis. Alguns descriptors havien estat formulats prèviament a l’anàlisi, en base a altres investigacions i al background de la investigadora; altres s’han anat descobrint durant l’anàlisi. Els descriptors i subdescriptors de la conducta estan relacionats amb l'intent dels estats anímics i enèrgics dels diferents participants. L'anàlisi s'ha realitzat com un estudi de casos, fent un anàlisi exhaustiu persona per persona amb l'objectiu de trobar patrons de reacció intrapersonals i intrapersonals. Els patrons observats s'utilitzaran com a contrast amb la informació quantitativa, tot realitzant triangulació amb les dades per trobar-ne possibles recolzaments o contradiccions entre sí. Els resultats preliminars indiquen relació entre el tipus de música i el comportament, sent que la música d'emotivitat negativa està associada a un tancament de la persona, però quan la música és enèrgica els participants s'activen (conductualment observat) i somriuen si aquesta és positiva.

Embedding Emotional Context in Recommender Systems

Emotions are crucial for user's decision making in recommendation processes. We first introduce ambient recommender systems, which arise from the analysis of new trends on the exploitation of the emotional context in the next generation of recommender systems. We then explain some results of these new trends in real-world applications through the smart prediction assistant (SPA) platform in an intelligent learning guide with more than three million users. While most approaches to recommending have focused on algorithm performance. SPA makes recommendations to users on the basis of emotional information acquired in an incremental way. This article provides a cross-disciplinary perspective to achieve this goal in such recommender systems through a SPA platform. The methodology applied in SPA is the result of a bunch of technology transfer projects for large real-world rccommender systems

Emotional attention as a modulatory system of perception

Report for the scientific sojourn carried out at the University Medical Center, Swiss, from 2010 to 2012. Abundant evidence suggests that negative emotional stimuli are prioritized in the perceptual systems, eliciting enhanced neural responses in early sensory regions as compared with neutral information. This facilitated detection is generally paralleled by larger neural responses in early sensory areas, relative to the processing of neutral information. In this sense, the amygdala and other limbic regions, such as the orbitofrontal cortex, may play a critical role by sending modulatory projections onto the sensory cortices via direct or indirect feedback.The present project aimed at investigating two important issues regarding these mechanisms of emotional attention, by means of functional magnetic resonance imaging. In Study I, we examined the modulatory effects of visual emotion signals on the processing of task-irrelevant visual, auditory, and somatosensory input, that is, the intramodal and crossmodal effects of emotional attention. We observed that brain responses to auditory and tactile stimulation were enhanced during the processing of visual emotional stimuli, as compared to neutral, in bilateral primary auditory and somatosensory cortices, respectively. However, brain responses to visual task-irrelevant stimulation were diminished in left primary and secondary visual cortices in the same conditions. The results also suggested the existence of a multimodal network associated with emotional attention, presumably involving mediofrontal, temporal and orbitofrontal regions Finally, Study II examined the different brain responses along the low-level visual pathways and limbic regions, as a function of the number of retinal spikes during visual emotional processing. The experiment used stimuli resulting from an algorithm that simulates how the visual system perceives a visual input after a given number of retinal spikes. The results validated the visual model in human subjects and suggested differential emotional responses in the amygdala and visual regions as a function of spike-levels. A list of publications resulting from work in the host laboratory is included in the report.

Emotional attention as a modulatory system of perception

Report for the scientific sojourn carried out at the University Medical Center, Swiss, from 2010 to 2012. Abundant evidence suggests that negative emotional stimuli are prioritized in the perceptual systems, eliciting enhanced neural responses in early sensory regions as compared with neutral information. This facilitated detection is generally paralleled by larger neural responses in early sensory areas, relative to the processing of neutral information. In this sense, the amygdala and other limbic regions, such as the orbitofrontal cortex, may play a critical role by sending modulatory projections onto the sensory cortices via direct or indirect feedback.The present project aimed at investigating two important issues regarding these mechanisms of emotional attention, by means of functional magnetic resonance imaging. In Study I, we examined the modulatory effects of visual emotion signals on the processing of task-irrelevant visual, auditory, and somatosensory input, that is, the intramodal and crossmodal effects of emotional attention. We observed that brain responses to auditory and tactile stimulation were enhanced during the processing of visual emotional stimuli, as compared to neutral, in bilateral primary auditory and somatosensory cortices, respectively. However, brain responses to visual task-irrelevant stimulation were diminished in left primary and secondary visual cortices in the same conditions. The results also suggested the existence of a multimodal network associated with emotional attention, presumably involving mediofrontal, temporal and orbitofrontal regions Finally, Study II examined the different brain responses along the low-level visual pathways and limbic regions, as a function of the number of retinal spikes during visual emotional processing. The experiment used stimuli resulting from an algorithm that simulates how the visual system perceives a visual input after a given number of retinal spikes. The results validated the visual model in human subjects and suggested differential emotional responses in the amygdala and visual regions as a function of spike-levels. A list of publications resulting from work in the host laboratory is included in the report.

Emotion and reason in everyday risk perception

Although research has documented the importance of emotion in risk perception, little is knownabout its prevalence in everyday life. Using the Experience Sampling Method, 94 part-timestudents were prompted at random via cellular telephones to report on mood state and threeemotions and to assess risk on thirty occasions during their working hours. The emotions valence, arousal, and dominance were measured using self-assessment manikins (Bradley &Lang, 1994). Hierarchical linear models (HLM) revealed that mood state and emotions explainedsignificant variance in risk perception. In addition, valence and arousal accounted for varianceover and above reason (measured by severity and possibility of risks). Six risks were reassessedin a post-experimental session and found to be lower than their real-time counterparts.The study demonstrates the feasibility and value of collecting representative samples of data withsimple technology. Evidence for the statistical consistency of the HLM estimates is provided inan Appendix.

Assessment of emotional reactivity produced by exposure to virtual environments in patients with eating disorders

The aim of this study was to assess the usefulness of virtual environments representing situations that are emotionally significant to subjects with eating disorders (ED). These environments may be applied with both evaluative and therapeutic aims and in simulation procedures to carry out a range of experimental studies. This paper is part of a wider research project analyzing the influence of the situation to which subjects are exposed on their performance on body image estimation tasks. Thirty female patients with eating disorders were exposed to six virtual environments: a living-room (neutral situation), a kitchen with highcalorie food, a kitchen with low-calorie food, a restaurant with high-calorie food, a restaurant with low-calorie food, and a swimming-pool. After exposure to each environment the STAI-S (a measurement of state anxiety) and the CDB (a measurement of depression) were administered to all subjects. The results show that virtual reality instruments are particularly useful for simulating everyday situations that may provoke emotional reactions such as anxiety and depression, in patients with ED. Virtual environments in which subjects are obliged to ingest high-calorie food provoke the highest levels of state anxiety and depression.

On Automatic Diagnosis of Alzheimer's Disease based on Spontaneous Speech Analysis and Emotional Temperature

Alzheimer's disease is the most prevalent form of progressive degenerative dementia; it has a high socio-economic impact in Western countries. Therefore it is one of the most active research areas today. Alzheimer's is sometimes diagnosed by excluding other dementias, and definitive confirmation is only obtained through a post-mortem study of the brain tissue of the patient. The work presented here is part of a larger study that aims to identify novel technologies and biomarkers for early Alzheimer's disease detection, and it focuses on evaluating the suitability of a new approach for early diagnosis of Alzheimer’s disease by non-invasive methods. The purpose is to examine, in a pilot study, the potential of applying Machine Learning algorithms to speech features obtained from suspected Alzheimer sufferers in order help diagnose this disease and determine its degree of severity. Two human capabilities relevant in communication have been analyzed for feature selection: Spontaneous Speech and Emotional Response. The experimental results obtained were very satisfactory and promising for the early diagnosis and classification of Alzheimer’s disease patients.

Differences of Functional Connectivity Brain Network in Emotional Judgment

Using combined emotional stimuli, combining photos of faces and recording of voices, we investigated the neural dynamics of emotional judgment using scalp EEG recordings. Stimuli could be either combioned in a congruent, or a non-congruent way.. As many evidences show the major role of alpha in emotional processing, the alpha band was subjected to be analyzed. Analysis was performed by computing the synchronization of the EEGs and the conditions congruent vs. non-congruent were compared using statistical tools. The obtained results demonstrate that scalp EEG ccould be used as a tool to investigate the neural dynamics of emotional valence and discriminate various emotions (angry, happy and neutral stimuli).

Emotional intelligence and social interest : are they related constructs?

In the last 15 years, a new psychological construct has emerged in the field of psychology: Emotional Intelligence. Some models of Emotional Intelligence bear ressemblence with aspects of one of the core constructs of Adlerian Psychology: Social Interest. The authors investigated, if both constructs are also empirically related and which is their capacity to predict psychiatric symptoms and antisocial behavior. Results indicate that Social Interest and Emotional Intelligence are empirically different constructs; Social Interest was negatively correlated to aspects of antisocial attitudes (but not to antisocial behavior). Social Interest also failed to predict symptoms of psychological distress. Emotional Intelligence, in change, was a better predictor for mental problems than Social Interest. The results are discussed in view of the validity of Social Interest measurement.

Spontaneous Speech and Emotional Response modeling based on One-class classifier oriented to Alzheimer Disease diagnosis

The purpose of our project is to contribute to earlier diagnosis of AD and better estimates of its severity by using automatic analysis performed through new biomarkers extracted from non-invasive intelligent methods. The methods selected in this case are speech biomarkers oriented to Sponta-neous Speech and Emotional Response Analysis. Thus the main goal of the present work is feature search in Spontaneous Speech oriented to pre-clinical evaluation for the definition of test for AD diagnosis by One-class classifier. One-class classifi-cation problem differs from multi-class classifier in one essen-tial aspect. In one-class classification it is assumed that only information of one of the classes, the target class, is available. In this work we explore the problem of imbalanced datasets that is particularly crucial in applications where the goal is to maximize recognition of the minority class as in medical diag-nosis. The use of information about outlier and Fractal Dimen-sion features improves the system performance.

Feature selection for automatic analysis of emotional response based on nonlinear speech modeling suitable for diagnosis of Alzheimer׳s disease

Alzheimer׳s disease (AD) is the most common type of dementia among the elderly. This work is part of a larger study that aims to identify novel technologies and biomarkers or features for the early detection of AD and its degree of severity. The diagnosis is made by analyzing several biomarkers and conducting a variety of tests (although only a post-mortem examination of the patients’ brain tissue is considered to provide definitive confirmation). Non-invasive intelligent diagnosis techniques would be a very valuable diagnostic aid. This paper concerns the Automatic Analysis of Emotional Response (AAER) in spontaneous speech based on classical and new emotional speech features: Emotional Temperature (ET) and fractal dimension (FD). This is a pre-clinical study aiming to validate tests and biomarkers for future diagnostic use. The method has the great advantage of being non-invasive, low cost, and without any side effects. The AAER shows very promising results for the definition of features useful in the early diagnosis of AD.

A Markov chain analysis of emotional exchange in voice-to-voice communication: Testing for the mimicry hypothesis of emotional contagion

Mimicry is a central plank of the emotional contagion theory; however, it was only tested with facial and postural emotional stimuli. This study explores the existence of mimicry in voice-to-voice communication by analyzing 8,747 sequences of emotional displays between customers and employees in a call-center context. We listened live to 967 telephone inter-actions, registered the sequences of emotional displays, and analyzed them with a Markov chain. We also explored other propositions of emotional contagion theory that were yet to be tested in vocal contexts. Results supported that mimicry is significantly present at all levels. Our findings fill an important gap in the emotional contagion theory; have practical implications regarding voice-to-voice interactions; and open doors for future vocal mimicry research.

An fMRI Study of Emotional engagement in decicion-making

It has been suggested that decisionmaking depends on sensitive feelings associatedwith cognitive processing rather than cognitiveprocessing alone. From human lesions, we knowthe medial anterior inferior-ventral prefrontalcortex processes the sensitivity associated withcognitive processing, it being essentiallyresponsible for decision making.In this fMRI (functional Magnetic ResonanceImage) study 15 subjects were analyzed usingmoral dilemmas as probes to investigate the neuralbasis for painful-emotional sensitivity associatedwith decision making. We found that a networkcomprising the posterior and anterior cingulateand the medial anterior prefrontal cortex wassignificantly and specifically activated by painfulmoral dilemmas, but not by non-painful dilemmas.These findings provide new evidence that thecingulate and medial anterior prefrontal areinvolved in processing painful emotionalsensibility, in particular, when decision makingtakes place. We speculate that decision makinghas a cognitive component processed by cognitivebrain areas and a sensitivity component processedby emotional brain areas. The structures activatedsuggest that decision making depends on painfulemotional feeling processing rather than cognitiveprocessing when painful feeling processinghappens

The factorial structure of the 41-item version of the Screen for Child Anxiety Related Emotional Disorders (SCARED) in a Spanish population of 8 to 12 years-old

On this instrumental study we intend to analyse the factorial structure of the Screen for Child Anxiety Related Emotional Disorders (SCARED) in a Spanish sample using exploratory and confirmatory factorial analysis. As a second objective we intend to develop a short form of it for rapid screening and, finally, to analyze the reliabilities of both questionnaires. The SCARED was administered to a community sample of 1,508 children aged between 8 and 12 years. The sample was randomly split using half for the exploratory analysis and the other half for the confirmatory study. Furthermore a reduced version of the SCARED was developed using the SchmidLeiman procedure. Exploratory Factor Analysis yielded a four factor structure comprised of Somatic/panic, Generalized anxiety, Separation anxiety and Social phobia factors This structure was confirmed using Confirmatory Factor Analysis. The four factors, the full scale and the short scale showed good reliabilities. The results obtained seem to indicate that the Spanish version of the SCARED has good internal consistency, and along with other recent results, has a structure of four related factors that replicates the dimensions proposed for anxiety disorders by the DSM-IV-TR