Impact of cognitive load and frustration on drivers’ speech [Abstract]

Secondary tasks such as cell phone calls or interaction with automated speech dialog systems (SDSs) increase the driver’s cognitive load as well as the probability of driving errors. This study analyzes speech production variations due to cognitive load and emotional state of drivers in real driving conditions. Speech samples were acquired from 24 female and 17 male subjects (approximately 8.5 h of data) while talking to a co-driver and communicating with two automated call centers, with emotional states (neutral, negative) and the number of necessary SDS query repetitions also labeled. A consistent shift in a number of speech production parameters (pitch, first format center frequency, spectral center of gravity, spectral energy spread, and duration of voiced segments) was observed when comparing SDS interaction against co-driver interaction; further increases were observed when considering negative emotion segments and the number of requested SDS query repetitions. A mel frequency cepstral coefficient based Gaussian mixture classifier trained on 10 male and 10 female sessions provided 91% accuracy in the open test set task of distinguishing co-driver interactions from SDS interactions, suggesting—together with the acoustic analysis—that it is possible to monitor the level of driver distraction directly from their speech.

Analysis and detection of cognitive load and frustration in drivers' speech

Non-driving related cognitive load and variations of emotional state may impact a driver’s capability to control a vehicle and introduces driving errors. Availability of reliable cognitive load and emotion detection in drivers would benefit the design of active safety systems and other intelligent in-vehicle interfaces. In this study, speech produced by 68 subjects while driving in urban areas is analyzed. A particular focus is on speech production differences in two secondary cognitive tasks, interactions with a co-driver and calls to automated spoken dialog systems (SDS), and two emotional states during the SDS interactions - neutral/negative. A number of speech parameters are found to vary across the cognitive/emotion classes. Suitability of selected cepstral- and production-based features for automatic cognitive task/emotion classification is investigated. A fusion of GMM/SVM classifiers yields an accuracy of 94.3% in cognitive task and 81.3% in emotion classification.

The subjective experience of negative symptoms : characteristics of emotional withdrawal

According to the diagnosis of schizophrenia in the DSM-IV-TR (American Psychiatric Association, 2000), negative symptoms are those personal characteristics that are thought to be reduced from normal functioning, while positive symptoms are aspects of functioning that exist as an excess or distortion of normal functioning. Negative symptoms are generally considered to be a core feature of people diagnosed with schizophrenia. However, negative symptoms are not always present in those diagnosed, and a diagnosis can be made with only negative or only positive symptoms, or with a combination of both. Negative symptoms include an observed loss of emotional expression (affective flattening), loss of motivation or self directedness (avolition), loss of speech (alogia), and also a loss of interests and pleasures (anhedonia). Positive symptoms include the perception of things that others do not perceive (hallucinations), and extraordinary explanations for ordinary events (delusions) (American Psychiatric Association, 2000). Both negative and positive symptoms are derived from watching the patient and thus do not consider the patient’s subjective experience. However, aspects of negative symptoms, such as observed affective flattening are highly contended. Within conventional psychiatry, the absence of emotional expression is assumed to coincide with an absence of emotional experience. Contrasting research findings suggests that patients who were observed to score low on displayed emotional expression, scored high on self ratings of emotional experience. Patients were also observed to be significantly lower on emotional expression when compared with others (Aghevli, Blanchard, & Horan, 2003; Selton, van der Bosch, & Sijben, 1998). It appears that there is little correlation between emotional experience and emotional expression in patients, and that observer ratings cannot help us to understand the subjective experience of the negative symptoms. This chapter will focus on research into the subjective experiences of negative symptoms. A framework for these experiences will be used from the qualitative research findings of the primary author (Le Lievre, 2010). In this study, the primary author found that subjective experiences of the negative symptoms belonged to one of the two phases of the illness experience; “transitioning into emotional shutdown” or “recovering from emotional shutdown”. This chapter will use the six themes from the phase of “transitioning into emotional shutdown”. This phase described the experience of turning the focus of attention away from the world and onto the self and the past, thus losing contact with the world and others (emotional shutdown). Transitioning into emotional shutdown involved; “not being acknowledged”, “relational confusion”, “not being expressive”, “reliving the past”, “detachment”, and “no sense of direction” (Le Lievre, 2010). Detail will be added to this framework of experience from other qualitative research in this area. We will now review the six themes that constitute a “transition into emotional shutdown” and corresponding previous research findings.

On the Selection of Non-Invasive Methods Based on Speech Analysis Oriented to Automatic Alzheimer Disease Diagnosis

The work presented here is part of a larger study to identify novel technologies and biomarkers for early Alzheimer disease (AD) detection and it focuses on evaluating the suitability of a new approach for early AD diagnosis by non-invasive methods. The purpose is to examine in a pilot study the potential of applying intelligent algorithms to speech features obtained from suspected patients in order to contribute to the improvement of diagnosis of AD and its degree of severity. In this sense, Artificial Neural Networks (ANN) have been used for the automatic classification of the two classes (AD and control subjects). Two human issues have been analyzed for feature selection: Spontaneous Speech and Emotional Response. Not only linear features but also non-linear ones, such as Fractal Dimension, have been explored. The approach is non invasive, low cost and without any side effects. Obtained experimental results were very satisfactory and promising for early diagnosis and classification of AD patients.

Intonation modelling and adaptation for emotional prosody generation

This paper proposes an HMM-based approach to generating emotional intonation patterns. A set of models were built to represent syllable-length intonation units. In a classification framework, the models were able to detect a sequence of intonation units from raw fundamental frequency values. Using the models in a generative framework, we were able to synthesize smooth and natural sounding pitch contours. As a case study for emotional intonation generation, Maximum Likelihood Linear Regression (MLLR) adaptation was used to transform the neutral model parameters with a small amount of happy and sad speech data. Perceptual tests showed that listeners could identify the speech with the sad intonation 80% of the time. On the other hand, listeners formed a bimodal distribution in their ability to detect the system generated happy intontation and on average listeners were able to detect happy intonation only 46% of the time. © Springer-Verlag Berlin Heidelberg 2005.

Major and minor music compared to excited and subdued speech.

The affective impact of music arises from a variety of factors, including intensity, tempo, rhythm, and tonal relationships. The emotional coloring evoked by intensity, tempo, and rhythm appears to arise from association with the characteristics of human behavior in the corresponding condition; however, how and why particular tonal relationships in music convey distinct emotional effects are not clear. The hypothesis examined here is that major and minor tone collections elicit different affective reactions because their spectra are similar to the spectra of voiced speech uttered in different emotional states. To evaluate this possibility the spectra of the intervals that distinguish major and minor music were compared to the spectra of voiced segments in excited and subdued speech using fundamental frequency and frequency ratios as measures. Consistent with the hypothesis, the spectra of major intervals are more similar to spectra found in excited speech, whereas the spectra of particular minor intervals are more similar to the spectra of subdued speech. These results suggest that the characteristic affective impact of major and minor tone collections arises from associations routinely made between particular musical intervals and voiced speech.

Towards Real-time Speech Emotion Recognition for Affective E-Learning

The original article is available as an open access file on the Springer website in the following link: http://link.springer.com/article/10.1007/s10639-015-9388-2

A socio friendly approach to the analysis of emotive speech

This paper describes certain findings of intonation and intensity study of emotive speech with the minimal use of signal processing algorithms. This study was based on six basic emotions and the neutral, elicited from 1660 English utterances obtained from the speech recordings of six Indian women. The correctness of the emotional content was verified through perceptual listening tests. Marked similarity was noted among pitch contours of like-worded, positive valence emotions, though no such similarity was observed among the four negative valence emotional expressions. The intensity patterns were also studied. The results of the study were validated using arbitrary television recordings for four emotions. The findings are useful to technical researchers, social psychologists and to the common man interested in the dynamics of vocal expression of emotions

Neural response to emotional prosody in schizophrenia and in bipolar affective disorder

Background Evidence suggests a reversal of the normal left-lateralised response to speech in schizophrenia. Aims To test the brain's response to emotional prosody in schizophrenia and bipolar disorder. Method BOLD contrast functional magnetic resonance imaging of subjects while they passively listened or attended to sentences that differed in emotional prosody Results Patients with schizophrenia exhibited normal right-lateralisation of the passive response to 'pure' emotional prosody and relative left-lateralisation of the response to unfiltered emotional prosody When attending to emotional prosody, patients with schizophrenia activated the left insula more than healthy controls. When listening passively, patients with bipolar disorder demonstrated less activation of the bilateral superior temporal gyri in response to pure emotional prosody, and greater activation of the left superior temporal gyrus in response to unfiltered emotional prosody In both passive experiments, the patient groups activated different lateral temporal lobe regions. Conclusions Patients with schizophrenia and bipolar disorder may display some left-lateralisation of the normal right-lateralised temporal lobe response to emotional prosody. Declaration of interest R.M. received a studentship from Neuraxis,, and funding from the Neuroscience and Psychiatry Unit, University of Manchester.

The neural response to emotional prosody, as revealed by functional magnetic resonance imaging

Prosody is an important feature of language, comprising intonation, loudness, and tempo. Emotional prosodic processing forms an integral part of our social interactions. The main aim of this study was to use bold contrast fMRI to clarify the normal functional neuroanatomy of emotional prosody, in passive and active contexts. Subjects performed six separate scanning studies, within which two different conditions were contrasted: (1) "pure" emotional prosody versus rest; (2) congruent emotional prosody versus 'neutral' sentences; (3) congruent emotional prosody versus rest; (4) incongruent emotional prosody versus rest; (5) congruent versus incongruent emotional prosody; and (6) an active experiment in which subjects were instructed to either attend to the emotion conveyed by semantic content or that conveyed by tone of voice. Data resulting from these contrasts were analysed using SPM99. Passive listening to emotional prosody consistently activated the lateral temporal lobe (superior and/or middle temporal gyri). This temporal lobe response was relatively right-lateralised with or without semantic information. Both the separate and direct comparisons of congruent and incongruent emotional prosody revealed that subjects used fewer brain regions to process incongruent emotional prosody than congruent. The neural response to attention to semantics, was left lateralised, and recruited an extensive network not activated by attention to emotional prosody. Attention to emotional prosody modulated the response to speech, and induced right-lateralised activity, including the middle temporal gyrus. In confirming the results of lesion and neuropsychological studies, the current study emphasises the importance of the right hemisphere in the processing of emotional prosody, specifically the lateral temporal lobes. (C) 2003 Elsevier Science Ltd. All rights reserved.

Differentiation in Male and Female Speech Styles

From the moment of their birth, a person's life is determined by their sex. Ms. Goroshko wants to know why this difference is so striking, why society is so concerned to sustain it, and how it is able to persist even when certain national or behavioural stereotypes are erased between people. She is convinced of the existence of not only social, but biological differences between men and women, and set herself the task, in a manuscript totalling 126 pages, written in Ukrainian and including extensive illustrations, of analysing these distinctions as they are manifested in language. She points out that, even before 1900, certain stylistic differences between the ways that men and women speak had been noted. Since then it has become possible, for instance in the case of Japanese, to point to examples of male and female sub-languages. In general, one can single out the following characteristics. Males tend to write with less fluency, to refer to events in a verb-phrase, to be time-oriented, to involve themselves more in their references to events, to locate events in their personal sphere of activity, and to refer less to others. Therefore, concludes Ms Goroshko, the male is shown to be more active, more ego-involved in what he does, and less concerned about others. Women, in contrast, were more fluent, referred to events in a noun-phrase, were less time-oriented, tended to be less involved in their event-references, locate events within their interactive community and refer more to others. They spent much more time discussing personal and domestic subjects, relationship problems, family, health and reproductive matters, weight, food and clothing, men, and other women. As regards discourse strategies, Ms Goroshko notes the following. Men more often begin a conversation, they make more utterances, these utterances are longer, they make more assertions, speak less carefully, generally determine the topic of conversation, speak more impersonally, use more vulgar expressions, and use fewer diminutives and more imperatives. Women's speech strategies, apart from being the opposite of those enumerated above, also contain more euphemisms, polite forms, apologies, laughter and crying. All of the above leads Ms. Goroshko to conclude that the differences between male and female speech forms are more striking than the similarities. Furthermore she is convinced that the biological divergence between the sexes is what generates the verbal divergence, and that social factors can only intensify or diminish the differentiation in verbal behaviour established by the sex of a person. Bearing all this in mind, Ms Goroshko set out to construct a grammar of male and female styles of speaking within Russian. One of her most important research tools was a certain type of free association test. She took a list comprising twelve stimuli (to love, to have, to speak, to fuck, a man, a woman, a child, the sky, a prayer, green, beautiful) and gave it to a group of participants specially selected, according to a preliminary psychological testing, for the high levels of masculinity or femininity they displayed. Preliminary responses revealed that the female reactions were more diverse than the male ones, there were more sentences and word combinations in the female reactions, men gave more negative responses to the stimulus and sometimes didn't want to react at all, women reacted more to adjectives and men to nouns, and that, surprisingly, women coloured more negatively their reactions to the words man, to love and a child (Ms. Goroshko is inclined to attribute this to the present economic situation in Russia). Another test performed by Ms. Goroshko was the so-called "defective text" developed by A.A. Brudny. All participants were distributed with packets of complete sentences, which had been taken from a text and then mixed at random. The task was to reconstruct the original text. There were three types of test, the first descriptive, the second narrative, and the third logical. Ms. Goroshko created computer programmes to analyse the results. She found that none of the reconstructed texts was coincident with the original, differing both from the original text and amongst themselves and that there were many more disparities in the male than the female texts. In the descriptive and logical texts the differences manifested themselves more clearly in the male texts, and in the narrative texts in the female texts. The widest dispersal of values was observed at the outset, while the female text ending was practically coincident with the original (in contrast to the male ending). The greatest differences in text reconstruction for both males and females were registered in the middle of the texts. Women, Ms. Goroshko claims, were more sensitive to the semantic structure of the texts, since they assembled the narrative text much more accurately than the other two, while the men assembled more accurately the logical text. Texts written by women were assembled more accurately by women and texts by men by men. On the basis of computer analysis, Ms. Goroshko found that female speech was substantially more emotional. It was expressed by various means, hyperbole, metaphor, comparisons, epithets, ways of enumeration, and with the aid of interjections, rhetorical questions, exclamations. The level of literacy was higher for female speech, and there were fewer mistakes in grammar and spelling in female texts. The last stage of Ms Goroshko's research concerned the social stereotypes of beliefs about men and women in Russian society today. A large number of respondents were asked questions such as "What merits must a woman possess?", "What are male vices and virtues?", etc. After statistical manipulation, an image of modern man and woman, as it exists in the minds of modern Russian men and women, emerged. Ms. Goroshko believes that her findings are significant not only within the field of linguistics. She has already successfully worked on anonymous texts and been able to decide on the sex of the author and consequently believes that in the future her research may even be of benefit to forensic science.

Affective and linguistic processing of speech prosody: DC potential studies

Speech melody or prosody subserves linguistic, emotional, and pragmatic functions in speech communication. Prosodic perception is based on the decoding of acoustic cues with a predominant function of frequency-related information perceived as speaker's pitch. Evaluation of prosodic meaning is a cognitive function implemented in cortical and subcortical networks that generate continuously updated affective or linguistic speaker impressions. Various brain-imaging methods allow delineation of neural structures involved in prosody processing. In contrast to functional magnetic resonance imaging techniques, DC (direct current, slow) components of the EEG directly measure cortical activation without temporal delay. Activation patterns obtained with this method are highly task specific and intraindividually reproducible. Studies presented here investigated the topography of prosodic stimulus processing in dependence on acoustic stimulus structure and linguistic or affective task demands, respectively. Data obtained from measuring DC potentials demonstrated that the right hemisphere has a predominant role in processing emotions from the tone of voice, irrespective of emotional valence. However, right hemisphere involvement is modulated by diverse speech and language-related conditions that are associated with a left hemisphere participation in prosody processing. The degree of left hemisphere involvement depends on several factors such as (i) articulatory demands on the perceiver of prosody (possibly, also the poser), (ii) a relative left hemisphere specialization in processing temporal cues mediating prosodic meaning, and (iii) the propensity of prosody to act on the segment level in order to modulate word or sentence meaning. The specific role of top-down effects in terms of either linguistically or affectively oriented attention on lateralization of stimulus processing is not clear and requires further investigations.

Unattended emotional intonations modulate linguistic prosody processing

Prosody or speech melody subserves linguistic (e.g., question intonation) and emotional functions in speech communication. Findings from lesion studies and imaging experiments suggest that, depending on function or acoustic stimulus structure, prosodic speech components are differentially processed in the right and left hemispheres. This direct current (DC) potential study investigated the linguistic processing of digitally manipulated pitch contours of sentences that carried an emotional or neutral intonation. Discrimination of linguistic prosody was better for neutral stimuli as compared to happily as well as fearfully spoken sentences. Brain activation was increased during the processing of happy sentences as compared to neutral utterances. Neither neutral nor emotional stimuli evoked lateralized processing in the left or right hemisphere, indicating bilateral mechanisms of linguistic processing for pitch direction. Acoustic stimulus analysis suggested that prosodic components related to emotional intonation, such as pitch variability, interfered with linguistic processing of pitch course direction.

Formal speech stylistics and type A behavior in 38 subjects during nonstress interviews

Open-ended interviews of 90 min length of 38 patients were analyzed with respect to speech stylistics, shown by Schucker and Jacobs to differentiate individuals with type A personality features from those with type B. In our patients, Type A/B had been assessed by the Bortner Personality Inventory. The stylistics studied were: repeated words swallowed words, interruptions, simultaneous speech, silence latency (between question and answer) (SL), speed of speech, uneven speed of speech (USS), explosive words (PW), uneven speech volume (USV), and speech volume. Correlations between both raters for all speech categories were high. Positive correlations between extent of type A and SL (r = 0.33; p = 0.022), USS (r = 0.51; p = 0.002), PW (r = 0.46; p = 0.003) and USV (r = 0.39; p = 0.012) were found. Our results indicate that the speech in nonstress open-ended interviews of type A individuals tends to show a higher emotional tension (positive correlations for USS PW and USV) and is more controlled in conversation (positive correlation for SL).