Auditory-visual multisensory interactions in humans: timing, topography, directionality, and sources.

Current models of brain organization include multisensory interactions at early processing stages and within low-level, including primary, cortices. Embracing this model with regard to auditory-visual (AV) interactions in humans remains problematic. Controversy surrounds the application of an additive model to the analysis of event-related potentials (ERPs), and conventional ERP analysis methods have yielded discordant latencies of effects and permitted limited neurophysiologic interpretability. While hemodynamic imaging and transcranial magnetic stimulation studies provide general support for the above model, the precise timing, superadditive/subadditive directionality, topographic stability, and sources remain unresolved. We recorded ERPs in humans to attended, but task-irrelevant stimuli that did not require an overt motor response, thereby circumventing paradigmatic caveats. We applied novel ERP signal analysis methods to provide details concerning the likely bases of AV interactions. First, nonlinear interactions occur at 60-95 ms after stimulus and are the consequence of topographic, rather than pure strength, modulations in the ERP. AV stimuli engage distinct configurations of intracranial generators, rather than simply modulating the amplitude of unisensory responses. Second, source estimations (and statistical analyses thereof) identified primary visual, primary auditory, and posterior superior temporal regions as mediating these effects. Finally, scalar values of current densities in all of these regions exhibited functionally coupled, subadditive nonlinear effects, a pattern increasingly consistent with the mounting evidence in nonhuman primates. In these ways, we demonstrate how neurophysiologic bases of multisensory interactions can be noninvasively identified in humans, allowing for a synthesis across imaging methods on the one hand and species on the other.

Organizing principles across the senses

ABSTRACT This thesis is composed of two main parts. The first addressed the question of whether the auditory and somatosensory systems, like their visual counterpart, comprise parallel functional pathways for processing identity and spatial attributes (so-called `what' and `where' pathways, respectively). The second part examined the independence of control processes mediating task switching across 'what' and `where' pathways in the auditory and visual modalities. Concerning the first part, electrical neuroimaging of event-related potentials identified the spatio-temporal mechanisms subserving auditory (see Appendix, Study n°1) and vibrotactile (see Appendix, Study n°2) processing during two types of blocks of trials. `What' blocks varied stimuli in their frequency independently of their location.. `Where' blocks varied the same stimuli in their location independently of their frequency. Concerning the second part (see Appendix, Study n°3), a psychophysical task-switching paradigm was used to investigate the hypothesis that the efficacy of control processes depends on the extent of overlap between the neural circuitry mediating the different tasks at hand, such that more effective task preparation (and by extension smaller switch costs) is achieved when the anatomical/functional overlap of this circuitry is small. Performance costs associated with switching tasks and/or switching sensory modalities were measured. Tasks required the analysis of either the identity or spatial location of environmental objects (`what' and `where' tasks, respectively) that were presented either visually or acoustically on any given trial. Pretrial cues informed participants of the upcoming task, but not of the sensory modality. - In the audio-visual domain, the results showed that switch costs between tasks were significantly smaller when the sensory modality of the task switched versus when it repeated. In addition, switch costs between the senses were correlated only when the sensory modality of the task repeated across trials and not when it switched. The collective evidence not only supports the independence of control processes mediating task switching and modality switching, but also the hypothesis that switch costs reflect competitive interterence between neural circuits that in turn can be diminished when these neural circuits are distinct. - In the auditory and somatosensory domains, the findings show that a segregation of location vs. recognition information is observed across sensory systems and that these happen around 100ms for both sensory modalities. - Also, our results show that functionally specialized pathways for audition and somatosensation involve largely overlapping brain regions, i.e. posterior superior and middle temporal cortices and inferior parietal areas. Both these properties (synchrony of differential processing and overlapping brain regions) probably optimize the relationships across sensory modalities. - Therefore, these results may be indicative of a computationally advantageous organization for processing spatial anal identity information.

Spontaneous pre-stimulus fluctuations in the activity of right fronto-parietal areas influence inhibitory control performance.

Inhibitory control refers to the ability to suppress planned or ongoing cognitive or motor processes. Electrophysiological indices of inhibitory control failure have been found to manifest even before the presentation of the stimuli triggering the inhibition, suggesting that pre-stimulus brain-states modulate inhibition performance. However, previous electrophysiological investigations on the state-dependency of inhibitory control were based on averaged event-related potentials (ERPs), a method eliminating the variability in the ongoing brain activity not time-locked to the event of interest. These studies thus left unresolved whether spontaneous variations in the brain-state immediately preceding unpredictable inhibition-triggering stimuli also influence inhibitory control performance. To address this question, we applied single-trial EEG topographic analyses on the time interval immediately preceding NoGo stimuli in conditions where the responses to NoGo trials were correctly inhibited [correct rejection (CR)] vs. committed [false alarms (FAs)] during an auditory spatial Go/NoGo task. We found a specific configuration of the EEG voltage field manifesting more frequently before correctly inhibited responses to NoGo stimuli than before FAs. There was no evidence for an EEG topography occurring more frequently before FAs than before CR. The visualization of distributed electrical source estimations of the EEG topography preceding successful response inhibition suggested that it resulted from the activity of a right fronto-parietal brain network. Our results suggest that the fluctuations in the ongoing brain activity immediately preceding stimulus presentation contribute to the behavioral outcomes during an inhibitory control task. Our results further suggest that the state-dependency of sensory-cognitive processing might not only concern perceptual processes, but also high-order, top-down inhibitory control mechanisms.

Effect of Emotion and Personality on Deviation from Purely Rational Decision-Making

Human decision-making has consistently demonstrated deviation from "pure" rationality. Emotions are a primary driver of human actions and the current study investigates how perceived emotions and personality traits may affect decision-making during the Ultimatum Game (UG). We manipulated emotions by showing images with emotional connotation while participants decided how to split money with a second player. Event-related potentials (ERPs) from scalp electrodes were recorded during the whole decision-making process. We observed significant differences in the activity of central and frontal areas when participants offered money with respect to when they accepted or rejected an offer. We found that participants were more likely to offer a higher amount of money when making their decision in association with negative emotions. Furthermore, participants were more likely to accept offers when making their decision in association with positive emotions. Honest, conscientious, and introverted participants were more likely to accept offers. Our results suggest that factors others than a rational strategy may predict economic decision-making in the UG.

Early attentional processes distinguish selective from global motor inhibitory control : An electrical neuroimaging study

The rapid stopping of specific parts of movements is frequently required in daily life. Yet, whether selective inhibitory control of movements is mediated by a specific neural pathway or by the combination between a global stopping of all ongoing motor activity followed by the re-initiation of task-relevant movements remains unclear. To address this question, we applied time-wise statistical analyses of the topography, global field power and electrical sources of the event-related potentials to the global vs selective inhibition stimuli presented during a Go/NoGo task. Participants (n = 18) had to respond as fast as possible with their two hands to Go stimuli and to withhold the response from the two hands (global inhibition condition, GNG) or from only one hand (selective inhibition condition, SNG) when specific NoGo stimuli were presented. Behaviorally, we replicated previous evidence for slower response times in the SNG than in the Go condition. Electrophysiologically, there were two distinct phases of event-related potentials modulations between the GNG and the SNG conditions. At 110âeuro"150 ms post-stimulus onset, there was a difference in the strength of the electric field without concomitant topographic modulation, indicating the differential engagement of statistically indistinguishable configurations of neural generators for selective and global inhibitory control. At 150âeuro"200 ms, there was topographic modulation, indicating the engagement of distinct brain networks. Source estimations localized these effects within bilateral temporo-parieto-occipital and within parieto-central networks, respectively. Our results suggest that while both types of motor inhibitory control depend on global stopping mechanisms, selective and global inhibition still differ quantitatively at early attention-related processing phases.

Context- and Prosody-Driven ERP Markers for Dialog Focus Perception in Children.

The development of language proficiency extends late into childhood and includes not only producing or comprehending sounds, words and sentences, but likewise larger utterances spanning beyond sentence borders like dialogs. Dialogs consist of information units whose value constantly varies within a verbal exchange. While information is focused when introduced for the first time or corrected in order to alter the knowledge state of communication partners, the same information turns into shared knowledge during the further course of a verbal exchange. In many languages, prosodic means are used by speakers to highlight the informational value of information foci. Our study investigated the developmental pattern of event-related potentials (ERPs) in three age groups (12, 8 and 5 years) when perceiving two information focus types (news and corrections) embedded in short question-answer dialogs. The information foci contained in the answer sentences were either adequately marked by prosodic means or not. In so doing, we questioned to what extent children depend on prosodic means to recognize information foci or whether contextual means as provided by dialog questions are sufficient to guide focus processing.Only 12-year-olds yield prosody-independent ERPs when encountering new and corrective information foci, resembling previous findings in adults. Focus processing in the 8-year-olds relied upon prosodic highlighting, and differing ERP responses as a function of focus type were observed. In the 5-year-olds, merely prosody-driven ERP responses were apparent, but no distinctive ERP indicating information focus recognition. Our findings reveal substantial alterations in information focus perception throughout childhood that are likely related to long-lasting maturational changes during brain development.

Ventral and dorsal pathways of speech perception: An intracerebral ERP study.

Recent theory of physiology of language suggests a dual stream dorsal/ventral organization of speech perception. Using intra-cerebral Event-related potentials (ERPs) during pre-surgical assessment of twelve drug-resistant epileptic patients, we aimed to single out electrophysiological patterns during both lexical-semantic and phonological monitoring tasks involving ventral and dorsal regions respectively. Phonological information processing predominantly occurred in the left supra-marginal gyrus (dorsal stream) and lexico-semantic information occurred in anterior/middle temporal and fusiform gyri (ventral stream). Similar latencies were identified in response to phonological and lexico-semantic tasks, suggesting parallel processing. Typical ERP components were strongly left lateralized since no evoked responses were recorded in homologous right structures. Finally, ERP patterns suggested the inferior frontal gyrus as the likely final common pathway of both dorsal and ventral streams. These results brought out detailed evidence of the spatial-temporal information processing in the dual pathways involved in speech perception.

Different colors of light lead to different adaptation and activation as determined by high-density EEG.

Light adaptation is crucial for coping with the varying levels of ambient light. Using high-density electroencephalography (EEG), we investigated how adaptation to light of different colors affects brain responsiveness. In a within-subject design, sixteen young participants were adapted first to dim white light and then to blue, green, red, or white bright light (one color per session in a randomized order). Immediately after both dim and bright light adaptation, we presented brief light pulses and recorded event-related potentials (ERPs). We analyzed ERP response strengths and brain topographies and determined the underlying sources using electrical source imaging. Between 150 and 261ms after stimulus onset, the global field power (GFP) was higher after dim than bright light adaptation. This effect was most pronounced with red light and localized in the frontal lobe, the fusiform gyrus, the occipital lobe and the cerebellum. After bright light adaptation, within the first 100ms after light onset, stronger responses were found than after dim light adaptation for all colors except for red light. Differences between conditions were localized in the frontal lobe, the cingulate gyrus, and the cerebellum. These results indicate that very short-term EEG brain responses are influenced by prior light adaptation and the spectral quality of the light stimulus. We show that the early EEG responses are differently affected by adaptation to different colors of light which may contribute to known differences in performance and reaction times in cognitive tests.

Topographic ERP analyses: a step-by-step tutorial review

In this tutorial review, we detail both the rationale for as well as the implementation of a set of analyses of surface-recorded event-related potentials (ERPs) that uses the reference-free spatial (i.e. topographic) information available from high-density electrode montages to render statistical information concerning modulations in response strength, latency, and topography both between and within experimental conditions. In these and other ways these topographic analysis methods allow the experimenter to glean additional information and neurophysiologic interpretability beyond what is available from canonical waveform analyses. In this tutorial we present the example of somatosensory evoked potentials (SEPs) in response to stimulation of each hand to illustrate these points. For each step of these analyses, we provide the reader with both a conceptual and mathematical description of how the analysis is carried out, what it yields, and how to interpret its statistical outcome. We show that these topographic analysis methods are intuitive and easy-to-use approaches that can remove much of the guesswork often confronting ERP researchers and also assist in identifying the information contained within high-density ERP datasets

Plastic modifications within inhibitory control networks induced by practicing a stop-signal task: an electrical neuroimaging study.

INTRODUCTION: Inhibitory control refers to our ability to suppress ongoing motor, affective or cognitive processes and mostly depends on a fronto-basal brain network. Inhibitory control deficits participate in the emergence of several prominent psychiatric conditions, including attention deficit/hyperactivity disorder or addiction. The rehabilitation of these pathologies might therefore benefit from training-based behavioral interventions aiming at improving inhibitory control proficiency and normalizing the underlying neurophysiological mechanisms. The development of an efficient inhibitory control training regimen first requires determining the effects of practicing inhibition tasks. METHODS: We addressed this question by contrasting behavioral performance and electrical neuroimaging analyses of event-related potentials (ERPs) recorded from humans at the beginning versus the end of 1 h of practice on a stop-signal task (SST) involving the withholding of responses when a stop signal was presented during a speeded auditory discrimination task. RESULTS: Practicing a short SST improved behavioral performance. Electrophysiologically, ERPs differed topographically at 200 msec post-stimulus onset, indicative of the engagement of distinct brain network with learning. Source estimations localized this effect within the inferior frontal gyrus, the pre-supplementary motor area and the basal ganglia. CONCLUSION: Our collective results indicate that behavioral and brain responses during an inhibitory control task are subject to fast plastic changes and provide evidence that high-order fronto-basal executive networks can be modified by practicing a SST.

Comparing ICA-based and single-trial topographic ERP analyses.

Single-trial analysis of human electroencephalography (EEG) has been recently proposed for better understanding the contribution of individual subjects to a group-analysis effect as well as for investigating single-subject mechanisms. Independent Component Analysis (ICA) has been repeatedly applied to concatenated single-trial responses and at a single-subject level in order to extract those components that resemble activities of interest. More recently we have proposed a single-trial method based on topographic maps that determines which voltage configurations are reliably observed at the event-related potential (ERP) level taking advantage of repetitions across trials. Here, we investigated the correspondence between the maps obtained by ICA versus the topographies that we obtained by the single-trial clustering algorithm that best explained the variance of the ERP. To do this, we used exemplar data provided from the EEGLAB website that are based on a dataset from a visual target detection task. We show there to be robust correspondence both at the level of the activation time courses and at the level of voltage configurations of a subset of relevant maps. We additionally show the estimated inverse solution (based on low-resolution electromagnetic tomography) of two corresponding maps occurring at approximately 300 ms post-stimulus onset, as estimated by the two aforementioned approaches. The spatial distribution of the estimated sources significantly correlated and had in common a right parietal activation within Brodmann's Area (BA) 40. Despite their differences in terms of theoretical bases, the consistency between the results of these two approaches shows that their underlying assumptions are indeed compatible.

A Threat to a Virtual Hand Elicits Motor Cortex Activation

We report an experiment where participants observed an attack on their virtual body as experienced in an immersive virtual reality (IVR) system. Participants sat by a table with their right hand resting upon it. In IVR, they saw a virtual table that was registered with the real one, and they had a virtual body that substituted their real body seen from a first person perspective. The virtual right hand was collocated with their real right hand. Event-related brain potentials were recorded in two conditions, one where the participant"s virtual hand was attacked with a knife and a control condition where the knife only struck the virtual table. Significantly greater P450 potentials were obtained in the attack condition confirming our expectations that participants had a strong illusion of the virtual hand being their own, which was also strongly supported by questionnaire responses. Higher levels of subjective virtual hand ownership correlated with larger P450 amplitudes. Mu-rhythm event-related desynchronization in the motor cortex and readiness potential (C3C4) negativity were clearly observed when the virtual hand was threatened as would be expected, if the real hand was threatened and the participant tried to avoid harm. Our results support the idea that event-related potentials may provide a promising non-subjective measure of virtual embodiment. They also support previous experiments on pain observation and are placed into context of similar experiments and studies of body perception and body ownership within cognitive neuroscience.

Normative Topographic ERP Analyses of Speed of Speech Processing and Grammar Before and After Grammatical Treatment.

Typically developing (TD) preschoolers and age-matched preschoolers with specific language impairment (SLI) received event-related potentials (ERPs) to four monosyllabic speech sounds prior to treatment and, in the SLI group, after 6 months of grammatical treatment. Before treatment, the TD group processed speech sounds faster than the SLI group. The SLI group increased the speed of their speech processing after treatment. Posttreatment speed of speech processing predicted later impairment in comprehending phrase elaboration in the SLI group. During the treatment phase, change in speed of speech processing predicted growth rate of grammar in the SLI group.

Early, low-level auditory-somatosensory multisensory interactions impact reaction time speed.

Several lines of research have documented early-latency non-linear response interactions between audition and touch in humans and non-human primates. That these effects have been obtained under anesthesia, passive stimulation, as well as speeded reaction time tasks would suggest that some multisensory effects are not directly influencing behavioral outcome. We investigated whether the initial non-linear neural response interactions have a direct bearing on the speed of reaction times. Electrical neuroimaging analyses were applied to event-related potentials in response to auditory, somatosensory, or simultaneous auditory-somatosensory multisensory stimulation that were in turn averaged according to trials leading to fast and slow reaction times (using a median split of individual subject data for each experimental condition). Responses to multisensory stimulus pairs were contrasted with each unisensory response as well as summed responses from the constituent unisensory conditions. Behavioral analyses indicated that neural response interactions were only implicated in the case of trials producing fast reaction times, as evidenced by facilitation in excess of probability summation. In agreement, supra-additive non-linear neural response interactions between multisensory and the sum of the constituent unisensory stimuli were evident over the 40-84 ms post-stimulus period only when reaction times were fast, whereas subsequent effects (86-128 ms) were observed independently of reaction time speed. Distributed source estimations further revealed that these earlier effects followed from supra-additive modulation of activity within posterior superior temporal cortices. These results indicate the behavioral relevance of early multisensory phenomena.

Multisensory context portends object memory.

Multisensory processes facilitate perception of currently-presented stimuli and can likewise enhance later object recognition. Memories for objects originally encountered in a multisensory context can be more robust than those for objects encountered in an exclusively visual or auditory context [1], upturning the assumption that memory performance is best when encoding and recognition contexts remain constant [2]. Here, we used event-related potentials (ERPs) to provide the first evidence for direct links between multisensory brain activity at one point in time and subsequent object discrimination abilities. Across two experiments we found that individuals showing a benefit and those impaired during later object discrimination could be predicted by their brain responses to multisensory stimuli upon their initial encounter. These effects were observed despite the multisensory information being meaningless, task-irrelevant, and presented only once. We provide critical insights into the advantages associated with multisensory interactions; they are not limited to the processing of current stimuli, but likewise encompass the ability to determine the benefit of one's memories for object recognition in later, unisensory contexts.