Seeing Spaces: An Eye-Tracking Study Of Speech Segmentation

Speech is often a multimodal process, presented audiovisually through a talking face. One area of speech perception influenced by visual speech is speech segmentation, or the process of breaking a stream of speech into individual words. Mitchel and Weiss (2013) demonstrated that a talking face contains specific cues to word boundaries and that subjects can correctly segment a speech stream when given a silent video of a speaker. The current study expanded upon these results, using an eye tracker to identify highly attended facial features of the audiovisual display used in Mitchel and Weiss (2013). In Experiment 1, subjects were found to spend the most time watching the eyes and mouth, with a trend suggesting that the mouth was viewed more than the eyes. Although subjects displayed significant learning of word boundaries, performance was not correlated with gaze duration on any individual feature, nor was performance correlated with a behavioral measure of autistic-like traits. However, trends suggested that as autistic-like traits increased, gaze duration of the mouth increased and gaze duration of the eyes decreased, similar to significant trends seen in autistic populations (Boratston & Blakemore, 2007). In Experiment 2, the same video was modified so that a black bar covered the eyes or mouth. Both videos elicited learning of word boundaries that was equivalent to that seen in the first experiment. Again, no correlations were found between segmentation performance and SRS scores in either condition. These results, taken with those in Experiment, suggest that neither the eyes nor mouth are critical to speech segmentation and that perhaps more global head movements indicate word boundaries (see Graf, Cosatto, Strom, & Huang, 2002). Future work will elucidate the contribution of individual features relative to global head movements, as well as extend these results to additional types of speech tasks.

Multimodal Integration in Statistical Learning: Evidence from the McGurk Illusion

Recent advances in the field of statistical learning have established that learners are able to track regularities of multimodal stimuli, yet it is unknown whether the statistical computations are performed on integrated representations or on separate, unimodal representations. In the present study, we investigated the ability of adults to integrate audio and visual input during statistical learning. We presented learners with a speech stream synchronized with a video of a speaker's face. In the critical condition, the visual (e.g., /gi/) and auditory (e.g., /mi/) signals were occasionally incongruent, which we predicted would produce the McGurk illusion, resulting in the perception of an audiovisual syllable (e.g., /ni/). In this way, we used the McGurk illusion to manipulate the underlying statistical structure of the speech streams, such that perception of these illusory syllables facilitated participants' ability to segment the speech stream. Our results therefore demonstrate that participants can integrate audio and visual input to perceive the McGurk illusion during statistical learning. We interpret our findings as support for modality-interactive accounts of statistical learning.

Different duration of at-risk mental state associated with neurofunctional abnormalities. A multimodal imaging study

Objectives: Neurofunctional alterations are correlates of vulnerability to psychosis, as well as of the disorder itself. How these abnormalities relate to different probabilities for later transition to psychosis is unclear. We investigated vulnerability- versus disease-related versus resilience biomarkers of psychosis during working memory (WM) processing in individuals with an at-risk mental state (ARMS). Experimental design: Patients with “first-episode psychosis” (FEP, n = 21), short-term ARMS (ARMS-ST, n = 17), long-term ARMS (ARMS-LT, n = 16), and healthy controls (HC, n = 20) were investigated with an n-back WM task. We examined functional magnetic resonance imaging (fMRI) and structural magnetic resonance imaging (sMRI) data in conjunction using biological parametric mapping (BPM) toolbox. Principal observations: There were no differences in accuracy, but the FEP and the ARMS-ST group had longer reaction times compared with the HC and the ARMS-LT group. With the 2-back > 0-back contrast, we found reduced functional activation in ARMS-ST and FEP compared with the HC group in parietal and middle frontal regions. Relative to ARMS-LT individuals, FEP patients showed decreased activation in the bilateral inferior frontal gyrus and insula, and in the left prefrontal cortex. Compared with the ARMS-LT, the ARMS-ST subjects showed reduced activation in the right inferior frontal gyrus and insula. Reduced insular and prefrontal activation was associated with gray matter volume reduction in the same area in the ARMS-LT group. Conclusions: These findings suggest that vulnerability to psychosis was associated with neurofunctional alterations in fronto-temporo-parietal networks in a WM task. Neurofunctional differences within the ARMS were related to different duration of the prodromal state and resilience factors

End-tidal CO(2). An important parameter for a correct interpretation in functional brain studies using speech tasks

The aim was to investigate the effect of different speech tasks, i.e. recitation of prose (PR), alliteration (AR) and hexameter (HR) verses and a control task (mental arithmetic (MA) with voicing of the result on end-tidal CO2 (PETCO2), cerebral hemodynamics and oxygenation. CO2 levels in the blood are known to strongly affect cerebral blood flow. Speech changes breathing pattern and may affect CO2 levels. Measurements were performed on 24 healthy adult volunteers during the performance of the 4 tasks. Tissue oxygen saturation (StO2) and absolute concentrations of oxyhemoglobin ([O2Hb]), deoxyhemoglobin ([HHb]) and total hemoglobin ([tHb]) were measured by functional near-infrared spectroscopy (fNIRS) and PETCO2 by a gas analyzer. Statistical analysis was applied to the difference between baseline before the task, 2 recitation and 5 baseline periods after the task. The 2 brain hemispheres and 4 tasks were tested separately. A significant decrease in PETCO2 was found during all 4 tasks with the smallest decrease during the MA task. During the recitation tasks (PR, AR and HR) a statistically significant (p < 0.05) decrease occurred for StO2 during PR and AR in the right prefrontal cortex (PFC) and during AR and HR in the left PFC. [O2Hb] decreased significantly during PR, AR and HR in both hemispheres. [HHb] increased significantly during the AR task in the right PFC. [tHb] decreased significantly during HR in the right PFC and during PR, AR and HR in the left PFC. During the MA task, StO2 increased and [HHb] decreased significantly during the MA task. We conclude that changes in breathing (hyperventilation) during the tasks led to lower CO2 pressure in the blood (hypocapnia), predominantly responsible for the measured changes in cerebral hemodynamics and oxygenation. In conclusion, our findings demonstrate that PETCO2 should be monitored during functional brain studies investigating speech using neuroimaging modalities, such as fNIRS, fMRI to ensure a correct interpretation of changes in hemodynamics and oxygenation.

Multimodal augmented reality system for surgical microscopy

Image-guided, computer-assisted neurosurgery has emerged to improve localization and targeting, to provide a better anatomic definition of the surgical field, and to decrease invasiveness. Usually, in image-guided surgery, a computer displays the surgical field in a CT/MR environment, using axial, coronal or sagittal views, or even a 3D representation of the patient. Such a system forces the surgeon to look away from the surgical scene to the computer screen. Moreover, this kind of information, being pre-operative imaging, can not be modified during the operation, so it remains valid for guidance in the first stage of the surgical procedure, and mainly for rigid structures like bones. In order to solve the two constraints mentioned before, we are developing an ultrasoundguided surgical microscope. Such a system takes the advantage that surgical microscopy and ultrasound systems are already used in neurosurgery, so it does not add more complexity to the surgical procedure. We have integrated an optical tracking device in the microscope and an augmented reality overlay system with which we avoid the need to look away from the scene, providing correctly aligned surgical images with sub-millimeter accuracy. In addition to the standard CT and 3D views, we are able to track an ultrasound probe, and using a previous calibration and registration of the imaging, the image obtained is correctly projected to the overlay system, so the surgeon can always localize the target and verify the effects of the intervention. Several tests of the system have been already performed to evaluate the accuracy, and clinical experiments are currently in progress in order to validate the clinical usefulness of the system.

Multimodal imaging in macular diagnostics: combined OCT-SLO improves therapeutical monitoring

BACKGROUND: Digital imaging methods are a centrepiece for diagnosis and management of macular disease. A recently developed imaging device is composed of simultaneous confocal scanning laser ophthalmoscopy (SLO) and optical coherence tomography (OCT). By means of clinical samples the benefit of this technique concerning diagnostic and therapeutic follow-up will be assessed. METHODS: The combined OCT-SLO-System (Ophthalmic Technologies Inc., Toronto, Canada) allows for confocal en-face fundus imaging and high resolution OCT scanning at the same time. OCT images are obtained from transversal line scans. One light source and the identical scanning rate yield a pixel-to-pixel correspondence of images. Three-dimensional thickness maps are derived from C-scan stacking. RESULTS: We followed-up patients with cystoid macular edema, pigment epithelium detachment, macular hole, venous branch occlusion, and vitreoretinal tractions during their course of therapy. The new imaging method illustrates the reduction of cystoid volume, e.g. after intravitreal injections of either angiostatic drugs or steroids. C-scans are used for appreciation of lesion diameters, visualisation of pathologies involving the vitreoretinal interface, and quantification of retinal thickness change. CONCLUSION: The combined OCT-SLO system creates both topographic and tomographic images of the retina. New therapeutic options can be followed-up closely by observing changes in lesion thickness and cyst volumes. For clinical use further studies are needed.

Movement control of manipulative tasks in patients with Gilles de la Tourette syndrome

When a hand-held object is moved, grip and load force are accurately coordinated for establishing grasp stability. In the present work, the question was raised whether patients with Gilles de la Tourette syndrome (TS), who show tic-like movements, are impaired in grip-load force control when executing a manipulative task. To this end, we assessed force regulation during action patterns that required rhythmical unimanual or bimanual (iso-directional/anti-directional) movements. Results showed that the profile of grip-load force ratio was characterized by maxima and minima that were realized at upward and downward hand positions, respectively. TS patients showed increased force ratios during unimanual and bimanual movements, compared with control subjects, indicative of an inaccurate specification of the precision grip. Functional imaging data complemented the behavioural results and revealed that secondary motor areas showed no (or greatly reduced) activation in TS patients when executing the movement tasks as compared with baseline conditions. This indicates that the metabolic level in the secondary motor areas was equal during rest and task performance. At the neuronal level, this observation suggests that these cortical areas were continuously involved in movement preparation. Based on these data, we conclude that the ongoing activation of secondary motor areas may be explained by the TS patients' involuntary urges to move. Accordingly, interference will prevent an accurate planning of voluntary behaviour. Together, these findings reveal modulations in movement organization in patients with TS and exemplify degrading consequences for manual function.

Multimodal approach in the use of clinical scoring, morphological MRI and biochemical T2-mapping and diffusion-weighted imaging in their ability to assess differences between cartilage repair tissue after microfracture therapy and matrix-associated autologous chondrocyte transplantation: a pilot study

OBJECTIVE: The aim of the present pilot study is to show initial results of a multimodal approach using clinical scoring, morphological magnetic resonance imaging (MRI) and biochemical T2-relaxation and diffusion-weighted imaging (DWI) in their ability to assess differences between cartilage repair tissue after microfracture therapy (MFX) and matrix-associated autologous chondrocyte transplantation (MACT). METHOD: Twenty patients were cross-sectionally evaluated at different post-operative intervals from 12 to 63 months after MFX and 12-59 months after MACT. The two groups were matched by age (MFX: 36.0+/-10.4 years; MACT: 35.1+/-7.7 years) and post-operative interval (MFX: 32.6+/-16.7 months; MACT: 31.7+/-18.3 months). After clinical evaluation using the Lysholm score, 3T-MRI was performed obtaining the MR observation of cartilage repair tissue (MOCART) score as well as T2-mapping and DWI for multi-parametric MRI. Quantitative T2-relaxation was achieved using a multi-echo spin-echo sequence; semi-quantitative diffusion-quotient (signal intensity without diffusion-weighting divided by signal intensity with diffusion weighting) was prepared by a partially balanced, steady-state gradient-echo pulse sequence. RESULTS: No differences in Lysholm (P=0.420) or MOCART (P=0.209) score were observed between MFX and MACT. T2-mapping showed lower T2 values after MFX compared to MACT (P=0.039). DWI distinguished between healthy cartilage and cartilage repair tissue in both procedures (MFX: P=0.001; MACT: P=0.007). Correlations were found between the Lysholm and the MOCART score (Pearson: 0.484; P=0.031), between the Lysholm score and DWI (Pearson:-0.557; P=0.011) and a trend between the Lysholm score and T2 (Person: 0.304; P=0.193). CONCLUSION: Using T2-mapping and DWI, additional information could be gained compared to clinical scoring or morphological MRI. In combination clinical, MR-morphological and MR-biochemical parameters can be seen as a promising multimodal tool in the follow-up of cartilage repair.