Cognitive and Visual Speech Contributions to Speech Perception in Challenging Listening Conditions

Speech perception routinely takes place in noisy or degraded listening environments, leading to ambiguity in the identity of the speech token. Here, I present one review paper and two experimental papers that highlight cognitive and visual speech contributions to the listening process, particularly in challenging listening environments. First, I survey the literature linking audiometric age-related hearing loss and cognitive decline and review the four proposed causal mechanisms underlying this link. I argue that future research in this area requires greater consideration of the functional overlap between hearing and cognition. I also present an alternative framework for understanding causal relationships between age-related declines in hearing and cognition, with emphasis on the interconnected nature of hearing and cognition and likely contributions from multiple causal mechanisms. I also provide a number of testable hypotheses to examine how impairments in one domain may affect the other. In my first experimental study, I examine the direct contribution of working memory (through a cognitive training manipulation) on speech in noise comprehension in older adults. My results challenge the efficacy of cognitive training more generally, and also provide support for the contribution of sentence context in reducing working memory load. My findings also challenge the ubiquitous use of the Reading Span test as a pure test of working memory. In a second experimental (fMRI) study, I examine the role of attention in audiovisual speech integration, particularly when the acoustic signal is degraded. I demonstrate that attentional processes support audiovisual speech integration in the middle and superior temporal gyri, as well as the fusiform gyrus. My results also suggest that the superior temporal sulcus is sensitive to intelligibility enhancement, regardless of how this benefit is obtained (i.e., whether it is obtained through visual speech information or speech clarity). In addition, I also demonstrate that both the cingulo-opercular network and motor speech areas are recruited in difficult listening conditions. Taken together, these findings augment our understanding of cognitive contributions to the listening process and demonstrate that memory, working memory, and executive control networks may flexibly be recruited in order to meet listening demands in challenging environments.

The neuroscience of cognitive enhancement : enhanced attention, working memory and visual information processing speed using 3D-MOT

Des interventions ciblant l’amélioration cognitive sont de plus en plus à l’intérêt dans nombreux domaines, y compris la neuropsychologie. Bien qu'il existe de nombreuses méthodes pour maximiser le potentiel cognitif de quelqu’un, ils sont rarement appuyé par la recherche scientifique. D’abord, ce mémoire examine brièvement l'état des interventions d'amélioration cognitives. Il décrit premièrement les faiblesses observées dans ces pratiques et par conséquent il établit un modèle standard contre lequel on pourrait et devrait évaluer les diverses techniques ciblant l'amélioration cognitive. Une étude de recherche est ensuite présenté qui considère un nouvel outil de l'amélioration cognitive, une tâche d’entrainement perceptivo-cognitive : 3-dimensional multiple object tracking (3D-MOT). Il examine les preuves actuelles pour le 3D-MOT auprès du modèle standard proposé. Les résultats de ce projet démontrent de l’augmentation dans les capacités d’attention, de mémoire de travail visuel et de vitesse de traitement d’information. Cette étude représente la première étape dans la démarche vers l’établissement du 3D-MOT comme un outil d’amélioration cognitive.

Distraction by task-irrelevant stimuli: the effects of endogenous spatial attention and visual working memory load

Salient stimuli, like sudden changes in the environment or emotional stimuli, generate a priority signal that captures attention even if they are task-irrelevant. However, to achieve goal-driven behavior, we need to ignore them and to avoid being distracted. It is generally agreed that top-down factors can help us to filter out distractors. A fundamental question is how and at which stage of processing the rejection of distractors is achieved. Two circumstances under which the allocation of attention to distractors is supposed to be prevented are represented by the case in which distractors occur at an unattended location (as determined by the deployment of endogenous spatial attention) and when the amount of visual working memory resources is reduced by an ongoing task. The present thesis is focused on the impact of these factors on three sources of distraction, namely auditory and visual onsets (Experiments 1 and 2, respectively) and pleasant scenes (Experiment 3). In the first two studies we recorded neural correlates of distractor processing (i.e., Event-Related Potentials), whereas in the last study we used interference effects on behavior (i.e., a slowing down of response times on a simultaneous task) to index distraction. Endogenous spatial attention reduced distraction by auditory stimuli and eliminated distraction by visual onsets. Differently, visual working memory load only affected the processing of visual onsets. Emotional interference persisted even when scenes occurred always at unattended locations and when visual working memory was loaded. Altogether, these findings indicate that the ability to detect the location of salient task-irrelevant sounds and identify the affective significance of natural scenes is preserved even when the amount of visual working memory resources is reduced by an ongoing task and when endogenous attention is elsewhere directed. However, these results also indicate that the processing of auditory and visual distractors is not entirely automatic.

1° level of automation: the effectiveness of adaptive cruise control on driving and visual behaviour

The research activities have allowed the analysis of the driver assistance systems, called Advanced Driver Assistance Systems (ADAS) in relation to road safety. The study is structured according to several evaluation steps, related to definite on-site tests that have been carried out with different samples of users, according to their driving experience with the ACC. The evaluation steps concern: •The testing mode and the choice of suitable instrumentation to detect the driver’s behaviour in relation to the ACC. •The analysis modes and outputs to be obtained, i.e.: - Distribution of attention and inattention; - Mental workload; - The Perception-Reaction Time (PRT), the Time To Collision (TTC) and the Time Headway (TH). The main purpose is to assess the interaction between vehicle drivers and ADAS, highlighting the inattention and variation of the workloads they induce regarding the driving task. The research project considered the use of a system for monitoring visual behavior (ASL Mobile Eye-XG - ME), a powerful GPS that allowed to record the kinematic data of the vehicle (Racelogic Video V-BOX) and a tool for reading brain activity (Electroencephalographic System - EEG). Just during the analytical phase, a second and important research objective was born: the creation of a graphical interface that would allow exceeding the frame count limit, making faster and more effective the labeling of the driver’s points of view. The results show a complete and exhaustive picture of the vehicle-driver interaction. It has been possible to highlight the main sources of criticalities related to the user and the vehicle, in order to concretely reduce the accident rate. In addition, the use of mathematical-computational methodologies for the analysis of experimental data has allowed the optimization and verification of analytical processes with neural networks that have made an effective comparison between the manual and automatic methodology.

Visual Tracking, Pose Estimation, and Control for Aerial Vehicles

El principal objetivo de esta tesis es dotar a los vehículos aéreos no tripulados (UAVs, por sus siglas en inglés) de una fuente de información adicional basada en visión. Esta fuente de información proviene de cámaras ubicadas a bordo de los vehículos o en el suelo. Con ella se busca que los UAVs realicen tareas de aterrizaje o inspección guiados por visión, especialmente en aquellas situaciones en las que no haya disponibilidad de estimar la posición del vehículo con base en GPS, cuando las estimaciones de GPS no tengan la suficiente precisión requerida por las tareas a realizar, o cuando restricciones de carga de pago impidan añadir sensores a bordo de los vehículos. Esta tesis trata con tres de las principales áreas de la visión por computador: seguimiento visual y estimación visual de la pose (posición y orientación), que a su vez constituyen la base de la tercera, denominada control servo visual, que en nuestra aplicación se enfoca en el empleo de información visual para controlar los UAVs. Al respecto, esta tesis se ocupa de presentar propuestas novedosas que permitan solucionar problemas relativos al seguimiento de objetos mediante cámaras ubicadas a bordo de los UAVs, se ocupa de la estimación de la pose de los UAVs basada en información visual obtenida por cámaras ubicadas en el suelo o a bordo, y también se ocupa de la aplicación de las técnicas propuestas para solucionar diferentes problemas, como aquellos concernientes al seguimiento visual para tareas de reabastecimiento autónomo en vuelo o al aterrizaje basado en visión, entre otros. Las diversas técnicas de visión por computador presentadas en esta tesis se proponen con el fin de solucionar dificultades que suelen presentarse cuando se realizan tareas basadas en visión con UAVs, como las relativas a la obtención, en tiempo real, de estimaciones robustas, o como problemas generados por vibraciones. Los algoritmos propuestos en esta tesis han sido probados con información de imágenes reales obtenidas realizando pruebas on-line y off-line. Diversos mecanismos de evaluación han sido empleados con el propósito de analizar el desempeño de los algoritmos propuestos, entre los que se incluyen datos simulados, imágenes de vuelos reales, estimaciones precisas de posición empleando el sistema VICON y comparaciones con algoritmos del estado del arte. Los resultados obtenidos indican que los algoritmos de visión por computador propuestos tienen un desempeño que es comparable e incluso mejor al de algoritmos que se encuentran en el estado del arte. Los algoritmos propuestos permiten la obtención de estimaciones robustas en tiempo real, lo cual permite su uso en tareas de control visual. El desempeño de estos algoritmos es apropiado para las exigencias de las distintas aplicaciones examinadas: reabastecimiento autónomo en vuelo, aterrizaje y estimación del estado del UAV. Abstract The main objective of this thesis is to provide Unmanned Aerial Vehicles (UAVs) with an additional vision-based source of information extracted by cameras located either on-board or on the ground, in order to allow UAVs to develop visually guided tasks, such as landing or inspection, especially in situations where GPS information is not available, where GPS-based position estimation is not accurate enough for the task to develop, or where payload restrictions do not allow the incorporation of additional sensors on-board. This thesis covers three of the main computer vision areas: visual tracking and visual pose estimation, which are the bases the third one called visual servoing, which, in this work, focuses on using visual information to control UAVs. In this sense, the thesis focuses on presenting novel solutions for solving the tracking problem of objects when using cameras on-board UAVs, on estimating the pose of the UAVs based on the visual information collected by cameras located either on the ground or on-board, and also focuses on applying these proposed techniques for solving different problems, such as visual tracking for aerial refuelling or vision-based landing, among others. The different computer vision techniques presented in this thesis are proposed to solve some of the frequently problems found when addressing vision-based tasks in UAVs, such as obtaining robust vision-based estimations at real-time frame rates, and problems caused by vibrations, or 3D motion. All the proposed algorithms have been tested with real-image data in on-line and off-line tests. Different evaluation mechanisms have been used to analyze the performance of the proposed algorithms, such as simulated data, images from real-flight tests, publicly available datasets, manually generated ground truth data, accurate position estimations using a VICON system and a robotic cell, and comparison with state of the art algorithms. Results show that the proposed computer vision algorithms obtain performances that are comparable to, or even better than, state of the art algorithms, obtaining robust estimations at real-time frame rates. This proves that the proposed techniques are fast enough for vision-based control tasks. Therefore, the performance of the proposed vision algorithms has shown to be of a standard appropriate to the different explored applications: aerial refuelling and landing, and state estimation. It is noteworthy that they have low computational overheads for vision systems.

Invariance of brain-wave representations of simple visual images and their names

In two experiments, electric brain waves of 14 subjects were recorded under several different conditions to study the invariance of brain-wave representations of simple patches of colors and simple visual shapes and their names, the words blue, circle, etc. As in our earlier work, the analysis consisted of averaging over trials to create prototypes and test samples, to both of which Fourier transforms were applied, followed by filtering and an inverse transformation to the time domain. A least-squares criterion of fit between prototypes and test samples was used for classification. The most significant results were these. By averaging over different subjects, as well as trials, we created prototypes from brain waves evoked by simple visual images and test samples from brain waves evoked by auditory or visual words naming the visual images. We correctly recognized from 60% to 75% of the test-sample brain waves. The general conclusion is that simple shapes such as circles and single-color displays generate brain waves surprisingly similar to those generated by their verbal names. These results, taken together with extensive psychological studies of auditory and visual memory, strongly support the solution proposed for visual shapes, by Bishop Berkeley and David Hume in the 18th century, to the long-standing problem of how the mind represents simple abstract ideas.

Magnetoencephalography:technical improvements in image co-registration and studies of visual cortical oscillations

The work presented in this thesis is divided into two distinct sections. In the first, the functional neuroimaging technique of Magnetoencephalography (MEG) is described and a new technique is introduced for accurate combination of MEG and MRI co-ordinate systems. In the second part of this thesis, MEG and the analysis technique of SAM are used to investigate responses of the visual system in the context of functional specialisation within the visual cortex. In chapter one, the sources of MEG signals are described, followed by a brief description of the necessary instrumentation for accurate MEG recordings. This chapter is concluded by introducing the forward and inverse problems of MEG, techniques to solve the inverse problem, and a comparison of MEG with other neuroimaging techniques. Chapter two provides an important contribution to the field of research with MEG. Firstly, it is described how MEG and MRI co-ordinate systems are combined for localisation and visualisation of activated brain regions. A previously used co-registration methods is then described, and a new technique is introduced. In a series of experiments, it is demonstrated that using fixed fiducial points provides a considerable improvement in the accuracy and reliability of co-registration. Chapter three introduces the visual system starting from the retina and ending with the higher visual rates. The functions of the magnocellular and the parvocellular pathways are described and it is shown how the parallel visual pathways remain segregated throughout the visual system. The structural and functional organisation of the visual cortex is then described. Chapter four presents strong evidence in favour of the link between conscious experience and synchronised brain activity. The spatiotemporal responses of the visual cortex are measured in response to specific gratings. It is shown that stimuli that induce visual discomfort and visual illusions share their physical properties with those that induce highly synchronised gamma frequency oscillations in the primary visual cortex. Finally chapter five is concerned with localization of colour in the visual cortex. In this first ever use of Synthetic Aperture Magnetometry to investigate colour processing in the visual cortex, it is shown that in response to isoluminant chromatic gratings, the highest magnitude of cortical activity arise from area V2.

Applications Of Visual Evoked Potentials And Fourier-domain Optical Coherence Tomography In Parkinson's Disease: A Controlled Study.

The goal of this cross-sectional observational study was to quantify the pattern-shift visual evoked potentials (VEP) and the thickness as well as the volume of retinal layers using optical coherence tomography (OCT) across a cohort of Parkinson's disease (PD) patients and age-matched controls. Forty-three PD patients and 38 controls were enrolled. All participants underwent a detailed neurological and ophthalmologic evaluation. Idiopathic PD cases were included. Cases with glaucoma or increased intra-ocular pressure were excluded. Patients were assessed by VEP and high-resolution Fourier-domain OCT, which quantified the inner and outer thicknesses of the retinal layers. VEP latencies and the thicknesses of the retinal layers were the main outcome measures. The mean age, with standard deviation (SD), of the PD patients and controls were 63.1 (7.5) and 62.4 (7.2) years, respectively. The patients were predominantly in the initial Hoehn-Yahr (HY) disease stages (34.8% in stage 1 or 1.5, and 55.8 % in stage 2). The VEP latencies and the thicknesses as well as the volumes of the retinal inner and outer layers of the groups were similar. A negative correlation between the retinal thickness and the age was noted in both groups. The thickness of the retinal nerve fibre layer (RNFL) was 102.7 μm in PD patients vs. 104.2 μm in controls. The thicknesses of retinal layers, VEP, and RNFL of PD patients were similar to those of the controls. Despite the use of a representative cohort of PD patients and high-resolution OCT in this study, further studies are required to establish the validity of using OCT and VEP measurements as the anatomic and functional biomarkers for the evaluation of retinal and visual pathways in PD patients.

Neuronal Ceroid-lipofuscinosis with prominent chorea and without visual manifestations: a case report

A case of neuronal ceroid-lipofuscinosis (NCL) is reported in a 11-year-old girl, whose main symptoms were progressive dementia since the age of 4 years and choreic movements since age 10. Seizures, myoclonus and visual deterioration were absent and optic fundi were normal. A cerebral biopsy disclosed two basic types of stored substance in the cytoplasm of neurons: a) severely balloned nerve cells in cortical layers HI and V contained a non-autofluorescent material, which stained with PAS and Sudan Black B in frozen, but not in paraffin sections; ultrastructurally, these neurons showed abundant corpuscles similar to the membranous cytoplasmic bodies of Tay-Sachs disease and, in smaller amounts, also zebra bodies; b) slightly distended or non-distended neurons in all layers contained lipopigment granules, which were autofluorescent, PAS-positive and sudanophil in both frozen and paraffin sections; their ultrastructure was closely comparable to that of lipofuscin. Similar bodies were found in the swollen segments of axons and in a few astrocytes and endothelial cells. The histochemical and ultrastructural demonstration of large amounts of lipopigments allows a presumptive classification of the case as NCL. However, the presence of involuntary movements, the absence of visual disturbances and the unusual ultrastructural features place the patient into a small heterogeneous group within the NCL. A better classification of such unique instances of the disease must await elucidation of the basic enzymatic defects.

Perfil antropometrico e desempenho fisico-motor de crianças e jovens com deficiencia visual participantes do atletismo nos 1. Jogos Escolares da Confederação Brasileira de Desportos para Cegos

Universidade Estadual de Campinas . Faculdade de Educação Física

Mães de crianças com deficiência visual: percepções, conduta e constribuição do atendimento em grupo

A presente pesquisa teve quatro objetivos: 1) identificar reações de mães em relação ao diagnóstico da deficiência visual; 2) identificar o responsável pela detecção da deficiência; 3) verificar dificuldades da criança no processo de escolarização, e 4) verificar possíveis contribuições de atividades terapêuticas direcionadas ao grupo de mães. Foi realizado um "survey" descritivo com as mães de crianças com deficiência visual, atendidas no CEPRE-FCM-Unicamp. Para a coleta de dados foi utilizado um questionário aplicado por entrevista, desenvolvido após estudo exploratório. Compôs-se uma amostra não probabilística, constituída por 14 mães. Entre os resultados obtidos, com relação aos sentimentos apontados pelas mães em relação ao diagnóstico, destacaram-se: a tristeza (71,0 por cento); o medo (64,0 por cento) e a decepção (42,0 por cento). O problema visual foi percebido por mães em 53,0 por cento dos casos, por pediatras em 26,0 por cento e por familiares em 21,0 por cento. Entre as dificuldades da criança no processo de escolarização foram apontadas: medo de não conseguirem acompanhar as exigências escolares (75,0 por cento) e discriminação (63,0 por cento). A maioria das mães (78,0 por cento) acredita que as atividades do grupo contribuem para o esclarecimento de dúvidas, e as atividades terapêuticas contribuíram para que aprendessem a lidar com as dificuldades de seus filhos (78,0 por cento). Os resultados obtidos contribuíram para concluir: que os sentimentos de tristeza, medo e decepção mostraram-se mais evidentes; que, na maioria dos casos, a deficiência visual foi detectada pela mãe; que na opinião das mães, as crianças teriam dificuldades em acompanhar as atividades escolares e que o grupo contribuiu para esclarecer dúvidas e favorecer troca de experiências

Predictive factors for the development of visual loss in patients with pituitary macroadenomas and for visual recovery after optic pathway decompression

Objective: To investigate clinical and MRI findings that are predictive of both visual loss in patients with pituitary adenomas and visual recovery after treatment. Design: Cohort study. Participants: Thirty patients (60 eyes) with pituitary adenoma. Methods: Patients underwent neuro-ophthalmic examination and MRI before and after optic chiasm decompression. Visual field (VF) was assessed using the mean deviation in standard automated perimetry (SAP) and temporal mean defect, the average of 22 temporal values of the total deviation plot. Tumour size was measured on sagittal and coronal cuts. Results: Visual loss was found in 47 eyes; 35 had optic atrophy (subtle in 9, moderate in 14, and severe in 12). Before treatment, the average SAP mean deviation and temporal mean defect were -11.78 (SD 8.56) dB and -18.66 (SD 11.20) dB, respectively. The chiasm was 17.3 (SD 6.2, range 10-34) mm above the reference line on the sagittal and 21.8 (SD 8.3, range 12-39) mm on the coronal images. Tumour size correlated with the severity of VF defect. VF improvement occurred in 80% of eyes after treatment. The degree of optic atrophy, visual loss, and tumour size were significantly associated with improvement after treatment. Conclusions: The best predictive factor for visual loss was tumour size, and factors related to visual recovery were the degree of optic atrophy, the severity of VF defect, and the tumour size. Diagnosing pituitary adenomas before optic atrophy becomes severe may be related to a better prognosis in such patients.

Variations in cone photoreceptor abundance and the visual ecology of birds

The relative abundance and topographical distribution of retinal cone photoreceptors was measured in 19 bird species to identify possible correlations between photoreceptor complement and visual ecology. In contrast to previous studies, all five types of cone photoreceptor were distinguished, using bright field and epifluorescent light microscopy, in four retinal quadrants. Land birds tended to show either posterior dorsal to anterior ventral or anterior dorsal to posterior ventral gradients in cone photoreceptor distribution, fundus coloration and oil droplet pigmentation across the retina. Marine birds tended to show dorsal to ventral gradients instead. Statistical analyses showed that the proportions of the different cone types varied significantly across the retinae of all species investigated. Cluster analysis was performed on the data to identify groups or clusters of species on the basis of their oil droplet complement. Using the absolute percentages of each oil droplet type in each quadrant for the analysis produced clusters that tended to reflect phylogenetic relatedness between species rather than similarities in their visual ecology. Repeating the analysis after subtracting the mean percentage of a given oil droplet type across the whole retina (the 'eye mean') from the percentage of that oil droplet type in each quadrant, i.e. to give a measure of the variation about the mean, resulted in clusters that reflected diet, feeding behaviour and habitat to a greater extent than phylogeny.

Visual and verbal processing effects on human memory

In this study in the field of Consumer Behavior, brand name memory of consumers with regard to verbal and visual incongruent and congruent information such as memory structure of brands was tested. Hence, four experimental groups with different constellations of verbal and visual congruity and incongruity were created to compare their brand name memory performance. The experiment was conducted in several classes with 128 students, each group with 32 participants. It was found that brands, which are presented in a congruent or moderately incongruent relation to their brand schema, result in a better brand recall than their incongruent counterparts. A difference between visual congruity and moderately incongruity could not be confirmed. In contrast to visual incongruent information, verbal incongruent information does not result in a worse brand recall performance.

Detection of road accident accumulation zones with a visual analytics approach

Nowadays, road accidents are a major public health problem, which increase is forecasted if road safety is not treated properly, dying about 1.2 million people every year around the globe. In 2012, Portugal recorded 573 fatalities in road accidents, on site, revealing the largest decreasing of the European Union for 2011, along with Denmark. Beyond the impact caused by fatalities, it was calculated that the economic and social costs of road accidents weighted about 1.17% of the Portuguese gross domestic product in 2010. Visual Analytics allows the combination of data analysis techniques with interactive visualizations, which facilitates the process of knowledge discovery in sets of large and complex data, while the Geovisual Analytics facilitates the exploration of space-time data through maps with different variables and parameters that are under analysis. In Portugal, the identification of road accident accumulation zones, in this work named black spots, has been restricted to annual fixed windows. In this work, it is presented a dynamic approach based on Visual Analytics techniques that is able to identify the displacement of black spots on sliding windows of 12 months. Moreover, with the use of different parameterizations in the formula usually used to detect black spots, it is possible to identify zones that are almost becoming black spots. Through the proposed visualizations, the study and identification of countermeasures to this social and economic problem can gain new grounds and thus the decision- making process is supported and improved.