COMP6234 Week 6

Visual perception, information design for the brain, discussion of good and bad visualisations

Perception Quality Evaluation with Visual and Infrared Cameras in Challenging Environmental Conditions

This work aims to contribute to the reliability and integrity of perceptual systems of unmanned ground vehicles (UGV). A method is proposed to evaluate the quality of sensor data prior to its use in a perception system by utilising a quality metric applied to heterogeneous sensor data such as visual and infrared camera images. The concept is illustrated specifically with sensor data that is evaluated prior to the use of the data in a standard SIFT feature extraction and matching technique. The method is then evaluated using various experimental data sets that were collected from a UGV in challenging environmental conditions, represented by the presence of airborne dust and smoke. In the first series of experiments, a motionless vehicle is observing a ’reference’ scene, then the method is extended to the case of a moving vehicle by compensating for its motion. This paper shows that it is possible to anticipate degradation of a perception algorithm by evaluating the input data prior to any actual execution of the algorithm.

Visual metrics for the evaluation of sensor data quality in outdoor perception

This paper proposes an experimental study of quality metrics that can be applied to visual and infrared images acquired from cameras onboard an unmanned ground vehicle (UGV). The relevance of existing metrics in this context is discussed and a novel metric is introduced. Selected metrics are evaluated on data collected by a UGV in clear and challenging environmental conditions, represented in this paper by the presence of airborne dust or smoke. An example of application is given with monocular SLAM estimating the pose of the UGV while smoke is present in the environment. It is shown that the proposed novel quality metric can be used to anticipate situations where the quality of the pose estimate will be significantly degraded due to the input image data. This leads to decisions of advantageously switching between data sources (e.g. using infrared images instead of visual images).

Visual metrics for the evaluation of sensor data quality in outdoor perception

This paper proposes an experimental study of quality metrics that can be applied to visual and infrared images acquired from cameras onboard an unmanned ground vehicle (UGV). The relevance of existing metrics in this context is discussed and a novel metric is introduced. Selected metrics are evaluated on data collected by a UGV in clear and challenging environmental conditions, represented in this paper by the presence of airborne dust or smoke.

Investigating human audio-visual object perception with a combination of hypothesis-generating and hypothesis-testing fMRI analysis tools

Primate multisensory object perception involves distributed brain regions. To investigate the network character of these regions of the human brain, we applied data-driven group spatial independent component analysis (ICA) to a functional magnetic resonance imaging (fMRI) data set acquired during a passive audio-visual (AV) experiment with common object stimuli. We labeled three group-level independent component (IC) maps as auditory (A), visual (V), and AV, based on their spatial layouts and activation time courses. The overlap between these IC maps served as definition of a distributed network of multisensory candidate regions including superior temporal, ventral occipito-temporal, posterior parietal and prefrontal regions. During an independent second fMRI experiment, we explicitly tested their involvement in AV integration. Activations in nine out of these twelve regions met the max-criterion (A < AV > V) for multisensory integration. Comparison of this approach with a general linear model-based region-of-interest definition revealed its complementary value for multisensory neuroimaging. In conclusion, we estimated functional networks of uni- and multisensory functional connectivity from one dataset and validated their functional roles in an independent dataset. These findings demonstrate the particular value of ICA for multisensory neuroimaging research and using independent datasets to test hypotheses generated from a data-driven analysis.

Auditory-visual speech integration by prelinguistic infants: Perception of an emergent consonant in the McGurk effect

The McGurk effect, in which auditory [ba] dubbed onto [go] lip movements is perceived as da or tha, was employed in a real-time task to investigate auditory-visual speech perception in prelingual infants. Experiments 1A and 1B established the validity of real-time dubbing for producing the effect. In Experiment 2, 4(1)/(2)-month-olds were tested in a habituation-test paradigm, in which 2 an auditory-visual stimulus was presented contingent upon visual fixation of a live face. The experimental group was habituated to a McGurk stimulus (auditory [ba] visual [ga]), and the control group to matching auditory-visual [ba]. Each group was then presented with three auditory-only test trials, [ba], [da], and [deltaa] (as in then). Visual-fixation durations in test trials showed that the experimental group treated the emergent percept in the McGurk effect, [da] or [deltaa], as familiar (even though they had not heard these sounds previously) and [ba] as novel. For control group infants [da] and [deltaa] were no more familiar than [ba]. These results are consistent with infants'perception of the McGurk effect, and support the conclusion that prelinguistic infants integrate auditory and visual speech information. (C) 2004 Wiley Periodicals, Inc.

Cognitive and Visual Speech Contributions to Speech Perception in Challenging Listening Conditions

Speech perception routinely takes place in noisy or degraded listening environments, leading to ambiguity in the identity of the speech token. Here, I present one review paper and two experimental papers that highlight cognitive and visual speech contributions to the listening process, particularly in challenging listening environments. First, I survey the literature linking audiometric age-related hearing loss and cognitive decline and review the four proposed causal mechanisms underlying this link. I argue that future research in this area requires greater consideration of the functional overlap between hearing and cognition. I also present an alternative framework for understanding causal relationships between age-related declines in hearing and cognition, with emphasis on the interconnected nature of hearing and cognition and likely contributions from multiple causal mechanisms. I also provide a number of testable hypotheses to examine how impairments in one domain may affect the other. In my first experimental study, I examine the direct contribution of working memory (through a cognitive training manipulation) on speech in noise comprehension in older adults. My results challenge the efficacy of cognitive training more generally, and also provide support for the contribution of sentence context in reducing working memory load. My findings also challenge the ubiquitous use of the Reading Span test as a pure test of working memory. In a second experimental (fMRI) study, I examine the role of attention in audiovisual speech integration, particularly when the acoustic signal is degraded. I demonstrate that attentional processes support audiovisual speech integration in the middle and superior temporal gyri, as well as the fusiform gyrus. My results also suggest that the superior temporal sulcus is sensitive to intelligibility enhancement, regardless of how this benefit is obtained (i.e., whether it is obtained through visual speech information or speech clarity). In addition, I also demonstrate that both the cingulo-opercular network and motor speech areas are recruited in difficult listening conditions. Taken together, these findings augment our understanding of cognitive contributions to the listening process and demonstrate that memory, working memory, and executive control networks may flexibly be recruited in order to meet listening demands in challenging environments.

Seeing music performance : visual influences on perception and experience

Drawing from ethnographic, empirical, and historical / cultural perspectives, we examine the extent to which visual aspects of music contribute to the communication that takes place between performers and their listeners. First, we introduce a framework for understanding how media and genres shape aural and visual experiences of music. Second, we present case studies of two performances, and describe the relation between visual and aural aspects of performance. Third, we report empirical evidence that visual aspects of performance reliably influence perceptions of musical structure (pitch related features) and affective interpretations of music. Finally, we trace new and old media trajectories of aural and visual dimensions of music, and highlight how our conceptions, perceptions and appreciation of music are intertwined with technological innovation and media deployment strategies.

Availability of advance visual information constrains association-football goalkeeping performance during penalty kicks

A pressing concern within the literature on anticipatory perceptual-motor behaviour is the lack of clarity on the applicability of data, observed under video-simulation task constraints, to actual performance in which actions are coupled to perception, as captured during in-situ experimental conditions. We developed an in-situ experimental paradigm which manipulated the duration of anticipatory visual information from a penalty taker’s actions to examine experienced goalkeepers’ vulnerability to deception for the penalty kick in association football. Irrespective of the penalty taker’s kick strategy, goalkeepers initiated movement responses earlier across consecutively earlier presentation points. Overall goalkeeping performance was better in non-deception trials than in deception conditions. In deception trials, the kinematic information presented up until the penalty taker initiated his/her kicking action had a negative effect on goalkeepers’ performance. It is concluded that goalkeepers are likely to benefit from not anticipating a penalty taker’s performance outcome based on information from the run-up, in preference to later information that emerges just before the initiation of the penalty taker’s kicking action.

Selective combination of visual and thermal imaging for resilient localization in adverse conditions : day and night, smoke and fire

Long-term autonomy in robotics requires perception systems that are resilient to unusual but realistic conditions that will eventually occur during extended missions. For example, unmanned ground vehicles (UGVs) need to be capable of operating safely in adverse and low-visibility conditions, such as at night or in the presence of smoke. The key to a resilient UGV perception system lies in the use of multiple sensor modalities, e.g., operating at different frequencies of the electromagnetic spectrum, to compensate for the limitations of a single sensor type. In this paper, visual and infrared imaging are combined in a Visual-SLAM algorithm to achieve localization. We propose to evaluate the quality of data provided by each sensor modality prior to data combination. This evaluation is used to discard low-quality data, i.e., data most likely to induce large localization errors. In this way, perceptual failures are anticipated and mitigated. An extensive experimental evaluation is conducted on data sets collected with a UGV in a range of environments and adverse conditions, including the presence of smoke (obstructing the visual camera), fire, extreme heat (saturating the infrared camera), low-light conditions (dusk), and at night with sudden variations of artificial light. A total of 240 trajectory estimates are obtained using five different variations of data sources and data combination strategies in the localization method. In particular, the proposed approach for selective data combination is compared to methods using a single sensor type or combining both modalities without preselection. We show that the proposed framework allows for camera-based localization resilient to a large range of low-visibility conditions.

Medium and mode : relationships between visual experience and perception in an installation arts practice

Working primarily within the natural landscape, this practice-led research project explored connections between the artist's visual and perceptual experience of a journey or place while simultaneously emphasizing the capacity for digital media to create a perceptual dissonance. By exploring concepts of time, viewpoint, duration of sequences and the manipulation of traditional constructs of stop-frame animation, the practical work created a cognitive awareness of the elements of the journey through optical sensations. The work allowed an opportunity to reflect on the nature of visual experience and its mediation through images. The project recontextualized the selected mediums of still photography, animation and projection within contemporary display modes of multiple screen installations by analysing relationships between the experienced and the perceived. The resulting works added to current discourse on the interstices between still and moving imagery in a digital world.

The N170 observed ‘in the wild’: robust event-related potentials to faces in cluttered dynamic visual scenes

As a social species in a constantly changing environment, humans rely heavily on the informational richness and communicative capacity of the face. Thus, understanding how the brain processes information about faces in real-time is of paramount importance. The N170 is a high temporal resolution electrophysiological index of the brain's early response to visual stimuli that is reliably elicited in carefully controlled laboratory-based studies. Although the N170 has often been reported to be of greatest amplitude to faces, there has been debate regarding whether this effect might be an artifact of certain aspects of the controlled experimental stimulation schedules and materials. To investigate whether the N170 can be identified in more realistic conditions with highly variable and cluttered visual images and accompanying auditory stimuli we recorded EEG 'in the wild', while participants watched pop videos. Scene-cuts to faces generated a clear N170 response, and this was larger than the N170 to transitions where the videos cut to non-face stimuli. Within participants, wild-type face N170 amplitudes were moderately correlated to those observed in a typical laboratory experiment. Thus, we demonstrate that the face N170 is a robust and ecologically valid phenomenon and not an artifact arising as an unintended consequence of some property of the more typical laboratory paradigm.

La participación de los microRNAs en la regulación de la plasticidad en la corteza visual en un modelo de ambliopía experimental

La aplicaci�n de campos geomagn�ticos con la polaridad cambiante promueve la plasticidad de la corteza visual de animales privados visualmente mediante la cr�a en oscuridad, o bien a trav�s de la sutura palpebral de un ojo.Esto es debido a que la estimulaci�n magn�tica produce la down-regulaci�n de la expresi�n de los microRNAs let-7b*, miR-330, miR-338* y miR-376c en la corteza visual de animales sometidos a la cr�a en oscuridad, as� como de animales sujetos a sutura palpebral de un ojo. Como consecuencia de la down-regulaci�n de la expresi�n de los microRNAs citados anteriormente, se ven incrementados los niveles de expresi�n de sus correspondientes mol�culas diana, Conexina 26 para el caso de let-7b*, Tenascina R en el caso de miR-330, Contactina 4 para el caso de miR-338* y Matriz metalopeptidasa 9 y �-sinucle�na, ambas mol�culas diana de miR-376c. El aumento de expresi�n de estas mol�culas diana a nivel de RNA mensajero, as� como a nivel de prote�na en la corteza visual promueve la capacidad pl�stica de la corteza visual, ya que estas mol�culas diana se encuentran implicadas en procesos de crecimiento/elongaci�n de las neuritas y en la regulaci�n de la morfolog�a de las espinas dendr�ticas.