Perception of co-speech gestures in aphasic patients: A visual exploration study during the observation of dyadic conversations

Background: Co-speech gestures are part of nonverbal communication during conversations. They either support the verbal message or provide the interlocutor with additional information. Furthermore, they prompt as nonverbal cues the cooperative process of turn taking. In the present study, we investigated the influence of co-speech gestures on the perception of dyadic dialogue in aphasic patients. In particular, we analysed the impact of co-speech gestures on gaze direction (towards speaker or listener) and fixation of body parts. We hypothesized that aphasic patients, who are restricted in verbal comprehension, adapt their visual exploration strategies. Methods: Sixteen aphasic patients and 23 healthy control subjects participated in the study. Visual exploration behaviour was measured by means of a contact-free infrared eye-tracker while subjects were watching videos depicting spontaneous dialogues between two individuals. Cumulative fixation duration and mean fixation duration were calculated for the factors co-speech gesture (present and absent), gaze direction (to the speaker or to the listener), and region of interest (ROI), including hands, face, and body. Results: Both aphasic patients and healthy controls mainly fixated the speaker’s face. We found a significant co-speech gesture x ROI interaction, indicating that the presence of a co-speech gesture encouraged subjects to look at the speaker. Further, there was a significant gaze direction x ROI x group interaction revealing that aphasic patients showed reduced cumulative fixation duration on the speaker’s face compared to healthy controls. Conclusion: Co-speech gestures guide the observer’s attention towards the speaker, the source of semantic input. It is discussed whether an underlying semantic processing deficit or a deficit to integrate audio-visual information may cause aphasic patients to explore less the speaker’s face. Keywords: Gestures, visual exploration, dialogue, aphasia, apraxia, eye movements

Dynamic population coding in primary visual cortex

More than a century ago Ramon y Cajal pioneered the description of neural circuits. Currently, new techniques are being developed to streamline the characterization of entire neural circuits. Even if this 'connectome' approach is successful, it will represent only a static description of neural circuits. Thus, a fundamental question in neuroscience is to understand how information is dynamically represented by neural populations. In this thesis, I studied two main aspects of dynamical population codes. ^ First, I studied how the exposure or adaptation, for a fraction of a second to oriented gratings dynamically changes the population response of primary visual cortex neurons. The effects of adaptation to oriented gratings have been extensively explored in psychophysical and electrophysiological experiments. However, whether rapid adaptation might induce a change in the primary visual cortex's functional connectivity to dynamically impact the population coding accuracy is currently unknown. To address this issue, we performed multi-electrode recordings in primary visual cortex, where adaptation has been previously shown to induce changes in the selectivity and response amplitude of individual neurons. We found that adaptation improves the population coding accuracy. The improvement was more prominent for iso- and orthogonal orientation adaptation, consistent with previously reported psychophysical experiments. We propose that selective decorrelation is a metabolically inexpensive mechanism that the visual system employs to dynamically adapt the neural responses to the statistics of the input stimuli to improve coding efficiency. ^ Second, I investigated how ongoing activity modulates orientation coding in single neurons, neural populations and behavior. Cortical networks are never silent even in the absence of external stimulation. The ongoing activity can account for up to 80% of the metabolic energy consumed by the brain. Thus, a fundamental question is to understand the functional role of ongoing activity and its impact on neural computations. I studied how the orientation coding by individual neurons and cell populations in primary visual cortex depend on the spontaneous activity before stimulus presentation. We hypothesized that since the ongoing activity of nearby neurons is strongly correlated, it would influence the ability of the entire population of orientation-selective cells to process orientation depending on the prestimulus spontaneous state. Our findings demonstrate that ongoing activity dynamically filters incoming stimuli to shape the accuracy of orientation coding by individual neurons and cell populations and this interaction affects behavioral performance. In summary, this thesis is a contribution to the study of how dynamic internal states such as rapid adaptation and ongoing activity modulate the population code accuracy. ^

On nonlinearity in neural encoding models applied to the primary visual cortex

Within the regression framework, we show how different levels of nonlinearity influence the instantaneous firing rate prediction of single neurons. Nonlinearity can be achieved in several ways. In particular, we can enrich the predictor set with basis expansions of the input variables (enlarging the number of inputs) or train a simple but different model for each area of the data domain. Spline-based models are popular within the first category. Kernel smoothing methods fall into the second category. Whereas the first choice is useful for globally characterizing complex functions, the second is very handy for temporal data and is able to include inner-state subject variations. Also, interactions among stimuli are considered. We compare state-of-the-art firing rate prediction methods with some more sophisticated spline-based nonlinear methods: multivariate adaptive regression splines and sparse additive models. We also study the impact of kernel smoothing. Finally, we explore the combination of various local models in an incremental learning procedure. Our goal is to demonstrate that appropriate nonlinearity treatment can greatly improve the results. We test our hypothesis on both synthetic data and real neuronal recordings in cat primary visual cortex, giving a plausible explanation of the results from a biological perspective.

A Visual AGV-Urban Car using Fuzzy Control

The goal of the work described in this paper is to develop a visual line guided system for being used on-board an Autonomous Guided Vehicle (AGV) commercial car, controlling the steering and using just the visual information of a line painted below the car. In order to implement the control of the vehicle, a Fuzzy Logic controller has been implemented, that has to be robust against curvature changes and velocity changes. The only input information for the controller is the visual distance from the image center captured by a camera pointing downwards to the guiding line on the road, at a commercial frequency of 30Hz. The good performance of the controller has successfully been demonstrated in a real environment at urban velocities. The presented results demonstrate the capability of the Fuzzy controller to follow a circuit in urban environments without previous information about the path or any other information from additional sensors

Image characterisation based on the mammalian visual cortex

A proposal for a model of the primary visual cortex is reported. It is structured with the basis of a simple unit cell able to perform fourteen pairs of different boolean functions corresponding to the two possible inputs. As a first step, a model of the retina is presented. Different types of responses, according to the different possibilities of interconnecting the building blocks, have been obtained. These responses constitute the basis for an initial configuration of the mammalian primary visual cortex. Some qualitative functions, as symmetry or size of an optical input, have been obtained. A proposal to extend this model to some higher functions, concludes the paper.

Photonic processing subsystem based on visual cortex architecture

Optical signal processing in any living being is more complex than the one obtained in artificial systems. Cortex architecture, although only partly known, gives some useful ideas to be employed in communications. To analyze some of these structures is the objective of this paper. One of the main possibilities reported is handling signals in a parallel way. As it is shown, according to the signal characteristics each signal impinging onto a single input may be routed to a different output. At the same time, identical signals, coming to different inputs, may be routed to the same output without internal conflicts. This is due to the change of some of their characteristics in the way out when going through the intermediate levels. The simulation of this architecture is based on simple logic cells. The basis for the proposed architecture is the five layers of the mammalian retina and the first levels of the visual cortex.

Human-computer interaction based on visual hand-gesture recognition using volumetric spatiograms of local binary patterns

A more natural, intuitive, user-friendly, and less intrusive Human–Computer interface for controlling an application by executing hand gestures is presented. For this purpose, a robust vision-based hand-gesture recognition system has been developed, and a new database has been created to test it. The system is divided into three stages: detection, tracking, and recognition. The detection stage searches in every frame of a video sequence potential hand poses using a binary Support Vector Machine classifier and Local Binary Patterns as feature vectors. These detections are employed as input of a tracker to generate a spatio-temporal trajectory of hand poses. Finally, the recognition stage segments a spatio-temporal volume of data using the obtained trajectories, and compute a video descriptor called Volumetric Spatiograms of Local Binary Patterns (VS-LBP), which is delivered to a bank of SVM classifiers to perform the gesture recognition. The VS-LBP is a novel video descriptor that constitutes one of the most important contributions of the paper, which is able to provide much richer spatio-temporal information than other existing approaches in the state of the art with a manageable computational cost. Excellent results have been obtained outperforming other approaches of the state of the art.

Visual stimuli induce waves of electrical activity in turtle cortex

The computations involved in the processing of a visual scene invariably involve the interactions among neurons throughout all of visual cortex. One hypothesis is that the timing of neuronal activity, as well as the amplitude of activity, provides a means to encode features of objects. The experimental data from studies on cat [Gray, C. M., Konig, P., Engel, A. K. & Singer, W. (1989) Nature (London) 338, 334–337] support a view in which only synchronous (no phase lags) activity carries information about the visual scene. In contrast, theoretical studies suggest, on the one hand, the utility of multiple phases within a population of neurons as a means to encode independent visual features and, on the other hand, the likely existence of timing differences solely on the basis of network dynamics. Here we use widefield imaging in conjunction with voltage-sensitive dyes to record electrical activity from the virtually intact, unanesthetized turtle brain. Our data consist of single-trial measurements. We analyze our data in the frequency domain to isolate coherent events that lie in different frequency bands. Low frequency oscillations (<5 Hz) are seen in both ongoing activity and activity induced by visual stimuli. These oscillations propagate parallel to the afferent input. Higher frequency activity, with spectral peaks near 10 and 20 Hz, is seen solely in response to stimulation. This activity consists of plane waves and spiral-like waves, as well as more complex patterns. The plane waves have an average phase gradient of ≈π/2 radians/mm and propagate orthogonally to the low frequency waves. Our results show that large-scale differences in neuronal timing are present and persistent during visual processing.

Brain activity in visual cortex predicts individual differences in reading performance

The relationship between brain activity and reading performance was examined to test the hypothesis that dyslexia involves a deficit in a specific visual pathway known as the magnocellular (M) pathway. Functional magnetic resonance imaging was used to measure brain activity in dyslexic and control subjects in conditions designed to preferentially stimulate the M pathway. Dyslexics showed reduced activity compared with controls both in the primary visual cortex and in a secondary cortical visual area (MT+) that is believed to receive a strong M pathway input. Most importantly, significant correlations were found between individual differences in reading rate and brain activity. These results support the hypothesis for an M pathway abnormality in dyslexia and imply a strong relationship between the integrity of the M pathway and reading ability.

Surgically created neural pathways mediate visual pattern discrimination

Combined lesions of retinal targets and ascending auditory pathways can induce, in developing animals, permanent retinal projections to auditory thalamic nuclei and to visual thalamic nuclei that normally receive little direct retinal input. Neurons in the auditory cortex of such animals have visual response properties that resemble those of neurons in the primary visual cortex of normal animals. Therefore, we investigated the behavioral function of the surgically induced retino-thalamo-cortical pathways. We showed that both surgically induced pathways can mediate visually guided behaviors whose normal substrate, the pathway from the retina to the primary visual cortex via the primary thalamic visual nucleus, is missing.

Computational models of cortical visual processing.

The visual responses of neurons in the cerebral cortex were first adequately characterized in the 1960s by D. H. Hubel and T. N. Wiesel [(1962) J. Physiol. (London) 160, 106-154; (1968) J. Physiol. (London) 195, 215-243] using qualitative analyses based on simple geometric visual targets. Over the past 30 years, it has become common to consider the properties of these neurons by attempting to make formal descriptions of these transformations they execute on the visual image. Most such models have their roots in linear-systems approaches pioneered in the retina by C. Enroth-Cugell and J. R. Robson [(1966) J. Physiol. (London) 187, 517-552], but it is clear that purely linear models of cortical neurons are inadequate. We present two related models: one designed to account for the responses of simple cells in primary visual cortex (V1) and one designed to account for the responses of pattern direction selective cells in MT (or V5), an extrastriate visual area thought to be involved in the analysis of visual motion. These models share a common structure that operates in the same way on different kinds of input, and instantiate the widely held view that computational strategies are similar throughout the cerebral cortex. Implementations of these models for Macintosh microcomputers are available and can be used to explore the models' properties.

Theory of orientation tuning in visual cortex.

The role of intrinsic cortical connections in processing sensory input and in generating behavioral output is poorly understood. We have examined this issue in the context of the tuning of neuronal responses in cortex to the orientation of a visual stimulus. We analytically study a simple network model that incorporates both orientation-selective input from the lateral geniculate nucleus and orientation-specific cortical interactions. Depending on the model parameters, the network exhibits orientation selectivity that originates from within the cortex, by a symmetry-breaking mechanism. In this case, the width of the orientation tuning can be sharp even if the lateral geniculate nucleus inputs are only weakly anisotropic. By using our model, several experimental consequences of this cortical mechanism of orientation tuning are derived. The tuning width is relatively independent of the contrast and angular anisotropy of the visual stimulus. The transient population response to changing of the stimulus orientation exhibits a slow "virtual rotation." Neuronal cross-correlations exhibit long time tails, the sign of which depends on the preferred orientations of the cells and the stimulus orientation.

Active Visual Features Based on Events to Guide Robot Manipulators in Tracking Tasks

Traditional visual servoing systems do not deal with the topic of moving objects tracking. When these systems are employed to track a moving object, depending on the object velocity, visual features can go out of the image, causing the fail of the tracking task. This occurs specially when the object and the robot are both stopped and then the object starts the movement. In this work, we have employed a retina camera based on Address Event Representation (AER) in order to use events as input in the visual servoing system. The events launched by the camera indicate a pixel movement. Event visual information is processed only at the moment it occurs, reducing the response time of visual servoing systems when they are used to track moving objects.