View-Invariant Object Category Learning, Recognition, and Search: How Spatial and Object Attention Are Coordinated Using Surface-Based Attentional Shrouds

Air Force Office of Scientific Research (F49620-01-1-0397); National Science Foundation (SBE-0354378); Office of Naval Research (N00014-01-1-0624)

ART-EMAP: A Neural Network Architecture for Object Recognition by Evidence Accumulation

A new neural network architecture is introduced for the recognition of pattern classes after supervised and unsupervised learning. Applications include spatio-temporal image understanding and prediction and 3-D object recognition from a series of ambiguous 2-D views. The architecture, called ART-EMAP, achieves a synthesis of adaptive resonance theory (ART) and spatial and temporal evidence integration for dynamic predictive mapping (EMAP). ART-EMAP extends the capabilities of fuzzy ARTMAP in four incremental stages. Stage 1 introduces distributed pattern representation at a view category field. Stage 2 adds a decision criterion to the mapping between view and object categories, delaying identification of ambiguous objects when faced with a low confidence prediction. Stage 3 augments the system with a field where evidence accumulates in medium-term memory (MTM). Stage 4 adds an unsupervised learning process to fine-tune performance after the limited initial period of supervised network training. Each ART-EMAP stage is illustrated with a benchmark simulation example, using both noisy and noise-free data. A concluding set of simulations demonstrate ART-EMAP performance on a difficult 3-D object recognition problem.

A Neural Theory of Attentive Visual Search: Interactions of Boundary, Surface, Spatial, and Object Representations

Visual search data are given a unified quantitative explanation by a model of how spatial maps in the parietal cortex and object recognition categories in the inferotemporal cortex deploy attentional resources as they reciprocally interact with visual representations in the prestriate cortex. The model visual representations arc organized into multiple boundary and surface representations. Visual search in the model is initiated by organizing multiple items that lie within a given boundary or surface representation into a candidate search grouping. These items arc compared with object recognition categories to test for matches or mismatches. Mismatches can trigger deeper searches and recursive selection of new groupings until a target object io identified. This search model is algorithmically specified to quantitatively simulate search data using a single set of parameters, as well as to qualitatively explain a still larger data base, including data of Aks and Enns (1992), Bravo and Blake (1990), Chellazzi, Miller, Duncan, and Desimone (1993), Egeth, Viri, and Garbart (1984), Cohen and Ivry (1991), Enno and Rensink (1990), He and Nakayarna (1992), Humphreys, Quinlan, and Riddoch (1989), Mordkoff, Yantis, and Egeth (1990), Nakayama and Silverman (1986), Treisman and Gelade (1980), Treisman and Sato (1990), Wolfe, Cave, and Franzel (1989), and Wolfe and Friedman-Hill (1992). The model hereby provides an alternative to recent variations on the Feature Integration and Guided Search models, and grounds the analysis of visual search in neural models of preattentive vision, attentive object learning and categorization, and attentive spatial localization and orientation.

Texture Segregation By Visual Cortex: Perceptual Grouping, Attention, and Learning

A neural model is proposed of how laminar interactions in the visual cortex may learn and recognize object texture and form boundaries. The model brings together five interacting processes: region-based texture classification, contour-based boundary grouping, surface filling-in, spatial attention, and object attention. The model shows how form boundaries can determine regions in which surface filling-in occurs; how surface filling-in interacts with spatial attention to generate a form-fitting distribution of spatial attention, or attentional shroud; how the strongest shroud can inhibit weaker shrouds; and how the winning shroud regulates learning of texture categories, and thus the allocation of object attention. The model can discriminate abutted textures with blurred boundaries and is sensitive to texture boundary attributes like discontinuities in orientation and texture flow curvature as well as to relative orientations of texture elements. The model quantitatively fits a large set of human psychophysical data on orientation-based textures. Object boundar output of the model is compared to computer vision algorithms using a set of human segmented photographic images. The model classifies textures and suppresses noise using a multiple scale oriented filterbank and a distributed Adaptive Resonance Theory (dART) classifier. The matched signal between the bottom-up texture inputs and top-down learned texture categories is utilized by oriented competitive and cooperative grouping processes to generate texture boundaries that control surface filling-in and spatial attention. Topdown modulatory attentional feedback from boundary and surface representations to early filtering stages results in enhanced texture boundaries and more efficient learning of texture within attended surface regions. Surface-based attention also provides a self-supervising training signal for learning new textures. Importance of the surface-based attentional feedback in texture learning and classification is tested using a set of textured images from the Brodatz micro-texture album. Benchmark studies vary from 95.1% to 98.6% with attention, and from 90.6% to 93.2% without attention.

Facial and expression recognition for the blind using computer vision

Gemstone Team FACE

Division of labor in frontal eye field neurons during presaccadic remapping of visual receptive fields.

Our percept of visual stability across saccadic eye movements may be mediated by presaccadic remapping. Just before a saccade, neurons that remap become visually responsive at a future field (FF), which anticipates the saccade vector. Hence, the neurons use corollary discharge of saccades. Many of the neurons also decrease their response at the receptive field (RF). Presaccadic remapping occurs in several brain areas including the frontal eye field (FEF), which receives corollary discharge of saccades in its layer IV from a collicular-thalamic pathway. We studied, at two levels, the microcircuitry of remapping in the FEF. At the laminar level, we compared remapping between layers IV and V. At the cellular level, we compared remapping between different neuron types of layer IV. In the FEF in four monkeys (Macaca mulatta), we identified 27 layer IV neurons with orthodromic stimulation and 57 layer V neurons with antidromic stimulation from the superior colliculus. With the use of established criteria, we classified the layer IV neurons as putative excitatory (n = 11), putative inhibitory (n = 12), or ambiguous (n = 4). We found that just before a saccade, putative excitatory neurons increased their visual response at the RF, putative inhibitory neurons showed no change, and ambiguous neurons increased their visual response at the FF. None of the neurons showed presaccadic visual changes at both RF and FF. In contrast, neurons in layer V showed full remapping (at both the RF and FF). Our data suggest that elemental signals for remapping are distributed across neuron types in early cortical processing and combined in later stages of cortical microcircuitry.

Frontal eye field neurons assess visual stability across saccades.

The image on the retina may move because the eyes move, or because something in the visual scene moves. The brain is not fooled by this ambiguity. Even as we make saccades, we are able to detect whether visual objects remain stable or move. Here we test whether this ability to assess visual stability across saccades is present at the single-neuron level in the frontal eye field (FEF), an area that receives both visual input and information about imminent saccades. Our hypothesis was that neurons in the FEF report whether a visual stimulus remains stable or moves as a saccade is made. Monkeys made saccades in the presence of a visual stimulus outside of the receptive field. In some trials, the stimulus remained stable, but in other trials, it moved during the saccade. In every trial, the stimulus occupied the center of the receptive field after the saccade, thus evoking a reafferent visual response. We found that many FEF neurons signaled, in the strength and timing of their reafferent response, whether the stimulus had remained stable or moved. Reafferent responses were tuned for the amount of stimulus translation, and, in accordance with human psychophysics, tuning was better (more prevalent, stronger, and quicker) for stimuli that moved perpendicular, rather than parallel, to the saccade. Tuning was sometimes present as well for nonspatial transaccadic changes (in color, size, or both). Our results indicate that FEF neurons evaluate visual stability during saccades and may be general purpose detectors of transaccadic visual change.

A subcortical source of visual input to the frontal eye field

Many neurons in the frontal eye field (FEF) exhibit visual responses and are thought to play important roles in visuosaccadic behavior. The FEF, however, is far removed from striate cortex. Where do the FEF's visual signals come from? Usually they are reasonably assumed to enter the FEF through afferents from extrastriate cortex. Here we show that, surprisingly, visual signals also enter the FEF through a subcortical route: a disynaptic, ascending pathway originating in the intermediate layers of the superior colliculus (SC). We recorded from identified neurons at all three stages of this pathway (n=30-40 in each sample): FEF recipient neurons, orthodromically activated from the SC; mediodorsal thalamus (MD) relay neurons, antidromically activated from FEF and orthodromically activated from SC; and SC source neurons, antidromically activated from MD. We studied the neurons while monkeys performed delayed saccade tasks designed to temporally resolve visual responses from presaccadic discharges. We found, first, that most neurons at every stage in the pathway had visual responses, presaccadic bursts, or both. Second, we found marked similarities between the SC source neurons and MD relay neurons: in both samples, about 15% of the neurons had only a visual response, 10% had only a presaccadic burst, and 75% had both. In contrast, FEF recipient neurons tended to be more visual in nature: 50% had only a visual response, none had only a presaccadic burst, and 50% had both a visual response and a presaccadic burst. This suggests that in addition to their subcortical inputs, these FEF neurons also receive other visual inputs, e.g. from extrastriate cortex. We conclude that visual activity in the FEF results not only from cortical afferents but also from subcortical inputs. Intriguingly, this implies that some of the visual signals in FEF are pre-processed by the SC.

The frontal eye field sends predictively remapped visual signals to the superior colliculus

We perceive a stable visual world even though saccades often move our retinas. One way the brain may achieve a stable visual percept is through predictive remapping of visual receptive fields: just before a saccade, the receptive field of many neurons moves from its current location ("current receptive field") to the location it is expected to occupy after the saccade ("future receptive field"). Goldberg and colleagues found such remapping in cortical areas, e.g. in the frontal eye field (FEF), as well as in the intermediate layers of the superior colliculus (SC). In the present study we investigated the source of the SC's remapped visual signals. Do some of them come from the FEF? We identified FEF neurons that project to the SC using antidromic stimulation. For neurons with a visual response, we tested whether the receptive field shifted just prior to making a saccade. Saccadic amplitudes were chosen to be as small as possible while clearly separating the current and future receptive fields; they ranged from 5-30 deg. in amplitude and were directed contraversively. The saccadic target was a small red spot. We probed visual responsiveness at the current and future receptive field locations using a white spot flashed at various times before or after the saccade. Predictive remapping was indicated by a visual response to a probe flashed in the future receptive field just before the saccade began. We found that many FEF neurons projecting to the SC exhibited predictive remapping. Moreover, the remapping was as fast and strong as any previously reported for FEF or SC. It is clear, therefore, that remapped visual signals are sent from FEF to SC, providing direct evidence that the FEF is one source of the SC's remapped visual signals. Because remapping requires information about an imminent saccade, we hypothesize that remapping in FEF depends on corollary discharge signals such as those ascending from the SC through MD thalamus (Sommer and Wurtz 2002).

Influence of the thalamus on spatial visual processing in frontal cortex.

Each of our movements activates our own sensory receptors, and therefore keeping track of self-movement is a necessary part of analysing sensory input. One way in which the brain keeps track of self-movement is by monitoring an internal copy, or corollary discharge, of motor commands. This concept could explain why we perceive a stable visual world despite our frequent quick, or saccadic, eye movements: corollary discharge about each saccade would permit the visual system to ignore saccade-induced visual changes. The critical missing link has been the connection between corollary discharge and visual processing. Here we show that such a link is formed by a corollary discharge from the thalamus that targets the frontal cortex. In the thalamus, neurons in the mediodorsal nucleus relay a corollary discharge of saccades from the midbrain superior colliculus to the cortical frontal eye field. In the frontal eye field, neurons use corollary discharge to shift their visual receptive fields spatially before saccades. We tested the hypothesis that these two components-a pathway for corollary discharge and neurons with shifting receptive fields-form a circuit in which the corollary discharge drives the shift. First we showed that the known spatial and temporal properties of the corollary discharge predict the dynamic changes in spatial visual processing of cortical neurons when saccades are made. Then we moved from this correlation to causation by isolating single cortical neurons and showing that their spatial visual processing is impaired when corollary discharge from the thalamus is interrupted. Thus the visual processing of frontal neurons is spatiotemporally matched with, and functionally dependent on, corollary discharge input from the thalamus. These experiments establish the first link between corollary discharge and visual processing, delineate a brain circuit that is well suited for mediating visual stability, and provide a framework for studying corollary discharge in other sensory systems.

Towards Real-time Speech Emotion Recognition for Affective E-Learning

The original article is available as an open access file on the Springer website in the following link: http://link.springer.com/article/10.1007/s10639-015-9388-2

Neurobiological model of visuo-spatial cognition in young Williams Syndrome children: measures of dorsal-stream and frontal function

We examine hypotheses for the neural basis of the profile of visual cognition in young children with Williams syndrome (WS). These are: (a) that it is a consequence of anomalies in sensory visual processing; (b) that it is a deficit of the dorsal relative to the ventral cortical stream; (c) that it reflects deficit of frontal function, in particular of fronto-parietal interaction; (d) that it is related to impaired function in the right hemisphere relative to the left. The tests reported here are particularly relevant to (b) and (c). They form part of a more extensive programme of investigating visual, visuospatial, and cognitive function in large group of children with WS children, aged 8 months to 15 years. To compare performance across tests, avoiding floor and ceiling effects, we have measured performance in children with WS in terms of the ‘age equivalence’ for typically developing children. In this paper the relation between dorsal and ventral function was tested by motion and form coherence thresholds respectively. We confirm the presence of a subgroup of children with WS who perform particularly poorly on the motion (dorsal) task. However, such performance is also characteristic of normally developingchildren up to 5 years: thus the WS performance may reflect an overall persisting immaturity of visuospatial processing which is particularly evident in the dorsal stream. Looking at the performance on the global coherence tasks of the entire WS group, we find that there is also a subgroup who have both high form and motion coherence thresholds, relative to the performance of children of the same chronological age and verbal age on the BPVS, suggesting a more general global processing deficit. Frontal function was tested by a counterpointing task, ability to retrieve a ball from a ‘detour box’, and the Stroop-like ‘day-night’ task, all of which require inhibition of a familiar response. When considered in relation to overall development as indexed by vocabulary, the day-night task shows little specific impairment, the detour box shows a significant delay relative to controls, and the counterpointing task shows a marked and persistent deficit in many children. We conclude that frontal control processes show most impairment in WS when they are associated with spatially directed responses, reflecting a deficit of fronto-parietal processing. However, children with WS may successfully reduce the effect of this impairment by verbally mediated strategies. On all these tasks we find a range of difficulties across individual children and a small subset of WS who show very good performance, equivalent to chronological age norms of typically developing children. Neurobiological models of visuo-spatial cognition in children with WS p.4 Overall, we conclude that children with WS have specific processing difficulties with tasks involving frontoparietal circuits within the spatial domain. However, some children with WS can achieve similar performance to typically developing children on some tasks involving the dorsal stream, although the strategies and processing may be different in the two groups.

The visual representation of the “indignados”. A visual analysis about the characterization of the participants of the 15-M movement.

The aim of this work is to show the type of media coverage done by the newspapers La Razón, El País and Público about the 15-M social movement during the time that the camping at Sol took place. Specifically, in terms of how the characterization of the “indignados” (outraged) got made. Based on our previous descriptive observations, we approached a visual analysis of the photographs published on the paper editions of those mainstream media from May 15-June 12 of the 2011. we started from a total sample of 379 items, developing:1) A content analysis of La Razón, El País, and Público; the most frequents words of each media, articles classifications from the reviews found on them (expositive, positive-evaluation, negative-evaluation).2) An analysis of the 408 images obtained from the total sample, which establishes a clear evolution of the “indignados” profile and how differently each media took the movement as such. That’s, when they stop naming them “indignados”, and recognize its nature as social movement by calling it: “Movimiento 15-M"...

Places making: participatory construction of scientifi c culture cities

In the framework of the European project Platform of Local Authorities and Communicators Engaged in Science (PLACES), we analyse the articulations between scientifi c communication, public perception of science, processes of citizen participation and apropiation of space, based on a case study of the inhabitants of Teruel city, Autonomous Community of Aragon, Spain. On the interrelationships between these issues, there are a number of contradictions, such as the difference between a high interest for information about science and technology and a low level of recognition and interaction with local institutions involved in those activities, the complex conceptualization of scientifi c space in relation to the “public-private” pair, or an articulation of a claiming civic rethoric and an insuffi cient co-responsibility. We conclude that, in a local context, the dimension of territoriality and, in particular, the identifi cation with the town, is a central mediation for activating citizen participation as part of processes of appropriation of space for setting up cities of scientifi c culture.