Semantic feature-based visual attention model for pedestrian detection

Objective
Pedestrian detection under video surveillance systems has always been a hot topic in computer vision research. These systems are widely used in train stations, airports, large commercial plazas, and other public places. However, pedestrian detection remains difficult because of complex backgrounds. Given its development in recent years, the visual attention mechanism has attracted increasing attention in object detection and tracking research, and previous studies have achieved substantial progress and breakthroughs. We propose a novel pedestrian detection method based on the semantic features under the visual attention mechanism.
Method
The proposed semantic feature-based visual attention model is a spatial-temporal model that consists of two parts: the static visual attention model and the motion visual attention model. The static visual attention model in the spatial domain is constructed by combining bottom-up with top-down attention guidance. Based on the characteristics of pedestrians, the bottom-up visual attention model of Itti is improved by intensifying the orientation vectors of elementary visual features to make the visual saliency map suitable for pedestrian detection. In terms of pedestrian attributes, skin color is selected as a semantic feature for pedestrian detection. The regional and Gaussian models are adopted to construct the skin color model. Skin feature-based visual attention guidance is then proposed to complete the top-down process. The bottom-up and top-down visual attentions are linearly combined using the proper weights obtained from experiments to construct the static visual attention model in the spatial domain. The spatial-temporal visual attention model is then constructed via the motion features in the temporal domain. Based on the static visual attention model in the spatial domain, the frame difference method is combined with optical flowing to detect motion vectors. Filtering is applied to process the field of motion vectors. The saliency of motion vectors can be evaluated via motion entropy to make the selected motion feature more suitable for the spatial-temporal visual attention model.
Result
Standard datasets and practical videos are selected for the experiments. The experiments are performed on a MATLAB R2012a platform. The experimental results show that our spatial-temporal visual attention model demonstrates favorable robustness under various scenes, including indoor train station surveillance videos and outdoor scenes with swaying leaves. Our proposed model outperforms the visual attention model of Itti, the graph-based visual saliency model, the phase spectrum of quaternion Fourier transform model, and the motion channel model of Liu in terms of pedestrian detection. The proposed model achieves a 93% accuracy rate on the test video.
Conclusion
This paper proposes a novel pedestrian method based on the visual attention mechanism. A spatial-temporal visual attention model that uses low-level and semantic features is proposed to calculate the saliency map. Based on this model, the pedestrian targets can be detected through focus of attention shifts. The experimental results verify the effectiveness of the proposed attention model for detecting pedestrians.

Pseudoneglect and visual attention networks

Pseudoneglect represents the tendency for healthy individuals to show a slight but consistent bias in favour of stimuli appearing in the left visual field. The bias is often measured using variants of the line bisection task. An accurate model of the functional architecture of the visuospatial attention system must account for this widely observed phenomenon, as well as for modulation of the direction and magnitude of the bias within individuals by a variety of factors relating to the state of the participant and/or stimulus characteristics. To date, the neural correlates of pseudoneglect remain relatively unmapped. In the current thesis, I employed a combination of psychophysical measurements, electroencephalography (EEG) recording and transcranial direct current stimulation (tDCS) in an attempt to probe the neural generator(s) of pseudoneglect. In particular, I wished to utilise and investigate some of the factors known to modulate the bias (including age, time-on-task and the length of the to-be-bisected line) in order to identify neural processes and activity that are necessary and sufficient for the lateralized bias to arise. Across four experiments utilising a computerized version of a perceptual line bisection task, pseudoneglect was consistently observed at baseline in healthy young participants. However, decreased line length (experiments 1, 2 and 3), time-on-task (experiment 1) and healthy aging (experiment 3) were all found to modulate the bias. Specifically, all three modulations induced a rightward shift in subjective midpoint estimation. Additionally, the line length and time-on-task effects (experiment 1) and the line length and aging effects (experiment 3) were found to have additive relationships. In experiment 2, EEG measurements revealed the line length effect to be reflected in neural activity 100 – 200ms post-stimulus onset over source estimated posterior regions of the right hemisphere (RH: temporo-parietal junction (TPJ)). Long lines induced a hemispheric asymmetry in processing (in favour of the RH) during this period that was absent in short lines. In experiment 4, bi-parietal tDCS (Left Anodal/Right Cathodal) induced a polarity-specific rightward shift in bias, highlighting the crucial role played by parietal cortex in the genesis of pseudoneglect. The opposite polarity (Left Cathodal/Right Anodal) did not induce a change in bias. The combined results from the four experiments of the current thesis provide converging evidence as to the crucial role played by the RH in the genesis of pseudoneglect and in the processing of visual input more generally. The reduction in pseudoneglect with decreased line length, increased time-on-task and healthy aging may be explained by a reduction in RH function, and hence contribution to task processing, induced by each of these modulations. I discuss how behavioural and neuroimaging studies of pseudoneglect (and its various modulators) can provide empirical data upon which accurate formal models of visuospatial attention networks may be based and further tested.

An exploration of varieties of visual attention: ERP findings

A set of five tasks was designed to examine dynamic aspects of visual attention: selective attention to color, selective attention to pattern, dividing and switching attention between color and pattern, and selective attention to pattern with changing target. These varieties of visual attention were examined using the same set of stimuli under different instruction sets; thus differences between tasks cannot be attributed to differences in the perceptual features of the stimuli. ERP data are presented for each of these tasks. A within-task analysis of different stimulus types varying in similarity to the attended target feature revealed that an early frontal selection positivity (FSP) was evident in selective attention tasks, regardless of whether color was the attended feature. The scalp distribution of a later posterior selection negativity (SN) was affected by whether the attended feature was color or pattern. The SN was largely unaffected by dividing attention across color and pattern. A large widespread positivity was evident in most conditions, consisting of at least three subcomponents which were differentially affected by the attention conditions. These findings are discussed in relation to prior research and the time course of visual attention processes in the brain. (C) 1999 Elsevier Science B.V. All rights reserved.

Comparative effects of ageing and dementia of the Alzheimer type on orientation of visual attention

Age-related changes and the effects of dementia of the Alzheimer type (DAT) were investigated during a visual orienting attention task in which attention was pre-cued to one or other hemifields. Central cues were either valid, neutral, invalid or NoGo (inhibitory). The response time cost-benefit analysis showed a decreased benefit after valid cueing in the old compared with the young group with no change in the cost of invalid cueing. The older group were also slower over all cue types. These results suggest there is an age-related reduced ability to covertly orient attention in a visual hemifield before target onset. In contrast, the DAT group showed an increased response time benefit and showed a trend for a decreased cost in response time compared with controls. This was due to slowest response times after neutral cues. They also made significantly more response errors particularly following neutral cueing, and were less able to inhibit responses on NoGo trials than controls. The increased benefit and reduced cost found in the DAT group was interpreted as an impairment in dividing attention between left and right target locations.

spatiotemporal profile of visual attention

Visual attention, focus of attention, mexican hat profile, surround inhibition

Dyslexia in a French-Spanish bilingual girl: behavioural and neural modulations following a visual attention span intervention.

We report the case study of a French-Spanish bilingual dyslexic girl, MP, who exhibited a severe visual attention (VA) span deficit but preserved phonological skills. Behavioural investigation showed a severe reduction of reading speed for both single items (words and pseudo-words) and texts in the two languages. However, performance was more affected in French than in Spanish. MP was administered an intensive VA span intervention programme. Pre-post intervention comparison revealed a positive effect of intervention on her VA span abilities. The intervention further transferred to reading. It primarily resulted in faster identification of the regular and irregular words in French. The effect of intervention was rather modest in Spanish that only showed a tendency for faster word reading. Text reading improved in the two languages with a stronger effect in French but pseudo-word reading did not improve in either French or Spanish. The overall results suggest that VA span intervention may primarily enhance the fast global reading procedure, with stronger effects in French than in Spanish. MP underwent two fMRI sessions to explore her brain activations before and after VA span training. Prior to the intervention, fMRI assessment showed that the striate and extrastriate visual cortices alone were activated but none of the regions typically involved in VA span. Post-training fMRI revealed increased activation of the superior and inferior parietal cortices. Comparison of pre- and post-training activations revealed significant activation increase of the superior parietal lobes (BA 7) bilaterally. Thus, we show that a specific VA span intervention not only modulates reading performance but further results in increased brain activity within the superior parietal lobes known to housing VA span abilities. Furthermore, positive effects of VA span intervention on reading suggest that the ability to process multiple visual elements simultaneously is one cause of successful reading acquisition.

Visual attention and attribute attendance in multi-attribute choice experiments

Decision strategies in multi-attribute Choice Experiments are investigated using eye-tracking. The visual attention towards, and attendance of, attributes is examined. Stated attendance is found to diverge substantively from visual attendance of attributes. However, stated and visual attendance are shown to be informative, non-overlapping sources of information about respondent utility functions when incorporated into model estimation. Eye-tracking also reveals systematic nonattendance of attributes only by a minority of respondents. Most respondents visually attend most attributes most of the time. We find no compelling evidence that the level of attention is related to respondent certainty, or that higher or lower value attributes receive more or less attention

Endogenous-like orienting of visual attention in rats

This study investigated the orienting of visual attention in rats using a 3-hole nose-poke task analogous to Posner, Information processing in cognition: the Loyola Symposium, Erlbaum, Hillsdale, (1980) covert attention task for humans. The effects of non-predictive (50% valid and 50% invalid) and predictive (80% valid and 20% invalid) peripheral visual cues on reaction times and response accuracy to a target stimulus, using Stimuli-Onset Asynchronies (SOAs) varying between 200 and 1,200 ms, were investigated. The results showed shorter reaction times in valid trials relative to invalid trials for both subjects trained in the non-predictive and predictive conditions, particularly when the SOAs were 200 and 400 ms. However, the magnitude of this validity effect was significantly greater for subjects exposed to predictive cues, when the SOA was 800 ms. Subjects exposed to invalid predictive cues exhibited an increase in omission errors relative to subjects exposed to invalid non-predictive cues. In contrast, valid cues reduced the proportion of omission errors for subjects trained in the predictive condition relative to subjects trained in the non-predictive condition. These results are congruent with those usually reported for humans and indicate that, in addition to the exogenous capture of attention promoted by both predictive and non-predictive peripheral cues, rats exposed to predictive cues engaged an additional slower process equivalent to human`s endogenous orienting of attention. To our knowledge, this is the first demonstration of an endogenous-like process of covert orienting of visual attention in rats.

The role of the right frontal eye field in overt visual attention deployment as assessed by free visual exploration

The frontal eye field (FEF) is known to be involved in saccade generation and visual attention control. Studies applying covert attentional orienting paradigms have shown that the right FEF is involved in attentional shifts to both the left and the right hemifield. In the current study, we aimed at examining the effects of inhibitory continuous theta burst (cTBS) transcranial magnetic stimulation over the right FEF on overt attentional orienting, as measured by a free visual exploration paradigm. In forty-two healthy subjects, free visual exploration of naturalistic pictures was tested in three conditions: (1) after cTBS over the right FEF; (2) after cTBS over a control site (vertex); and, (3) without any stimulation. The results showed that cTBS over the right FEF-but not cTBS over the vertex-triggered significant changes in the spatial distribution of the cumulative fixation duration. Compared to the group without stimulation and the group with cTBS over the vertex, cTBS over the right FEF decreased cumulative fixation duration in the left and in the right peripheral regions, and increased cumulative fixation duration in the central region. The present study supports the view that the right FEF is involved in the bilateral control of not only covert, but also of overt, peripheral visual attention.

The influence of naturalistic, directionally non-specific motion on the spatial deployment of visual attention in right-hemispheric stroke

An impairment of the spatial deployment of visual attention during exploration of static (i.e., motionless) stimuli is a common finding after an acute, right-hemispheric stroke. However, less is known about how these deficits: a) are modulated through naturalistic motion (i.e., without directional, specific spatial features); and, b) evolve in the subacute/chronic post-stroke phase. In the present study, we investigated free visual exploration in three patient groups with subacute/chronic right-hemispheric stroke and in healthy subjects. The first group included patients with left visual neglect and a left visual field defect (VFD), the second patients with a left VFD but no neglect, and the third patients without neglect or VFD. Eye movements were measured in all participants while they freely explored a traffic scene without (static condition) and with (dynamic condition) naturalistic motion, i.e., cars moving from the right or left. In the static condition, all patient groups showed similar deployment of visual exploration (i.e., as measured by the cumulative fixation duration) as compared to healthy subjects, suggesting that recovery processes took place, with normal spatial allocation of attention. However, the more demanding dynamic condition with moving cars elicited different re-distribution patterns of visual attention, quite similar to those typically observed in acute stroke. Neglect patients with VFD showed a significant decrease of visual exploration in the contralesional space, whereas patients with VFD but no neglect showed a significant increase of visual exploration in the contralesional space. No differences, as compared to healthy subjects, were found in patients without neglect or VFD. These results suggest that naturalistic motion, without directional, specific spatial features, may critically influence the spatial distribution of visual attention in subacute/chronic stroke patients.

Monitoring visual attention on a neurorehabilitation environment based on interactive video

The use of new technologies in neurorehabilitation has led to higher intensity rehabilitation processes, extending therapies in an economically sustainable way. Interactive Video (IV) technology allows therapists to work with virtual environments that reproduce real situations. In this way, patients deal with Activities of the Daily Living (ADL) immersed within enhanced environments [1]. These rehabilitation exercises, which focus in re-learning lost functions, will try to modulate the neural plasticity processes [2]. This research presents a system where a neurorehabilitation IV-based environment has been integrated with an eye-tracker device in order to monitor and to interact using visual attention. While patients are interacting with the neurorehabilitation environment, their visual behavior is closely related with their cognitive state, which in turn mirrors the brain damage condition suffered by them [3] [4]. Patients’ gaze data can provide knowledge on their attention focus and their cognitive state, as well as on the validity of the rehabilitation tasks proposed [5].

Visual attention and perception models for assessing quality in 2D and 3D stereoscopic video

La medida de calidad de vídeo sigue siendo necesaria para definir los criterios que caracterizan una señal que cumpla los requisitos de visionado impuestos por el usuario. Las nuevas tecnologías, como el vídeo 3D estereoscópico o formatos más allá de la alta definición, imponen nuevos criterios que deben ser analizadas para obtener la mayor satisfacción posible del usuario. Entre los problemas detectados durante el desarrollo de esta tesis doctoral se han determinado fenómenos que afectan a distintas fases de la cadena de producción audiovisual y tipo de contenido variado. En primer lugar, el proceso de generación de contenidos debe encontrarse controlado mediante parámetros que eviten que se produzca el disconfort visual y, consecuentemente, fatiga visual, especialmente en lo relativo a contenidos de 3D estereoscópico, tanto de animación como de acción real. Por otro lado, la medida de calidad relativa a la fase de compresión de vídeo emplea métricas que en ocasiones no se encuentran adaptadas a la percepción del usuario. El empleo de modelos psicovisuales y diagramas de atención visual permitirían ponderar las áreas de la imagen de manera que se preste mayor importancia a los píxeles que el usuario enfocará con mayor probabilidad. Estos dos bloques se relacionan a través de la definición del término saliencia. Saliencia es la capacidad del sistema visual para caracterizar una imagen visualizada ponderando las áreas que más atractivas resultan al ojo humano. La saliencia en generación de contenidos estereoscópicos se refiere principalmente a la profundidad simulada mediante la ilusión óptica, medida en términos de distancia del objeto virtual al ojo humano. Sin embargo, en vídeo bidimensional, la saliencia no se basa en la profundidad, sino en otros elementos adicionales, como el movimiento, el nivel de detalle, la posición de los píxeles o la aparición de caras, que serán los factores básicos que compondrán el modelo de atención visual desarrollado. Con el objetivo de detectar las características de una secuencia de vídeo estereoscópico que, con mayor probabilidad, pueden generar disconfort visual, se consultó la extensa literatura relativa a este tema y se realizaron unas pruebas subjetivas preliminares con usuarios. De esta forma, se llegó a la conclusión de que se producía disconfort en los casos en que se producía un cambio abrupto en la distribución de profundidades simuladas de la imagen, aparte de otras degradaciones como la denominada “violación de ventana”. A través de nuevas pruebas subjetivas centradas en analizar estos efectos con diferentes distribuciones de profundidades, se trataron de concretar los parámetros que definían esta imagen. Los resultados de las pruebas demuestran que los cambios abruptos en imágenes se producen en entornos con movimientos y disparidades negativas elevadas que producen interferencias en los procesos de acomodación y vergencia del ojo humano, así como una necesidad en el aumento de los tiempos de enfoque del cristalino. En la mejora de las métricas de calidad a través de modelos que se adaptan al sistema visual humano, se realizaron también pruebas subjetivas que ayudaron a determinar la importancia de cada uno de los factores a la hora de enmascarar una determinada degradación. Los resultados demuestran una ligera mejora en los resultados obtenidos al aplicar máscaras de ponderación y atención visual, los cuales aproximan los parámetros de calidad objetiva a la respuesta del ojo humano. ABSTRACT Video quality assessment is still a necessary tool for defining the criteria to characterize a signal with the viewing requirements imposed by the final user. New technologies, such as 3D stereoscopic video and formats of HD and beyond HD oblige to develop new analysis of video features for obtaining the highest user’s satisfaction. Among the problems detected during the process of this doctoral thesis, it has been determined that some phenomena affect to different phases in the audiovisual production chain, apart from the type of content. On first instance, the generation of contents process should be enough controlled through parameters that avoid the occurrence of visual discomfort in observer’s eye, and consequently, visual fatigue. It is especially necessary controlling sequences of stereoscopic 3D, with both animation and live-action contents. On the other hand, video quality assessment, related to compression processes, should be improved because some objective metrics are adapted to user’s perception. The use of psychovisual models and visual attention diagrams allow the weighting of image regions of interest, giving more importance to the areas which the user will focus most probably. These two work fields are related together through the definition of the term saliency. Saliency is the capacity of human visual system for characterizing an image, highlighting the areas which result more attractive to the human eye. Saliency in generation of 3DTV contents refers mainly to the simulated depth of the optic illusion, i.e. the distance from the virtual object to the human eye. On the other hand, saliency is not based on virtual depth, but on other features, such as motion, level of detail, position of pixels in the frame or face detection, which are the basic features that are part of the developed visual attention model, as demonstrated with tests. Extensive literature involving visual comfort assessment was looked up, and the development of new preliminary subjective assessment with users was performed, in order to detect the features that increase the probability of discomfort to occur. With this methodology, the conclusions drawn confirmed that one common source of visual discomfort was when an abrupt change of disparity happened in video transitions, apart from other degradations, such as window violation. New quality assessment was performed to quantify the distribution of disparities over different sequences. The results confirmed that abrupt changes in negative parallax environment produce accommodation-vergence mismatches derived from the increasing time for human crystalline to focus the virtual objects. On the other side, for developing metrics that adapt to human visual system, additional subjective tests were developed to determine the importance of each factor, which masks a concrete distortion. Results demonstrated slight improvement after applying visual attention to objective metrics. This process of weighing pixels approximates the quality results to human eye’s response.

Visual attention to surfaces in three-dimensional space.

Although attention plays a significant role in vision, its spatial deployment and spread in the third dimension is not well understood. In visual search experiments we show that we cannot easily focus attention across isodepth loci unless they are part of a well-formed surface with locally coplanar elements. Yet we can easily spread our attention selectively across well-formed surfaces that span an extreme range of stereoscopic depths. In cueing experiments, we show that this spread of attention is, in part, obligatory. Attentional selectivity is reduced when targets and distractors are coplanar with or rest on a common receding stereoscopic plane. We conclude that attention cannot be efficiently allocated to arbitrary depths and extents in space but is linked to and spreads automatically across perceived surfaces.

Behavioural and neuroimaging investigations of the relationship between visual attention, affordance and action

In this thesis the relationship between visual attention, affordance and action was investigated using a combination of neuroimaging and behavioural studies. Neuronal activity and movement construction were assessed when individuals passively viewed or produced action towards stimuli varying in their affordance and/or attentional attributes. The main findings were: (i) the passive perception of both object and abstract visual patterns was associated with decreased alpha and/or beta activity in sensori-motor cortex, occipito-temporal cortex and cerebellum. These are brain regions associated with the planning and production of visually guided action; (ii) for object patterns, decreased alpha and beta activity was also observed in regions of superior parietal and premotor cortex. These regions contain neurons argued to be essential for matching hand kinematics with manipulate objects; and (iii) in both control participants and a deafferented individual, studies of planned and unplanned pointing manoeuvres revealed that the attentional bias of a stimulus was critical for fast, efficient action production whereas the affordance bias was critical in determining end-point accuracy. Taken together, these findings demonstrate that affordance is not a necessary prerequisite for the potential of motor codes. Rather, affordance enables the construction of motor responses that reflect object functionality and/or manipulability. They further demonstrate that visual attention is associated with the potentiation of motor codes. Indeed, directed visual attention would appear critical for speeded responses. These findings provide new insights into the roles of directed visual attention and affordance upon action.

A comparison between static and kinetic visual attention as a means of detecting age-related deterioration of the visual system and driving performance

A critical review of previous research revealed that visual attention tests, such as the Useful Field of View (UFOV) test, provided the best means of detecting age-related changes to the visual system that could potentially increase crash risk. However, the question was raised as to whether the UFOV, which was regarded as a static visual attention test, could be improved by inclusion of kinetic targets that more closely represent the driving task. A computer program was written to provide more information about the derivation of UFOV test scores. Although this investigation succeeded in providing new information, some of the commercially protected UFOV test procedures still remain unknown. Two kinetic visual attention tests (DRTS1 and 2), developed at Aston University to investigate inclusion of kinetic targets in visual attention tests, were introduced. The UFOV was found to be more repeatable than either of the kinetic visual attention tests and learning effects or age did not influence these findings. Determinants of static and kinetic visual attention were explored. Increasing target eccentricity led to reduced performance on the UFOV and DRTS1 tests. The DRTS2 was not affected by eccentricity but this may have been due to the style of presentation of its targets. This might also have explained why only the DRTS2 showed laterality effects (i.e. better performance to targets presented on the left hand side of the road). Radial location, explored using the UFOV test, showed that subjects responded best to targets positioned to the horizontal meridian. Distraction had opposite effects on static and kinetic visual attention. While UFOV test performance declined with distraction, DRTS1 performance increased. Previous research had shown that this striking difference was to be expected. Whereas the detection of static targets is attenuated in the presence of distracting stimuli, distracting stimuli that move in a structured flow field enhances the detection of moving targets. Subjects reacted more slowly to kinetic compared to static targets, longitudinal motion compared to angular motion and to increased self-motion. However, the effects of longitudinal motion, angular motion, self-motion and even target eccentricity were caused by target edge speed variations arising because of optic flow field effects. The UFOV test was more able to detect age-related changes to the visual system than were either of the kinetic visual attention tests. The driving samples investigated were too limited to draw firm conclusions. Nevertheless, the results presented showed that neither the DRTS2 nor the UFOV tests were powerful tools for the identification of drivers prone to crashes or poor driving performance.