Visual attention and perception models for assessing quality in 2D and 3D stereoscopic video

La medida de calidad de vídeo sigue siendo necesaria para definir los criterios que caracterizan una señal que cumpla los requisitos de visionado impuestos por el usuario. Las nuevas tecnologías, como el vídeo 3D estereoscópico o formatos más allá de la alta definición, imponen nuevos criterios que deben ser analizadas para obtener la mayor satisfacción posible del usuario. Entre los problemas detectados durante el desarrollo de esta tesis doctoral se han determinado fenómenos que afectan a distintas fases de la cadena de producción audiovisual y tipo de contenido variado. En primer lugar, el proceso de generación de contenidos debe encontrarse controlado mediante parámetros que eviten que se produzca el disconfort visual y, consecuentemente, fatiga visual, especialmente en lo relativo a contenidos de 3D estereoscópico, tanto de animación como de acción real. Por otro lado, la medida de calidad relativa a la fase de compresión de vídeo emplea métricas que en ocasiones no se encuentran adaptadas a la percepción del usuario. El empleo de modelos psicovisuales y diagramas de atención visual permitirían ponderar las áreas de la imagen de manera que se preste mayor importancia a los píxeles que el usuario enfocará con mayor probabilidad. Estos dos bloques se relacionan a través de la definición del término saliencia. Saliencia es la capacidad del sistema visual para caracterizar una imagen visualizada ponderando las áreas que más atractivas resultan al ojo humano. La saliencia en generación de contenidos estereoscópicos se refiere principalmente a la profundidad simulada mediante la ilusión óptica, medida en términos de distancia del objeto virtual al ojo humano. Sin embargo, en vídeo bidimensional, la saliencia no se basa en la profundidad, sino en otros elementos adicionales, como el movimiento, el nivel de detalle, la posición de los píxeles o la aparición de caras, que serán los factores básicos que compondrán el modelo de atención visual desarrollado. Con el objetivo de detectar las características de una secuencia de vídeo estereoscópico que, con mayor probabilidad, pueden generar disconfort visual, se consultó la extensa literatura relativa a este tema y se realizaron unas pruebas subjetivas preliminares con usuarios. De esta forma, se llegó a la conclusión de que se producía disconfort en los casos en que se producía un cambio abrupto en la distribución de profundidades simuladas de la imagen, aparte de otras degradaciones como la denominada “violación de ventana”. A través de nuevas pruebas subjetivas centradas en analizar estos efectos con diferentes distribuciones de profundidades, se trataron de concretar los parámetros que definían esta imagen. Los resultados de las pruebas demuestran que los cambios abruptos en imágenes se producen en entornos con movimientos y disparidades negativas elevadas que producen interferencias en los procesos de acomodación y vergencia del ojo humano, así como una necesidad en el aumento de los tiempos de enfoque del cristalino. En la mejora de las métricas de calidad a través de modelos que se adaptan al sistema visual humano, se realizaron también pruebas subjetivas que ayudaron a determinar la importancia de cada uno de los factores a la hora de enmascarar una determinada degradación. Los resultados demuestran una ligera mejora en los resultados obtenidos al aplicar máscaras de ponderación y atención visual, los cuales aproximan los parámetros de calidad objetiva a la respuesta del ojo humano. ABSTRACT Video quality assessment is still a necessary tool for defining the criteria to characterize a signal with the viewing requirements imposed by the final user. New technologies, such as 3D stereoscopic video and formats of HD and beyond HD oblige to develop new analysis of video features for obtaining the highest user’s satisfaction. Among the problems detected during the process of this doctoral thesis, it has been determined that some phenomena affect to different phases in the audiovisual production chain, apart from the type of content. On first instance, the generation of contents process should be enough controlled through parameters that avoid the occurrence of visual discomfort in observer’s eye, and consequently, visual fatigue. It is especially necessary controlling sequences of stereoscopic 3D, with both animation and live-action contents. On the other hand, video quality assessment, related to compression processes, should be improved because some objective metrics are adapted to user’s perception. The use of psychovisual models and visual attention diagrams allow the weighting of image regions of interest, giving more importance to the areas which the user will focus most probably. These two work fields are related together through the definition of the term saliency. Saliency is the capacity of human visual system for characterizing an image, highlighting the areas which result more attractive to the human eye. Saliency in generation of 3DTV contents refers mainly to the simulated depth of the optic illusion, i.e. the distance from the virtual object to the human eye. On the other hand, saliency is not based on virtual depth, but on other features, such as motion, level of detail, position of pixels in the frame or face detection, which are the basic features that are part of the developed visual attention model, as demonstrated with tests. Extensive literature involving visual comfort assessment was looked up, and the development of new preliminary subjective assessment with users was performed, in order to detect the features that increase the probability of discomfort to occur. With this methodology, the conclusions drawn confirmed that one common source of visual discomfort was when an abrupt change of disparity happened in video transitions, apart from other degradations, such as window violation. New quality assessment was performed to quantify the distribution of disparities over different sequences. The results confirmed that abrupt changes in negative parallax environment produce accommodation-vergence mismatches derived from the increasing time for human crystalline to focus the virtual objects. On the other side, for developing metrics that adapt to human visual system, additional subjective tests were developed to determine the importance of each factor, which masks a concrete distortion. Results demonstrated slight improvement after applying visual attention to objective metrics. This process of weighing pixels approximates the quality results to human eye’s response.

Mechanisms of migraine aura revealed by functional MRI in human visual cortex

Cortical spreading depression (CSD) has been suggested to underlie migraine visual aura. However, it has been challenging to test this hypothesis in human cerebral cortex. Using high-field functional MRI with near-continuous recording during visual aura in three subjects, we observed blood oxygenation level-dependent (BOLD) signal changes that demonstrated at least eight characteristics of CSD, time-locked to percept/onset of the aura. Initially, a focal increase in BOLD signal (possibly reflecting vasodilation), developed within extrastriate cortex (area V3A). This BOLD change progressed contiguously and slowly (3.5 ± 1.1 mm/min) over occipital cortex, congruent with the retinotopy of the visual percept. Following the same retinotopic progression, the BOLD signal then diminished (possibly reflecting vasoconstriction after the initial vasodilation), as did the BOLD response to visual activation. During periods with no visual stimulation, but while the subject was experiencing scintillations, BOLD signal followed the retinotopic progression of the visual percept. These data strongly suggest that an electrophysiological event such as CSD generates the aura in human visual cortex.

Requirement of the nicotinic acetylcholine receptor β2 subunit for the anatomical and functional development of the visual system

In the mammalian visual system the formation of eye-specific layers at the thalamic level depends on retinal waves of spontaneous activity, which rely on nicotinic acetylcholine receptor activation. We found that in mutant mice lacking the β2 subunit of the neuronal nicotinic receptor, but not in mice lacking the α4 subunit, retinofugal projections do not segregate into eye-specific areas, both in the dorso-lateral geniculate nucleus and in the superior colliculus. Moreover, β2−/− mice show an expansion of the binocular subfield of the primary visual cortex and a decrease in visual acuity at the cortical level but not in the retina. We conclude that the β2 subunit of the nicotinic acetylcholine receptor is necessary for the anatomical and functional development of the visual system.

Faithful representation of similarities among three-dimensional shapes in human vision.

Efficient and reliable classification of visual stimuli requires that their representations reside a low-dimensional and, therefore, computationally manageable feature space. We investigated the ability of the human visual system to derive such representations from the sensory input-a highly nontrivial task, given the million or so dimensions of the visual signal at its entry point to the cortex. In a series of experiments, subjects were presented with sets of parametrically defined shapes; the points in the common high-dimensional parameter space corresponding to the individual shapes formed regular planar (two-dimensional) patterns such as a triangle, a square, etc. We then used multidimensional scaling to arrange the shapes in planar configurations, dictated by their experimentally determined perceived similarities. The resulting configurations closely resembled the original arrangements of the stimuli in the parameter space. This achievement of the human visual system was replicated by a computational model derived from a theory of object representation in the brain, according to which similarities between objects, and not the geometry of each object, need to be faithfully represented.

Expression and analysis of presenilin 1 in a human neuronal system: localization in cell bodies and dendrites.

Mutations in the recently identified presenilin 1 gene on chromosome 14 cause early onset familial Alzheimer disease (FAD). Herein we describe the expression and analysis of the protein coded by presenilin 1 (PS1) in NT2N neurons, a human neuronal model system. PS1 was expressed using recombinant Semliki Forest virions and detected by introduced antigenic tags or antisera to PS1-derived peptides. Immunoprecipitation revealed two major PS1 bands of approximately 43 and 50 kDa, neither of which were N-glycosylated or O-glycosylated. Immunoreactive PS1 was detected in cell bodies and dendrites of NT2N neurons but not in axons or on the cell surface. PS1 was also detected in BHK cells, where it was also intracellular and colocalized with calnexin, a marker for the rough endoplasmic reticulum. A mutant form of PS1 linked to FAD did not differ from the wild-type protein at the light microscopic level. The model system described here will enable studies of the function of PS1 in human neurons and the role of mutant PS1 in FAD.

Emergence of order in visual system development.

Neural connections in the adult central nervous system are highly precise. In the visual system, retinal ganglion cells send their axons to target neurons in the lateral geniculate nucleus (LGN) in such a way that axons originating from the two eyes terminate in adjacent but nonoverlapping eye-specific layers. During development, however, inputs from the two eyes are intermixed, and the adult pattern emerges gradually as axons from the two eyes sort out to form the layers. Experiments indicate that the sorting-out process, even though it occurs in utero in higher mammals and always before vision, requires retinal ganglion cell signaling; blocking retinal ganglion cell action potentials with tetrodotoxin prevents the formation of the layers. These action potentials are endogenously generated by the ganglion cells, which fire spontaneously and synchronously with each other, generating "waves" of activity that travel across the retina. Calcium imaging of the retina shows that the ganglion cells undergo correlated calcium bursting to generate the waves and that amacrine cells also participate in the correlated activity patterns. Physiological recordings from LGN neurons in vitro indicate that the quasiperiodic activity generated by the retinal ganglion cells is transmitted across the synapse between ganglion cells to drive target LGN neurons. These observations suggest that (i) a neural circuit within the immature retina is responsible for generating specific spatiotemporal patterns of neural activity; (ii) spontaneous activity generated in the retina is propagated across central synapses; and (iii) even before the photoreceptors are present, nerve cell function is essential for correct wiring of the visual system during early development. Since spontaneously generated activity is known to be present elsewhere in the developing CNS, this process of activity-dependent wiring could be used throughout the nervous system to help refine early sets of neural connections into their highly precise adult patterns.

A behavioral screen for isolating zebrafish mutants with visual system defects.

Optokinetic and phototactic behaviors of zebrafish larvae were examined for their usefulness in screening for recessive defects in the visual system. The optokinetic response can be reliably and rapidly detected in 5-day larvae, whereas the phototactic response of larvae is variable and not robust enough to be useful for screening. We therefore measured optokinetic responses of mutagenized larvae as a genetic screen for visual system defects. Third-generation larvae, representing 266 mutagenized genomes, were examined for abnormal optokinetic responses. Eighteen optokinetic-defective mutants were identified and two mutants that did not show obvious morphological defects, no optokinetic response a (noa) and partial optokinetic response a (poa), were studied further. We recorded the electroretinogram (ERG) to determine whether these two mutations affect the retina. The b-wave of noa larvae was grossly abnormal, being delayed in onset and significantly reduced in amplitude. In contrast, the ERG waveform of poa larvae was normal, although the b-wave was reduced in amplitude in bright light. Histologically, the retinas of noa and poa larvae appeared normal. We conclude that noa larvae have a functional defect in the outer retina, whereas the outer retina of poa larvae is likely to be normal.

System for robot control inspired by the functionality and organisation of the human nervous system

This paper presents a model of a control system for robot systems inspired by the functionality and organisation of human neuroregulatory system. Our model was specified using software agents within a formal framework and implemented through Web Services. This approach allows the implementation of the control logic of a robot system with relative ease, in an incremental way, using the addition of new control centres to the system as its behaviour is observed or needs to be detailed with greater precision, without the need to modify existing functionality. The tests performed verify that the proposed model has the general characteristics of biological systems together with the desirable features of software, such as robustness, flexibility, reuse and decoupling.

Driver visibility quality: comparison of visibility quality meter data with human visual performance. Final report.

National Highway Traffic Safety Administration, Washington, D.C.

Stimuli of varying spatial scale induce gamma activity with distinct temporal characteristics in human visual cortex

Gamma activity to stationary grating stimuli was studied non-invasively using MEG recordings in humans. Using a spatial filtering technique, we localized gamma activity to primary visual cortex. We tested the hypothesis that spatial frequency properties of visual stimuli may be related to the temporal frequency characteristics of the associated cortical responses. We devised a method to assess temporal frequency differences between stimulus-related responses that typically exhibit complex spectral shapes. We applied this methodology to either single-trial (induced) or time-averaged (evoked) responses in four frequency ranges (0-40, 20-60, 40-80 and 60-100 Hz) and two time windows (either the entire duration of stimulus presentation or the first second following stimulus onset). Our results suggest that stimuli of varying spatial frequency induce responses that exhibit significantly different temporal frequency characteristics. These effects were particularly accentuated for induced responses in the classical gamma frequency band (20-60 Hz) analyzed over the entire duration of stimulus presentation. Strikingly, examining the first second of the responses following stimulus onset resulted in significant loss in stimulus specificity, suggesting that late signal components contain functionally relevant information. These findings advocate a functional role of gamma activity in sensory representation. We suggest that stimulus specific frequency characteristics of MEG signals can be mapped to processes of neuronal synchronization within the framework of coupled dynamical systems.

Interocular suppression and contrast gain control in human vision

The human visual system combines contrast information from the two eyes to produce a single cyclopean representation of the external world. This task requires both summation of congruent images and inhibition of incongruent images across the eyes. These processes were explored psychophysically using narrowband sinusoidal grating stimuli. Initial experiments focussed on binocular interactions within a single detecting mechanism, using contrast discrimination and contrast matching tasks. Consistent with previous findings, dichoptic presentation produced greater masking than monocular or binocular presentation. Four computational models were compared, two of which performed well on all data sets. Suppression between mechanisms was then investigated, using orthogonal and oblique stimuli. Two distinct suppressive pathways were identified, corresponding to monocular and dichoptic presentation. Both pathways impact prior to binocular summation of signals, and differ in their strengths, tuning, and response to adaptation, consistent with recent single-cell findings in cat. Strikingly, the magnitude of dichoptic masking was found to be spatiotemporally scale invariant, whereas monocular masking was dependent on stimulus speed. Interocular suppression was further explored using a novel manipulation, whereby stimuli were presented in dichoptic antiphase. Consistent with the predictions of a computational model, this produced weaker masking than in-phase presentation. This allowed the bandwidths of suppression to be measured without the complicating factor of additive combination of mask and test. Finally, contrast vision in strabismic amblyopia was investigated. Although amblyopes are generally believed to have impaired binocular vision, binocular summation was shown to be intact when stimuli were normalized for interocular sensitivity differences. An alternative account of amblyopia was developed, in which signals in the affected eye are subject to attenuation and additive noise prior to binocular combination.

Attentional changes in pre-stimulus oscillatory activity within early visual cortex are predictive of human visual performance

Physiological and neuroimaging studies provide evidence to suggest that attentional mechanisms operating within the fronto-parietal network may exert top–down control on early visual areas, priming them for forthcoming sensory events. The believed consequence of such priming is enhanced task performance. Using the technique of magnetoencephalography (MEG), we investigated this possibility by examining whether attention-driven changes in cortical activity are correlated with performance on a line-orientation judgment task. We observed that, approximately 200 ms after a covert attentional shift towards the impending visual stimulus, the level of phase-resetting (transient neural coherence) within the calcarine significantly increased for 2–10 Hz activity. This was followed by a suppression of alpha activity (near 10 Hz) which persisted until the onset of the stimulus. The levels of phase-resetting, alpha suppression and subsequent behavioral performance varied between subjects in a systematic fashion. The magnitudes of phase-resetting and alpha-band power were negatively correlated, with high levels of coherence associated with high levels of performance. We propose that top–down attentional control mechanisms exert their initial effects within the calcarine through a phase-resetting within the 2–10 Hz band, which in turn triggers a suppression of alpha activity, priming early visual areas for incoming information and enhancing behavioral performance.

Attentional modulation of oscillatory activity in human visual cortex

The effects of attentional modulation on activity within the human visual cortex were investigated using magnetoencephalography. Chromatic sinusoidal stimuli were used to evoke activity from the occipital cortex, with attention directed either toward or away from the stimulus using a bar-orientation judgment task. For five observers, global magnetic field power was plotted as a function of time from stimulus onset. The major peak of each function occurred at about 120 ms latency and was well modeled by a current dipole near the calcarine sulcus. Independent component analysis (ICA) on the non-averaged data for each observer also revealed one component of calcarine origin, the location of which matched that of the dipolar source determined from the averaged data. For two observers, ICA revealed a second component near the parieto-occipital sulcus. Although no effects of attention were evident using standard averaging procedures, time-varying spectral analyses of single trials revealed that the main effect of attention was to alter the level of oscillatory activity. Most notably, a sustained increase in alpha-band (7-12 Hz) activity of both calcarine and parieto-occipital origin was evident. In addition, calcarine activity in the range of 13-21 Hz was enhanced, while calcarine activity in the range of 5-6 Hz was reduced. Our results are consistent with the hypothesis that attentional modulation affects neural processing within the calcarine and parieto-occipital cortex by altering the amplitude of alpha-band activity and other natural brain rhythms. © 2003 Elsevier Inc. All rights reserved.

Colour and luminance interactions in the visual perception of motion

We sought to determine the extent to which red–green, colour–opponent mechanisms in the human visual system play a role in the perception of drifting luminance–modulated targets. Contrast sensitivity for the directional discrimination of drifting luminance–modulated (yellow–black) test sinusoids was measured following adaptation to isoluminant red–green sinusoids drifting in either the same or opposite direction. When the test and adapt stimuli drifted in the same direction, large sensitivity losses were evident at all test temporal frequencies employed (1–16 Hz). The magnitude of the loss was independent of temporal frequency. When adapt and test stimuli drifted in opposing directions, large sensitivity losses were evident at lower temporal frequencies (1–4 Hz) and declined with increasing temporal frequency. Control studies showed that this temporal–frequency–dependent effect could not reflect the activity of achromatic units. Our results provide evidence that chromatic mechanisms contribute to the perception of luminance–modulated motion targets drifting at speeds of up to at least 32°s-1. We argue that such mechanisms most probably lie within a parvocellular–dominated cortical visual pathway, sensitive to both chromatic and luminance modulation, but only weakly selective for the direction of stimulus motion.

The human immune system: Part 1

The immune system protects the human body against infectious and maligant disease. The concept of an immune system arose because of the observation that an attack of measles or mumps, two common childhood diseases, conferred an immunity on the individual, the immunity being specific to the disease. It was only much later that it was discovered that a system in the body conferred this immunity.