Multi-scale 3D Scene Flow from Binocular Stereo Sequences

Scene flow methods estimate the three-dimensional motion field for points in the world, using multi-camera video data. Such methods combine multi-view reconstruction with motion estimation approaches. This paper describes an alternative formulation for dense scene flow estimation that provides convincing results using only two cameras by fusing stereo and optical flow estimation into a single coherent framework. To handle the aperture problems inherent in the estimation task, a multi-scale method along with a novel adaptive smoothing technique is used to gain a regularized solution. This combined approach both preserves discontinuities and prevents over-regularization-two problems commonly associated with basic multi-scale approaches. Internally, the framework generates probability distributions for optical flow and disparity. Taking into account the uncertainty in the intermediate stages allows for more reliable estimation of the 3D scene flow than standard stereo and optical flow methods allow. Experiments with synthetic and real test data demonstrate the effectiveness of the approach.

Multi-Scale 3D Scene Flow from Binocular Stereo Sequences

Scene flow methods estimate the three-dimensional motion field for points in the world, using multi-camera video data. Such methods combine multi-view reconstruction with motion estimation. This paper describes an alternative formulation for dense scene flow estimation that provides reliable results using only two cameras by fusing stereo and optical flow estimation into a single coherent framework. Internally, the proposed algorithm generates probability distributions for optical flow and disparity. Taking into account the uncertainty in the intermediate stages allows for more reliable estimation of the 3D scene flow than previous methods allow. To handle the aperture problems inherent in the estimation of optical flow and disparity, a multi-scale method along with a novel region-based technique is used within a regularized solution. This combined approach both preserves discontinuities and prevents over-regularization – two problems commonly associated with the basic multi-scale approaches. Experiments with synthetic and real test data demonstrate the strength of the proposed approach.

Foreground Object Segmentation from Binocular Stereo Video

Moving cameras are needed for a wide range of applications in robotics, vehicle systems, surveillance, etc. However, many foreground object segmentation methods reported in the literature are unsuitable for such settings; these methods assume that the camera is fixed and the background changes slowly, and are inadequate for segmenting objects in video if there is significant motion of the camera or background. To address this shortcoming, a new method for segmenting foreground objects is proposed that utilizes binocular video. The method is demonstrated in the application of tracking and segmenting people in video who are approximately facing the binocular camera rig. Given a stereo image pair, the system first tries to find faces. Starting at each face, the region containing the person is grown by merging regions from an over-segmented color image. The disparity map is used to guide this merging process. The system has been implemented on a consumer-grade PC, and tested on video sequences of people indoors obtained from a moving camera rig. As can be expected, the proposed method works well in situations where other foreground-background segmentation methods typically fail. We believe that this superior performance is partly due to the use of object detection to guide region merging in disparity/color foreground segmentation, and partly due to the use of disparity information available with a binocular rig, in contrast with most previous methods that assumed monocular sequences.

On Interpreting Stereo Disparity

The problems under consideration center around the interpretation of binocular stereo disparity. In particular, the goal is to establish a set of mappings from stereo disparity to corresponding three-dimensional scene geometry. An analysis has been developed that shows how disparity information can be interpreted in terms of three-dimensional scene properties, such as surface depth, discontinuities, and orientation. These theoretical developments have been embodied in a set of computer algorithms for the recovery of scene geometry from input stereo disparity. The results of applying these algorithms to several disparity maps are presented. Comparisons are made to the interpretation of stereo disparity by biological systems.

目标位姿测量中的三维视觉方法

要测量出一组特征点分别在两个空间坐标系下的坐标 ,就可以求解两个空间目标间的位姿关系 .实现上述目标位姿测量方法的前提条件是要保证该组特征点在不同坐标系下 ,其位置关系相同 ,但计算误差的存在却破坏了这种固定的位置关系 .为此 ,提出了两种基于模型的三维视觉方法——基于模型的单目视觉和基于模型的双目视觉 ,前者从视觉计算的物理意义入手 ,通过简单的约束迭代求解实现模型约束 ;后者则将简单的约束最小二乘法和基于模型的单目视觉方法融合在一起来实现模型约束 .引入模型约束后 ,单目视觉方法可以达到很高的测量精度 .而基于模型的双目视觉较传统的无模型立体视觉方法位移精度提高有限 ,但姿态精度提高很多

How are Three-Deminsional Objects Represented in the Brain?

We discuss a variety of object recognition experiments in which human subjects were presented with realistically rendered images of computer-generated three-dimensional objects, with tight control over stimulus shape, surface properties, illumination, and viewpoint, as well as subjects' prior exposure to the stimulus objects. In all experiments recognition performance was: (1) consistently viewpoint dependent; (2) only partially aided by binocular stereo and other depth information, (3) specific to viewpoints that were familiar; (4) systematically disrupted by rotation in depth more than by deforming the two-dimensional images of the stimuli. These results are consistent with recently advanced computational theories of recognition based on view interpolation.

Bilayer segmentation of live video in uncontrolled environments for background substitution: an overview and main challenges

Bilayer segmentation of live video in uncontrolled environments is an essential task for home applications in which the original background of the scene must be replaced, as in videochats or traditional videoconference. The main challenge in such conditions is overcome all difficulties in problem-situations (e. g., illumination change, distract events such as element moving in the background and camera shake) that may occur while the video is being captured. This paper presents a survey of segmentation methods for background substitution applications, describes the main concepts and identifies events that may cause errors. Our analysis shows that although robust methods rely on specific devices (multiple cameras or sensors to generate depth maps) which aid the process. In order to achieve the same results using conventional devices (monocular video cameras), most current research relies on energy minimization frameworks, in which temporal and spacial information are probabilistically combined with those of color and contrast.

Stereo display with time-multiplexed focal adjustment

In stereo displays, binocular disparity creates a striking impression of depth. However, such displays present focus cues - blur and accommodation - that specify a different depth than disparity, thereby causing a conflict. This conflict causes several problems including misperception of the 3D layout, difficulty fusing binocular images, and visual fatigue. To address these problems, we developed a display that preserves the advantages of conventional stereo displays, while presenting correct or nearly correct focus cues. In our new stereo display each eye views a display through a lens that switches between four focal distances at very high rate. The switches are synchronized to the display, so focal distance and the distance being simulated on the display are consistent or nearly consistent with one another. Focus cues for points in-between the four focal planes are simulated by using a depth-weighted blending technique. We will describe the design of the new display, discuss the retinal images it forms under various conditions, and describe an experiment that illustrates the effectiveness of the display in maximizing visual performance while minimizing visual fatigue. © 2009 SPIE-IS&T.

A population study of binocular function

As part of a genome-wide association study (GWAS) of perceptual traits in healthy adults, we measured stereo acuity, the duration of alternative percepts in binocular rivalry and the extent of dichoptic masking in 1060 participants. We present the distributions of the measures, the correlations between measures, and their relationships to other psychophysical traits. We report sex differences, and correlations with age, interpupillary distance, eye dominance, phorias, visual acuity and personality. The GWAS, using data from 988 participants, yielded one genetic association that passed a permutation test for significance: The variant rs1022907 in the gene VTI1A was associated with self-reported ability to see autostereograms. We list a number of other suggestive genetic associations (p<10^-5).

Stereo and motion parallax cues in human 3D vision: can they vanish without a trace?

In an immersive virtual reality environment, subjects fail to notice when a scene expands or contracts around them, despite correct and consistent information from binocular stereopsis and motion parallax, resulting in gross failures of size constancy (A. Glennerster, L. Tcheang, S. J. Gilson, A. W. Fitzgibbon, & A. J. Parker, 2006). We determined whether the integration of stereopsis/motion parallax cues with texture-based cues could be modified through feedback. Subjects compared the size of two objects, each visible when the room was of a different size. As the subject walked, the room expanded or contracted, although subjects failed to notice any change. Subjects were given feedback about the accuracy of their size judgments, where the “correct” size setting was defined either by texture-based cues or (in a separate experiment) by stereo/motion parallax cues. Because of feedback, observers were able to adjust responses such that fewer errors were made. For texture-based feedback, the pattern of responses was consistent with observers weighting texture cues more heavily. However, for stereo/motion parallax feedback, performance in many conditions became worse such that, paradoxically, biases moved away from the point reinforced by the feedback. This can be explained by assuming that subjects remap the relationship between stereo/motion parallax cues and perceived size or that they develop strategies to change their criterion for a size match on different trials. In either case, subjects appear not to have direct access to stereo/motion parallax cues.

Humans ignore motion and stereo cues in favor of a fictional stable world

As we move through the world, our eyes acquire a sequence of images. The information from this sequence is sufficient to determine the structure of a three-dimensional scene, up to a scale factor determined by the distance that the eyes have moved [1, 2]. Previous evidence shows that the human visual system accounts for the distance the observer has walked [3,4] and the separation of the eyes [5-8] when judging the scale, shape, and distance of objects. However, in an immersive virtual-reality environment, observers failed to notice when a scene expanded or contracted, despite having consistent information about scale from both distance walked and binocular vision. This failure led to large errors in judging the size of objects. The pattern of errors cannot be explained by assuming a visual reconstruction of the scene with an incorrect estimate of interocular separation or distance walked. Instead, it is consistent with a Bayesian model of cue integration in which the efficacy of motion and disparity cues is greater at near viewing distances. Our results imply that observers are more willing to adjust their estimate of interocular separation or distance walked than to accept that the scene has changed in size.

Análise do comportamento de técnicas de correspondência binocular para imagens em alta definição

This work presents an analysis of the behavior of some algorithms usually available in stereo correspondence literature, with full HD images (1920x1080 pixels) to establish, within the precision dilemma versus runtime applications which these methods can be better used. The images are obtained by a system composed of a stereo camera coupled to a computer via a capture board. The OpenCV library is used for computer vision operations and processing images involved. The algorithms discussed are an overall method of search for matching blocks with the Sum of the Absolute Value of the difference (Sum of Absolute Differences - SAD), a global technique based on cutting energy graph cuts, and a so-called matching technique semi -global. The criteria for analysis are processing time, the consumption of heap memory and the mean absolute error of disparity maps generated.

Estudo comparativo da estereopsia com três testes diferentes: TNO®, Fly Stereo Acuity Test® e StereoTAB® em estudantes do ensino superior

Estereopsia define-se como a perceção de profundidade baseada na disparidade retiniana. A estereopsia global depende do processamento de estímulos de pontos aleatórios e a estereopsia local depende da perceção de contornos. O objetivo deste estudo é correlacionar três testes de estereopsia: TNO®, StereoTAB® e Fly Stereo Acuity Test® e verificar a sensibilidade e correlação entre eles, tendo o TNO® como gold standard. Incluíram-se 49 estudantes da Escola Superior de Tecnologia da Saúde de Lisboa (ESTeSL) entre os 18 e 26 anos. As variáveis ponto próximo de convergência (ppc), vergências, sintomatologia e correção ótica foram correlacionadas com os três testes. Os valores médios (desvios-padrão) de estereopsia foram: TNO® = 87,04’’ ±84,09’’; FlyTest® = 38,18’’ ±34,59’’; StereoTAB® = 124,89’’ ±137,38’’. Coeficiente de determinação: TNO® e StereoTAB® com R2=0,6 e TNO® e FlyTest® com R2=0,2. O coeficiente de correlação de Pearson mostra uma correlação positiva de entre o TNO® e o StereoTAB® (r=0,784 com α=0,01). O coeficiente de associação de Phi mostrou uma relação positiva forte entre o TNO® e StereoTAB® (Φ=0,848 com α=0,01). Na curva ROC, o StereoTAB® possui uma área sob a curva maior que o FlyTest®, apresentando valor de sensibilidade de 92,3% para uma especificidade de 94,4%, tornando-o num teste sensível e com bom poder discriminativo.

Binocular functional architecture for detection of contrast-modulated gratings

Combination of signals from the two eyes is the gateway to stereo vision. To gain insight into binocular signal processing, we studied binocular summation for luminance-modulated gratings (L or LM) and contrast-modulated gratings (CM). We measured 2AFC detection thresholds for a signal grating (0.75 c/deg, 216msec) shown to one eye, both eyes, or both eyes out-of-phase. For LM and CM, the carrier noise was in both eyes, even when the signal was monocular. Mean binocular thresholds for luminance gratings (L) were 5.4dB better than monocular thresholds - close to perfect linear summation (6dB). For LM and CM the binocular advantage was again 5-6dB, even when the carrier noise was uncorrelated, anti-correlated, or at orthogonal orientations in the two eyes. Binocular combination for CM probably arises from summation of envelope responses, and not from summation of these conflicting carrier patterns. Antiphase signals produced no binocular advantage, but thresholds were about 1-3dB higher than monocular ones. This is not consistent with simple linear summation, which should give complete cancellation and unmeasurably high thresholds. We propose a three-channel model in which noisy monocular responses to the envelope are binocularly combined in a contrast-weighted sum, but also remain separately available to perception via a max operator. Vision selects the largest of the three responses. With in-phase gratings the binocular channel dominates, but antiphase gratings cancel in the binocular channel and the monocular channels mediate detection. The small antiphase disadvantage might be explained by a subtle influence of background responses on binocular and monocular detection.

Visual suppression in intermittent exotropia during binocular alignment.

PURPOSE To investigate the cortical mechanisms that prevent diplopia in intermittent exotropia (X(T)) during binocular alignment (orthotropia). METHODS The authors studied 12 X(T) patients aged 5 to 22 years. Seventy-five percent had functional stereo vision with stereoacuity similar to that of 12 age-matched controls (0.2-3.7 min arc). Identical face images were presented to the two eyes for 400 ms. In one eye, the face was presented at the fovea; in the other, offset along the horizontal axis with up to 12° eccentricity. The task was to indicate whether one or two faces were perceived. RESULTS All X(T) patients showed normal diplopia when the nonfoveal face was presented to nasal hemiretina, though with a slightly larger fusional range than age-matched controls. However, 10 of 12 patients never experienced diplopia when the nonfoveal face was presented to temporal hemiretina (i.e., when the stimulus simulated exodeviation). Patients showed considerable variability when the single image was perceived. Some patients suppressed the temporal stimulus regardless of which eye viewed it, whereas others suppressed a particular eye even when it viewed the foveal stimulus. In two patients, the simulated exodeviation might have triggered a shift from normal to anomalous retinal correspondence. CONCLUSIONS Antidiplopic mechanisms in X(T) can be reliably triggered by purely retinal information during orthotropia, but the nature of these mechanisms varies between patients.