A Video Database for the Development of Stereo-3D Post-Production Algorithms

This paper introduces a database of freely available stereo-3D content designed to facilitate research in stereo post-production. It describes the structure and content of the database and provides some details about how the material was gathered. The database includes examples of many of the scenarios characteristic to broadcast footage. Material was gathered at different locations including a studio with controlled lighting and both indoor and outdoor on-location sites with more restricted lighting control. The database also includes video sequences with accompanying 3D audio data recorded in an Ambisonics format. An intended consequence of gathering the material is that the database contains examples of degradations that would be commonly present in real-world scenarios. This paper describes one such artefact caused by uneven exposure in the stereo views, causing saturation in the over-exposed view. An algorithm for the restoration of this artefact is proposed in order to highlight the usefuiness of the database.

Stereo and motion parallax cues in human 3D vision: can they vanish without a trace?

In an immersive virtual reality environment, subjects fail to notice when a scene expands or contracts around them, despite correct and consistent information from binocular stereopsis and motion parallax, resulting in gross failures of size constancy (A. Glennerster, L. Tcheang, S. J. Gilson, A. W. Fitzgibbon, & A. J. Parker, 2006). We determined whether the integration of stereopsis/motion parallax cues with texture-based cues could be modified through feedback. Subjects compared the size of two objects, each visible when the room was of a different size. As the subject walked, the room expanded or contracted, although subjects failed to notice any change. Subjects were given feedback about the accuracy of their size judgments, where the “correct” size setting was defined either by texture-based cues or (in a separate experiment) by stereo/motion parallax cues. Because of feedback, observers were able to adjust responses such that fewer errors were made. For texture-based feedback, the pattern of responses was consistent with observers weighting texture cues more heavily. However, for stereo/motion parallax feedback, performance in many conditions became worse such that, paradoxically, biases moved away from the point reinforced by the feedback. This can be explained by assuming that subjects remap the relationship between stereo/motion parallax cues and perceived size or that they develop strategies to change their criterion for a size match on different trials. In either case, subjects appear not to have direct access to stereo/motion parallax cues.

Regularizing a set of unstructured 3D points from a sequence of stereo images

[EN] In this paper we present a method for the regularization of a set of unstructured 3D points obtained from a sequence of stereo images. This method takes into account the information supplied by the disparity maps computed between pairs of images to constraint the regularization of the set of 3D points. We propose a model based on an energy which is composed of two terms: an attachment term that minimizes the distance from 3D points to the projective lines of camera points, and a second term that allows for the regularization of the set of 3D points by preserving discontinuities presented on the disparity maps. We embed this energy in a 2D finite element method. After minimizing, this method results in a large system of equations that can be optimized for fast computations. We derive an efficient implicit numerical scheme which reduces the number of calculations and memory allocations.

Experimental Prototype Merging Stereo Panoramic Video and Interactive 3D Content in a 5-sided CAVETM

Immersion and interaction have been identified as key factors influencing the quality of experience in stereoscopic video systems. An experimental prototype designed to explore the influence of these factors in 3D video applications is described here1. The focus is on the real-time insertion algorithm of new 3D models into the original video streams. Using this algorithm, our prototype is aimed to explore a new interaction paradigm ? similar to the augmented reality approach ? with 3D video applications.

Self-calibrated, multi-spectral photometric stereo for 3D face capture

This paper addresses the problem of obtaining 3d detailed reconstructions of human faces in real-time and with inexpensive hardware. We present an algorithm based on a monocular multi-spectral photometric-stereo setup. This system is known to capture high-detailed deforming 3d surfaces at high frame rates and without having to use any expensive hardware or synchronized light stage. However, the main challenge of such a setup is the calibration stage, which depends on the lights setup and how they interact with the specific material being captured, in this case, human faces. For this purpose we develop a self-calibration technique where the person being captured is asked to perform a rigid motion in front of the camera, maintaining a neutral expression. Rigidity constrains are then used to compute the head's motion with a structure-from-motion algorithm. Once the motion is obtained, a multi-view stereo algorithm reconstructs a coarse 3d model of the face. This coarse model is then used to estimate the lighting parameters with a stratified approach: In the first step we use a RANSAC search to identify purely diffuse points on the face and to simultaneously estimate this diffuse reflectance model. In the second step we apply non-linear optimization to fit a non-Lambertian reflectance model to the outliers of the previous step. The calibration procedure is validated with synthetic and real data.

Practical 3D reconstruction based on photometric stereo

Photometric Stereo is a powerful image based 3D reconstruction technique that has recently been used to obtain very high quality reconstructions. However, in its classic form, Photometric Stereo suffers from two main limitations: Firstly, one needs to obtain images of the 3D scene under multiple different illuminations. As a result the 3D scene needs to remain static during illumination changes, which prohibits the reconstruction of deforming objects. Secondly, the images obtained must be from a single viewpoint. This leads to depth-map based 2.5 reconstructions, instead of full 3D surfaces. The aim of this Chapter is to show how these limitations can be alleviated, leading to the derivation of two practical 3D acquisition systems: The first one, based on the powerful Coloured Light Photometric Stereo method can be used to reconstruct moving objects such as cloth or human faces. The second, permits the complete 3D reconstruction of challenging objects such as porcelain vases. In addition to algorithmic details, the Chapter pays attention to practical issues such as setup calibration, detection and correction of self and cast shadows. We provide several evaluation experiments as well as reconstruction results. © 2010 Springer-Verlag Berlin Heidelberg.

Multi-robot cooperative stereo for outdoor scenarios

13th International Conference on Autonomous Robot Systems (Robotica), 2013

Recovering Euclidean deformable models from stereo-motion

In this paper we present a novel structure from motion (SfM) approach able to infer 3D deformable models from uncalibrated stereo images. Using a stereo setup dramatically improves the 3D model estimation when the observed 3D shape is mostly deforming without undergoing strong rigid motion. Our approach first calibrates the stereo system automatically and then computes a single metric rigid structure for each frame. Afterwards, these 3D shapes are aligned to a reference view using a RANSAC method in order to compute the mean shape of the object and to select the subset of points on the object which have remained rigid throughout the sequence without deforming. The selected rigid points are then used to compute frame-wise shape registration and to extract the motion parameters robustly from frame to frame. Finally, all this information is used in a global optimization stage with bundle adjustment which allows to refine the frame-wise initial solution and also to recover the non-rigid 3D model. We show results on synthetic and real data that prove the performance of the proposed method even when there is no rigid motion in the original sequence

Reconstruction of 3D Planar Objects with Partial Occlusion

Kolmiulotteisten kappaleiden rekonstruktio on yksi konenäön haastavimmista ongelmista, koska kappaleiden kolmiulotteisia etäisyyksiä ei voida selvittää yhdestä kaksiulotteisesta kuvasta. Ongelma voidaan ratkaista stereonäön avulla, jossa näkymän kolmiulotteinen rakenne päätellään usean kuvan perusteella. Tämä lähestymistapa mahdollistaa kuitenkin vain rekonstruktion niille kappaleiden osille, jotka näkyvät vähintään kahdessa kuvassa. Piilossa olevien osien rekonstruktio ei ole mahdollista pelkästään stereonäön avulla. Tässä työssä on kehitetty uusi menetelmä osittain piilossa olevien kolmiulotteisten tasomaisten kappaleiden rekonstruktioon. Menetelmän avulla voidaan selvittää hyvällä tarkkuudella tasomaisista pinnoista koostuvan kappaleen muoto ja paikka käyttäen kahta kuvaa kappaleesta. Menetelmä perustuu epipolaarigeometriaan, jonka avulla selvitetään molemmissa kuvissa näkyvät kappaleiden osat. Osittain piilossa olevien piirteiden rekonstruointi suoritetaan käyttämäen stereonäköä sekä tietoa kappaleen rakenteesta. Esitettyä ratkaisua voitaisiin käyttää esimerkiksi kolmiulotteisten kappaleiden visualisointiin, robotin navigointiin tai esineentunnistukseen.

Visual SLAM for 3D large-scale seabed acquisition employing underwater vehicles

This paper presents a novel technique to align partial 3D reconstructions of the seabed acquired by a stereo camera mounted on an autonomous underwater vehicle. Vehicle localization and seabed mapping is performed simultaneously by means of an Extended Kalman Filter. Passive landmarks are detected on the images and characterized considering 2D and 3D features. Landmarks are re-observed while the robot is navigating and data association becomes easier but robust. Once the survey is completed, vehicle trajectory is smoothed by a Rauch-Tung-Striebel filter obtaining an even better alignment of the 3D views and yet a large-scale acquisition of the seabed

3D Surface Modeling From Point Clouds

The goal of this thesis is to implement software for creating 3D models from point clouds. Point clouds are acquired with stereo cameras, monocular systems or laser scanners. The created 3D models are triangular models or NURBS (Non-Uniform Rational B-Splines) models. Triangular models are constructed from selected areas from the point clouds and resulted triangular models are translated into a set of quads. The quads are further translated into an estimated grid structure and used for NURBS surface approximation. Finally, we have a set of NURBS surfaces which represent the whole model. The problem wasn’t so easy to solve. The selected triangular surface reconstruction algorithm did not deal well with noise in point clouds. To handle this problem, a clustering method is introduced for simplificating the model and removing noise. As we had better results with the smaller point clouds produced by clustering, we used points in clusters to better estimate the grids for NURBS models. The overall results were good when the point cloud did not have much noise. The point clouds with small amount of error had good results as the triangular model was solid. NURBS surface reconstruction performed well on solid models.

Performance evaluation of 3D computer vision techniques

This work presents the implementation and comparison of three different techniques of three-dimensional computer vision as follows: • Stereo vision - correlation between two 2D images • Sensorial fusion - use of different sensors: camera 2D + ultrasound sensor (1D); • Structured light The computer vision techniques herein presented took into consideration the following characteristics: • Computational effort ( elapsed time for obtain the 3D information); • Influence of environmental conditions (noise due to a non uniform lighting, overlighting and shades); • The cost of the infrastructure for each technique; • Analysis of uncertainties, precision and accuracy. The option of using the Matlab software, version 5.1, for algorithm implementation of the three techniques was due to the simplicity of their commands, programming and debugging. Besides, this software is well known and used by the academic community, allowing the results of this work to be obtained and verified. Examples of three-dimensional vision applied to robotic assembling tasks ("pick-and-place") are presented.

L’impact de la stéréoscopie dans la reconnaissance, la perception et la constance de forme 3D

Les buts des recherches présentées dans cette thèse étaient d’évaluer le rôle de la stéréoscopie dans la reconnaissance de forme, dans la perception du relief et dans la constance de forme. La première étude a examiné le rôle de la stéréoscopie dans la perception des formes visuelles en utilisant une tâche de reconnaissance de formes. Les stimuli pouvaient être présentés en 2D, avec disparité normale (3D) ou avec disparité inversée. La performance de reconnaissance était meilleure avec les modes de présentation 2D et 3D qu’avec la 3D inversée. Cela indique que la stéréoscopie contribue à la reconnaissance de forme. La deuxième étude s’est intéressée à la contribution conjointe de l’ombrage et de la stéréoscopie dans la perception du relief des formes. Les stimuli étaient des images d’une forme 3D convexe synthétique présentée sous un point de vue menant à une ambigüité quant à sa convexité. L’illumination pouvait provenir du haut ou du bas et de la gauche ou de la droite, et les stimuli étaient présentés dichoptiquement avec soit de la disparité binoculaire normale, de la disparité inversée ou sans disparité entre les vues. Les participants ont répondu que les formes étaient convexes plus souvent lorsque la lumière provenait du haut que du bas, plus souvent avec la disparité normale qu’en 2D, et plus souvent avec absence de disparité qu’avec disparité inversée. Les effets de direction d’illumination et du mode de présentation étaient additifs, c’est-à-dire qu’ils n’interagissaient pas. Cela indique que l’ombrage et la stéréoscopie contribuent indépendamment à la perception du relief des formes. La troisième étude a évalué la contribution de la stéréoscopie à la constance de forme, et son interaction avec l’expertise perceptuelle. Elle a utilisé trois tâches de discrimination séquentielle de trombones tordus ayant subi des rotations en profondeur. Les stimuli pouvaient être présentés sans stéréoscopie, avec stéréoscopie normale ou avec stéréoscopie inversée. Dans la première moitié de l’Exp. 1, dans laquelle les variations du mode de présentation étaient intra-sujets, les performances étaient meilleures en 3D qu’en 2D et qu’en 3D inversée. Ces effets ont été renversés dans la seconde moitié de l’expérience, et les coûts de rotation sont devenus plus faibles pour la 2D et la 3D inversée que pour la 3D. Dans les Exps. 2 (variations intra-sujets du mode de présentation, avec un changement de stimuli au milieu de l’expérience) et 3 (variations inter-sujets du mode de présentation), les effets de rotation étaient en tout temps plus faibles avec stéréoscopie qu’avec stéréoscopie inversée et qu’en 2D, et plus faibles avec stéréoscopie inversée que sans stéréoscopie. Ces résultats indiquent que la stéréoscopie contribue à la constance de forme. Toutefois, cela demande qu’elle soit valide avec un niveau minimal de consistance, sinon elle devient stratégiquement ignorée. En bref, les trois études présentées dans cette thèse ont permis de montrer que la stéréoscopie contribue à la reconnaissance de forme, à la perception du relief et à la constance de forme. De plus, l’ombrage et la stéréoscopie sont intégrés linéairement.

Stereo-Based Head Pose Tracking Using Iterative Closest Point and Normal Flow Constraint

In this text, we present two stereo-based head tracking techniques along with a fast 3D model acquisition system. The first tracking technique is a robust implementation of stereo-based head tracking designed for interactive environments with uncontrolled lighting. We integrate fast face detection and drift reduction algorithms with a gradient-based stereo rigid motion tracking technique. Our system can automatically segment and track a user's head under large rotation and illumination variations. Precision and usability of this approach are compared with previous tracking methods for cursor control and target selection in both desktop and interactive room environments. The second tracking technique is designed to improve the robustness of head pose tracking for fast movements. Our iterative hybrid tracker combines constraints from the ICP (Iterative Closest Point) algorithm and normal flow constraint. This new technique is more precise for small movements and noisy depth than ICP alone, and more robust for large movements than the normal flow constraint alone. We present experiments which test the accuracy of our approach on sequences of real and synthetic stereo images. The 3D model acquisition system we present quickly aligns intensity and depth images, and reconstructs a textured 3D mesh. 3D views are registered with shape alignment based on our iterative hybrid tracker. We reconstruct the 3D model using a new Cubic Ray Projection merging algorithm which takes advantage of a novel data structure: the linked voxel space. We present experiments to test the accuracy of our approach on 3D face modelling using real-time stereo images.