Augmenting depth camera output using photometric stereo

We present a system for augmenting depth camera output using multispectral photometric stereo. The technique is demonstrated using a Kinect sensor and is able to produce geometry independently for each frame. Improved reconstruction is demonstrated using the Kinect's inbuilt RGB camera and further improvements are achieved by introducing an additional high resolution camera. As well as qualitative improvements in reconstruction a quantitative reduction in temporal noise is shown. As part of the system an approach is presented for relaxing the assumption of multispectral photometric stereo that scenes are of constant chromaticity to the assumption that scenes contain multiple piecewise constant chromaticities.

Localization and tracking of phonating finless porpoises using towed stereo acoustic data-loggers

Cetaceans produce sound signals frequently. Usually, acoustic localization of cetaceans was made by cable hydrophone arrays and multichannel recording systems. In this study, a simple and relatively inexpensive towed acoustic system consisting of two miniature stereo acoustic data-loggers is described for localization and tracking of finless porpoises in a mobile survey. Among 204 porpoises detected acoustically, 34 individuals (similar to 17%) were localized, and 4 of the 34 localized individuals were tracked. The accuracy of the localization is considered to be fairly high, as the upper bounds of relative distance errors were less than 41% within 173 m. With the location information, source levels of finless porpoise clicks were estimated to range from 180 to 209 dB re 1 mu Pa pp at 1 m with an average of 197 dB (N=34), which is over 20 dB higher than that estimated previously from animals in enclosed waters. For the four tracked porpoises, two-dimensional swimming trajectories relative to the moving survey boat, absolute swimming speed, and absolute heading direction are deduced by assuming the animal movements are straight and at constant speed in the segment between two consecutive locations.

Off-axis sonar beam pattern of free-ranging finless porpoises measured by a stereo pulse event data logger

The off-axis sonar beam patterns of eight free-ranging finless porpoises were measured using attached data logger systems. The transmitted sound pressure level at each beam angle was calculated from the animal's body angle, the water surface echo level, and the swimming depth. The beam pattern of the off-axis signals between 45 and 115 (where 0 corresponds to the on-axis direction) was nearly constant. The sound pressure level of the off-axis signals reached 162 dB re 1 mPa peak-to-peak. The surface echo level received at the animal was over 140 dB, much higher than the auditory threshold level of small odontocetes. Finless porpoises are estimated to be able to receive the surface echoes of off-axis signals even at 50-m depth. Shallow water systems (less than 50-m depth) are the dominant habitat of both oceanic and freshwater populations of this species. Surface echoes may provide porpoises not only with diving depth information but also with information about surface direction and location of obstacles (including prey items) outside the on-axis sector of the sonar beam. 2005 Acoustical Society of America.

Direct Estimation of Motion and Extended Scene Structure from a Moving Stereo Rig

We describe a new method for motion estimation and 3D reconstruction from stereo image sequences obtained by a stereo rig moving through a rigid world. We show that given two stereo pairs one can compute the motion of the stereo rig directly from the image derivatives (spatial and temporal). Correspondences are not required. One can then use the images from both pairs combined to compute a dense depth map. The motion estimates between stereo pairs enable us to combine depth maps from all the pairs in the sequence to form an extended scene reconstruction and we show results from a real image sequence. The motion computation is a linear least squares computation using all the pixels in the image. Areas with little or no contrast are implicitly weighted less so one does not have to explicitly apply a confidence measure.

Seeing 'Ghost' Solutions in Stereo Vision

A unique matching is a stated objective of most computational theories of stereo vision. This report describes situations where humans perceive a small number of surfaces carried by non-unique matching of random dot patterns, although a unique solution exists and is observed unambiguously in the perception of isolated features. We find both cases where non-unique matchings compete and suppress each other and cases where they are all perceived as transparent surfaces. The circumstances under which each behavior occurs are discussed and a possible explanation is sketched. It appears that matching reduces many false targets to a few, but may still yield multiple solutions in some cases through a (possibly different) process of surface interpolation.

Structure from Stereo and Motion

Stereopsis and motion parallax are two methods for recovering three dimensional shape. Theoretical analyses of each method show that neither alone can recover rigid 3D shapes correctly unless other information, such as perspective, is included. The solutions for recovering rigid structure from motion have a reflection ambiguity; the depth scale of the stereoscopic solution will not be known unless the fixation distance is specified in units of interpupil separation. (Hence the configuration will appear distorted.) However, the correct configuration and the disposition of a rigid 3D shape can be recovered if stereopsis and motion are integrated, for then a unique solution follows from a set of linear equations. The correct interpretation requires only three points and two stereo views.

Robust Photo-topography by Fusing Shape-from-Shading and Stereo

Methods for fusing two computer vision methods are discussed and several example algorithms are presented to illustrate the variational method of fusing algorithms. The example algorithms seek to determine planet topography given two images taken from two different locations with two different lighting conditions. The algorithms each employ assingle cost function that combines the computer vision methods of shape-from-shading and stereo in different ways. The algorithms are closely coupled and take into account all the constraints of the photo-topography problem. The algorithms are run on four synthetic test image sets of varying difficulty.

On Interpreting Stereo Disparity

The problems under consideration center around the interpretation of binocular stereo disparity. In particular, the goal is to establish a set of mappings from stereo disparity to corresponding three-dimensional scene geometry. An analysis has been developed that shows how disparity information can be interpreted in terms of three-dimensional scene properties, such as surface depth, discontinuities, and orientation. These theoretical developments have been embodied in a set of computer algorithms for the recovery of scene geometry from input stereo disparity. The results of applying these algorithms to several disparity maps are presented. Comparisons are made to the interpretation of stereo disparity by biological systems.

Multi-scale 3D Scene Flow from Binocular Stereo Sequences

Scene flow methods estimate the three-dimensional motion field for points in the world, using multi-camera video data. Such methods combine multi-view reconstruction with motion estimation approaches. This paper describes an alternative formulation for dense scene flow estimation that provides convincing results using only two cameras by fusing stereo and optical flow estimation into a single coherent framework. To handle the aperture problems inherent in the estimation task, a multi-scale method along with a novel adaptive smoothing technique is used to gain a regularized solution. This combined approach both preserves discontinuities and prevents over-regularization-two problems commonly associated with basic multi-scale approaches. Internally, the framework generates probability distributions for optical flow and disparity. Taking into account the uncertainty in the intermediate stages allows for more reliable estimation of the 3D scene flow than standard stereo and optical flow methods allow. Experiments with synthetic and real test data demonstrate the effectiveness of the approach.

Multi-Scale 3D Scene Flow from Binocular Stereo Sequences

Scene flow methods estimate the three-dimensional motion field for points in the world, using multi-camera video data. Such methods combine multi-view reconstruction with motion estimation. This paper describes an alternative formulation for dense scene flow estimation that provides reliable results using only two cameras by fusing stereo and optical flow estimation into a single coherent framework. Internally, the proposed algorithm generates probability distributions for optical flow and disparity. Taking into account the uncertainty in the intermediate stages allows for more reliable estimation of the 3D scene flow than previous methods allow. To handle the aperture problems inherent in the estimation of optical flow and disparity, a multi-scale method along with a novel region-based technique is used within a regularized solution. This combined approach both preserves discontinuities and prevents over-regularization – two problems commonly associated with the basic multi-scale approaches. Experiments with synthetic and real test data demonstrate the strength of the proposed approach.

Foreground Object Segmentation from Binocular Stereo Video

Moving cameras are needed for a wide range of applications in robotics, vehicle systems, surveillance, etc. However, many foreground object segmentation methods reported in the literature are unsuitable for such settings; these methods assume that the camera is fixed and the background changes slowly, and are inadequate for segmenting objects in video if there is significant motion of the camera or background. To address this shortcoming, a new method for segmenting foreground objects is proposed that utilizes binocular video. The method is demonstrated in the application of tracking and segmenting people in video who are approximately facing the binocular camera rig. Given a stereo image pair, the system first tries to find faces. Starting at each face, the region containing the person is grown by merging regions from an over-segmented color image. The disparity map is used to guide this merging process. The system has been implemented on a consumer-grade PC, and tested on video sequences of people indoors obtained from a moving camera rig. As can be expected, the proposed method works well in situations where other foreground-background segmentation methods typically fail. We believe that this superior performance is partly due to the use of object detection to guide region merging in disparity/color foreground segmentation, and partly due to the use of disparity information available with a binocular rig, in contrast with most previous methods that assumed monocular sequences.