Improving the Accuracy of 3-D Reconstruction in Robotic Vision Applications

Thesis (Ph.D.)--University of Washington, 2016-08

Three-dimensional (3D) surface reconstruction is a process for retrieving the 3D shape and appearance of real objects or scenes. The generated 3D point clouds can be used in many fields, including entertainment, measurement, design, reverse engineering, homeland security and {\it etc}. Over the past decades, 3D reconstruction has been widely used in clinical diagnosis and surgical treatment of diseases, such as X-ray, ultrasound, computed tomography (CT) and magnetic resonance imaging (MRI). However, all of these technologies are radiography-based volumetric 3D reconstruction instead of 3D surface reconstruction. With the growing need of tissue texture information for clinical purposes, the 3D reconstruction based on endoscopic images plays a more vital role than ever, especially in tumor diagnosis and surveillance of esophagus, lung, stomach, bladder and etc. In this work, new algorithms were developed to solve specific 3D reconstruction problems in biomedical applications. To reconstruct the 3D internal surface of a human organ, such as bladder or stomach, a sequence of 2D endoscopic images were captured by rotating and moving the scope around inside of the organs. This 3D reconstruction solely based on images is called Structure-from-Motion (SfM). To overcome the problems of insufficient features in medical images and short camera baselines, the camera poses were initially estimated by constraining the surface on a spherical shape at the first step. The more realistic organ surface was then reconstructed by releasing the spherical constraints. Extra features were built to handle multiple scanning videos and recover the physical scale of the 3D surface with reference lesion target. To reduce the human error and surgical operating time in removing the tumor/cancer in brain, a semi-automated surgical robotic system with 3D vision is being developed. By providing an accurate 3D surface model of the surgical field based on a RGB (red, green and blue) camera attached to the surgical tool, the robot could perform tedious operation of residual tumor tissue removal automatically. Camera position and orientation were also known throughout the surgery from the robotic system. This 3D reconstruction with known camera parameters is called Multi-view Stereo. Due to the mechanical limitation of robotic system, the camera pose parameters were with certain errors (tolerance). To utilize these inaccurate but bound constrained variables, Bound Constrained Bundle Adjustment (BCBA) algorithm was developed based on gradient projection to generate accurate 3D model efficiently. Besides biomedical applications, 3D computer vision is emerging in traditional industries, such as manufacture and quality control applications. To build a potential in-line 3D metrology tool for internal threads in automobile engine blocks, two 3D reconstruction algorithms were developed with forward-view and side-view cameras, respectively. Axial-stereo vision algorithm was proposed to create dense 3D point cloud of internal surface based on two forward-view images that are aligned on the optical axis. Feature-based panoramic 3D registration algorithm was developed to register different side-view image-generated 3D surface patches together, by taking advantage of the robustness and accuracy of SIFT features. Each side-view patch of the repeated geometry of a threaded hole was reconstructed by multi-view stereo. Comparing with traditional 3D point clouds registration algorithm Iterative Closest Point (ICP), our algorithm has the advantages of high-efficiency and high-accuracy, especially for the registration of repetitive geometries.