Improving the Accuracy of 3-D Reconstruction in Robotic Vision Applications

Thesis (Ph.D.)--University of Washington, 2016-08

Does countershading contribute to visual camouflage by concealing 3D shape?

This is an abstract of an invited talk presented at AVA Animal Vision Meeting / Camocon 2015, which took place in Liverpool, 23 August 2015.

Contributions to 3D Data Registration and Representation

Nowadays, new computers generation provides a high performance that enables to build computationally expensive computer vision applications applied to mobile robotics. Building a map of the environment is a common task of a robot and is an essential part to allow the robots to move through these environments. Traditionally, mobile robots used a combination of several sensors from different technologies. Lasers, sonars and contact sensors have been typically used in any mobile robotic architecture, however color cameras are an important sensor due to we want the robots to use the same information that humans to sense and move through the different environments. Color cameras are cheap and flexible but a lot of work need to be done to give robots enough visual understanding of the scenes. Computer vision algorithms are computational complex problems but nowadays robots have access to different and powerful architectures that can be used for mobile robotics purposes. The advent of low-cost RGB-D sensors like Microsoft Kinect which provide 3D colored point clouds at high frame rates made the computer vision even more relevant in the mobile robotics field. The combination of visual and 3D data allows the systems to use both computer vision and 3D processing and therefore to be aware of more details of the surrounding environment. The research described in this thesis was motivated by the need of scene mapping. Being aware of the surrounding environment is a key feature in many mobile robotics applications from simple robotic navigation to complex surveillance applications. In addition, the acquisition of a 3D model of the scenes is useful in many areas as video games scene modeling where well-known places are reconstructed and added to game systems or advertising where once you get the 3D model of one room the system can add furniture pieces using augmented reality techniques. In this thesis we perform an experimental study of the state-of-the-art registration methods to find which one fits better to our scene mapping purposes. Different methods are tested and analyzed on different scene distributions of visual and geometry appearance. In addition, this thesis proposes two methods for 3d data compression and representation of 3D maps. Our 3D representation proposal is based on the use of Growing Neural Gas (GNG) method. This Self-Organizing Maps (SOMs) has been successfully used for clustering, pattern recognition and topology representation of various kind of data. Until now, Self-Organizing Maps have been primarily computed offline and their application in 3D data has mainly focused on free noise models without considering time constraints. Self-organising neural models have the ability to provide a good representation of the input space. In particular, the Growing Neural Gas (GNG) is a suitable model because of its flexibility, rapid adaptation and excellent quality of representation. However, this type of learning is time consuming, specially for high-dimensional input data. Since real applications often work under time constraints, it is necessary to adapt the learning process in order to complete it in a predefined time. This thesis proposes a hardware implementation leveraging the computing power of modern GPUs which takes advantage of a new paradigm coined as General-Purpose Computing on Graphics Processing Units (GPGPU). Our proposed geometrical 3D compression method seeks to reduce the 3D information using plane detection as basic structure to compress the data. This is due to our target environments are man-made and therefore there are a lot of points that belong to a plane surface. Our proposed method is able to get good compression results in those man-made scenarios. The detected and compressed planes can be also used in other applications as surface reconstruction or plane-based registration algorithms. Finally, we have also demonstrated the goodness of the GPU technologies getting a high performance implementation of a CAD/CAM common technique called Virtual Digitizing.

Scene Mapping and Understanding by Robotic Vision

The first mechanical Automaton concept was found in a Chinese text written in the 3rd century BC, while Computer Vision was born in the late 1960s. Therefore, visual perception applied to machines (i.e. the Machine Vision) is a young and exciting alliance. When robots came in, the new field of Robotic Vision was born, and these terms began to be erroneously interchanged. In short, we can say that Machine Vision is an engineering domain, which concern the industrial use of Vision. The Robotic Vision, instead, is a research field that tries to incorporate robotics aspects in computer vision algorithms. Visual Servoing, for example, is one of the problems that cannot be solved by computer vision only. Accordingly, a large part of this work deals with boosting popular Computer Vision techniques by exploiting robotics: e.g. the use of kinematics to localize a vision sensor, mounted as the robot end-effector. The remainder of this work is dedicated to the counterparty, i.e. the use of computer vision to solve real robotic problems like grasping objects or navigate avoiding obstacles. Will be presented a brief survey about mapping data structures most widely used in robotics along with SkiMap, a novel sparse data structure created both for robotic mapping and as a general purpose 3D spatial index. Thus, several approaches to implement Object Detection and Manipulation, by exploiting the aforementioned mapping strategies, will be proposed, along with a completely new Machine Teaching facility in order to simply the training procedure of modern Deep Learning networks.

Neurogeometry of stereo vision

This work aims to develop a neurogeometric model of stereo vision, based on cortical architectures involved in the problem of 3D perception and neural mechanisms generated by retinal disparities. First, we provide a sub-Riemannian geometry for stereo vision, inspired by the work on the stereo problem by Zucker (2006), and using sub-Riemannian tools introduced by Citti-Sarti (2006) for monocular vision. We present a mathematical interpretation of the neural mechanisms underlying the behavior of binocular cells, that integrate monocular inputs. The natural compatibility between stereo geometry and neurophysiological models shows that these binocular cells are sensitive to position and orientation. Therefore, we model their action in the space R3xS2 equipped with a sub-Riemannian metric. Integral curves of the sub-Riemannian structure model neural connectivity and can be related to the 3D analog of the psychophysical association fields for the 3D process of regular contour formation. Then, we identify 3D perceptual units in the visual scene: they emerge as a consequence of the random cortico-cortical connection of binocular cells. Considering an opportune stochastic version of the integral curves, we generate a family of kernels. These kernels represent the probability of interaction between binocular cells, and they are implemented as facilitation patterns to define the evolution in time of neural population activity at a point. This activity is usually modeled through a mean field equation: steady stable solutions lead to consider the associated eigenvalue problem. We show that three-dimensional perceptual units naturally arise from the discrete version of the eigenvalue problem associated to the integro-differential equation of the population activity.

Understanding 3D structures from sketches

Sketches are a unique way to communicate: drawing a simple sketch does not require any training, sketches convey information that is hard to describe with words, they are powerful enough to represent almost any concept, and nowadays, it is possible to draw directly from mobile devices. Motivated from the unique characteristics of sketches and fascinated by the human ability to imagine 3D objects from drawings, this thesis focuses on automatically associating geometric information to sketches. The main research directions of the thesis can be summarized as obtaining geometric information from freehand scene sketches to improve 2D sketch-based tasks and investigating Vision-Language models to overcome 3D sketch-based tasks limitations. The first part of the thesis concerns geometric information prediction from scene sketches improving scene sketch to image generation and unlocking new creativity effects. The thesis proceeds showing a study conducted on the Vision-Language models embedding space considering sketches, line renderings and RGB renderings of 3D shape to overcome the use of supervised datasets for 3D sketch-based tasks, that are limited and hard to acquire. Following the obtained observations and results, Vision-Language models are applied to Sketch Based Shape Retrieval without the need of training on supervised datasets. We then analyze the use of Vision-Language models for sketch based 3D reconstruction in an unsupervised manner. In the final chapter we report the results obtained in an additional project carried during the PhD, which has lead to the development of a framework to learn an embedding space of neural networks that can be navigated to get ready-to-use models with desired characteristics.

An Embedded In-Cabin Lightweight High-Performance 3D Gaze Estimation System

Gaze estimation has gained interest in recent years for being an important cue to obtain information about the internal cognitive state of humans. Regardless of whether it is the 3D gaze vector or the point of gaze (PoG), gaze estimation has been applied in various fields, such as: human robot interaction, augmented reality, medicine, aviation and automotive. In the latter field, as part of Advanced Driver-Assistance Systems (ADAS), it allows the development of cutting-edge systems capable of mitigating road accidents by monitoring driver distraction. Gaze estimation can be also used to enhance the driving experience, for instance, autonomous driving. It also can improve comfort with augmented reality components capable of being commanded by the driver's eyes. Although, several high-performance real-time inference works already exist, just a few are capable of working with only a RGB camera on computationally constrained devices, such as a microcontroller. This work aims to develop a low-cost, efficient and high-performance embedded system capable of estimating the driver's gaze using deep learning and a RGB camera. The proposed system has achieved near-SOTA performances with about 90% less memory footprint. The capabilities to generalize in unseen environments have been evaluated through a live demonstration, where high performance and near real-time inference were obtained using a webcam and a Raspberry Pi4.

Optical And Nonoptical Aids For Reading And Writing In Individuals With Acquired Low Vision.

To evaluate the use of optical and nonoptical aids during reading and writing activities in individuals with acquired low vision. This study was performed using descriptive and cross-sectional surveys. The data collection instrument was created with structured questions that were developed from an exploratory study and a previous test based on interviews, and it evaluated the following variables: personal characteristics, use of optical and nonoptical aids, and activities that required the use of optical and nonoptical aids. The study population included 30 subjects with acquired low vision and visual acuities of 20/200-20/400. Most subjects reported the use of some optical aids (60.0%). Of these 60.0%, the majority (83.3%) cited spectacles as the most widely used optical aid. The majority (63.3%) of subjects also reported the use of nonoptical aids, the most frequent ones being letter magnification (68.4%), followed by bringing the objects closer to the eyes (57.8%). Subjects often used more than one nonoptical aid. The majority of participants reported the use of optical and nonoptical aids during reading activities, highlighting the use of spectacles, magnifying glasses, and letter magnification; however, even after the use of these aids, we found that the subjects often needed to read the text more than once to understand it. During writing activities, all subjects reported the use of optical aids, while most stated that they did not use nonoptical aids for such activities.

Structural and chemical basis for anticancer activity of a series of 'beta'-tubulin ligands: molecular modeling and 3D QSAR studies

An important approach to cancer therapy is the design of small molecule modulators that interfere with microtubule dynamics through their specific binding to the ²-subunit of tubulin. In the present work, comparative molecular field analysis (CoMFA) studies were conducted on a series of discodermolide analogs with antimitotic properties. Significant correlation coefficients were obtained (CoMFA(i), q² =0.68, r²=0.94; CoMFA(ii), q² = 0.63, r²= 0.91), indicating the good internal and external consistency of the models generated using two independent structural alignment strategies. The models were externally validated employing a test set, and the predicted values were in good agreement with the experimental results. The final QSAR models and the 3D contour maps provided important insights into the chemical and structural basis involved in the molecular recognition process of this family of discodermolide analogs, and should be useful for the design of new specific ²-tubulin modulators with potent anticancer activity.

Stress distribution in the cervical region of an upper central incisor in a 3D finite element model

The aim of this study was to evaluate the stress distribution in the cervical region of a sound upper central incisor in two clinical situations, standard and maximum masticatory forces, by means of a 3D model with the highest possible level of fidelity to the anatomic dimensions. Two models with 331,887 linear tetrahedral elements that represent a sound upper central incisor with periodontal ligament, cortical and trabecular bones were loaded at 45º in relation to the tooth's long axis. All structures were considered to be homogeneous and isotropic, with the exception of the enamel (anisotropic). A standard masticatory force (100 N) was simulated on one of the models, while on the other one a maximum masticatory force was simulated (235.9 N). The software used were: PATRAN for pre- and post-processing and Nastran for processing. In the cementoenamel junction area, tensile forces reached 14.7 MPa in the 100 N model, and 40.2 MPa in the 235.9 N model, exceeding the enamel's tensile strength (16.7 MPa). The fact that the stress concentration in the amelodentinal junction exceeded the enamel's tensile strength under simulated conditions of maximum masticatory force suggests the possibility of the occurrence of non-carious cervical lesions such as abfractions.

Perception of difficulties with vision-related activities of daily living among patients undergoing unilateral posterior capsulotomy

OBJECTIVES: To assess the influence of Nd:YAG (neodymium: yttrium-aluminum- garnet) laser unilateral posterior capsulotomy on visual acuity and patients' perception of difficulties with vision-related activities of daily life. METHODS: We conducted an interventional survey that included 48 patients between 40 and 80 years of age with uni- or bilateral pseudophakia, posterior capsule opacification, and visual acuity <0.30 (logMAR) in one eye who were seen at a Brazilian university hospital. All patients underwent posterior capsulotomy using an Nd:YAG laser. Before and after the intervention, patients were asked to complete a questionnaire that was developed in an exploratory study. RESULTS: Before posterior capsulotomy, the median visual acuity (logMAR) of the included patients was 0.52 (range 0.30-1.60). After posterior capsulotomy, the median visual acuity of the included patients improved to 0.10 (range 0.0-0.52). According to the subjects' perceptions, their ability to perform most of their daily life activities improved after the intervention (p<0.05). CONCLUSIONS: After patients underwent posterior capsulotomy with an Nd:YAG laser, a significant improvement in the visual acuity of the treated eye was observed. Additionally, subjects felt that they experienced less difficulty performing most of their vision-dependent activities of daily living.

Dynamical Lorentz symmetry breaking in 3D and charge fractionalization

We analyze the breaking of Lorentz invariance in a 3D model of fermion fields self-coupled through four-fermion interactions. The low-energy limit of the theory contains various submodels which are similar to those used in the study of graphene or in the description of irrational charge fractionalization.

Density functional theory investigation of 3d, 4d, and 5d 13-atom metal clusters

The knowledge of the atomic structure of clusters composed by few atoms is a basic prerequisite to obtain insights into the mechanisms that determine their chemical and physical properties as a function of diameter, shape, surface termination, as well as to understand the mechanism of bulk formation. Due to the wide use of metal systems in our modern life, the accurate determination of the properties of 3d, 4d, and 5d metal clusters poses a huge problem for nanoscience. In this work, we report a density functional theory study of the atomic structure, binding energies, effective coordination numbers, average bond lengths, and magnetic properties of the 3d, 4d, and 5d metal (30 elements) clusters containing 13 atoms, M(13). First, a set of lowest-energy local minimum structures (as supported by vibrational analysis) were obtained by combining high-temperature first- principles molecular-dynamics simulation, structure crossover, and the selection of five well-known M(13) structures. Several new lower energy configurations were identified, e. g., Pd(13), W(13), Pt(13), etc., and previous known structures were confirmed by our calculations. Furthermore, the following trends were identified: (i) compact icosahedral-like forms at the beginning of each metal series, more opened structures such as hexagonal bilayerlike and double simple-cubic layers at the middle of each metal series, and structures with an increasing effective coordination number occur for large d states occupation. (ii) For Au(13), we found that spin-orbit coupling favors the three-dimensional (3D) structures, i.e., a 3D structure is about 0.10 eV lower in energy than the lowest energy known two-dimensional configuration. (iii) The magnetic exchange interactions play an important role for particular systems such as Fe, Cr, and Mn. (iv) The analysis of the binding energy and average bond lengths show a paraboliclike shape as a function of the occupation of the d states and hence, most of the properties can be explained by the chemistry picture of occupation of the bonding and antibonding states.

Use of peripheral vision in the interception of moving targets

Use of peripheral vision to organize and reorganize an interceptive action was investigated in young adults. Temporal errors and kinematic variables were evaluated in the interception of a virtual moving target, in situations in which its initial velocity was kept unchanged or was unexpectedly decreased. Observation of target approach was made through continuous visual pursuit (focal vision) or keeping visual focus at the origin of the trajectory or at the contact spot (peripheral vision). Results showed that visual focus at the contact spot led to temporal errors similar to focal vision, although showing a distinct kinematic profile, while focus at the origin led to an impoverished performance

Monocular vision and increased distance reducing the effects of visual manipulation on body sway

The goal of this study was to examine the coupling between visual information and body sway with binocular and monocular vision at two distances from the front wall of a moving room. Ten participants stood as still as possible inside of a moving room facing the front wall in conditions that combined room movement with monocular/binocular vision and distance from the front wall (75 and 150cm). Visual information effect on body sway decreased with monocular vision and with increased distance from the front wall. In addition, the combination of monocular vision with the farther distance resulted in the smallest body sway response to the driving stimulus provided by the moving room. These results suggest that binocularvision near the front wall provides visual information of a better quality than the monocular vision far from the front wall. We discuss the results with respect to two modes of visual detection of body sway: ocular and extraocular. (C) 2009 Elsevier Ireland Ltd. All rights reserved.