Learning and Exploiting Camera Geometry for Computer Vision

This work explores the use of statistical methods in describing and estimating camera poses, as well as the information feedback loop between camera pose and object detection. Surging development in robotics and computer vision has pushed the need for algorithms that infer, understand, and utilize information about the position and orientation of the sensor platforms when observing and/or interacting with their environment.

The first contribution of this thesis is the development of a set of statistical tools for representing and estimating the uncertainty in object poses. A distribution for representing the joint uncertainty over multiple object positions and orientations is described, called the mirrored normal-Bingham distribution. This distribution generalizes both the normal distribution in Euclidean space, and the Bingham distribution on the unit hypersphere. It is shown to inherit many of the convenient properties of these special cases: it is the maximum-entropy distribution with fixed second moment, and there is a generalized Laplace approximation whose result is the mirrored normal-Bingham distribution. This distribution and approximation method are demonstrated by deriving the analytical approximation to the wrapped-normal distribution. Further, it is shown how these tools can be used to represent the uncertainty in the result of a bundle adjustment problem.

Another application of these methods is illustrated as part of a novel camera pose estimation algorithm based on object detections. The autocalibration task is formulated as a bundle adjustment problem using prior distributions over the 3D points to enforce the objects' structure and their relationship with the scene geometry. This framework is very flexible and enables the use of off-the-shelf computational tools to solve specialized autocalibration problems. Its performance is evaluated using a pedestrian detector to provide head and foot location observations, and it proves much faster and potentially more accurate than existing methods.

Finally, the information feedback loop between object detection and camera pose estimation is closed by utilizing camera pose information to improve object detection in scenarios with significant perspective warping. Methods are presented that allow the inverse perspective mapping traditionally applied to images to be applied instead to features computed from those images. For the special case of HOG-like features, which are used by many modern object detection systems, these methods are shown to provide substantial performance benefits over unadapted detectors while achieving real-time frame rates, orders of magnitude faster than comparable image warping methods.

The statistical tools and algorithms presented here are especially promising for mobile cameras, providing the ability to autocalibrate and adapt to the camera pose in real time. In addition, these methods have wide-ranging potential applications in diverse areas of computer vision, robotics, and imaging.

On Provable Algorithms for Determination of Continuous Protein Interdomain Motions from Residual Dipolar Couplings

Dynamics of biomolecules over various spatial and time scales are essential for biological functions such as molecular recognition, catalysis and signaling. However, reconstruction of biomolecular dynamics from experimental observables requires the determination of a conformational probability distribution. Unfortunately, these distributions cannot be fully constrained by the limited information from experiments, making the problem an ill-posed one in the terminology of Hadamard. The ill-posed nature of the problem comes from the fact that it has no unique solution. Multiple or even an infinite number of solutions may exist. To avoid the ill-posed nature, the problem needs to be regularized by making assumptions, which inevitably introduce biases into the result.

Here, I present two continuous probability density function approaches to solve an important inverse problem called the RDC trigonometric moment problem. By focusing on interdomain orientations we reduced the problem to determination of a distribution on the 3D rotational space from residual dipolar couplings (RDCs). We derived an analytical equation that relates alignment tensors of adjacent domains, which serves as the foundation of the two methods. In the first approach, the ill-posed nature of the problem was avoided by introducing a continuous distribution model, which enjoys a smoothness assumption. To find the optimal solution for the distribution, we also designed an efficient branch-and-bound algorithm that exploits the mathematical structure of the analytical solutions. The algorithm is guaranteed to find the distribution that best satisfies the analytical relationship. We observed good performance of the method when tested under various levels of experimental noise and when applied to two protein systems. The second approach avoids the use of any model by employing maximum entropy principles. This 'model-free' approach delivers the least biased result which presents our state of knowledge. In this approach, the solution is an exponential function of Lagrange multipliers. To determine the multipliers, a convex objective function is constructed. Consequently, the maximum entropy solution can be found easily by gradient descent methods. Both algorithms can be applied to biomolecular RDC data in general, including data from RNA and DNA molecules.

Behavioral Ecology of the Western Atlantic Short-finned Pilot Whale (Globicephala macrorhynchus)

Social structure is a key determinant of population biology and is central to the way animals exploit their environment. The risk of predation is often invoked as an important factor influencing the evolution of social structure in cetaceans and other mammals, but little direct information is available about how cetaceans actually respond to predators or other perceived threats. The playback of sounds to an animal is a powerful tool for assessing behavioral responses to predators, but quantifying behavioral responses to playback experiments requires baseline knowledge of normal behavioral patterns and variation. The central goal of my dissertation is to describe baseline foraging behavior for the western Atlantic short-finnned pilot whales (Globicephala macrohynchus) and examine the role of social organization in their response to predators. To accomplish this I used multi-sensor digital acoustic tags (DTAGs), satellite-linked time-depth recorders (SLTDR), and playback experiments to study foraging behavior and behavioral response to predators in pilot whales. Fine scale foraging strategies and population level patterns were identified by estimating the body size and examining the location and movement around feeding events using data collected with DTAGs deployed on 40 pilot whales in summers of 2008-2014 off the coast of Cape Hatteras, North Carolina. Pilot whales were found to forage throughout the water column and performed feeding buzzes at depths ranging from 29-1176 meters. The results indicated potential habitat segregation in foraging depth in short-finned pilot whales with larger individuals foraging on average at deeper depths. Calculated aerobic dive limit for large adult males was approximately 6 minutes longer than that of females and likely facilitated the difference in foraging depth. Furthermore, the buzz frequency and speed around feeding attempts indicate this population pilot whales are likely targeting multiple small prey items. Using these results, I built decision trees to inform foraging dive classification in coarse, long-term dive data collected with SLTDRs deployed on 6 pilot whales in the summers of 2014 and 2015 in the same area off the coast of North Carolina. I used these long term foraging records to compare diurnal foraging rates and depths, as well as classify bouts with a maximum likelihood method, and evaluate behavioral aerobic dive limits (ADLB) through examination of dive durations and inter-dive intervals. Dive duration was the best predictor of foraging, with dives >400.6 seconds classified as foraging, and a 96% classification accuracy. There were no diurnal patterns in foraging depth or rates and average duration of bouts was 2.94 hours with maximum bout durations lasting up to 14 hours. The results indicated that pilot whales forage in relatively long bouts and the ADLB indicate that pilot whales rarely, if ever exceed their aerobic limits. To evaluate the response to predators I used controlled playback experiments to examine the behavioral responses of 10 of the tagged short-finned pilot whales off Cape Hatteras, North Carolina and 4 Risso’s dolphins (Grampus griseus) off Southern California to the calls of mammal-eating killer whales (MEK). Both species responded to a subset of MEK calls with increased movement, swim speed and increased cohesion of the focal groups, but the two species exhibited different directional movement and vocal responses. Pilot whales increased their call rate and approached the sound source, but Risso’s dolphins exhibited no change in their vocal behavior and moved in a rapid, directed manner away from the source. Thus, at least to a sub-set of mammal-eating killer whale calls, these two study species reacted in a manner that is consistent with their patterns of social organization. Pilot whales, which live in relatively permanent groups bound by strong social bonds, responded in a manner that built on their high levels of social cohesion. In contrast, Risso’s dolphins exhibited an exaggerated flight response and moved rapidly away from the sound source. The fact that both species responded strongly to a select number of MEK calls, suggests that structural features of signals play critical contextual roles in the probability of response to potential threats in odontocete cetaceans.

Dosimetric Evaluation of Metal Artefact Reduction using Metal Artefact Reduction (MAR) Algorithm and Dual-energy Computed Tomography (CT) Method

Purpose: Computed Tomography (CT) is one of the standard diagnostic imaging modalities for the evaluation of a patient’s medical condition. In comparison to other imaging modalities such as Magnetic Resonance Imaging (MRI), CT is a fast acquisition imaging device with higher spatial resolution and higher contrast-to-noise ratio (CNR) for bony structures. CT images are presented through a gray scale of independent values in Hounsfield units (HU). High HU-valued materials represent higher density. High density materials, such as metal, tend to erroneously increase the HU values around it due to reconstruction software limitations. This problem of increased HU values due to metal presence is referred to as metal artefacts. Hip prostheses, dental fillings, aneurysm clips, and spinal clips are a few examples of metal objects that are of clinical relevance. These implants create artefacts such as beam hardening and photon starvation that distort CT images and degrade image quality. This is of great significance because the distortions may cause improper evaluation of images and inaccurate dose calculation in the treatment planning system. Different algorithms are being developed to reduce these artefacts for better image quality for both diagnostic and therapeutic purposes. However, very limited information is available about the effect of artefact correction on dose calculation accuracy. This research study evaluates the dosimetric effect of metal artefact reduction algorithms on severe artefacts on CT images. This study uses Gemstone Spectral Imaging (GSI)-based MAR algorithm, projection-based Metal Artefact Reduction (MAR) algorithm, and the Dual-Energy method.

Materials and Methods: The Gemstone Spectral Imaging (GSI)-based and SMART Metal Artefact Reduction (MAR) algorithms are metal artefact reduction protocols embedded in two different CT scanner models by General Electric (GE), and the Dual-Energy Imaging Method was developed at Duke University. All three approaches were applied in this research for dosimetric evaluation on CT images with severe metal artefacts. The first part of the research used a water phantom with four iodine syringes. Two sets of plans, multi-arc plans and single-arc plans, using the Volumetric Modulated Arc therapy (VMAT) technique were designed to avoid or minimize influences from high-density objects. The second part of the research used projection-based MAR Algorithm and the Dual-Energy Method. Calculated Doses (Mean, Minimum, and Maximum Doses) to the planning treatment volume (PTV) were compared and homogeneity index (HI) calculated.

Results: (1) Without the GSI-based MAR application, a percent error between mean dose and the absolute dose ranging from 3.4-5.7% per fraction was observed. In contrast, the error was decreased to a range of 0.09-2.3% per fraction with the GSI-based MAR algorithm. There was a percent difference ranging from 1.7-4.2% per fraction between with and without using the GSI-based MAR algorithm. (2) A range of 0.1-3.2% difference was observed for the maximum dose values, 1.5-10.4% for minimum dose difference, and 1.4-1.7% difference on the mean doses. Homogeneity indexes (HI) ranging from 0.068-0.065 for dual-energy method and 0.063-0.141 with projection-based MAR algorithm were also calculated.

Conclusion: (1) Percent error without using the GSI-based MAR algorithm may deviate as high as 5.7%. This error invalidates the goal of Radiation Therapy to provide a more precise treatment. Thus, GSI-based MAR algorithm was desirable due to its better dose calculation accuracy. (2) Based on direct numerical observation, there was no apparent deviation between the mean doses of different techniques but deviation was evident on the maximum and minimum doses. The HI for the dual-energy method almost achieved the desirable null values. In conclusion, the Dual-Energy method gave better dose calculation accuracy to the planning treatment volume (PTV) for images with metal artefacts than with or without GE MAR Algorithm.