Learning and Exploiting Camera Geometry for Computer Vision

This work explores the use of statistical methods in describing and estimating camera poses, as well as the information feedback loop between camera pose and object detection. Surging development in robotics and computer vision has pushed the need for algorithms that infer, understand, and utilize information about the position and orientation of the sensor platforms when observing and/or interacting with their environment.

The first contribution of this thesis is the development of a set of statistical tools for representing and estimating the uncertainty in object poses. A distribution for representing the joint uncertainty over multiple object positions and orientations is described, called the mirrored normal-Bingham distribution. This distribution generalizes both the normal distribution in Euclidean space, and the Bingham distribution on the unit hypersphere. It is shown to inherit many of the convenient properties of these special cases: it is the maximum-entropy distribution with fixed second moment, and there is a generalized Laplace approximation whose result is the mirrored normal-Bingham distribution. This distribution and approximation method are demonstrated by deriving the analytical approximation to the wrapped-normal distribution. Further, it is shown how these tools can be used to represent the uncertainty in the result of a bundle adjustment problem.

Another application of these methods is illustrated as part of a novel camera pose estimation algorithm based on object detections. The autocalibration task is formulated as a bundle adjustment problem using prior distributions over the 3D points to enforce the objects' structure and their relationship with the scene geometry. This framework is very flexible and enables the use of off-the-shelf computational tools to solve specialized autocalibration problems. Its performance is evaluated using a pedestrian detector to provide head and foot location observations, and it proves much faster and potentially more accurate than existing methods.

Finally, the information feedback loop between object detection and camera pose estimation is closed by utilizing camera pose information to improve object detection in scenarios with significant perspective warping. Methods are presented that allow the inverse perspective mapping traditionally applied to images to be applied instead to features computed from those images. For the special case of HOG-like features, which are used by many modern object detection systems, these methods are shown to provide substantial performance benefits over unadapted detectors while achieving real-time frame rates, orders of magnitude faster than comparable image warping methods.

The statistical tools and algorithms presented here are especially promising for mobile cameras, providing the ability to autocalibrate and adapt to the camera pose in real time. In addition, these methods have wide-ranging potential applications in diverse areas of computer vision, robotics, and imaging.

Axis Mapping: The Estimation of Surface Orientations and its Applications in Vehicle Localization and Structural Geology

The map representation of an environment should be selected based on its intended application. For example, a geometrically accurate map describing the Euclidean space of an environment is not necessarily the best choice if only a small subset its features are required. One possible subset is the orientations of the flat surfaces in the environment, represented by a special parameterization of normal vectors called axes. Devoid of positional information, the entries of an axis map form a non-injective relationship with the flat surfaces in the environment, which results in physically distinct flat surfaces being represented by a single axis. This drastically reduces the complexity of the map, but retains important information about the environment that can be used in meaningful applications in both two and three dimensions. This thesis presents axis mapping, which is an algorithm that accurately and automatically estimates an axis map of an environment based on sensor measurements collected by a mobile platform. Furthermore, two major applications of axis maps are developed and implemented. First, the LiDAR compass is a heading estimation algorithm that compares measurements of axes with an axis map of the environment. Pairing the LiDAR compass with simple translation measurements forms the basis for an accurate two-dimensional localization algorithm. It is shown that this algorithm eliminates the growth of heading error in both indoor and outdoor environments, resulting in accurate localization over long distances. Second, in the context of geotechnical engineering, a three-dimensional axis map is called a stereonet, which is used as a tool to examine the strength and stability of a rock face. Axis mapping provides a novel approach to create accurate stereonets safely, rapidly, and inexpensively compared to established methods. The non-injective property of axis maps is leveraged to probabilistically describe the relationships between non-sequential measurements of the rock face. The automatic estimation of stereonets was tested in three separate outdoor environments. It is shown that axis mapping can accurately estimate stereonets while improving safety, requiring significantly less time and effort, and lowering costs compared to traditional and current state-of-the-art approaches.

Requirements, Issues, and Challenges for Sense and Avoid in Unmanned Aircraft Systems

The sense and avoid capability is one of the greatest challenges that has to be addressed to safely integrate unmanned aircraft systems into civil and nonsegregated airspace. This paper gives a review of existing regulations, recommended practices, and standards in sense and avoid for unmanned aircraft systems. Gaps and issues are identified, as are the different factors that are likely to affect actual sense and avoid requirements. It is found that the operational environment (flight altitude, meteorological conditions, and class of airspace) plays an important role when determining the type of flying hazards that the unmanned aircraft system might encounter. In addition, the automation level and the data-link architecture of the unmanned aircraft system are key factors that will definitely determine the sense and avoid system requirements. Tactical unmanned aircraft, performing similar missions to general aviation, are found to be the most challenging systems from an sense and avoid point of view, and further research and development efforts are still needed before their seamless integration into nonsegregated airspace

Autonomous Vehicles and Commercial Real Estate

Hardly a day goes by without the release of a handful of news stories about autonomous vehicles (or AVs for short). The proverbial “tipping point” of awareness has been reached in the public consciousness as AV technology is quickly becoming the new focus of firms from Silicon Valley to Detroit and beyond. Automation has, and will continue to have far-reaching implications for many human activities, but for driving, the technology is here. Google has been in talks with automaker Ford (1), Elon Musk has declared that Tesla will have the appropriate technology in two years (2), GM is paired-up with Lyft (3), Uber is in development-mode (4), Microsoft and Volvo have announced a partnership (5), Apple has been piloting its top-secret project “Titan” (6), Toyota is working on its own technology (7), as is BMW (8). Audi (9) made a splash by sending a driverless A7 concept car 550 miles from San Francisco to Las Vegas just in time to roll-into the 2016 Consumer Electronics Show. Clearly, the race is on.

Applications of a Collaborative Open Object Oriented Learning Environment to Electronics, Computing and Mathematics Education

A lightweight Java application suite has been developed and deployed allowing collaborative learning between students and tutors at remote locations. Students can engage in group activities online and also collaborate with tutors. A generic Java framework has been developed and applied to electronics, computing and mathematics education. The applications are respectively: (a) a digital circuit simulator, which allows students to collaborate in building simple or complex electronic circuits; (b) a Java programming environment where the paradigm is behavioural-based robotics, and (c) a differential equation solver useful in modelling of any complex and nonlinear dynamic system. Each student sees a common shared window on which may be added text or graphical objects and which can then be shared online. A built-in chat room supports collaborative dialogue. Students can work either in collaborative groups or else in teams as directed by the tutor. This paper summarises the technical architecture of the system as well as the pedagogical implications of the suite. A report of student evaluation is also presented distilled from use over a period of twelve months. We intend this suite to facilitate learning between groups at one or many institutions and to facilitate international collaboration. We also intend to use the suite as a tool to research the establishment and behaviour of collaborative learning groups. We shall make our software freely available to interested researchers.

The Inverse Jacobian Control Method Applied to a Four Degree of Freedom Confined Space Robot

Thesis (Master's)--University of Washington, 2016-06

Techniques for batch reinforcement learning in robotics

This thesis addresses the Batch Reinforcement Learning methods in Robotics. This sub-class of Reinforcement Learning has shown promising results and has been the focus of recent research. Three contributions are proposed that aim to extend the state-of-art methods allowing for a faster and more stable learning process, such as required for learning in Robotics. The Q-learning update-rule is widely applied, since it allows to learn without the presence of a model of the environment. However, this update-rule is transition-based and does not take advantage of the underlying episodic structure of collected batch of interactions. The Q-Batch update-rule is proposed in this thesis, to process experiencies along the trajectories collected in the interaction phase. This allows a faster propagation of obtained rewards and penalties, resulting in faster and more robust learning. Non-parametric function approximations are explored, such as Gaussian Processes. This type of approximators allows to encode prior knowledge about the latent function, in the form of kernels, providing a higher level of exibility and accuracy. The application of Gaussian Processes in Batch Reinforcement Learning presented a higher performance in learning tasks than other function approximations used in the literature. Lastly, in order to extract more information from the experiences collected by the agent, model-learning techniques are incorporated to learn the system dynamics. In this way, it is possible to augment the set of collected experiences with experiences generated through planning using the learned models. Experiments were carried out mainly in simulation, with some tests carried out in a physical robotic platform. The obtained results show that the proposed approaches are able to outperform the classical Fitted Q Iteration.

Database-backed library web sites: a case study of the use of PHP and MySQL at the University of Nottingham

This paper discusses the advantages of database-backed websites and describes the model for a library website implemented at the University of Nottingham using open source software, PHP and MySQL. As websites continue to grow in size and complexity it becomes increasingly important to introduce automation to help manage them. It is suggested that a database-backed website offers many advantages over one built from static HTML pages. These include a consistency of style and content, the ability to present different views of the same data, devolved editing and enhanced security. The University of Nottingham Library Services website is described and issues surrounding its design, technological implementation and management are explored.

Multirobot Tethering for Localization and Control

Particle filtering has proven to be an effective localization method for wheeled autonomous vehicles. For a given map, a sensor model, and observations, occasions arise where the vehicle could equally likely be in many locations of the map. Because particle filtering algorithms may generate low confidence pose estimates under these conditions, more robust localization strategies are required to produce reliable pose estimates. This becomes more critical if the state estimate is an integral part of system control. We investigate the use of particle filter estimation techniques on a hovercraft vehicle. The marginally stable dynamics of a hovercraft require reliable state estimates for proper stability and control. We use the Monte Carlo localization method, which implements a particle filter in a recursive state estimate algorithm. An H-infinity controller, designed to accommodate the latency inherent in our state estimation, provides stability and controllability to the hovercraft. In order to eliminate the low confidence estimates produced in certain environments, a multirobot system is designed to introduce mobile environment features. By tracking and controlling the secondary robot, we can position the mobile feature throughout the environment to ensure a high confidence estimate, thus maintaining stability in the system. A laser rangefinder is the sensor the hovercraft uses to track the secondary robot, observe the environment, and facilitate successful localization and stability in motion.

DESIGN, FABRICATION, AND PERFORMANCE CHARACTERIZATION OF MULTIFUNCTIONAL STRUCTURES TO HARVEST SOLAR ENERGY FOR FLAPPING WING AERIAL VEHICLES

Flapping Wing Aerial Vehicles (FWAVs) have the capability to combine the benefits of both fixed wing vehicles and rotary vehicles. However, flight time is limited due to limited on-board energy storage capacity. For most Unmanned Aerial Vehicle (UAV) operators, frequent recharging of the batteries is not ideal due to lack of nearby electrical outlets. This imposes serious limitations on FWAV flights. The approach taken to extend the flight time of UAVs was to integrate photovoltaic solar cells onto different structures of the vehicle to harvest and use energy from the sun. Integration of the solar cells can greatly improve the energy capacity of an UAV; however, this integration does effect the performance of the UAV and especially FWAVs. The integration of solar cells affects the ability of the vehicle to produce the aerodynamic forces necessary to maintain flight. This PhD dissertation characterizes the effects of solar cell integration on the performance of a FWAV. Robo Raven, a recently developed FWAV, is used as the platform for this work. An additive manufacturing technique was developed to integrate photovoltaic solar cells into the wing and tail structures of the vehicle. An approach to characterizing the effects of solar cell integration to the wings, tail, and body of the UAV is also described. This approach includes measurement of aerodynamic forces generated by the vehicle and measurements of the wing shape during the flapping cycle using Digital Image Correlation. Various changes to wing, body, and tail design are investigated and changes in performance for each design are measured. The electrical performance from the solar cells is also characterized. A new multifunctional performance model was formulated that describes how integration of solar cells influences the flight performance. Aerodynamic models were developed to describe effects of solar cell integration force production and performance of the FWAV. Thus, performance changes can be predicted depending on changes in design. Sensing capabilities of the solar cells were also discovered and correlated to the deformation of the wing. This demonstrated that the solar cells were capable of: (1) Lightweight and flexible structure to generate aerodynamic forces, (2) Energy harvesting to extend operational time and autonomy, (3) Sensing of an aerodynamic force associated with wing deformation. Finally, different flexible photovoltaic materials with higher efficiencies are investigated, which enable the multifunctional wings to provide enough solar power to keep the FWAV aloft without batteries as long as there is enough sunlight to power the vehicle.

SOCIAL AND ECOLOGICAL FACTORS INFLUENCING COLLECTIVE BEHAVIOR IN GIANT DANIO

A fundamental problem in biology is understanding how and why things group together. Collective behavior is observed on all organismic levels - from cells and slime molds, to swarms of insects, flocks of birds, and schooling fish, and in mammals, including humans. The long-term goal of this research is to understand the functions and mechanisms underlying collective behavior in groups. This dissertation focuses on shoaling (aggregating) fish. Shoaling behaviors in fish confer foraging and anti-predator benefits through social cues from other individuals in the group. However, it is not fully understood what information individuals receive from one another or how this information is propagated throughout a group. It is also not fully understood how the environmental conditions and perturbations affect group behaviors. The specific research objective of this dissertation is to gain a better understanding of how certain social and environmental factors affect group behaviors in fish. I focus on two ecologically relevant decision-making behaviors: (i) rheotaxis, or orientation with respect to a flow, and (ii) startle response, a rapid response to a perceived threat. By integrating behavioral and engineering paradigms, I detail specifics of behavior in giant danio Devario aequipinnatus (McClelland 1893), and numerically analyze mathematical models that may be extended to group behavior for fish in general, and potentially other groups of animals as well. These models that predict behavior data, as well as generate additional, testable hypotheses. One of the primary goals of neuroethology is to study an organism's behavior in the context of evolution and ecology. Here, I focus on studying ecologically relevant behaviors in giant danio in order to better understand collective behavior in fish. The experiments in this dissertation provide contributions to fish ecology, collective behavior, and biologically-inspired robotics.

Evaluating alternative gait strategies using evolutionary robotics

Evolutionary robitics is a branch of artificial intelligence concerned with the automatic generation of autonomous robots. Usually the form of the robit is predefined an various computational techniques are used to control the machine's behaviour. One aspect is the spontaneous generation of walking in legged robots and this can be used to investigate the mechanical requiements for efficient walking in bipeds. This paper demonstrates a bipedal simulator that spontaneously generates walking and running gaits. The model can be customized to represent a range of hominoid morphologies and used to predict performance paramets such as preferred speed and metabolic energy cost. Because it does not require any motion capture data it is particularly suitable for investigating locomotion in fossil animals. The predictoins for modern humans are highly accurate in terms of energy cost for a given speend and thus the values predicted for other bipeds are likely to be good estimates. To illustrate this the cost of transport is calculated for Australopithecus afarensis. The model allows the degree of maximum extension at the knee to be varied causing the model to adopt walking gaits varying from chimpanzee-like to human=like. The energy costs associated with these gait choices can thus be calculated and this information used to evaluate possible locomotor strategies in early hominids

Predicting the metabolic energy costs of bipedalism using evolutionary robotics

To understand the evolution of bipedalism among the homnoids in an ecological context we need to be able to estimate theenerrgetic cost of locomotion in fossil forms. Ideally such an estimate would be based entirely on morphology since, except for the rare instances where footprints are preserved, this is hte only primary source of evidence available. In this paper we use evolutionary robotics techniques (genetic algoritms, pattern generators and mechanical modeling) to produce a biomimentic simulation of bipedalism based on human body dimensions. The mechnaical simulation is a seven-segment, two-dimensional model with motive force provided by tension generators representing the major muscle groups acting around the lower-limb joints. Metabolic energy costs are calculated from the muscel model, and bipedal gait is generated using a finite-state pattern generator whose parameters are produced using a genetic algorithm with locomotor economy (maximum distance for a fixed energy cost) as the fitness criterion. The model is validated by comparing the values it generates with those for modern humans. The result (maximum efficiency of 200 J m-1) is within 15% of the experimentally derived value, which is very encouraging and suggests that this is a useful analytic technique for investigating the locomotor behaviour of fossil forms. Initial work suggests that in the future this technique could be used to estimate other locomotor parameters such as top speed. In addition, the animations produced by this technique are qualitatively very convincing, which suggests that this may also be a useful technique for visualizing bipedal locomotion.

Dynamically balanced and plausible trajectory planning for human-like characters

International audience