Visual Tracking, Pose Estimation, and Control for Aerial Vehicles

El principal objetivo de esta tesis es dotar a los vehículos aéreos no tripulados (UAVs, por sus siglas en inglés) de una fuente de información adicional basada en visión. Esta fuente de información proviene de cámaras ubicadas a bordo de los vehículos o en el suelo. Con ella se busca que los UAVs realicen tareas de aterrizaje o inspección guiados por visión, especialmente en aquellas situaciones en las que no haya disponibilidad de estimar la posición del vehículo con base en GPS, cuando las estimaciones de GPS no tengan la suficiente precisión requerida por las tareas a realizar, o cuando restricciones de carga de pago impidan añadir sensores a bordo de los vehículos. Esta tesis trata con tres de las principales áreas de la visión por computador: seguimiento visual y estimación visual de la pose (posición y orientación), que a su vez constituyen la base de la tercera, denominada control servo visual, que en nuestra aplicación se enfoca en el empleo de información visual para controlar los UAVs. Al respecto, esta tesis se ocupa de presentar propuestas novedosas que permitan solucionar problemas relativos al seguimiento de objetos mediante cámaras ubicadas a bordo de los UAVs, se ocupa de la estimación de la pose de los UAVs basada en información visual obtenida por cámaras ubicadas en el suelo o a bordo, y también se ocupa de la aplicación de las técnicas propuestas para solucionar diferentes problemas, como aquellos concernientes al seguimiento visual para tareas de reabastecimiento autónomo en vuelo o al aterrizaje basado en visión, entre otros. Las diversas técnicas de visión por computador presentadas en esta tesis se proponen con el fin de solucionar dificultades que suelen presentarse cuando se realizan tareas basadas en visión con UAVs, como las relativas a la obtención, en tiempo real, de estimaciones robustas, o como problemas generados por vibraciones. Los algoritmos propuestos en esta tesis han sido probados con información de imágenes reales obtenidas realizando pruebas on-line y off-line. Diversos mecanismos de evaluación han sido empleados con el propósito de analizar el desempeño de los algoritmos propuestos, entre los que se incluyen datos simulados, imágenes de vuelos reales, estimaciones precisas de posición empleando el sistema VICON y comparaciones con algoritmos del estado del arte. Los resultados obtenidos indican que los algoritmos de visión por computador propuestos tienen un desempeño que es comparable e incluso mejor al de algoritmos que se encuentran en el estado del arte. Los algoritmos propuestos permiten la obtención de estimaciones robustas en tiempo real, lo cual permite su uso en tareas de control visual. El desempeño de estos algoritmos es apropiado para las exigencias de las distintas aplicaciones examinadas: reabastecimiento autónomo en vuelo, aterrizaje y estimación del estado del UAV. Abstract The main objective of this thesis is to provide Unmanned Aerial Vehicles (UAVs) with an additional vision-based source of information extracted by cameras located either on-board or on the ground, in order to allow UAVs to develop visually guided tasks, such as landing or inspection, especially in situations where GPS information is not available, where GPS-based position estimation is not accurate enough for the task to develop, or where payload restrictions do not allow the incorporation of additional sensors on-board. This thesis covers three of the main computer vision areas: visual tracking and visual pose estimation, which are the bases the third one called visual servoing, which, in this work, focuses on using visual information to control UAVs. In this sense, the thesis focuses on presenting novel solutions for solving the tracking problem of objects when using cameras on-board UAVs, on estimating the pose of the UAVs based on the visual information collected by cameras located either on the ground or on-board, and also focuses on applying these proposed techniques for solving different problems, such as visual tracking for aerial refuelling or vision-based landing, among others. The different computer vision techniques presented in this thesis are proposed to solve some of the frequently problems found when addressing vision-based tasks in UAVs, such as obtaining robust vision-based estimations at real-time frame rates, and problems caused by vibrations, or 3D motion. All the proposed algorithms have been tested with real-image data in on-line and off-line tests. Different evaluation mechanisms have been used to analyze the performance of the proposed algorithms, such as simulated data, images from real-flight tests, publicly available datasets, manually generated ground truth data, accurate position estimations using a VICON system and a robotic cell, and comparison with state of the art algorithms. Results show that the proposed computer vision algorithms obtain performances that are comparable to, or even better than, state of the art algorithms, obtaining robust estimations at real-time frame rates. This proves that the proposed techniques are fast enough for vision-based control tasks. Therefore, the performance of the proposed vision algorithms has shown to be of a standard appropriate to the different explored applications: aerial refuelling and landing, and state estimation. It is noteworthy that they have low computational overheads for vision systems.

A General Purpose Configurable Controller for Indoors and Outdoors GPS-Denied Navigation for Multirotor Unmanned Aerial Vehicles

This research on odometry based GPS-denied navigation on multirotor Unmanned Aerial Vehicles is focused among the interactions between the odometry sensors and the navigation controller. More precisely, we present a controller architecture that allows to specify a speed specified flight envelope where the quality of the odometry measurements is guaranteed. The controller utilizes a simple point mass kinematic model, described by a set of configurable parameters, to generate a complying speed plan. For experimental testing, we have used down-facing camera optical-flow as odometry measurement. This work is a continuation of prior research to outdoors environments using an AR Drone 2.0 vehicle, as it provides reliable optical flow on a wide range of flying conditions and floor textures. Our experiments show that the architecture is realiable for outdoors flight on altitudes lower than 9 m. A prior version of our code was utilized to compete in the International Micro Air Vehicle Conference and Flight Competition IMAV 2012. The code will be released as an open-source ROS stack hosted on GitHub.

Evaluation of mission planning strategies of a robotic aerial vehicle for the AIRC international competition

The International Aerial Robotics Competition (IARC) is an important event where teams from universities design flying autonomous vehicles to overcome the last challenges in the field. The goal of the Seventh Mission proposed by the IARC is to guide several mobile ground robots to a target area. The scenario is complex and not determinist due to the random behavior of the ground robots movement. The UAV must select efficient strategies to complete the mission. The goal of this work has been evaluating different alternative mission planning strategies of a UAV for this competition. The Mission Planner component is in charge of taking the UAV decisions. Different strategies have been developed and evaluated for the component, achieving a better performance Mission Planner and valuable knowledge about the mission. For this purpose, it was necessary to develop a simulator to evaluate the different strategies. The simulator was built as an improvement of an existing previous version.

A Motor-less and Gear-less Bio-mimetic Robotic Fish Design

In this paper, we describe our current work on bio-inspired locomotion systems using a deformable structure and smart materials, concretely Shape Memory Alloys, exploring the possibility of building motor-less and gear-less robots. A swimming underwater robot has been developed whose movements are generated using such actuators, used for bending the backbone of the fish, which in turn causes a change on the curvature of the body. This paper focuses on how standard swimming patterns can be reproduced with the proposed design, using an actuation dynamics model identified in prior work.

Improving Speech Interaction in Vehicles Using Context-Aware Information through A SCXML Framework

Speech Technologies can provide important benefits for the development of more usable and safe in-vehicle human-machine interactive systems (HMIs). However mainly due robustness issues, the use of spoken interaction can entail important distractions to the driver. In this challenging scenario, while speech technologies are evolving, further research is necessary to explore how they can be complemented with both other modalities (multimodality) and information from the increasing number of available sensors (context-awareness). The perceived quality of speech technologies can significantly be increased by implementing such policies, which simply try to make the best use of all the available resources; and the in vehicle scenario is an excellent test-bed for this kind of initiatives. In this contribution we propose an event-based HMI design framework which combines context modelling and multimodal interaction using a W3C XML language known as SCXML. SCXML provides a general process control mechanism that is being considered by W3C to improve both voice interaction (VoiceXML) and multimodal interaction (MMI). In our approach we try to anticipate and extend these initiatives presenting a flexible SCXML-based approach for the design of a wide range of multimodal context-aware HMI in-vehicle interfaces. The proposed framework for HMI design and specification has been implemented in an automotive OSGi service platform, and it is being used and tested in the Spanish research project MARTA for the development of several in-vehicle interactive applications.

3D Object following based on visual information for Unmanned Aerial Vehicles

This article presents a novel system and a control strategy for visual following of a 3D moving object by an Unmanned Aerial Vehicle UAV. The presented strategy is based only on the visual information given by an adaptive tracking method based on the color information, which jointly with the dynamics of a camera fixed to a rotary wind UAV are used to develop an Image-based visual servoing IBVS system. This system is focused on continuously following a 3D moving target object, maintaining it with a fixed distance and centered on the image plane. The algorithm is validated on real flights on outdoors scenarios, showing the robustness of the proposed systems against winds perturbations, illumination and weather changes among others. The obtained results indicate that the proposed algorithms is suitable for complex controls task, such object following and pursuit, flying in formation, as well as their use for indoor navigation

Precision Agriculture without borders: Practical issues and improvements in farmland coverage with aerial vehicles.

This work presents a solution for the aerial coverage of a field by using a fleet of aerial vehicles. The use of Unmanned Aerial Vehicles allows to obtain high resolution mosaics to be used in Precision Agriculture techniques. This report is focus on providing a solution for the full simultaneous coverage problem taking into account restrictions as the required spatial resolution and overlap while maintaining similar light conditions and safety operation of the drones. Results obtained from real field tests are finally reported

Safety assessment of underground vehicles passing over highly resilient straight tracks in the presence of a broken rail

In this paper, vehicle-track interaction for a new slab track design, conceived to reduce noise and vibration levels has been analyzed, assessing the derailment risk for trains running on curved track when encountering a broken rail. Two different types of rail fastening systems with different elasticities have been analysed and compared. Numerical methods were used in order to simulate the dynamic behaviour of the train-track interaction. Multibody system (MBS) modelling techniques were combined with techniques based on the finite element method (FEM). MBS modelling was used for modelling the vehicle and FEM for simulating the elastic track. The simulation model was validated by comparing simulated results to experimental data obtained in field testing. During the simulations various safety indices, characteristic of derailment risk, were analysed. The simulations realised at the maximum running velocity of 110 km/h showed a similar behaviour for several track types. When reducing the running speed, the safety indices worsened for both cases. Although the worst behaviour was observed for the track with a greater elasticity, in none of the simulations did a derailment occur when running over the broken rail.

Safety assessment of underground vehicles passing over highly resilient curved tracks in the presence of a broken rail

Vehicle–track interaction for a new resilient slab track designed to reduce noise and vibration levels was analysed, in order to assess the derailment risk on a curved track when encountering a broken rail. Sensitivity of the rail support spacing of the relative position of the rail breakage between two adjacent rail supports and of running speed were analysed for two different elasticities of the rail fastening system. In none of the cases analysed was observed an appreciable difference between either of the elastic systems. As was expected, the most unfavourable situations were those with greater rail support spacing and those with greater distance from the breakage to the nearest rail support, although in none of the simulations performed did a derailment occur when running over the broken rail. When varying the running speed, the most favourable condition was obtained for an intermediate speed, due to the superposition of two antagonistic effects.

Reverse logistics systems for waste generated throughout vehicles life-cycle

Waste produced during the service life of automobiles has received much less attention than end-of-life vehicles themselves. In this paper, we deal with the set up of a reverse logistics system for the collection and treatment of use-phase residues. First, the type of waste arising during vehicles? service life is characterized. Data were collected in collaboration with SIGRAUTO, the product stewardship organization in charge of vehicles? recovery in Spain. Next, three organizational models are proposed. The three alternatives are benchmarked and assessed from a double organizational and operational perspective for the particular case of the Madrid region in Spain

A Multiresolution Image Alignment Technique Based on Direct Methods for Pose Estimation of Aerial Vehicles

In this paper, we seek to expand the use of direct methods in real-time applications by proposing a vision-based strategy for pose estimation of aerial vehicles. The vast majority of approaches make use of features to estimate motion. Conversely, the strategy we propose is based on a MR (Multi-Resolution) implementation of an image registration technique (Inverse Compositional Image Alignment ICIA) using direct methods. An on-board camera in a downwards-looking configuration, and the assumption of planar scenes, are the bases of the algorithm. The motion between frames (rotation and translation) is recovered by decomposing the frame-to-frame homography obtained by the ICIA algorithm applied to a patch that covers around the 80% of the image. When the visual estimation is required (e.g. GPS drop-out), this motion is integrated with the previous known estimation of the vehicles' state, obtained from the on-board sensors (GPS/IMU), and the subsequent estimations are based only on the vision-based motion estimations. The proposed strategy is tested with real flight data in representative stages of a flight: cruise, landing, and take-off, being two of those stages considered critical: take-off and landing. The performance of the pose estimation strategy is analyzed by comparing it with the GPS/IMU estimations. Results show correlation between the visual estimation obtained with the MR-ICIA and the GPS/IMU data, that demonstrate that the visual estimation can be used to provide a good approximation of the vehicle's state when it is required (e.g. GPS drop-outs). In terms of performance, the proposed strategy is able to maintain an estimation of the vehicle's state for more than one minute, at real-time frame rates based, only on visual information.

On-board and Ground Visual Pose Estimation Techniques for UAV Control

In this paper, two techniques to control UAVs (Unmanned Aerial Vehicles), based on visual information are presented. The first one is based on the detection and tracking of planar structures from an on-board camera, while the second one is based on the detection and 3D reconstruction of the position of the UAV based on an external camera system. Both strategies are tested with a VTOL (Vertical take-off and landing) UAV, and results show good behavior of the visual systems (precision in the estimation and frame rate) when estimating the helicopter¿s position and using the extracted information to control the UAV.

Bending continuous structures with SMAs: a novel robotic fish design

In this paper, we describe our research on bio-inspired locomotion systems using deformable structures and smart materials, concretely shape memory alloys (SMAs). These types of materials allow us to explore the possibility of building motor-less and gear-less robots. A swimming underwater fish-like robot has been developed whose movements are generated using SMAs. These actuators are suitable for bending the continuous backbone of the fish, which in turn causes a change in the curvature of the body. This type of structural arrangement is inspired by fish red muscles, which are mainly recruited during steady swimming for the bending of a flexible but nearly incompressible structure such as the fishbone. This paper reviews the design process of these bio-inspired structures, from the motivations and physiological inspiration to the mechatronics design, control and simulations, leading to actual experimental trials and results. The focus of this work is to present the mechanisms by which standard swimming patterns can be reproduced with the proposed design. Moreover, the performance of the SMA-based actuators’ control in terms of actuation speed and position accuracy is also addressed.

Automatic lateral control for unmanned vehicles via genetic algorithms

It is known that the techniques under the topic of Soft Computing have a strong capability of learning and cognition, as well as a good tolerance to uncertainty and imprecision. Due to these properties they can be applied successfully to Intelligent Vehicle Systems; ITS is a broad range of technologies and techniques that hold answers to many transportation problems. The unmannedcontrol of the steering wheel of a vehicle is one of the most important challenges facing researchers in this area. This paper presents a method to adjust automatically a fuzzy controller to manage the steering wheel of a mass-produced vehicle; to reach it, information about the car state while a human driver is handling the car is taken and used to adjust, via iterative geneticalgorithms an appropriated fuzzy controller. To evaluate the obtained controllers, it will be considered the performance obtained in the track following task, as well as the smoothness of the driving carried out.

Cartography for cooperative manoeuvres with autonomous land vehicles

This article presents a cartographic system to facilitate cooperative manoeuvres among autonomous vehicles in a well-known environment. The main objective is to design an extended cartographic system to help in the navigation of autonomous vehicles. This system has to allow the vehicles not only to access the reference points needed for navigation, but also noticeable information such as the location and type of traffic signals, the proximity to a crossing, the streets en route, etc. To do this, a hierarchical representation of the information has been chosen, where the information has been stored in two levels. The lower level contains the archives with the Universal Traverse Mercator (UTM) coordinates of the points that define the reference segments to follow. The upper level contains a directed graph with the relational database in which streets, crossings, roundabouts and other points of interest are represented. Using this new system it is possible to know when the vehicle approaches a crossing, what other paths arrive at that crossing, and, should there be other vehicles circulating on those paths and arriving at the crossing, which one has the highest priority. The data obtained from the cartographic system is used by the autonomous vehicles for cooperative manoeuvres.