Robust Real-time Vision-based Aircraft Tracking From Unmanned Aerial Vehicles.

Aircraft tracking plays a key and important role in the Sense-and-Avoid system of Unmanned Aerial Vehicles (UAVs). This paper presents a novel robust visual tracking algorithm for UAVs in the midair to track an arbitrary aircraft at real-time frame rates, together with a unique evaluation system. This visual algorithm mainly consists of adaptive discriminative visual tracking method, Multiple-Instance (MI) learning approach, Multiple-Classifier (MC) voting mechanism and Multiple-Resolution (MR) representation strategy, that is called Adaptive M3 tracker, i.e. AM3. In this tracker, the importance of test sample has been integrated to improve the tracking stability, accuracy and real-time performances. The experimental results show that this algorithm is more robust, efficient and accurate against the existing state-of-art trackers, overcoming the problems generated by the challenging situations such as obvious appearance change, variant surrounding illumination, partial aircraft occlusion, blur motion, rapid pose variation and onboard mechanical vibration, low computation capacity and delayed information communication between UAVs and Ground Station (GS). To our best knowledge, this is the first work to present this tracker for solving online learning and tracking freewill aircraft/intruder in the UAVs.

A Ground-Truth Video Dataset for the Development and Evaluation of Vision-based Sense-and-Avoid systems

The importance of vision-based systems for Sense-and-Avoid is increasing nowadays as remotely piloted and autonomous UAVs become part of the non-segregated airspace. The development and evaluation of these systems demand flight scenario images which are expensive and risky to obtain. Currently Augmented Reality techniques allow the compositing of real flight scenario images with 3D aircraft models to produce useful realistic images for system development and benchmarking purposes at a much lower cost and risk. With the techniques presented in this paper, 3D aircraft models are positioned firstly in a simulated 3D scene with controlled illumination and rendering parameters. Realistic simulated images are then obtained using an image processing algorithm which fuses the images obtained from the 3D scene with images from real UAV flights taking into account on board camera vibrations. Since the intruder and camera poses are user-defined, ground truth data is available. These ground truth annotations allow to develop and quantitatively evaluate aircraft detection and tracking algorithms. This paper presents the software developed to create a public dataset of 24 videos together with their annotations and some tracking application results.

Online Learning-based Robust Visual Tracking for Autonomous Landing of Unmanned Aerial Vehicles

Autonomous landing is a challenging and important technology for both military and civilian applications of Unmanned Aerial Vehicles (UAVs). In this paper, we present a novel online adaptive visual tracking algorithm for UAVs to land on an arbitrary field (that can be used as the helipad) autonomously at real-time frame rates of more than twenty frames per second. The integration of low-dimensional subspace representation method, online incremental learning approach and hierarchical tracking strategy allows the autolanding task to overcome the problems generated by the challenging situations such as significant appearance change, variant surrounding illumination, partial helipad occlusion, rapid pose variation, onboard mechanical vibration (no video stabilization), low computational capacity and delayed information communication between UAV and Ground Control Station (GCS). The tracking performance of this presented algorithm is evaluated with aerial images from real autolanding flights using manually- labelled ground truth database. The evaluation results show that this new algorithm is highly robust to track the helipad and accurate enough for closing the vision-based control loop.

Real-Time Adaptive Multi-Classifier Multi-Resolution Visual Tracking Framework for Unmanned Aerial Vehicles

This paper presents a novel robust visual tracking framework, based on discriminative method, for Unmanned Aerial Vehicles (UAVs) to track an arbitrary 2D/3D target at real-time frame rates, that is called the Adaptive Multi-Classifier Multi-Resolution (AMCMR) framework. In this framework, adaptive Multiple Classifiers (MC) are updated in the (k-1)th frame-based Multiple Resolutions (MR) structure with compressed positive and negative samples, and then applied them in the kth frame-based Multiple Resolutions (MR) structure to detect the current target. The sample importance has been integrated into this framework to improve the tracking stability and accuracy. The performance of this framework was evaluated with the Ground Truth (GT) in different types of public image databases and real flight-based aerial image datasets firstly, then the framework has been applied in the UAV to inspect the Offshore Floating Platform (OFP). The evaluation and application results show that this framework is more robust, efficient and accurate against the existing state-of-art trackers, overcoming the problems generated by the challenging situations such as obvious appearance change, variant illumination, partial/full target occlusion, blur motion, rapid pose variation and onboard mechanical vibration, among others. To our best knowledge, this is the first work to present this framework for solving the online learning and tracking freewill 2D/3D target problems, and applied it in the UAVs.

Las Redes de Petri Interpretadas en el Diagnóstico de Fallos de Sistemas Híbridos. Aplicación a un Helicóptero no Tripulado

La seguridad y fiabilidad de los procesos industriales son la principal preocupación de los ingenieros encargados de las plantas industriales. Por lo tanto, desde un punto de vista económico, el objetivo principal es reducir el costo del mantenimiento, el tiempo de inactividad y las pérdidas causadas por los fallos. Por otra parte, la seguridad de los operadores, que afecta a los aspectos sociales y económicos, es el factor más relevante a considerar en cualquier sistema Debido a esto, el diagnóstico de fallos se ha convertido en un foco importante de interés para los investigadores de todo el mundo e ingenieros en la industria. Los principales trabajos enfocados en detección de fallos se basan en modelos de los procesos. Existen diferentes técnicas para el modelado de procesos industriales tales como máquinas de estado, árboles de decisión y Redes de Petri (RdP). Por lo tanto, esta tesis se centra en el modelado de procesos utilizando redes de petri interpretadas. Redes de Petri es una herramienta usada en el modelado gráfico y matemático con la habilidad para describir información de los sistemas de una manera concurrente, paralela, asincrona, distribuida y no determinística o estocástica. RdP son también una herramienta de comunicación visual gráfica útil como lo son las cartas de flujo o diagramas de bloques. Adicionalmente, las marcas de las RdP simulan la dinámica y concurrencia de los sistemas. Finalmente, ellas tienen la capacidad de definir ecuaciones de estado específicas, ecuaciones algebraicas y otros modelos que representan el comportamiento común de los sistemas. Entre los diferentes tipos de redes de petri (Interpretadas, Coloreadas, etc.), este trabajo de investigación trata con redes de petri interpretadas principalmente debido a características tales como sincronización, lugares temporizados, aparte de su capacidad para procesamiento de datos. Esta investigación comienza con el proceso para diseñar y construir el modelo y diagnosticador para detectar fallos definitivos, posteriormente, la dinámica temporal fue adicionada para detectar fallos intermitentes. Dos procesos industriales, concretamente un HVAC (Calefacción, Ventilación y Aire Acondicionado) y un Proceso de Envasado de Líquidos fueron usados como banco de pruebas para implementar la herramienta de diagnóstico de fallos (FD) creada. Finalmente, su capacidad de diagnóstico fue ampliada en orden a detectar fallos en sistemas híbridos. Finalmente, un pequeño helicóptero no tripulado fue elegido como ejemplo de sistema donde la seguridad es un desafío, y las técnicas de detección de fallos desarrolladas en esta tesis llevan a ser una herramienta valorada, desde que los accidentes de las aeronaves no tripuladas (UAVs) envuelven un alto costo económico y son la principal razón para introducir restricciones de volar sobre áreas pobladas. Así, este trabajo introduce un proceso sistemático para construir un Diagnosticador de Fallos del sistema mencionado basado en RdR Esta novedosa herramienta es capaz de detectar fallos definitivos e intermitentes. El trabajo realizado es discutido desde un punto de vista teórico y práctico. El procedimiento comienza con la división del sistema en subsistemas para seguido integrar en una RdP diagnosticadora global que es capaz de monitorear el sistema completo y mostrar las variables críticas al operador en orden a determinar la salud del UAV, para de esta manera prevenir accidentes. Un Sistema de Adquisición de Datos (DAQ) ha sido también diseñado para recoger datos durante los vuelos y alimentar la RdP diagnosticadora. Vuelos reales realizados bajo condiciones normales y de fallo han sido requeridos para llevar a cabo la configuración del diagnosticador y verificar su comportamiento. Vale la pena señalar que un alto riesgo fue asumido en la generación de fallos durante los vuelos, a pesar de eso esto permitió recoger datos básicos para desarrollar el diagnóstico de fallos, técnicas de aislamiento, protocolos de mantenimiento, modelos de comportamiento, etc. Finalmente, un resumen de la validación de resultados obtenidos durante las pruebas de vuelo es también incluido. Un extensivo uso de esta herramienta mejorará los protocolos de mantenimiento para UAVs (especialmente helicópteros) y permite establecer recomendaciones en regulaciones. El uso del diagnosticador usando redes de petri es considerado un novedoso enfoque. ABSTRACT Safety and reliability of industrial processes are the main concern of the engineers in charge of industrial plants. Thus, from an economic point of view, the main goal is to reduce the maintenance downtime cost and the losses caused by failures. Moreover, the safety of the operators, which affects to social and economic aspects, is the most relevant factor to consider in any system. Due to this, fault diagnosis has become a relevant focus of interest for worldwide researchers and engineers in the industry. The main works focused on failure detection are based on models of the processes. There are different techniques for modelling industrial processes such as state machines, decision trees and Petri Nets (PN). Thus, this Thesis is focused on modelling processes by using Interpreted Petri Nets. Petri Nets is a tool used in the graphic and mathematical modelling with ability to describe information of the systems in a concurrent, parallel, asynchronous, distributed and not deterministic or stochastic manner. PNs are also useful graphical visual communication tools as flow chart or block diagram. Additionally, the marks of the PN simulate the dynamics and concurrence of the systems. Finally, they are able to define specific state equations, algebraic equations and other models that represent the common behaviour of systems. Among the different types of PN (Interpreted, Coloured, etc.), this research work deals with the interpreted Petri Nets mainly due to features such as synchronization capabilities, timed places, apart from their capability for processing data. This Research begins with the process for designing and building the model and diagnoser to detect permanent faults, subsequently, the temporal dynamic was added for detecting intermittent faults. Two industrial processes, namely HVAC (Heating, Ventilation and Air Condition) and Liquids Packaging Process were used as testbed for implementing the Fault Diagnosis (FD) tool created. Finally, its diagnostic capability was enhanced in order to detect faults in hybrid systems. Finally, a small unmanned helicopter was chosen as example of system where safety is a challenge and fault detection techniques developed in this Thesis turn out to be a valuable tool since UAVs accidents involve high economic cost and are the main reason for setting restrictions to fly over populated areas. Thus, this work introduces a systematic process for building a Fault Diagnoser of the mentioned system based on Petri Nets. This novel tool is able to detect both intermittent and permanent faults. The work carried out is discussed from theoretical and practical point of view. The procedure begins with a division of the system into subsystems for further integration into a global PN diagnoser that is able to monitor the whole system and show critical variables to the operator in order to determine the UAV health, preventing accidents in this manner. A Data Acquisition System (DAQ) has been also designed for collecting data during the flights and feed PN Diagnoser. Real flights carried out under nominal and failure conditions have been required to perform the diagnoser setup and verify its performance. It is worth noting that a high risk was assumed in the generation of faults during the flights, nevertheless this allowed collecting basic data so as to develop fault diagnosis, isolations techniques, maintenance protocols, behaviour models, etc. Finally, a summary of the validation results obtained during real flight tests is also included. An extensive use of this tool will improve preventive maintenance protocols for UAVs (especially helicopters) and allow establishing recommendations in regulations. The use of the diagnoser by using Petri Nets is considered as novel approach.

Herramienta de interacción persona-ordenador para la operación de vehículos aéreos no tripulados

En la presente memoria se describe el trabajo de diseño de una herramienta de interacción persona-ordenador (HMI) para la operación y supervisión de vehículos aéreos no tripulados (UAV). En primer lugar se hace una introducción a los tipos de UAVs y aplicaciones más comunes, describiendo sus características técnicas y los componentes que integra en el sistema. Mediante la revisión y análisis de los diferentes niveles de autonomía y las diferentes soluciones de presentación existentes en el mercado, se identifican los modos de operación y componentes principales de la interfaz. A continuación se describe el diseño final del software de la interfaz y el proceso de desarrollo de la misma, para ello se hace un análisis previo del software robótico sobre el que opera el sistema abordo del UAV y se establecen los enlaces de comunicación entre cada uno de los componentes y los requisitos de integración con el sistema. Finalmente, se muestran las pruebas que se han realizado para validar la construcción de la herramienta. This report outlines the design and construction of a human-machine interface (HMI), designed to facilitate the supervision and operation with unmanned aerial vehicles (UAV). First, it is described an introduction to UAVs classification and application fields, reviewing the hardware features and software integration components. In order to define the basic components and operation modes in the general design, a brief review of the different presentation solutions and autonomous levels is described. As a result, it is presented the final software design, the components details and the system integration requirements. Finally, it is also concluded with some of the tests that have been conducted to validate the design and construction of the human-machine interface

Entorno de supervisión para vehículos aéreos no tripulados

La Universidad Politécnica de Madrid está investigando en el campo de la robótica inteligente, concretamente con el empleo de vehículos aéreos no tripulados (UAV). El objetivo final que se persigue con las investigaciones en este campo es el desarrollo de sistemas capaces de operar de forma más autónoma en un amplio espectro de situaciones. Dentro de este marco, este trabajo fin de grado se centra en el desarrollo de un sistema de supervisión para UAVs que persigue facilitar la monitorización de la ejecución de los procesos y facilitar la inclusión de procedimientos para incrementar la tolerancia a los fallos software. A lo largo de esta memoria se ofrece una revisión del estado del arte en el ámbito de la robótica, haciendo especial hincapié en la robótica inteligente con los métodos de desarrollo existentes y la definición de los distintos marcos de clasificación de la autonomía. También se ofrece una vista a las distintas técnicas existentes para lograr una mayor tolerancia a los fallos software, de entre las que han sido seleccionadas varias de ellas en la realización de este trabajo. Finalmente se describe el sistema de supervisión desarrollado, explicando primero el sistema desde un punto de vista funcional para más adelante adentrarse en la solución técnica elaborada. ---ABSTRACT--- The Universidad Politécnica de Madrid is currently handling several investigations regarding AI robotics, some of them are actually directing their efforts into the use of unmanned aerial vehicles (UAV). The goal in the long term for this investigations is the accomplishment of systems capable of operating autonomously, regardless of the situation the robot is place at. From this perspective, this final degree project focuses on de design and development of a supervision system for UAV’s, which function is to ease the monitoring of executing processes and the inclusion of fault tolerant procedures. During the development of this document a state of the art revision is offered, in which a thorough description through development methods and autonomy definitions for AI robotics is made. It is also offered a look around the different existing techniques for achieving a greater software fault tolerance, from which some of them were chosen for the development of this project. Finally the developed supervision system is described, first from a pure functional perspective of what the system should do and latter with a description of the actual technical solutions developed for this system.

Systematic Process for Building a Fault Diagnoser Based on Petri Nets Applied to a Helicopter

This work presents a systematic process for building a Fault Diagnoser (FD), based on Petri Nets (PNs) which has been applied to a small helicopter. This novel tool is able to detect both intermittent and permanent faults. The work carried out is discussed from theoretical and practical point of view. The procedure begins with a division of the whole system into subsystems, which are the devices that have to be modeled by using PN, considering both the normal and fault operations. Subsequently, the models are integrated into a global Petri Net diagnoser (PND) that is able to monitor a whole helicopter and show critical variables to the operator in order to determine the UAV health, preventing accidents in this manner. A Data Acquisition System (DAQ) has been designed for collecting data during the flights and feeding PN diagnoser with them. Several real flights (nominal or under failure) have been carried out to perform the diagnoser setup and verify its performance. A summary of the validation results obtained during real flight tests is also included. An extensive use of this tool will improve preventive maintenance protocols for UAVs (especially helicopters) and allow establishing recommendations in regulations

Multi-UAV Coordination and Control Interface

The interest in missions with multiple Unmanned Aerial Vehicles (UAVs) has increased significantly in last years. These missions take advantage of the use of fleets instead of single UAVs to ensure the success, reduce the duration or increase the goals of the mission. In addition, they allow performing tasks that require multiple agents and certain coordination (e.g. surveillance of large areas or transport of heavy loads). Nevertheless, these missions suppose a challenge in terms of control and monitoring. In fact, the workload of the operators rises with the utilization of multiple UAVs and payloads, since they have to analyze more information, make more decisions and generate more commands during the mission. This work addresses the operator workload problem in multi-UAV missions by reducing and selecting the information. Two approaches are considered: a first one that selects the information according to the mission state, and a second one that selects it according to the operator preferences. The result is an interface that is able to control the amount of information and show what is relevant for mission and operator at the time.