Vision-Based Tracking, Odometry and Control for UAV Autonomy

El principal objetivo de este trabajo es proporcionar una solución en tiempo real basada en visión estéreo o monocular precisa y robusta para que un vehículo aéreo no tripulado (UAV) sea autónomo en varios tipos de aplicaciones UAV, especialmente en entornos abarrotados sin señal GPS. Este trabajo principalmente consiste en tres temas de investigación de UAV basados en técnicas de visión por computador: (I) visual tracking, proporciona soluciones efectivas para localizar visualmente objetos de interés estáticos o en movimiento durante el tiempo que dura el vuelo del UAV mediante una aproximación adaptativa online y una estrategia de múltiple resolución, de este modo superamos los problemas generados por las diferentes situaciones desafiantes, tales como cambios significativos de aspecto, iluminación del entorno variante, fondo del tracking embarullado, oclusión parcial o total de objetos, variaciones rápidas de posición y vibraciones mecánicas a bordo. La solución ha sido utilizada en aterrizajes autónomos, inspección de plataformas mar adentro o tracking de aviones en pleno vuelo para su detección y evasión; (II) odometría visual: proporciona una solución eficiente al UAV para estimar la posición con 6 grados de libertad (6D) usando únicamente la entrada de una cámara estéreo a bordo del UAV. Un método Semi-Global Blocking Matching (SGBM) eficiente basado en una estrategia grueso-a-fino ha sido implementada para una rápida y profunda estimación del plano. Además, la solución toma provecho eficazmente de la información 2D y 3D para estimar la posición 6D, resolviendo de esta manera la limitación de un punto de referencia fijo en la cámara estéreo. Una robusta aproximación volumétrica de mapping basada en el framework Octomap ha sido utilizada para reconstruir entornos cerrados y al aire libre bastante abarrotados en 3D con memoria y errores correlacionados espacialmente o temporalmente; (III) visual control, ofrece soluciones de control prácticas para la navegación de un UAV usando Fuzzy Logic Controller (FLC) con la estimación visual. Y el framework de Cross-Entropy Optimization (CEO) ha sido usado para optimizar el factor de escala y la función de pertenencia en FLC. Todas las soluciones basadas en visión en este trabajo han sido probadas en test reales. Y los conjuntos de datos de imágenes reales grabados en estos test o disponibles para la comunidad pública han sido utilizados para evaluar el rendimiento de estas soluciones basadas en visión con ground truth. Además, las soluciones de visión presentadas han sido comparadas con algoritmos de visión del estado del arte. Los test reales y los resultados de evaluación muestran que las soluciones basadas en visión proporcionadas han obtenido rendimientos en tiempo real precisos y robustos, o han alcanzado un mejor rendimiento que aquellos algoritmos del estado del arte. La estimación basada en visión ha ganado un rol muy importante en controlar un UAV típico para alcanzar autonomía en aplicaciones UAV. ABSTRACT The main objective of this dissertation is providing real-time accurate robust monocular or stereo vision-based solution for Unmanned Aerial Vehicle (UAV) to achieve the autonomy in various types of UAV applications, especially in GPS-denied dynamic cluttered environments. This dissertation mainly consists of three UAV research topics based on computer vision technique: (I) visual tracking, it supplys effective solutions to visually locate interesting static or moving object over time during UAV flight with on-line adaptivity approach and multiple-resolution strategy, thereby overcoming the problems generated by the different challenging situations, such as significant appearance change, variant surrounding illumination, cluttered tracking background, partial or full object occlusion, rapid pose variation and onboard mechanical vibration. The solutions have been utilized in autonomous landing, offshore floating platform inspection and midair aircraft tracking for sense-and-avoid; (II) visual odometry: it provides the efficient solution for UAV to estimate the 6 Degree-of-freedom (6D) pose using only the input of stereo camera onboard UAV. An efficient Semi-Global Blocking Matching (SGBM) method based on a coarse-to-fine strategy has been implemented for fast depth map estimation. In addition, the solution effectively takes advantage of both 2D and 3D information to estimate the 6D pose, thereby solving the limitation of a fixed small baseline in the stereo camera. A robust volumetric occupancy mapping approach based on the Octomap framework has been utilized to reconstruct indoor and outdoor large-scale cluttered environments in 3D with less temporally or spatially correlated measurement errors and memory; (III) visual control, it offers practical control solutions to navigate UAV using Fuzzy Logic Controller (FLC) with the visual estimation. And the Cross-Entropy Optimization (CEO) framework has been used to optimize the scaling factor and the membership function in FLC. All the vision-based solutions in this dissertation have been tested in real tests. And the real image datasets recorded from these tests or available from public community have been utilized to evaluate the performance of these vision-based solutions with ground truth. Additionally, the presented vision solutions have compared with the state-of-art visual algorithms. Real tests and evaluation results show that the provided vision-based solutions have obtained real-time accurate robust performances, or gained better performance than those state-of-art visual algorithms. The vision-based estimation has played a critically important role for controlling a typical UAV to achieve autonomy in the UAV application.

Fluorescent RNA cytochemistry: tracking gene transcripts in living cells

The advent of jellyfish green fluorescent protein and its spectral variants, together with promising new fluorescent proteins from other classes of the Cnidarian phylum (coral and anemones), has greatly enhanced and promises to further boost the detection and localization of proteins in cell biology. It has been less widely appreciated that highly sensitive methods have also recently been developed for detecting the movement and localization in living cells of the very molecules that precede proteins in the gene expression pathway, i.e. RNAs. These approaches include the microinjection of fluorescent RNAs into living cells, the in vivo hybridization of fluorescent oligonucleotides to endogenous RNAs and the expression in cells of fluorescent RNA-binding proteins. This new field of ‘fluorescent RNA cytochemistry’ is summarized in this article, with emphasis on the biological insights it has already provided. These new techniques are likely to soon collaborate with other emerging approaches to advance the investigation of RNA birth, RNA–protein assembly and ribonucleoprotein particle transport in systems such as oocytes, embryos, neurons and other somatic cells, and may even permit the observation of viral replication and transcription pathways as they proceed in living cells, ushering in a new era of nucleic acids research in vivo.

Analysis of fixational eye micromovements through pupil tracking

Póster presentado en OPTYKA Optical Fair 2012, Poznan, Polonia, 9-10 noviembre 2012.

Method for targetless tracking subpixel in-plane movements

We present a targetless motion tracking method for detecting planar movements with subpixel accuracy. This method is based on the computation and tracking of the intersection of two nonparallel straight-line segments in the image of a moving object in a scene. The method is simple and easy to implement because no complex structures have to be detected. It has been tested and validated using a lab experiment consisting of a vibrating object that was recorded with a high-speed camera working at 1000 fps. We managed to track displacements with an accuracy of hundredths of pixel or even of thousandths of pixel in the case of tracking harmonic vibrations. The method is widely applicable because it can be used for distance measuring amplitude and frequency of vibrations with a vision system.

Visualization of Cerenkov radiation and the fields of a moving charge

For some physics students, the concept of a particle travelling faster than the speed of light holds endless fascination, and. Cerenkov radiation is a visible consequence of a charged particle travelling through a medium at locally superluminal velocities. The Heaviside-Feynman equations for calculating the magnetic and electric fields of a moving charge have been known for many decades, but it is only recently that the computing power to plot the fields of such a particle has become readily available for student use. This paper investigates and illustrates the calculation of Maxwell's D field in homogeneous isotropic media for arbitrary, including superluminal, constant velocity, and uses the results as a basis for discussing energy transfer in the electromagnetic field.

An indexing structure and application model for vehicles moving on road networks

Moving objects database systems are the most challenging sub-category among Spatio-Temporal database systems. A database system that updates in real-time the location information of GPS-equipped moving vehicles has to meet even stricter requirements. Currently existing data storage models and indexing mechanisms work well only when the number of moving objects in the system is relatively small. This dissertation research aimed at the real-time tracking and history retrieval of massive numbers of vehicles moving on road networks. A total solution has been provided for the real-time update of the vehicles' location and motion information, range queries on current and history data, and prediction of vehicles' movement in the near future. ^ To achieve these goals, a new approach called Segmented Time Associated to Partitioned Space (STAPS) was first proposed in this dissertation for building and manipulating the indexing structures for moving objects databases. ^ Applying the STAPS approach, an indexing structure of associating a time interval tree to each road segment was developed for real-time database systems of vehicles moving on road networks. The indexing structure uses affordable storage to support real-time data updates and efficient query processing. The data update and query processing performance it provides is consistent without restrictions such as a time window or assuming linear moving trajectories. ^ An application system design based on distributed system architecture with centralized organization was developed to maximally support the proposed data and indexing structures. The suggested system architecture is highly scalable and flexible. Finally, based on a real-world application model of vehicles moving in region-wide, main issues on the implementation of such a system were addressed. ^

An Indexing Structure and Application Model for Vehicles Moving on Road Networks

Moving objects database systems are the most challenging sub-category among Spatio-Temporal database systems. A database system that updates in real-time the location information of GPS-equipped moving vehicles has to meet even stricter requirements. Currently existing data storage models and indexing mechanisms work well only when the number of moving objects in the system is relatively small. This dissertation research aimed at the real-time tracking and history retrieval of massive numbers of vehicles moving on road networks. A total solution has been provided for the real-time update of the vehicles’ location and motion information, range queries on current and history data, and prediction of vehicles’ movement in the near future. To achieve these goals, a new approach called Segmented Time Associated to Partitioned Space (STAPS) was first proposed in this dissertation for building and manipulating the indexing structures for moving objects databases. Applying the STAPS approach, an indexing structure of associating a time interval tree to each road segment was developed for real-time database systems of vehicles moving on road networks. The indexing structure uses affordable storage to support real-time data updates and efficient query processing. The data update and query processing performance it provides is consistent without restrictions such as a time window or assuming linear moving trajectories. An application system design based on distributed system architecture with centralized organization was developed to maximally support the proposed data and indexing structures. The suggested system architecture is highly scalable and flexible. Finally, based on a real-world application model of vehicles moving in region-wide, main issues on the implementation of such a system were addressed.

DESEO: an active vision system for detection, tracking and recognition

[EN]In this paper, a basic conceptual architecture aimed at the design of Computer Vision System is qualitatively described. The proposed architecture addresses the design of vision systems in a modular fashion using modules with three distinct units or components: a processing network or diagnostics unit, a control unit and a communications unit. The control of the system at the modules level is designed based on a Discrete Events Model. This basic methodology has been used to design a realtime active vision system for detection, tracking and recognition of people. It is made up of three functional modules aimed at the detection, tracking, recognition of moving individuals plus a supervision module.

Real-time tracking system using C80 DSPs and a binocular robotic head

[EN]An active vision system to perform tracking of moving objects in real time is described. The main goal is to obtain a system integrating off-the-self components. These components includes a stereoscopic robotic-head, as active perception hardware; a DSP based board SDB C80, as massive data processor and image acquisition board; and finally, a Pentium PC running Windows NT that interconnects and manages the whole system. Real-time is achieved taking advantage of the special architecture of DSP. An evaluation of the performance is included.

Nuclei of Nearby Active Galaxies: NIR and Sub-mm Views and Finalizing the LINC-NIRVANA Fringe and Flexure Tracking System: Flexure and Temperature Behavior

The work presented in my thesis addresses the two cornerstones of modern astronomy: Observation and Instrumentation. Part I deals with the observation of two nearby active galaxies, the Seyfert 2 galaxy NGC 1433 and the Seyfert 1 galaxy NGC 1566, both at a distance of $\sim10$ Mpc, which are part of the Nuclei of Galaxies (NUGA) sample. It is well established that every galaxy harbors a super massive black hole (SMBH) at its center. Furthermore, there seems to be a fundamental correlation between the stellar bulge and SMBH masses. Simulations show that massive feedback, e.g., powerful outflows, in Quasi Stellar Objects (QSOs) has an impact on the mutual growth of bulge and SMBH. Nearby galaxies follow this relation but accrete mass at much lower rates. This gives rise to the following questions: Which mechanisms allow feeding of nearby Active Galactic Nuclei (AGN)? Is this feeding triggered by events, e.g., star formation, nuclear spirals, outflows, on $\sim500$ pc scales around the AGN? Does feedback on these scales play a role in quenching the feeding process? Does it have an effect on the star formation close to the nucleus? To answer these questions I have carried out observations with the Spectrograph for INtegral Field Observation in the Near Infrared (SINFONI) at the Very Large Telescope (VLT) situated on Cerro Paranal in Chile. I have reduced and analyzed the recorded data, which contain spatial and spectral information in the H-band ($1.45 \mic-1.85 \mic$) and K-band ($1.95 \mic-2.45 \mic$) on the central $10\arcsec\times10\arcsec$ of the observed galaxies. Additionally, Atacama Large Millimeter/Sub-millimeter Array (ALMA) data at $350$ GHz ($\sim0.87$ mm) as well as optical high resolution Hubble Space Telescope (HST) images are used for the analysis. For NGC 1433 I deduce from comparison of the distributions of gas, dust, and intensity of highly ionized emission lines that the galaxy center lies $\sim70$ pc north-northwest of the prior estimate. A velocity gradient is observed at the new center, which I interpret as a bipolar outflow, a circum nuclear disk, or a combination of both. At least one dust and gas arm leads from a $r\sim200$ pc ring towards the nucleus and might feed the SMBH. Two bright warm H$_2$ gas spots are detected that indicate hidden star formation or a spiral arm-arm interaction. From the stellar velocity dispersion (SVD) I estimate a SMBH mass of $\sim1.74\times10^7$ \msol. For NGC 1566 I observe a nuclear gas disk of $\sim150$ pc in radius with a spiral structure. I estimate the total mass of this disk to be $\sim5.4\times10^7$ \msol. What mechanisms excite the gas in the disk is not clear. Neither can the existence of outflows be proven nor is star formation detected over the whole disk. On one side of the spiral structure I detect a star forming region with an estimated star formation rate of $\sim2.6\times10^{-3}$ \msol\ yr$^{-1}$. From broad Br$\gamma$ emission and SVD I estimate a mean SMBH mass of $\sim5.3\times10^6$ \msol\ with an Eddington ratio of $\sim2\times10^{-3}$. Part II deals with the final tests of the Fringe and Flexure Tracker (FFTS) for LBT INterferometric Camera and the NIR/Visible Adaptive iNterferometer for Astronomy (LINC-NIRVANA) at the Large Binocular Telescope (LBT) in Arizona, USA, which I conducted. The FFTS is the subsystem that combines the two separate beams of the LBT and enables near-infrared interferometry with a significantly large field of view. The FFTS has a cryogenic system and an ambient temperature system which are separated by the baffle system. I redesigned this baffle to guarantee the functionality of the system after the final tests in the Cologne cryostat. The redesign did not affect any scientific performance of LINC-NIRVANA. I show in the final cooldown tests that the baffle fulfills the temperature requirement and stays $<110$ K whereas the moving stages in the ambient system stay $>273$ K, which was not given for the old baffle design. Additionally, I test the tilting flexure of the whole FFTS and show that accurate positioning of the detector and the tracking during observation can be guaranteed.

Enhancing SPH using moving least-squares and radial basis functions

In this paper we consider two sources of enhancement for the meshfree Lagrangian particle method smoothed particle hydrodynamics (SPH) by improving the accuracy of the particle approximation. Namely, we will consider shape functions constructed using: moving least-squares approximation (MLS); radial basis functions (RBF). Using MLS approximation is appealing because polynomial consistency of the particle approximation can be enforced. RBFs further appeal as they allow one to dispense with the smoothing-length - the parameter in the SPH method which governs the number of particles within the support of the shape function. Currently, only ad hoc methods for choosing the smoothing-length exist. We ensure that any enhancement retains the conservative and meshfree nature of SPH. In doing so, we derive a new set of variationally-consistent hydrodynamic equations. Finally, we demonstrate the performance of the new equations on the Sod shock tube problem.

Existence and weak-strong uniqueness for the Navier-Stokes-Smoluchowski system over moving domains

This dissertation concerns the well-posedness of the Navier-Stokes-Smoluchowski system. The system models a mixture of fluid and particles in the so-called bubbling regime. The compressible Navier-Stokes equations governing the evolution of the fluid are coupled to the Smoluchowski equation for the particle density at a continuum level. First, working on fixed domains, the existence of weak solutions is established using a three-level approximation scheme and based largely on the Lions-Feireisl theory of compressible fluids. The system is then posed over a moving domain. By utilizing a Brinkman-type penalization as well as penalization of the viscosity, the existence of weak solutions of the Navier-Stokes-Smoluchowski system is proved over moving domains. As a corollary the convergence of the Brinkman penalization is proved. Finally, a suitable relative entropy is defined. This relative entropy is used to establish a weak-strong uniqueness result for the Navier-Stokes-Smoluchowski system over moving domains, ensuring that strong solutions are unique in the class of weak solutions.

Automatic dynamic mask extraction for PIV images containing an unsteady interface, bubbles, and a moving structure

When performing Particle Image Velocimetry (PIV) measurements in complex fluid flows with moving interfaces and a two-phase flow, it is necessary to develop a mask to remove non-physical measurements. This is the case when studying, for example, the complex bubble sweep-down phenomenon observed in oceanographic research vessels. Indeed, in such a configuration, the presence of an unsteady free surface, of a solid–liquid interface and of bubbles in the PIV frame, leads to generate numerous laser reflections and therefore spurious velocity vectors. In this note, an image masking process is developed to successively identify the boundaries of the ship and the free surface interface. As the presence of the solid hull surface induces laser reflections, the hull edge contours are simply detected in the first PIV frame and dynamically estimated for consecutive ones. As for the unsteady surface determination, a specific process is implemented like the following: i) the edge detection of the gradient magnitude in the PIV frame, ii) the extraction of the particles by filtering high-intensity large areas related to the bubbles and/or hull reflections, iii) the extraction of the rough region containing these particles and their reflections, iv) the removal of these reflections. The unsteady surface is finally obtained with a fifth-order polynomial interpolation. The resulted free surface is successfully validated from the Fourier analysis and by visualizing selected PIV images containing numerous spurious high intensity areas. This paper demonstrates how this data analysis process leads to PIV images database without reflections and an automatic detection of both the free surface and the rigid body. An application of this new mask is finally detailed, allowing a preliminary analysis of the hydrodynamic flow.