Visual Tracking, Pose Estimation, and Control for Aerial Vehicles

El principal objetivo de esta tesis es dotar a los vehículos aéreos no tripulados (UAVs, por sus siglas en inglés) de una fuente de información adicional basada en visión. Esta fuente de información proviene de cámaras ubicadas a bordo de los vehículos o en el suelo. Con ella se busca que los UAVs realicen tareas de aterrizaje o inspección guiados por visión, especialmente en aquellas situaciones en las que no haya disponibilidad de estimar la posición del vehículo con base en GPS, cuando las estimaciones de GPS no tengan la suficiente precisión requerida por las tareas a realizar, o cuando restricciones de carga de pago impidan añadir sensores a bordo de los vehículos. Esta tesis trata con tres de las principales áreas de la visión por computador: seguimiento visual y estimación visual de la pose (posición y orientación), que a su vez constituyen la base de la tercera, denominada control servo visual, que en nuestra aplicación se enfoca en el empleo de información visual para controlar los UAVs. Al respecto, esta tesis se ocupa de presentar propuestas novedosas que permitan solucionar problemas relativos al seguimiento de objetos mediante cámaras ubicadas a bordo de los UAVs, se ocupa de la estimación de la pose de los UAVs basada en información visual obtenida por cámaras ubicadas en el suelo o a bordo, y también se ocupa de la aplicación de las técnicas propuestas para solucionar diferentes problemas, como aquellos concernientes al seguimiento visual para tareas de reabastecimiento autónomo en vuelo o al aterrizaje basado en visión, entre otros. Las diversas técnicas de visión por computador presentadas en esta tesis se proponen con el fin de solucionar dificultades que suelen presentarse cuando se realizan tareas basadas en visión con UAVs, como las relativas a la obtención, en tiempo real, de estimaciones robustas, o como problemas generados por vibraciones. Los algoritmos propuestos en esta tesis han sido probados con información de imágenes reales obtenidas realizando pruebas on-line y off-line. Diversos mecanismos de evaluación han sido empleados con el propósito de analizar el desempeño de los algoritmos propuestos, entre los que se incluyen datos simulados, imágenes de vuelos reales, estimaciones precisas de posición empleando el sistema VICON y comparaciones con algoritmos del estado del arte. Los resultados obtenidos indican que los algoritmos de visión por computador propuestos tienen un desempeño que es comparable e incluso mejor al de algoritmos que se encuentran en el estado del arte. Los algoritmos propuestos permiten la obtención de estimaciones robustas en tiempo real, lo cual permite su uso en tareas de control visual. El desempeño de estos algoritmos es apropiado para las exigencias de las distintas aplicaciones examinadas: reabastecimiento autónomo en vuelo, aterrizaje y estimación del estado del UAV. Abstract The main objective of this thesis is to provide Unmanned Aerial Vehicles (UAVs) with an additional vision-based source of information extracted by cameras located either on-board or on the ground, in order to allow UAVs to develop visually guided tasks, such as landing or inspection, especially in situations where GPS information is not available, where GPS-based position estimation is not accurate enough for the task to develop, or where payload restrictions do not allow the incorporation of additional sensors on-board. This thesis covers three of the main computer vision areas: visual tracking and visual pose estimation, which are the bases the third one called visual servoing, which, in this work, focuses on using visual information to control UAVs. In this sense, the thesis focuses on presenting novel solutions for solving the tracking problem of objects when using cameras on-board UAVs, on estimating the pose of the UAVs based on the visual information collected by cameras located either on the ground or on-board, and also focuses on applying these proposed techniques for solving different problems, such as visual tracking for aerial refuelling or vision-based landing, among others. The different computer vision techniques presented in this thesis are proposed to solve some of the frequently problems found when addressing vision-based tasks in UAVs, such as obtaining robust vision-based estimations at real-time frame rates, and problems caused by vibrations, or 3D motion. All the proposed algorithms have been tested with real-image data in on-line and off-line tests. Different evaluation mechanisms have been used to analyze the performance of the proposed algorithms, such as simulated data, images from real-flight tests, publicly available datasets, manually generated ground truth data, accurate position estimations using a VICON system and a robotic cell, and comparison with state of the art algorithms. Results show that the proposed computer vision algorithms obtain performances that are comparable to, or even better than, state of the art algorithms, obtaining robust estimations at real-time frame rates. This proves that the proposed techniques are fast enough for vision-based control tasks. Therefore, the performance of the proposed vision algorithms has shown to be of a standard appropriate to the different explored applications: aerial refuelling and landing, and state estimation. It is noteworthy that they have low computational overheads for vision systems.

Mecanismos de Fotopercepción No-Visual y Control Circadiano en la Retina Interna de Vertebrados en Condiciones Fisiológicas y en un Modelo Animal de Ceguera. Mechanisms of Non-visual Perception and Circadian Control in the Vertebrate Inner Retina under Physiological Conditions and in Animal Model of Blindness.

La retina juega un rol esencial en el funcionamiento del sistema circadiano de los vertebrados al ser la encargada de sensar las condiciones de iluminación ambiental que ajustan el reloj interno con el fotoperíodo exterior a través de un circuito no-visual. Este circuito es independiente de la vía de formación de imágenes e involucra a las células ganglionares retinianas (CGRs) que proyectan a varias estructuras no-visuales del cerebro; esta vía es la encargada de regular el reflejo pupilar, la sincronización de los ritmos diarios de actividad, el sueño y la supresión de melatonina pineal. La retina contiene además un reloj autónomo que genera ritmos diarios autosostenidos en distintas funciones bioquímicas y fisiológicas, que le confiere la capacidad de predecir el tiempo y anticiparse en su fisiología a los cambios lumínicos a lo largo del ciclo día-noche. Este laboratorio ha demostrado por 1ra vez que las CGRs de pollo poseen osciladores endógenos que generan variaciones diarias en la biosíntesis de fosfolípidos (Guido et al, J Neurochem. 2001; Garbarino et al., J Neurosci Res. 2004a) y de la hormona melatonina con niveles máximos durante el día (Garbarino et al., J Biol Chem 2004b). Aún más, cultivos primarios de CGRs responden a la luz a través de una cascada bioquímica de fototransducción similar a la de invertebrados y que involucra la activación de la enzima fosfolipasa C (PLC) (Contin et al., FASEB J 2006). Estos cultivos fueron obtenidos a estadios embrionarios muy tempranos en dónde solo las CGRs son postmitóticas y mayoritariamente maduras. A estos estadios, los cultivos expresan marcadores de especificación de células ganglionares (pax6, brn3), la proteina Gq y los fotopigmentos melanopsina y criptocromos con gran homología con marcadores descriptos para fotorreceptores rabdoméricos de invertebrados (Contin et al, 2006). Recientemente comenzamos a investigar la percepción de luz en pollos GUCY1*, un modelo de ceguera, en animales que carecen de células fotorreceptoras-conos y bastones-funcionales. Resultados preliminares indicarían que la retina interna, y potencialmente las CGRs de estos animales conservarían la capacidad de responder a la luz regulando el reflejo pupilar y sincronizando los ritmos diarios de alimentación. La convergencia de osciladores y fotopigmentos en la población de CGRs podría contribuir al control temporal de la fisiología del organismo y regulación de funciones no-visuales. Son objetivos de este proyecto: a) Investigar el rol de las CGRs en el sistema circadiano estudiando: i- su habilidad para sintetizar melatonina y, su regulación por luz y dopamina; ii- su capacidad fotorreceptora intrínseca, investigando la presencia de fotopigmentos y componentes de la cascada de fototransducción fundamentalmente la vía de los fosfoinosítidos y la activación de PLC, mediante ensayos moleculares, bioquímicos y farmacológicos; b) Extender estos estudios a cultivos primarios de CGRs inmunopurificadas midiendo la respuesta a la luz sobre la síntesis de melatonina, y los niveles de los mensajeros 2rios Ca2+ y AMP cíclico, la inducción de genes tempranos y la regulación de la actividad NAT, enzima clave en la síntesis de melatonina; y c) Investigar la percepción de luz en pollos GUCY1*(ciegos), sobre distintas funciones no-visuales tales como el reflejo pupilar, la sincronización de los ritmos diarios de alimentación, la síntesis de melatonina y la expresión génica en animales expuestos a estimulación lumínica de distintas intensidades y longitudes de onda. Estos estudios permitirán construir el espectro de acción de la respuesta a la luz en los pollos ciegos a fin de identificar el/los fotopigmentos intervinientes en este fenómeno. Este proyecto profundizará el conocimiento sobre la capacidad fotorreceptora-no visual de la retina interna y particularmente de las CGRs, de la naturaleza de la cascada bioquímica que opera en las mismas y de los mecanismos de regeneración del cromóforo utilizado.

Review of semiochemicals that mediate the oviposition of mosquitoes: a possible sustainable tool for the control and monitoring of Culicidae

The choice for suitable places for female mosquitoes to lay eggs is a key-factor for the survival of immature stages (eggs and larvae). This knowledge stands out in importance concerning the control of disease vectors. The selection of a place for oviposition requires a set of chemical, visual, olfactory and tactile cues that interact with the female before laying eggs, helping the localization of adequate sites for oviposition. The present paper presents a bibliographic revision on the main aspects of semiochemicals in regard to mosquitoes' oviposition, aiding the comprehension of their mechanisms and estimation of their potential as a tool for the monitoring and control of the Culicidae.

Postural control and automaticity in dyslexic children: The relationship between visual information and body sway

Difficulty with literacy acquisition is only one of the symptoms of developmental dyslexia. Dyslexic children also show poor motor coordination and postural control. Those problems could be associated with automaticity, i.e., difficulty in performing a task without dispending a fair amount of conscious efforts. If this is the case, dyslexic children would show difficulties in using "unperceived" sensory cues to control body sway. Therefore, the aim of the study was to examine postural control performance and the coupling between visual information and body sway in dyslexic children. Ten dyslexic children and 10 non-dyslexic children stood upright inside a moving room that remained stationary or oscillated back and forward at frequencies of 0.2 or 0.5 Hz. Body sway magnitude and the relationship between the room's movement and body sway were examined. The results indicated that dyslexic children oscillated more than non-dyslexic children in both stationary and oscillating conditions. Visual manipulation induced body sway in all children but the coupling between visual information and body sway was weaker and more variable in dyslexic children. Based upon these results, we can suggest that dyslexic children use visual information to postural control with the same underlying processes as non-dyslexic children; however, dyslexic children show poorer performance and more variability while relating visual information and motor action even in a task that does not require an active cognitive and conscious motor involvement, which may be a further evidence of automaticity problem. (C) 2011 Elsevier Ltd. All rights reserved.

Physical activity and postural control in the elderly: Coupling between visual information and body sway

Background: Aging is characterized by a decline in the postural control performance, which is based on a coherent and stable coupling between sensory information and motor action. Therefore, changes in postural control in elderlies can be related to changes in this coupling. In addition, it has been observed that physical activity seems to improve postural control performance in elderlies. These improvements can be due to changes in the coupling between sensory information and motor action related to postural control. Objective: the purpose of this study was to verify the coupling between visual information and body sway in active and sedentary elderlies. Methods: Sixteen sedentary elderlies ( SE), 16 active elderlies ( AE) and 16 young adults ( YA) were asked to stand upright inside a moving room in two experimental conditions: ( 1) discrete movement and ( 2) continuous movement of the room. Results: In the continuous condition, the results showed that the coupling between the movement of the room and body sway was stronger and more stable for SE and AE compared to YA. In the discrete condition, SE showed larger body displacement compared to AE and YA. Conclusions: SE have more difficulty to discriminate and to integrate sensory information than AE and YA indicating that physical activity may improve sensory integration. Copyright (C) 2005 S. Karger AG, Basel.

Age-related differences in postural control: effects of the complexity of visual manipulation and sensorimotor contribution to postural performance

Fundação de Amparo à Pesquisa do Estado de São Paulo (FAPESP)

Control framework for dexterous manipulation using dynamic visual servoing and tactile sensors’ feedback

Tactile sensors play an important role in robotics manipulation to perform dexterous and complex tasks. This paper presents a novel control framework to perform dexterous manipulation with multi-fingered robotic hands using feedback data from tactile and visual sensors. This control framework permits the definition of new visual controllers which allow the path tracking of the object motion taking into account both the dynamics model of the robot hand and the grasping force of the fingertips under a hybrid control scheme. In addition, the proposed general method employs optimal control to obtain the desired behaviour in the joint space of the fingers based on an indicated cost function which determines how the control effort is distributed over the joints of the robotic hand. Finally, authors show experimental verifications on a real robotic manipulation system for some of the controllers derived from the control framework.

Inclusion of immersive virtual learning environments and visual control systems to support the learning of students with Asperger syndrome

This paper presents the use of immersive virtual reality systems in the educational intervention with Asperger students. The starting points of this study are features of these students' cognitive style that requires an explicit teaching style supported by visual aids and highly structured environments. The proposed immersive virtual reality system, not only to assess the student's behavior and progress, but also is able to adapt itself to the student's specific needs. Additionally, the immersive reality system is equipped with sensors that can determine certain behaviors of the students. This paper determines the possible inclusion of immersive virtual reality as a support tool and learning strategy in these particular students' intervention. With this objective two task protocols have been defined with which the behavior and interaction situations performed by participant students are recorded. The conclusions from this study talks in favor of the inclusion of these virtual immersive environments as a support tool in the educational intervention of Asperger syndrome students as their social competences and executive functions have improved.

Plane-to-plane positioning from image-based visual servoing and structured light

In this paper we face the problem of positioning a camera attached to the end-effector of a robotic manipulator so that it gets parallel to a planar object. Such problem has been treated for a long time in visual servoing. Our approach is based on linking to the camera several laser pointers so that its configuration is aimed to produce a suitable set of visual features. The aim of using structured light is not only for easing the image processing and to allow low-textured objects to be treated, but also for producing a control scheme with nice properties like decoupling, stability, well conditioning and good camera trajectory

Environomics - A Financial Estimate of Environmental Pollution Control and Abatement Schemes in Eloor-Edayar Industrial Belt

The present study consists of nine chapters including the introductory chapter. Chapter II makes a brief review of environmental literature and examines various measures adopted at the global level to protect the environment. The environmental problems often transgress national sovereignity and geographical boundaries. Therefore, attempts must be made at the national and international levels to protect the environment, the resources of which are the common property of mankind. The protection of the national environment from the ancient till the present forms the content of Chapter III. These chapters together provide a background to understand the issues analysed in the subsequent chapters. Carefully worked out theoretical framework is a pre-requisite for the successful study of a complex subject. Some of the theoretical issues of ‘environomics’ are examined in Chapter IV. The theoretical issues involved in estimating the costs and benefits of environmental protection constitute the theme of Chapter V. The state of environment in Eloor-Edayar Industrial belt andthe impact analysis of pollution of the area are discussed in Chapter VI and VII respectively. Chapter VIII makes the financial estimate of environmental protection of the project And finally, Chapter IX presents the findings of the study

Plane-to-plane positioning from image-based visual servoing and structured light

In this paper we face the problem of positioning a camera attached to the end-effector of a robotic manipulator so that it gets parallel to a planar object. Such problem has been treated for a long time in visual servoing. Our approach is based on linking to the camera several laser pointers so that its configuration is aimed to produce a suitable set of visual features. The aim of using structured light is not only for easing the image processing and to allow low-textured objects to be treated, but also for producing a control scheme with nice properties like decoupling, stability, well conditioning and good camera trajectory

A novel technique for differentiation of proteins in the development of acid gel structure from control and heat treated milk using confocal scanning laser microscopy

The incorporation of caseins and whey proteins into acid gels produced from unheated and heat treated skimmed milk was studied by confocal scanning laser microscopy (CSLM) using fluorescent labelled proteins. Bovine casein micelles were labelled using Alexa Fluor 594, while whey proteins were labelled using Alexa Fluor 488. Samples of the labelled protein solutions were introduced into aliquots of pasteurised skim milk, and skim milk heated to 90 degrees C for 2 min and 95 degrees C for 8 min. The milk was acidified at 40 degrees C to a final pH of 4.4 using 20 g gluconodelta-lactone/l (GDL). The formation of gels was observed with CSLM at two wavelengths (488 nm and 594 nm), and also by visual and rheological methods. In the control milk, as pH decreased distinct casein aggregates appeared, and as further pH reduction occurred, the whey proteins could be seen to coat the casein aggregates. With the heated milks, the gel structure was formed of continuous strands consisting of both casein and whey protein. The formation of the gel network was correlated with an increase in the elastic modulus for all three treatments, in relation to the severity of heat treatment. This model system allows the separate observation of the caseins and whey proteins, and the study of the interactions between the two protein fractions during the formation of the acid gel structure, on a real-time basis. The system could therefore be a valuable tool in the study of structure formation in yoghurt and other dairy protein systems.

Controlling steering and judging heading: retinal flow, visual direction, and extraretinal information

The contribution of retinal flow (RF), extraretinal (ER), and egocentric visual direction (VD) information in locomotor control was explored. First, the recovery of heading from RF was examined when ER information was manipulated; results confirmed that ER signals affect heading judgments. Then the task was translated to steering curved paths, and the availability and veracity of VD were manipulated with either degraded or systematically biased RE Large steering errors resulted from selective manipulation of RF and VD, providing strong evidence for the combination of RF, ER, and VD. The relative weighting applied to RF and VD was estimated. A point-attractor model is proposed that combines redundant sources of information for robust locomotor control with flexible trajectory planning through active gaze.

Neural systems in the visual control of steering

Visual control of locomotion is essential for most mammals and requires coordination between perceptual processes and action systems. Previous research on the neural systems engaged by self-motion has focused on heading perception, which is only one perceptual subcomponent. For effective steering, it is necessary to perceive an appropriate future path and then bring about the required change to heading. Using function magnetic resonance imaging in humans, we reveal a role for the parietal eye fields (PEFs) in directing spatially selective processes relating to future path information. A parietal area close to PEFs appears to be specialized for processing the future path information itself. Furthermore, a separate parietal area responds to visual position error signals, which occur when steering adjustments are imprecise. A network of three areas, the cerebellum, the supplementary eye fields, and dorsal premotor cortex, was found to be involved in generating appropriate motor responses for steering adjustments. This may reflect the demands of integrating visual inputs with the output response for the control device.