Computer vision onboard UAVs for civilian tasks

Computer vision is much more than a technique to sense and recover environmental information from an UAV. It should play a main role regarding UAVs’ functionality because of the big amount of information that can be extracted, its possible uses and applications, and its natural connection to human driven tasks, taking into account that vision is our main interface to world understanding. Our current research’s focus lays on the development of techniques that allow UAVs to maneuver in spaces using visual information as their main input source. This task involves the creation of techniques that allow an UAV to maneuver towards features of interest whenever a GPS signal is not reliable or sufficient, e.g. when signal dropouts occur (which usually happens in urban areas, when flying through terrestrial urban canyons or when operating on remote planetary bodies), or when tracking or inspecting visual targets—including moving ones—without knowing their exact UMT coordinates. This paper also investigates visual serving control techniques that use velocity and position of suitable image features to compute the references for flight control. This paper aims to give a global view of the main aspects related to the research field of computer vision for UAVs, clustered in four main active research lines: visual serving and control, stereo-based visual navigation, image processing algorithms for detection and tracking, and visual SLAM. Finally, the results of applying these techniques in several applications are presented and discussed: this study will encompass power line inspection, mobile target tracking, stereo distance estimation, mapping and positioning.

Unmanned Aerial Vehicles UAVs attitude, height, motion estimation and control using visual systems

This paper presents an implementation of an aircraft pose and motion estimator using visual systems as the principal sensor for controlling an Unmanned Aerial Vehicle (UAV) or as a redundant system for an Inertial Measure Unit (IMU) and gyros sensors. First, we explore the applications of the unified theory for central catadioptric cameras for attitude and heading estimation, explaining how the skyline is projected on the catadioptric image and how it is segmented and used to calculate the UAV’s attitude. Then we use appearance images to obtain a visual compass, and we calculate the relative rotation and heading of the aerial vehicle. Additionally, we show the use of a stereo system to calculate the aircraft height and to measure the UAV’s motion. Finally, we present a visual tracking system based on Fuzzy controllers working in both a UAV and a camera pan and tilt platform. Every part is tested using the UAV COLIBRI platform to validate the different approaches, which include comparison of the estimated data with the inertial values measured onboard the helicopter platform and the validation of the tracking schemes on real flights.

Aerial object following using visual fuzzy servoing

This article presents a visual servoing system to follow a 3D moving object by a Micro Unmanned Aerial Vehicle (MUAV). The presented control strategy is based only on the visual information given by an adaptive tracking method based on the colour information. A visual fuzzy system has been developed for servoing the camera situated on a rotary wing MAUV, that also considers its own dynamics. This system is focused on continuously following of an aerial moving target object, maintaining it with a fixed safe distance and centred on the image plane. The algorithm is validated on real flights on outdoors scenarios, showing the robustness of the proposed systems against winds perturbations, illumination and weather changes among others. The obtained results indicate that the proposed algorithms is suitable for complex controls task, such object following and pursuit, flying in formation, as well as their use for indoor navigation

A multi-resolution image alignment technique based on direct methods for pose estimation of aerial vehicles

In this paper, we seek to expand the use of direct methods in real-time applications by proposing a vision-based strategy for pose estimation of aerial vehicles. The vast majority of approaches make use of features to estimate motion. Conversely, the strategy we propose is based on a MR (Multi- Resolution) implementation of an image registration technique (Inverse Compositional Image Alignment ICIA) using direct methods. An on-board camera in a downwards-looking configuration, and the assumption of planar scenes, are the bases of the algorithm. The motion between frames (rotation and translation) is recovered by decomposing the frame-to-frame homography obtained by the ICIA algorithm applied to a patch that covers around the 80% of the image. When the visual estimation is required (e.g. GPS drop-out), this motion is integrated with the previous known estimation of the vehicles’ state, obtained from the on-board sensors (GPS/IMU), and the subsequent estimations are based only on the vision-based motion estimations. The proposed strategy is tested with real flight data in representative stages of a flight: cruise, landing, and take-off, being two of those stages considered critical: take-off and landing. The performance of the pose estimation strategy is analyzed by comparing it with the GPS/IMU estimations. Results show correlation between the visual estimation obtained with the MR-ICIA and the GPS/IMU data, that demonstrate that the visual estimation can be used to provide a good approximation of the vehicle’s state when it is required (e.g. GPS drop-outs). In terms of performance, the proposed strategy is able to maintain an estimation of the vehicle’s state for more than one minute, at real-time frame rates based, only on visual information.

Omnidirectional bearing-only see-and-avoid for small aerial robots

Motivated by the growing interest in unmanned aerial system’s applications in indoor and outdoor settings and the standardisation of visual sensors as vehicle payload. This work presents a collision avoidance approach based on omnidirectional cameras that does not require the estimation of range between two platforms to resolve a collision encounter. It will achieve a minimum separation between the two vehicles involved by maximising the view-angle given by the omnidirectional sensor. Only visual information is used to achieve avoidance under a bearing-only visual servoing approach. We provide theoretical problem formulation, as well as results from real flight using small quadrotors.

Rapid prototyping framework for visual control of autonomous micro aerial vehicles

Rapid prototyping environments can speed up the research of visual control algorithms. We have designed and implemented a software framework for fast prototyping of visual control algorithms for Micro Aerial Vehicles (MAV). We have applied a combination of a proxy-based network communication architecture and a custom Application Programming Interface. This allows multiple experimental configurations, like drone swarms or distributed processing of a drone's video stream. Currently, the framework supports a low-cost MAV: the Parrot AR.Drone. Real tests have been performed on this platform and the results show comparatively low figures of the extra communication delay introduced by the framework, while adding new functionalities and flexibility to the selected drone. This implementation is open-source and can be downloaded from www.vision4uav.com/?q=VC4MAV-FW

See-and-avoid quadcopter using fuzzy control optimized by cross-entropy

In this work we present an optimized fuzzy visual servoing system for obstacle avoidance using an unmanned aerial vehicle. The cross-entropy theory is used to optimise the gains of our controllers. The optimization process was made using the ROS-Gazebo 3D simulation with purposeful extensions developed for our experiments. Visual servoing is achieved through an image processing front-end that uses the Camshift algorithm to detect and track objects in the scene. Experimental flight trials using a small quadrotor were performed to validate the parameters estimated from simulation. The integration of cross- entropy methods is a straightforward way to estimate optimal gains achieving excellent results when tested in real flights.

Quadcopter see and avoid using a fuzzy controller

Unmanned Aerial Vehicles (UAVs) industry is a fast growing sector. Nowadays, the market offers numerous possibilities for off-the-shelf UAVs such as quadrotors or fixed-wings. Until UAVs demonstrate advance capabilities such as autonomous collision avoidance they will be segregated and restricted to flight in controlled environments. This work presents a visual fuzzy servoing system for obstacle avoidance using UAVs. To accomplish this task we used the visual information from the front camera. Images are processed off-board and the result send to the Fuzzy Logic controller which then send commands to modify the orientation of the aircraft. Results from flight test are presented with a commercial off-the-shelf platform.

UAS See-and-Avoid using two different approaches of Fuzzy Control

This work presents two UAS See and Avoid approaches using Fuzzy Control. We compare the performance of each controller when a Cross-Entropy method is applied to optimase the parameters for one of the controllers. Each controller receive information from an image processing front-end that detect and track targets in the environment. Visual information is then used under a visual servoing approach to perform autonomous avoidance. Experimental flight trials using a small quadrotor were performed to validate and compare the behaviour of both controllers

Cross-entropy optimization for scaling factors of a fuzzy controller : a see-and-avoid approach for unmanned aerial systems

The Cross-Entropy (CE) is an efficient method for the estimation of rare-event probabilities and combinatorial optimization. This work presents a novel approach of the CE for optimization of a Soft-Computing controller. A Fuzzy controller was designed to command an unmanned aerial system (UAS) for avoiding collision task. The only sensor used to accomplish this task was a forward camera. The CE is used to reach a near-optimal controller by modifying the scaling factors of the controller inputs. The optimization was realized using the ROS-Gazebo simulation system. In order to evaluate the optimization a big amount of tests were carried out with a real quadcopter.

MAVwork : a framework for unified interfacing between micro aerial vehicles and visual controllers

Debugging control software for Micro Aerial Vehicles (MAV) can be risky out of the simulator, especially with professional drones that might harm people around or result in a high bill after a crash. We have designed a framework that enables a software application to communicate with multiple MAVs from a single unified interface. In this way, visual controllers can be first tested on a low-cost harmless MAV and, after safety is guaranteed, they can be moved to the production MAV at no additional cost. The framework is based on a distributed architecture over a network. This allows multiple configurations, like drone swarms or parallel processing of drones' video streams. Live tests have been performed and the results show comparatively low additional communication delays, while adding new functionalities and flexibility. This implementation is open-source and can be downloaded from github.com/uavster/mavwork

Vision based control for micro aerial vehicles : Application to sense and avoid

This work presents a collision avoidance approach based on omnidirectional cameras that does not require the estimation of range between two platforms to resolve a collision encounter. Our method achieves minimum separation between the two vehicles involved by maximising the view-angle given by the omnidirectional sensor. Only visual information is used to achieve avoidance under a bearing- only visual servoing approach. We provide theoretical problem formulation, as well as results from real flights using small quadrotors

Predicting invasion in grassland ecosystems: is exotic dominance the real embarrassment of richness?

Invasions have increased the size of regional species pools, but are typically assumed to reduce native diversity. However, global-scale tests of this assumption have been elusive because of the focus on exotic species richness, rather than relative abundance. This is problematic because low invader richness can indicate invasion resistance by the native community or, alternatively, dominance by a single exotic species. Here, we used a globally replicated study to quantify relationships between exotic richness and abundance in grass-dominated ecosystems in 13 countries on six continents, ranging from salt marshes to alpine tundra. We tested effects of human land use, native community diversity, herbivore pressure, and nutrient limitation on exotic plant dominance. Despite its widespread use, exotic richness was a poor proxy for exotic dominance at low exotic richness, because sites that contained few exotic species ranged from relatively pristine (low exotic richness and cover) to almost completely exotic-dominated ones (low exotic richness but high exotic cover). Both exotic cover and richness were predicted by native plant diversity (native grass richness) and land use (distance to cultivation). Although climate was important for predicting both exotic cover and richness, climatic factors predicting cover (precipitation variability) differed from those predicting richness (maximum temperature and mean temperature in the wettest quarter). Herbivory and nutrient limitation did not predict exotic richness or cover. Exotic dominance was greatest in areas with low native grass richness at the site- or regional-scale. Although this could reflect native grass displacement, a lack of biotic resistance is a more likely explanation, given that grasses comprise the most aggressive invaders. These findings underscore the need to move beyond richness as a surrogate for the extent of invasion, because this metric confounds monodominance with invasion resistance. Monitoring species' relative abundance will more rapidly advance our understanding of invasions.

Herbivores and nutrients control grassland plant diversity via light limitation

Human alterations to nutrient cycles1, 2 and herbivore communities3, 4, 5, 6, 7 are affecting global biodiversity dramatically2. Ecological theory predicts these changes should be strongly counteractive: nutrient addition drives plant species loss through intensified competition for light, whereas herbivores prevent competitive exclusion by increasing ground-level light, particularly in productive systems8, 9. Here we use experimental data spanning a globally relevant range of conditions to test the hypothesis that herbaceous plant species losses caused by eutrophication may be offset by increased light availability due to herbivory. This experiment, replicated in 40 grasslands on 6 continents, demonstrates that nutrients and herbivores can serve as counteracting forces to control local plant diversity through light limitation, independent of site productivity, soil nitrogen, herbivore type and climate. Nutrient addition consistently reduced local diversity through light limitation, and herbivory rescued diversity at sites where it alleviated light limitation. Thus, species loss from anthropogenic eutrophication can be ameliorated in grasslands where herbivory increases ground-level light.

Global dissemination of a multidrug resistant Escherichia coli clone

Escherichia coli sequence type 131 (ST131) is a globally disseminated, multidrug resistant (MDR) clone responsible for a high proportion of urinary tract and bloodstream infections. The rapid emergence and successful spread of E. coli ST131 is strongly associated with several factors, including resistance to fluoroquinolones, high virulence gene content, the possession of the type 1 fimbriae FimH30 allele, and the production of the CTX-M-15 extended spectrum β-lactamase (ESBL). Here, we used genome sequencing to examine the molecular epidemiology of a collection of E. coli ST131 strains isolated from six distinct geographical locations across the world spanning 2000–2011. The global phylogeny of E. coli ST131, determined from whole-genome sequence data, revealed a single lineage of E. coli ST131 distinct from other extraintestinal E. coli strains within the B2 phylogroup. Three closely related E. coli ST131 sublineages were identified, with little association to geographic origin. The majority of single-nucleotide variants associated with each of the sublineages were due to recombination in regions adjacent to mobile genetic elements (MGEs). The most prevalent sublineage of ST131 strains was characterized by fluoroquinolone resistance, and a distinct virulence factor and MGE profile. Four different variants of the CTX-M ESBL–resistance gene were identified in our ST131 strains, with acquisition of CTX-M-15 representing a defining feature of a discrete but geographically dispersed ST131 sublineage. This study confirms the global dispersal of a single E. coli ST131 clone and demonstrates the role of MGEs and recombination in the evolution of this important MDR pathogen.