Neural systems in the visual control of steering

Visual control of locomotion is essential for most mammals and requires coordination between perceptual processes and action systems. Previous research on the neural systems engaged by self-motion has focused on heading perception, which is only one perceptual subcomponent. For effective steering, it is necessary to perceive an appropriate future path and then bring about the required change to heading. Using function magnetic resonance imaging in humans, we reveal a role for the parietal eye fields (PEFs) in directing spatially selective processes relating to future path information. A parietal area close to PEFs appears to be specialized for processing the future path information itself. Furthermore, a separate parietal area responds to visual position error signals, which occur when steering adjustments are imprecise. A network of three areas, the cerebellum, the supplementary eye fields, and dorsal premotor cortex, was found to be involved in generating appropriate motor responses for steering adjustments. This may reflect the demands of integrating visual inputs with the output response for the control device.

Soft-computing based visual control for unmanned vehicles

El objetivo principal de esta Tesis es extender la utilización del “Soft- Computing” para el control de vehículos sin piloto utilizando visión. Este trabajo va más allá de los típicos sistemas de control utilizados en entornos altamente controlados, demonstrando la fuerza y versatilidad de la lógica difusa (Fuzzy Logic) para controlar vehículos aéreos y terrestres en un abanico de applicaciones diferentes. Para esta Tesis se ha realizado un gran número de pruebas reales en las cuales los controladores difusos han manejado una plataforma visual “pan-and-tilt”, un helicoptero, un coche comercial y hasta dos tipos de quadrirotores. El uso del método de optimización “Cross-Entropy” ha sido utilizado para mejorar el comportamiento de algunos de los controladores borrosos. Todos los controladores difusos presentados en ésta Tesis han sido implementados utilizando un código desarrollado por el candidato para tal efecto, llamado MOFS (Miguel Olivares’ Fuzzy Software). Diferentes algoritmos visuales han sido utilizados para adquirir la informaci´on visual del entorno, “Cmashift”, descomposición de la homografía y detección de marcas de realidad aumentada, entre otros. Dicha información visual ha sido utilizada como entrada de los controladores difusos para comandar los vehículos en las diferentes applicaciones autonomas. El volante de un vehículo comercial ha sido controlado para realizar pruebas de conducción autónoma en condiciones de tráfico similares a las de una ciudad. El sistema ha llegado a completar con éxito pruebas de más de 6 km sin ninguna interacción humana, mediante el seguimiento de una línea pintada en el suelo. El limitado campo visual del sistema no ha sido impedimento para alcanzar velocidades de hasta 48 km/h y ser guiado autonomamente en curvas de radio reducido. Objetos estáticos y móviles han sido seguidos desde un helicoptero no tripulado, mediante el control de una plataforma visual “pan-and-tilt”. ´Éste mismo helicoptero ha sido controlado completamente para su aterrizaje autonomo, mediante el control del movimiento lateral (roll), horizontal (pitch) y de altitud. El seguimiento de objetos volantes ha sido resulto mediante el control horizontal (pitch) y de orientación (heading) de un quadrirotor. Para tareas de evitación de obstáculos se ha implementado un controlador difuso para el manejo de la orientación (heading) de un quadrirotor. En el campo de la optimización de controladores se ha aportado al estado del arte una extensión del uso del método “Cross-Entropy”. Está Tesis presenta una novedosa implementación de dicho método para la optimización de las ganancias, la posición y medida de los conjuntos de las funciones de pertenecia y el peso de las reglas para mejorar el comportamiento de un controlador difuso. Dichos procesos de optimización se han realizado utilizando “ROS” y “Matlab Simulink” para obtener mejores resultados para la evitación de colisiones con vehículos aéreos no tripulados. Ésta Tesis demuestra que los controladores implementados con lógica difusa son altamente capaces de controlador sistemas sin tener en cuenta el modelo del vehículo a controlador en entornos altamente perturbables con un sensor de bajo coste como es una cámara. El ruido presentes causado por los cambios de iluminación en la adquisición de imágenes y la alta incertidumbre en la detección visual han sido manejados satisfactoriamente por ésta técnica de de “Soft-Computing” para distintas aplicaciones tanto con vehículos aéreos como terrestres.

Rapid prototyping framework for visual control of autonomous micro aerial vehicles

Rapid prototyping environments can speed up the research of visual control algorithms. We have designed and implemented a software framework for fast prototyping of visual control algorithms for Micro Aerial Vehicles (MAV). We have applied a combination of a proxy-based network communication architecture and a custom Application Programming Interface. This allows multiple experimental configurations, like drone swarms or distributed processing of a drone’s video stream. Currently, the framework supports a low-cost MAV: the Parrot AR.Drone. Real tests have been performed on this platform and the results show comparatively low figures of the extra communication delay introduced by the framework, while adding new functionalities and flexibility to the selected drone. This implementation is open-source and can be downloaded from www.vision4uav.com/?q=VC4MAV-FW

Inclusion of immersive virtual learning environments and visual control systems to support the learning of students with Asperger syndrome

This paper presents the use of immersive virtual reality systems in the educational intervention with Asperger students. The starting points of this study are features of these students' cognitive style that requires an explicit teaching style supported by visual aids and highly structured environments. The proposed immersive virtual reality system, not only to assess the student's behavior and progress, but also is able to adapt itself to the student's specific needs. Additionally, the immersive reality system is equipped with sensors that can determine certain behaviors of the students. This paper determines the possible inclusion of immersive virtual reality as a support tool and learning strategy in these particular students' intervention. With this objective two task protocols have been defined with which the behavior and interaction situations performed by participant students are recorded. The conclusions from this study talks in favor of the inclusion of these virtual immersive environments as a support tool in the educational intervention of Asperger syndrome students as their social competences and executive functions have improved.

Dynamic Visual Servoing With Chaos Control for Redundant Robots

This paper presents a new dynamic visual control system for redundant robots with chaos compensation. In order to implement the visual servoing system, a new architecture is proposed that improves the system maintainability and traceability. Furthermore, high performance is obtained as a result of parallel execution of the different tasks that compose the architecture. The control component of the architecture implements a new visual servoing technique for resolving the redundancy at the acceleration level in order to guarantee the correct motion of both end-effector and joints. The controller generates the required torques for the tracking of image trajectories. However, in order to guarantee the applicability of this technique, a repetitive path tracked by the robot-end must produce a periodic joint motion. A chaos controller is integrated in the visual servoing system and the correct performance is observed in low and high velocities. Furthermore, a method to adjust the chaos controller is proposed and validated using a real three-link robot.

Decision criteria for optimizing postemergence johnsongrass control in soybean crops in Argentina

Field studies were established in Zavalla and Oliveros, Argentina, during four years in order to optimize Johnsongrass (Sorghum halepense (L.) Pers.) chemical control by means of the thermal calendar model in comparison with other criteria (weed height or days after sowing). The effect of three application dates of postemergence herbicides was determined by visual control, density of tillers originated from rhizome bud regrowth, and from crown and shoot bud regrowth, and soybean yield. Following the thermal calendar model criterion, applications during the second date afforded the best control. Weed height for the first date showed little variability between experiments but was highly variable in the second and third application dates, achieving in some cases values greater than 120 cm. For all years, no significant differences were detected for crop yield between the first and second application dates, and yields were always lower for the third date. The greatest rhizome bud regrowth was observed for the earliest application date and the highest crown and shoot bud regrowth was determined for the last application date. Parameters associated with control efficiency showed the best behaviour for the second date. However, plant height at this moment may interfere with herbicide application and the variability exhibited by this parameter highlights the risk of determining control timing using only one decision criterion.

Eye-movements aid the control of locomotion

Eye-movements have long been considered a problem when trying to understand the visual control of locomotion. They transform the retinal image from a simple expanding pattern of moving texture elements (pure optic flow), into a complex combination of translation and rotation components (retinal flow). In this article we investigate whether there are measurable advantages to having an active free gaze, over a static gaze or tracking gaze, when steering along a winding path. We also examine patterns of free gaze behavior to determine preferred gaze strategies during active locomotion. Participants were asked to steer along a computer-simulated textured roadway with free gaze, fixed gaze, or gaze tracking the center of the roadway. Deviation of position from the center of the road was recorded along with their point of gaze. It was found that visually tracking the middle of the road produced smaller steering errors than for fixed gaze. Participants performed best at the steering task when allowed to sample naturally from the road ahead with free gaze. There was some variation in the gaze strategies used, but sampling was predominantly of areas proximal to the center of the road. These results diverge from traditional models of flow analysis.

The role of vision in Parkinson's disease locomotion control: Free walking task

Fundação de Amparo à Pesquisa do Estado de São Paulo (FAPESP)

Controle visual da freada em ciclistas: efeitos da incerteza em situação de curva

The purpose of the present study was to analyze the visual control of braking a bicycle when the cyclist is surprised by an obstacle in his way. According to Lee (1976), visually controlled braking based on time to collision information utilizes the optic variables tau and its first derivative in time, tau-dot, to initiating the braking action and regulating its intensity. Seven young adults performed a bicycle braking task in curvilinear trajectory under distinct velocity (high, medium, and low) and uncertainty (certainty and uncertainty) conditions. Results showed that, independently of velocity and uncertainty levels, participants utilized tau and tau-dot to initiating and regulating the braking action, avoiding collision with the obstacle. Cognitive, attentional, and other psychological factors resulting from both increased velocity and uncertainty were not capable of altering the use of time to collision information, corroborating the tested hypothesis

O tipo de trajetória não afeta o controle visual da freada em ciclistas

Braking visual control was studied in recreational cyclists through the manipulation of bicycle’s velocity at braking initiation (low, medium, and high) and approaching trajectory (straight and curved) with respect to a stationary obstacle. The hypothesis was that the type of trajectory, exclusively or interacting with initial velocity, would affect time to collision visual information (tau margin) and its fi rst derivative in time (tau-dot), respectively, in the onset and during braking. The results revealed that velocity affected signifi cantly tau margin while tau-dot remained unaltered independently of condition. The type of trajectory clearly did not affect the visual control of braking in cyclists.

Visual estimation of apertures for wheelchair locomotion in novices: perceptual judgment and motor practice

The aim of the present study was to determine the effects of motor practice on visual judgments of apertures for wheelchair locomotion and the visual control of wheelchair locomotion in wheelchair users who had no prior experience. Sixteen young adults, divided into motor practice and control groups, visually judged varying apertures as passable or impassable under walking, pre-practice, and post-practice conditions. The motor practice group underwent additional motor practice in 10 blocks of five trials each, moving the wheelchair through different apertures. The relative perceptual boundary was determined based on judgment data and kinematic variables that were calculated from videos of the motor practice trials. The participants overestimated the space needed under the walking condition and underestimated it under the wheelchair conditions, independent of group. The accuracy of judgments improved from the pre-practice to post-practice condition in both groups. During motor practice, the participants adaptively modulated wheelchair locomotion, adjusting it to the apertures available. The present findings from a priori visual judgments of space and the continuous judgments that are necessary for wheelchair approach and passage through apertures appear to support the dissociation between processes of perception and action.

Visual Servoing for the Robotenis System: a Strategy for a 3 DOF Parallel Robot to Hit a Ping-Pong Ball

This article describes a new visual servo control and strategies that are used to carry out dynamic tasks by the Robotenis platform. This platform is basically a parallel robot that is equipped with an acquisition and processing system of visual information, its main feature is that it has a completely open architecture control, and planned in order to design, implement, test and compare control strategies and algorithms (visual and actuated joint controllers). Following sections describe a new visual control strategy specially designed to track and intercept objects in 3D space. The results are compared with a controller shown in previous woks, where the end effector of the robot keeps a constant distance from the tracked object. In this work, the controller is specially designed in order to allow changes in the tracking reference. Changes in the tracking reference can be used to grip an object that is under movement, or as in this case, hitting a hanging Ping-Pong ball. Lyapunov stability is taken into account in the controller design.

Vision-Based Tracking, Odometry and Control for UAV Autonomy

El principal objetivo de este trabajo es proporcionar una solución en tiempo real basada en visión estéreo o monocular precisa y robusta para que un vehículo aéreo no tripulado (UAV) sea autónomo en varios tipos de aplicaciones UAV, especialmente en entornos abarrotados sin señal GPS. Este trabajo principalmente consiste en tres temas de investigación de UAV basados en técnicas de visión por computador: (I) visual tracking, proporciona soluciones efectivas para localizar visualmente objetos de interés estáticos o en movimiento durante el tiempo que dura el vuelo del UAV mediante una aproximación adaptativa online y una estrategia de múltiple resolución, de este modo superamos los problemas generados por las diferentes situaciones desafiantes, tales como cambios significativos de aspecto, iluminación del entorno variante, fondo del tracking embarullado, oclusión parcial o total de objetos, variaciones rápidas de posición y vibraciones mecánicas a bordo. La solución ha sido utilizada en aterrizajes autónomos, inspección de plataformas mar adentro o tracking de aviones en pleno vuelo para su detección y evasión; (II) odometría visual: proporciona una solución eficiente al UAV para estimar la posición con 6 grados de libertad (6D) usando únicamente la entrada de una cámara estéreo a bordo del UAV. Un método Semi-Global Blocking Matching (SGBM) eficiente basado en una estrategia grueso-a-fino ha sido implementada para una rápida y profunda estimación del plano. Además, la solución toma provecho eficazmente de la información 2D y 3D para estimar la posición 6D, resolviendo de esta manera la limitación de un punto de referencia fijo en la cámara estéreo. Una robusta aproximación volumétrica de mapping basada en el framework Octomap ha sido utilizada para reconstruir entornos cerrados y al aire libre bastante abarrotados en 3D con memoria y errores correlacionados espacialmente o temporalmente; (III) visual control, ofrece soluciones de control prácticas para la navegación de un UAV usando Fuzzy Logic Controller (FLC) con la estimación visual. Y el framework de Cross-Entropy Optimization (CEO) ha sido usado para optimizar el factor de escala y la función de pertenencia en FLC. Todas las soluciones basadas en visión en este trabajo han sido probadas en test reales. Y los conjuntos de datos de imágenes reales grabados en estos test o disponibles para la comunidad pública han sido utilizados para evaluar el rendimiento de estas soluciones basadas en visión con ground truth. Además, las soluciones de visión presentadas han sido comparadas con algoritmos de visión del estado del arte. Los test reales y los resultados de evaluación muestran que las soluciones basadas en visión proporcionadas han obtenido rendimientos en tiempo real precisos y robustos, o han alcanzado un mejor rendimiento que aquellos algoritmos del estado del arte. La estimación basada en visión ha ganado un rol muy importante en controlar un UAV típico para alcanzar autonomía en aplicaciones UAV. ABSTRACT The main objective of this dissertation is providing real-time accurate robust monocular or stereo vision-based solution for Unmanned Aerial Vehicle (UAV) to achieve the autonomy in various types of UAV applications, especially in GPS-denied dynamic cluttered environments. This dissertation mainly consists of three UAV research topics based on computer vision technique: (I) visual tracking, it supplys effective solutions to visually locate interesting static or moving object over time during UAV flight with on-line adaptivity approach and multiple-resolution strategy, thereby overcoming the problems generated by the different challenging situations, such as significant appearance change, variant surrounding illumination, cluttered tracking background, partial or full object occlusion, rapid pose variation and onboard mechanical vibration. The solutions have been utilized in autonomous landing, offshore floating platform inspection and midair aircraft tracking for sense-and-avoid; (II) visual odometry: it provides the efficient solution for UAV to estimate the 6 Degree-of-freedom (6D) pose using only the input of stereo camera onboard UAV. An efficient Semi-Global Blocking Matching (SGBM) method based on a coarse-to-fine strategy has been implemented for fast depth map estimation. In addition, the solution effectively takes advantage of both 2D and 3D information to estimate the 6D pose, thereby solving the limitation of a fixed small baseline in the stereo camera. A robust volumetric occupancy mapping approach based on the Octomap framework has been utilized to reconstruct indoor and outdoor large-scale cluttered environments in 3D with less temporally or spatially correlated measurement errors and memory; (III) visual control, it offers practical control solutions to navigate UAV using Fuzzy Logic Controller (FLC) with the visual estimation. And the Cross-Entropy Optimization (CEO) framework has been used to optimize the scaling factor and the membership function in FLC. All the vision-based solutions in this dissertation have been tested in real tests. And the real image datasets recorded from these tests or available from public community have been utilized to evaluate the performance of these vision-based solutions with ground truth. Additionally, the presented vision solutions have compared with the state-of-art visual algorithms. Real tests and evaluation results show that the provided vision-based solutions have obtained real-time accurate robust performances, or gained better performance than those state-of-art visual algorithms. The vision-based estimation has played a critically important role for controlling a typical UAV to achieve autonomy in the UAV application.

Control of Redundant Joint Structures Using Image Information During the Tracking of Non-Smooth Trajectories

Visual information is increasingly being used in a great number of applications in order to perform the guidance of joint structures. This paper proposes an image-based controller which allows the joint structure guidance when its number of degrees of freedom is greater than the required for the developed task. In this case, the controller solves the redundancy combining two different tasks: the primary task allows the correct guidance using image information, and the secondary task determines the most adequate joint structure posture solving the possible joint redundancy regarding the performed task in the image space. The method proposed to guide the joint structure also employs a smoothing Kalman filter not only to determine the moment when abrupt changes occur in the tracked trajectory, but also to estimate and compensate these changes using the proposed filter. Furthermore, a direct visual control approach is proposed which integrates the visual information provided by this smoothing Kalman filter. This last aspect permits the correct tracking when noisy measurements are obtained. All the contributions are integrated in an application which requires the tracking of the faces of Asperger children.

Augmented Reality for Airport Control Tower: Analysis and Implementation of Safety Nets

With the development of new technologies, Air Traffic Control, in the nearby of the airport, switched from a purely visual control to the use of radar, sensors and so on. As the industry is switching to the so-called Industry 4.0, also in this frame, it would be possible to implement some of the new tools that can facilitate the work of Air Traffic Controllers. The European Union proposed an innovative project to help the digitalization of the European Sky by means of the Single European Sky ATM Research (SESAR) program, which is the foundation on which the Single European Sky (SES) is based, in order to improve the already existing technologies to transform Air Traffic Management in Europe. Within this frame, the Resilient Synthetic Vision for Advanced Control Tower Air Navigation Service Provision (RETINA) project, which saw the light in 2016, studied the possibility to apply new tools within the conventional control tower to reduce the air traffic controller workload, thanks to the improvements in the augmented reality technologies. After the validation of RETINA, the Digital Technologies for Tower (DTT) project was established and the solution proposed by the University of Bologna aimed, among other things, to introduce Safety Nets in a Head-Up visualization. The aim of this thesis is to analyze the Safety Nets in use within the control tower and, by developing a working concept, implement them in a Head-Up view to be tested by Air Traffic Control Operators (ATCOs). The results, coming from the technical test, show that this concept is working and it could be leading to a future implementation in a real environment, as it improves the air traffic controller working conditions also when low visibility conditions apply.