Designing a Computer Vision System for Underwater Robotic Interventions

This thesis deals with the challenging problem of designing systems able to perceive objects in underwater environments. In the last few decades research activities in robotics have advanced the state of art regarding intervention capabilities of autonomous systems. State of art in fields such as localization and navigation, real time perception and cognition, safe action and manipulation capabilities, applied to ground environments (both indoor and outdoor) has now reached such a readiness level that it allows high level autonomous operations. On the opposite side, the underwater environment remains a very difficult one for autonomous robots. Water influences the mechanical and electrical design of systems, interferes with sensors by limiting their capabilities, heavily impacts on data transmissions, and generally requires systems with low power consumption in order to enable reasonable mission duration. Interest in underwater applications is driven by needs of exploring and intervening in environments in which human capabilities are very limited. Nowadays, most underwater field operations are carried out by manned or remotely operated vehicles, deployed for explorations and limited intervention missions. Manned vehicles, directly on-board controlled, expose human operators to risks related to the stay in field of the mission, within a hostile environment. Remotely Operated Vehicles (ROV) currently represent the most advanced technology for underwater intervention services available on the market. These vehicles can be remotely operated for long time but they need support from an oceanographic vessel with multiple teams of highly specialized pilots. Vehicles equipped with multiple state-of-art sensors and capable to autonomously plan missions have been deployed in the last ten years and exploited as observers for underwater fauna, seabed, ship wrecks, and so on. On the other hand, underwater operations like object recovery and equipment maintenance are still challenging tasks to be conducted without human supervision since they require object perception and localization with much higher accuracy and robustness, to a degree seldom available in Autonomous Underwater Vehicles (AUV). This thesis reports the study, from design to deployment and evaluation, of a general purpose and configurable platform dedicated to stereo-vision perception in underwater environments. Several aspects related to the peculiar environment characteristics have been taken into account during all stages of system design and evaluation: depth of operation and light conditions, together with water turbidity and external weather, heavily impact on perception capabilities. The vision platform proposed in this work is a modular system comprising off-the-shelf components for both the imaging sensors and the computational unit, linked by a high performance ethernet network bus. The adopted design philosophy aims at achieving high flexibility in terms of feasible perception applications, that should not be as limited as in case of a special-purpose and dedicated hardware. Flexibility is required by the variability of underwater environments, with water conditions ranging from clear to turbid, light backscattering varying with daylight and depth, strong color distortion, and other environmental factors. Furthermore, the proposed modular design ensures an easier maintenance and update of the system over time. Performance of the proposed system, in terms of perception capabilities, has been evaluated in several underwater contexts taking advantage of the opportunity offered by the MARIS national project. Design issues like energy power consumption, heat dissipation and network capabilities have been evaluated in different scenarios. Finally, real-world experiments, conducted in multiple and variable underwater contexts, including open sea waters, have led to the collection of several datasets that have been publicly released to the scientific community. The vision system has been integrated in a state of the art AUV equipped with a robotic arm and gripper, and has been exploited in the robot control loop to successfully perform underwater grasping operations.

The application of robotics to the turbine blade encapsulation process

A survey of the existing state-of-the-art of turbine blade manufacture highlights two operations that have not been automated namely that of loading of a turbine blade into an encapsulation die, and that of removing a machined blade from the encapsulation block. The automation of blade decapsulation has not been pursued. In order to develop a system to automate the loading of an encapsulation die a prototype mechanical handling robot has been designed together with a computer controlled encapsulation die. The robot has been designed as a mechanical handling robot of cylindrical geometry, suitable for use in a circular work cell. It is the prototype for a production model to be called `The Cybermate'. The prototype robot is mechanically complete but due to unforeseen circumstances the robot control system is not available (the development of the control system did not form a part of this project), hence it has not been possible to fully test and assess the robot mechanical design. Robot loading of the encapsulation die has thus been simulated. The research work with regard to the encapsulation die has focused on the development of computer controlled, hydraulically actuated, location pins. Such pins compensate for the inherent positional inaccuracy of the loading robot and reproduce the dexterity of the human operator. Each pin comprises a miniature hydraulic cylinder, controlled by a standard bidirectional flow control valve. The precision positional control is obtained through pulsing of the valves under software control, with positional feedback from an 8-bit transducer. A test-rig comprising one hydraulic location pin together with an opposing spring loaded pin has demonstrated that such a pin arrangement can be controlled with a repeatability of +/-.00045'. In addition this test-rig has demonstrated that such a pin arrangement can be used to gauge and compensate for the dimensional error of the component held between the pins, by offsetting the pin datum positions to allow for the component error. A gauging repeatability of +/- 0.00015' was demonstrated. This work has led to the design and manufacture of an encapsulation die comprising ten such pins and the associated computer software. All aspects of the control software except blade gauging and positional data storage have been demonstrated. Work is now required to achieve the accuracy of control demonstrated by the single pin test-rig, with each of the ten pins in the encapsulation die. This would allow trials of the complete loading cycle to take place.

Uma interface para o controle de robôs móveis por intermédio de gestos

The main objective of this work was to enable the recognition of human gestures through the development of a computer program. The program created captures the gestures executed by the user through a camera attached to the computer and sends it to the robot command referring to the gesture. They were interpreted in total ve gestures made by human hand. The software (developed in C ++) widely used the computer vision concepts and open source library OpenCV that directly impact the overall e ciency of the control of mobile robots. The computer vision concepts take into account the use of lters to smooth/blur the image noise reduction, color space to better suit the developer's desktop as well as useful information for manipulating digital images. The OpenCV library was essential in creating the project because it was possible to use various functions/procedures for complete control lters, image borders, image area, the geometric center of borders, exchange of color spaces, convex hull and convexity defect, plus all the necessary means for the characterization of imaged features. During the development of the software was the appearance of several problems, as false positives (noise), underperforming the insertion of various lters with sizes oversized masks, as well as problems arising from the choice of color space for processing human skin tones. However, after the development of seven versions of the control software, it was possible to minimize the occurrence of false positives due to a better use of lters combined with a well-dimensioned mask size (tested at run time) all associated with a programming logic that has been perfected over the construction of the seven versions. After all the development is managed software that met the established requirements. After the completion of the control software, it was observed that the overall e ectiveness of the various programs, highlighting in particular the V programs: 84.75 %, with VI: 93.00 % and VII with: 94.67 % showed that the nal program performed well in interpreting gestures, proving that it was possible the mobile robot control through human gestures without the need for external accessories to give it a better mobility and cost savings for maintain such a system. The great merit of the program was to assist capacity in demystifying the man set/machine therefore uses an easy and intuitive interface for control of mobile robots. Another important feature observed is that to control the mobile robot is not necessary to be close to the same, as to control the equipment is necessary to receive only the address that the Robotino passes to the program via network or Wi-Fi.

Haptic control methodologies for telerobotic stair traversal

Teleoperated mobile robots provide the ability for a human operator to safely explore and evaluate hazardous environments. This ability represents an important progression towards the preservation of human safety in the inevitable response to situations such as terrorist activities and urban search and rescue. The benefits of removing physical human presence from such environments are obvious, however challenges inhibiting task performance when remotely operating a mobile robotic system need to be addressed. The removal of physical human presence from the target environment introduces telepresence as a vital consideration in achieving the desired objective. Introducing haptic human-robotic interaction represents one approach towards improving operator performance in such a scenario. Teleoperative stair traversal proves to be a challenging task when undertaking threat response in an urban environment. This article investigates the teleoperation of an articulated track mobile robot designed for traversing stairs in a threat response scenario. Utilising a haptic medium for bilateral human-robotic interaction, the haptic cone methodology is introduced with the aim of providing the operator with a vision-independent, intuitive indication of the current commanded robot velocity. The haptic cone methodology operates synergistically with the introduced fuzzy-haptic augmentation for improving teleoperator performance in the stair traversal scenario.

A Novel Approach to Haptic Tele-operation of Aerial Robot Vehicles

We present a novel, simple and effective approach for tele-operation of aerial robotic vehicles with haptic feedback. Such feedback provides the remote pilot with an intuitive feel of the robot’s state and perceived local environment that will ensure simple and safe operation in cluttered 3D environments common in inspection and surveillance tasks. Our approach is based on energetic considerations and uses the concepts of network theory and port-Hamiltonian systems. We provide a general framework for addressing problems such as mapping the limited stroke of a ‘master’ joystick to the infinite stroke of a ‘slave’ vehicle, while preserving passivity of the closed-loop system in the face of potential time delays in communications links and limited sensor data

Minimum jerk trajectory control for rehabilitation and haptic applications

Smooth trajectories are essential for safe interaction in between human and a haptic interface. Different methods and strategies have been introduced to create such smooth trajectories. This paper studies the creation of human-like movements in haptic interfaces, based on the study of human arm motion. These motions are intended to retrain the upper limb movements of patients that lose manipulation functions following stroke. We present a model that uses higher degree polynomials to define a trajectory and control the robot arm to achieve minimum jerk movements. It also studies different methods that can be driven from polynomials to create more realistic human-like movements for therapeutic purposes.

Haptic control of multi-axis robotic systems

Control of tele-operated remote robot’s is nothing new; the public was introduced to this 'new' field in 1986 when the Chernobyl cleanup began. Pictures of weird and wonderful robotic workers pouring concrete or moving rubble flooded the world. Integration of force feedback or 'haptics' to remote robot's is a new development and one that is likely to make a big difference in man-machine interaction. Development of haptic capable tele-operation schema is a challenge. Often platform specific software is developed for one off tasks. This research focussed on the development of an open software platform for haptic control of multiple remote robotic platforms. The software utilises efficient server/client architecture for low data latency, while efficiently performing required kinematic transforms and data manipulation in real time. A description of the algorithm, software interface and hardware is presented in this paper. Preliminary results are encouraging as haptic control has been shown to greatly enhances remote positioning tasks.

3D virtual haptic cone for intuitive vehicle motion control

Haptic technology provides the ability for a system to recreate the sense of touch to a human operator, and as such offers wide reaching advantages. The ability to interact with the human's tactual modality introduces haptic human-machine interaction to replace or augment existing mediums such as visual and audible information. A distinct advantage of haptic human-machine interaction is the intrinsic bilateral nature, where information can be communicated in both directions simultaneously. This paper investigates the bilateral nature of the haptic interface in controlling the motion of a remote (or virtual) vehicle and presents the ability to provide an additional dimension of haptic information to the user over existing approaches [1-4]. The 3D virtual haptic cone offers the ability to not only provide the user with relevant haptic augmentation pertaining to the task at hand, as do existing approaches, however, to also simultaneously provide an intuitive indication of the current velocities being commanded.

Haptic control methodologies for rover teleoperation in non-deterministic remote environments

Haptic teleoperation allows human operators to interact with a remote mobile robot using their haptic sensory modality. This research introduces new haptic control methodologies allowing the teleoperator to overcome the limitations of existing techniques, ultimately facilitating improved mobile robotic control for the exploration of hazardous and remote environments.

Human-robot interaction for telemanipulation in large workspaces

Independientemente de la existencia de técnicas altamente sofisticadas y capacidades de cómputo cada vez más elevadas, los problemas asociados a los robots que interactúan con entornos no estructurados siguen siendo un desafío abierto en robótica. A pesar de los grandes avances de los sistemas robóticos autónomos, hay algunas situaciones en las que una persona en el bucle sigue siendo necesaria. Ejemplos de esto son, tareas en entornos de fusión nuclear, misiones espaciales, operaciones submarinas y cirugía robótica. Esta necesidad se debe a que las tecnologías actuales no pueden realizar de forma fiable y autónoma cualquier tipo de tarea. Esta tesis presenta métodos para la teleoperación de robots abarcando distintos niveles de abstracción que van desde el control supervisado, en el que un operador da instrucciones de alto nivel en la forma de acciones, hasta el control bilateral, donde los comandos toman la forma de señales de control de bajo nivel. En primer lugar, se presenta un enfoque para llevar a cabo la teleoperación supervisada de robots humanoides. El objetivo es controlar robots terrestres capaces de ejecutar tareas complejas en entornos de búsqueda y rescate utilizando enlaces de comunicación limitados. Esta propuesta incorpora comportamientos autónomos que el operador puede utilizar para realizar tareas de navegación y manipulación mientras se permite cubrir grandes áreas de entornos remotos diseñados para el acceso de personas. Los resultados experimentales demuestran la eficacia de los métodos propuestos. En segundo lugar, se investiga el uso de dispositivos rentables para telemanipulación guiada. Se presenta una aplicación que involucra un robot humanoide bimanual y un traje de captura de movimiento basado en sensores inerciales. En esta aplicación, se estudian las capacidades de adaptación introducidas por el factor humano y cómo estas pueden compensar la falta de sistemas robóticos de alta precisión. Este trabajo es el resultado de una colaboración entre investigadores del Biorobotics Laboratory de la Universidad de Harvard y el Centro de Automática y Robótica UPM-CSIC. En tercer lugar, se presenta un nuevo controlador háptico que combina velocidad y posición. Este controlador bilateral híbrido hace frente a los problemas relacionados con la teleoperación de un robot esclavo con un gran espacio de trabajo usando un dispositivo háptico pequeño como maestro. Se pueden cubrir amplias áreas de trabajo al cambiar automáticamente entre los modos de control de velocidad y posición. Este controlador háptico es ideal para sistemas maestro-esclavo con cinemáticas diferentes, donde los comandos se transmiten en el espacio de la tarea del entorno remoto. El método es validado para realizar telemanipulación hábil de objetos con un robot industrial. Por último, se introducen dos contribuciones en el campo de la manipulación robótica. Por un lado, se presenta un nuevo algoritmo de cinemática inversa, llamado método iterativo de desacoplamiento cinemático. Este método se ha desarrollado para resolver el problema cinemático inverso de un tipo de robot de seis grados de libertad donde una solución cerrada no está disponible. La eficacia del método se compara con métodos numéricos convencionales. Además, se ha diseñado una taxonomía robusta de agarres que permite controlar diferentes manos robóticas utilizando una correspondencia, basada en gestos, entre los espacios de trabajo de la mano humana y de la mano robótica. El gesto de la mano humana se identifica mediante la lectura de los movimientos relativos del índice, el pulgar y el dedo medio del usuario durante las primeras etapas del agarre. ABSTRACT Regardless of the availability of highly sophisticated techniques and ever increasing computing capabilities, the problems associated with robots interacting with unstructured environments remains an open challenge. Despite great advances in autonomous robotics, there are some situations where a humanin- the-loop is still required, such as, nuclear, space, subsea and robotic surgery operations. This is because the current technologies cannot reliably perform all kinds of task autonomously. This thesis presents methods for robot teleoperation strategies at different levels of abstraction ranging from supervisory control, where the operator gives high-level task actions, to bilateral teleoperation, where the commands take the form of low-level control inputs. These strategies contribute to improve the current human-robot interfaces specially in the case of slave robots deployed at large workspaces. First, an approach to perform supervisory teleoperation of humanoid robots is presented. The goal is to control ground robots capable of executing complex tasks in disaster relief environments under constrained communication links. This proposal incorporates autonomous behaviors that the operator can use to perform navigation and manipulation tasks which allow covering large human engineered areas of the remote environment. The experimental results demonstrate the efficiency of the proposed methods. Second, the use of cost-effective devices for guided telemanipulation is investigated. A case study involving a bimanual humanoid robot and an Inertial Measurement Unit (IMU) Motion Capture (MoCap) suit is introduced. Herein, it is corroborated how the adaptation capabilities offered by the human-in-the-loop factor can compensate for the lack of high-precision robotic systems. This work is the result of collaboration between researchers from the Harvard Biorobotics Laboratory and the Centre for Automation and Robotics UPM-CSIC. Thirdly, a new haptic rate-position controller is presented. This hybrid bilateral controller copes with the problems related to the teleoperation of a slave robot with large workspace using a small haptic device as master. Large workspaces can be covered by automatically switching between rate and position control modes. This haptic controller is ideal to couple kinematic dissimilar master-slave systems where the commands are transmitted in the task space of the remote environment. The method is validated to perform dexterous telemanipulation of objects with a robotic manipulator. Finally, two contributions for robotic manipulation are introduced. First, a new algorithm, the Iterative Kinematic Decoupling method, is presented. It is a numeric method developed to solve the Inverse Kinematics (IK) problem of a type of six-DoF robotic arms where a close-form solution is not available. The effectiveness of this IK method is compared against conventional numerical methods. Second, a robust grasp mapping has been conceived. It allows to control a wide range of different robotic hands using a gesture based correspondence between the human hand space and the robotic hand space. The human hand gesture is identified by reading the relative movements of the index, thumb and middle fingers of the user during the early stages of grasping.

Improved joint control using a genetic algorithm for a humanoid robot

This paper describes experiments conducted in order to simultaneously tune 15 joints of a humanoid robot. Two Genetic Algorithm (GA) based tuning methods were developed and compared against a hand-tuned solution. The system was tuned in order to minimise tracking error while at the same time achieve smooth joint motion. Joint smoothness is crucial for the accurate calculation of online ZMP estimation, a prerequisite for a closedloop dynamically stable humanoid walking gait. Results in both simulation and on a real robot are presented, demonstrating the superior smoothness performance of the GA based methods.

Fuzzy associative memory for humanoid robot joint control

Traditional approaches to joint control required accurate modelling of the system dynamic of the plant in question. Fuzzy Associative Memory (FAM) control schemes allow adequate control without a model of the system to be controlled. This paper presents a FAM based joint controller implemented on a humanoid robot. An empirically tuned PI velocity control loop is augmented with this feed forward FAM, with considerable reduction in joint position error achieved online and with minimal additional computational overhead.

Biologically inspired joint control for a humanoid robot

The GuRoo is a 1.2 m tall, 23 degree of freedom humanoid constructed at the University of Queensland for research into humanoid robotics. The key challenge being addressed by the GuRoo project is the development of appropriate learning strategies for control and coordination of the robot's many joints. The development of learning strategies is seen as a way to side-step the inherent intricacy of modeling a multi-DOF biped robot. This paper outlines the approach taken to generate an appropriate control scheme for the joints of the GuRoo. The paper demonstrates the determination of local feedback control parameters using a genetic algorithm. The feedback loop is then augmented by a predictive modulator that learns a form of feed-forward control to overcome the irregular loads experienced at each joint during the gait cycle. The predictive modulator is based on the CMAC architecture. Results from tests on the GuRoo platform show that both systems provide improvements in stability and tracking of joint control.

Modelling and control of a quad-rotor robot

To date, most quad-rotor aerial robots have been based on flying toys. Although such systems can be used as prototypes, they are not sufficiently robust to serve as experimental robotics platforms. We have developed the X-4 Flyer, a quad-rotor robot using custom-built chassis and avionics with off-the-shelf motors and batteries, to be a highly reliable experimental platform. The vehicle uses tuned plant dynamics with an onboard embedded attitude controller to stabilise flight. A linear SISO controller was designed to regulate flyer attitude.

Cerebellar augmented joint control for a humanoid robot

The joints of a humanoid robot experience disturbances of markedly different magnitudes during the course of a walking gait. Consequently, simple feedback control techniques poorly track desired joint trajectories. This paper explores the addition of a control system inspired by the architecture of the cerebellum to improve system response. This system learns to compensate the changes in load that occur during a cycle of motion. The joint compensation scheme, called Trajectory Error Learning, augments the existing feedback control loop on a humanoid robot. The results from tests on the GuRoo platform show an improvement in system response for the system when augmented with the cerebellar compensator.