3D Visual Data-Driven Spatiotemporal Deformations for Non-Rigid Object Grasping Using Robot Hands

Sensing techniques are important for solving problems of uncertainty inherent to intelligent grasping tasks. The main goal here is to present a visual sensing system based on range imaging technology for robot manipulation of non-rigid objects. Our proposal provides a suitable visual perception system of complex grasping tasks to support a robot controller when other sensor systems, such as tactile and force, are not able to obtain useful data relevant to the grasping manipulation task. In particular, a new visual approach based on RGBD data was implemented to help a robot controller carry out intelligent manipulation tasks with flexible objects. The proposed method supervises the interaction between the grasped object and the robot hand in order to avoid poor contact between the fingertips and an object when there is neither force nor pressure data. This new approach is also used to measure changes to the shape of an object’s surfaces and so allows us to find deformations caused by inappropriate pressure being applied by the hand’s fingers. Test was carried out for grasping tasks involving several flexible household objects with a multi-fingered robot hand working in real time. Our approach generates pulses from the deformation detection method and sends an event message to the robot controller when surface deformation is detected. In comparison with other methods, the obtained results reveal that our visual pipeline does not use deformations models of objects and materials, as well as the approach works well both planar and 3D household objects in real time. In addition, our method does not depend on the pose of the robot hand because the location of the reference system is computed from a recognition process of a pattern located place at the robot forearm. The presented experiments demonstrate that the proposed method accomplishes a good monitoring of grasping task with several objects and different grasping configurations in indoor environments.

Sensors and control for mining robotics

This paper discusses some of the sensing technologies and control approaches available for guiding robot manipulators for a class of underground mining tasks including drilling jumbos, bolting arms, shotcreters or explosive chargers. Data acquired with such sensors, in the laboratory and underground, is presented.

Optical micro- and nanofibres: Higher mode generation and particle manipulation studies

This thesis is centred on two experimental fields of optical micro- and nanofibre research; higher mode generation/excitation and evanescent field optical manipulation. Standard, commercial, single-mode silica fibre is used throughout most of the experiments; this generally produces high-quality, single-mode, micro- or nanofibres when tapered in a flame-heated, pulling rig in the laboratory. Single mode fibre can also support higher transverse modes, when transmitting wavelengths below that of their defined single-mode regime cut-off. To investigate this, a first-order Laguerre-Gaussian beam, LG01 of 1064 nm wavelength and doughnut-shaped intensity profile is generated free space via spatial light modulation. This technique facilitates coupling to the LP11 fibre mode in two-mode fibre, and convenient, fast switching to the fundamental mode via computer-generated hologram modulation. Following LP11 mode loss when exponentially tapering 125μm diameter fibre, two mode fibre with a cladding diameter of 80μm is selected fir testing since it is more suitable for satisfying the adiabatic criteria for fibre tapering. Proving a fruitful endeavour, experiments show a transmission of 55% of the original LP11 mode set (comprising TE01, TM01, HE21e,o true modes) in submicron fibres. Furthermore, by observing pulling dynamics and progressive mode-lass behaviour, it is possible to produce a nanofibre which supports only the TE01 and TM01 modes, while suppressing the HE21e,o elements of the LP11 group. This result provides a basis for experimental studies of atom trapping via mode-interference, and offers a new set of evanescent field geometries for sensing and particle manipulation applications. The thesis highlights the experimental results of the research unit’s Cold Atom subgroup, who successfully integrated one such higher-mode nanofibre into a cloud of cold Rubidium atoms. This led to the detection of stronger signals of resonance fluorescence coupling into the nanofibre and for light absorption by the atoms due to the presence of higher guided modes within the fibre. Theoretical work on the impact of the curved nanofibre surface on the atomic-surface van der Waals interaction is also presented, showing a clear deviation of the potential from the commonly-used flat-surface approximation. Optical micro- and nanofibres are also useful tools for evanescent-field mediated optical manipulation – this includes propulsion, defect-induced trapping, mass migration and size-sorting of micron-scale particles in dispersion. Similar early trapping experiments are described in this thesis, and resulting motivations for developing a targeted, site-specific particle induction method are given. The integration of optical nanofibres into an optical tweezers is presented, facilitating individual and group isolation of selected particles, and their controlled positioning and conveyance in the evanescent field. The effects of particle size and nanofibre diameter on pronounced scattering is experimentally investigated in this systems, as are optical binding effects between adjacent particles in the evanescent field. Such inter-particle interactions lead to regulated self-positioning and particle-chain speed enhancements.

Structure learning of graphical models for task-oriented robot grasping

In the collective imaginaries a robot is a human like machine as any androids in science fiction. However the type of robots that you will encounter most frequently are machinery that do work that is too dangerous, boring or onerous. Most of the robots in the world are of this type. They can be found in auto, medical, manufacturing and space industries. Therefore a robot is a system that contains sensors, control systems, manipulators, power supplies and software all working together to perform a task. The development and use of such a system is an active area of research and one of the main problems is the development of interaction skills with the surrounding environment, which include the ability to grasp objects. To perform this task the robot needs to sense the environment and acquire the object informations, physical attributes that may influence a grasp. Humans can solve this grasping problem easily due to their past experiences, that is why many researchers are approaching it from a machine learning perspective finding grasp of an object using information of already known objects. But humans can select the best grasp amongst a vast repertoire not only considering the physical attributes of the object to grasp but even to obtain a certain effect. This is why in our case the study in the area of robot manipulation is focused on grasping and integrating symbolic tasks with data gained through sensors. The learning model is based on Bayesian Network to encode the statistical dependencies between the data collected by the sensors and the symbolic task. This data representation has several advantages. It allows to take into account the uncertainty of the real world, allowing to deal with sensor noise, encodes notion of causality and provides an unified network for learning. Since the network is actually implemented and based on the human expert knowledge, it is very interesting to implement an automated method to learn the structure as in the future more tasks and object features can be introduced and a complex network design based only on human expert knowledge can become unreliable. Since structure learning algorithms presents some weaknesses, the goal of this thesis is to analyze real data used in the network modeled by the human expert, implement a feasible structure learning approach and compare the results with the network designed by the expert in order to possibly enhance it.

Contact Sensing: A Sequential Decision Approach to Sensing Manipulation Contact

This paper describes a new statistical, model-based approach to building a contact state observer. The observer uses measurements of the contact force and position, and prior information about the task encoded in a graph, to determine the current location of the robot in the task configuration space. Each node represents what the measurements will look like in a small region of configuration space by storing a predictive, statistical, measurement model. This approach assumes that the measurements are statistically block independent conditioned on knowledge of the model, which is a fairly good model of the actual process. Arcs in the graph represent possible transitions between models. Beam Viterbi search is used to match measurement history against possible paths through the model graph in order to estimate the most likely path for the robot. The resulting approach provides a new decision process that can be use as an observer for event driven manipulation programming. The decision procedure is significantly more robust than simple threshold decisions because the measurement history is used to make decisions. The approach can be used to enhance the capabilities of autonomous assembly machines and in quality control applications.

Robust sensing for a 3,500 tonne field robot

The mining industry is highly suitable for the application of robotics and automation technology since the work is arduous, dangerous and often repetitive. This paper discusses a robust sensing system developed to find and trade the position of the hoist ropes of a dragline. Draglines are large `walking cranes' used in open-pit coal mining to remove the material covering the coal seam. The rope sensing system developed uses two time-of-flight laser scanners. The finding algorithm uses a novel data association and tracking strategy based on pairing rope data.

Development of a robust framework for an outdoor mobile manipulation UAV

There is a growing interest to autonomously collect or manipulate objects in remote or unknown environments, such as mountains, gullies, bush-land, or rough terrain. There are several limitations of conventional methods using manned or remotely controlled aircraft. The capability of small Unmanned Aerial Vehicles (UAV) used in parallel with robotic manipulators could overcome some of these limitations. By enabling the autonomous exploration of both naturally hazardous environments, or areas which are biologically, chemically, or radioactively contaminated, it is possible to collect samples and data from such environments without directly exposing personnel to such risks. This paper covers the design, integration, and initial testing of a framework for outdoor mobile manipulation UAV. The framework is designed to allow further integration and testing of complex control theories, with the capability to operate outdoors in unknown environments. The results obtained act as a reference for the effectiveness of the integrated sensors and low-level control methods used for the preliminary testing, as well as identifying the key technologies needed for the development of an outdoor capable system.

Formal methods for control synthesis in partially observed environments : application to autonomous robotic manipulation

Modern robots are increasingly expected to function in uncertain and dynamically challenging environments, often in proximity with humans. In addition, wide scale adoption of robots requires on-the-fly adaptability of software for diverse application. These requirements strongly suggest the need to adopt formal representations of high level goals and safety specifications, especially as temporal logic formulas. This approach allows for the use of formal verification techniques for controller synthesis that can give guarantees for safety and performance. Robots operating in unstructured environments also face limited sensing capability. Correctly inferring a robot's progress toward high level goal can be challenging.

This thesis develops new algorithms for synthesizing discrete controllers in partially known environments under specifications represented as linear temporal logic (LTL) formulas. It is inspired by recent developments in finite abstraction techniques for hybrid systems and motion planning problems. The robot and its environment is assumed to have a finite abstraction as a Partially Observable Markov Decision Process (POMDP), which is a powerful model class capable of representing a wide variety of problems. However, synthesizing controllers that satisfy LTL goals over POMDPs is a challenging problem which has received only limited attention.

This thesis proposes tractable, approximate algorithms for the control synthesis problem using Finite State Controllers (FSCs). The use of FSCs to control finite POMDPs allows for the closed system to be analyzed as finite global Markov chain. The thesis explicitly shows how transient and steady state behavior of the global Markov chains can be related to two different criteria with respect to satisfaction of LTL formulas. First, the maximization of the probability of LTL satisfaction is related to an optimization problem over a parametrization of the FSC. Analytic computation of gradients are derived which allows the use of first order optimization techniques.

The second criterion encourages rapid and frequent visits to a restricted set of states over infinite executions. It is formulated as a constrained optimization problem with a discounted long term reward objective by the novel utilization of a fundamental equation for Markov chains - the Poisson equation. A new constrained policy iteration technique is proposed to solve the resulting dynamic program, which also provides a way to escape local maxima.

The algorithms proposed in the thesis are applied to the task planning and execution challenges faced during the DARPA Autonomous Robotic Manipulation - Software challenge.

Teaching from Examples in Assembly and Manipulation of Snack Food Ingredients by Robot

T. G. Williams, J.J. Rowland, and Lee M.H., Teaching from Examples in Assembly and Manipulation of Snack Food Ingredients by Robot, Proc. IEEE/RSJ Int. Conf. on Robots and Systems (IROS 2001), Nov., 2001, pp2300-2305.

Force distribution in manipulation by a robot hand with equality and inequality constraints

The problem of the appropriate distribution of forces among the fingers of a four-fingered robot hand is addressed. The finger-object interactions are modelled as point frictional contacts, hence the system is indeterminate and an optimal solution is required for controlling forces acting on an object. A fast and efficient method for computing the grasping and manipulation forces is presented, where computation has been based on using the true model of the nonlinear frictional cone of contact. Results are compared with previously employed methods of linearizing the cone constraints and minimizing the internal forces.

Situation assessment through multi-modal sensing of dynamic environments to support cognitive robot control

Awareness of emerging situations in a dynamic operational environment of a robotic assistive device is an essential capability of such a cognitive system, based on its effective and efficient assessment of the prevailing situation. This allows the system to interact with the environment in a sensible (semi)autonomous / pro-active manner without the need for frequent interventions from a supervisor. In this paper, we report a novel generic Situation Assessment Architecture for robotic systems directly assisting humans as developed in the CORBYS project. This paper presents the overall architecture for situation assessment and its application in proof-of-concept Demonstrators as developed and validated within the CORBYS project. These include a robotic human follower and a mobile gait rehabilitation robotic system. We present an overview of the structure and functionality of the Situation Assessment Architecture for robotic systems with results and observations as collected from initial validation on the two CORBYS Demonstrators.

Manipulation of the Quorum Sensing Signal AI-2 Affects the Antibiotic-Treated Gut Microbiota

The mammalian gut microbiota harbors a diverse ecosystem where hundreds of bacterial species interact with each other and their host. Given that bacteria use signals to communicate and regulate group behaviors (quorum sensing), we asked whether such communication between different commensal species can influence the interactions occurring in this environment. We engineered the enteric bacterium, Escherichia coli, to manipulate the levels of the interspecies quorum sensing signal, autoinducer-2 (AI-2), in the mouse intestine and investigated the effect upon antibiotic-induced gut microbiota dysbiosis. E. coli that increased intestinal AI-2 levels altered the composition of the antibiotic-treated gut microbiota, favoring the expansion of the Firmicutes phylum. This significantly increased the Firmicutes/Bacteroidetes ratio, to oppose the strong effect of the antibiotic, which had almost cleared the Firmicutes. This demonstrates that AI-2 levels influence the abundance of the major phyla of the gut microbiota, the balance of which is known to influence human health.

Waveguide microgripper for identification, sensing and manipulation

A Waveguide Microgripper utilizes flexible optical waveguides as gripping arms, which provide the physical means for grasping a microobject, while simultaneously enabling light to be delivered and collected. This unique capability allows extensive optical characterization of the structure being held such as transmission, reflection or fluorescence. One of the simplest capabilities of the waveguide microgripper is to be able to detect the presence of a microobject between the microgripper facets by monitoring the transmitted intensity of light coupled through the facets. The intensity of coupled light is expected to drop when there is an object obstructing the path of light. The optical sensing and characterization function of the microgripper is a strong function of the optical power incident on the structure of interest. Hence it is important to understand the factors affecting the power distribution across the facet. The microgripper is also capable of detecting the fluorescence. This capability of microgripper is expected to have applications in medical, bio-medical and related fields.

Air vehicle simulator : An application for a cable array robot

The development of autonomous air vehicles can be an expensive research pursuit. To alleviate some of the financial burden of this process, we have constructed a system consisting of four winches each attached to a central pod (the simulated air vehicle) via cables - a cable-array robot. The system is capable of precisely controlling the three dimensional position of the pod allowing effective testing of sensing and control strategies before experimentation on a free-flying vehicle. In this paper, we present a brief overview of the system and provide a practical control strategy for such a system. ©2005 IEEE.

A cable-array robot for air vehicle simulation

The development of autonomous air vehicles can be an expensive research pursuit. To alleviate some of the financial burden of this process, we have constructed a system consisting of four winches each attached to a central pod (the simulated air vehicle) via cables - a cable-array robot. The system is capable of precisely controlling the three dimensional position of the pod allowing effective testing of sensing and control strategies before experimentation on a free-flying vehicle. In this paper, we present a brief overview of the system and provide a practical control strategy for such a system.