Learning Robot Motions from Visual Sensing

Learning from demonstration becomes increasingly popular as an efficient way of robot programming. Not only a scientific interest acts as an inspiration in this case but also the possibility of producing the machines that would find application in different areas of life: robots helping with daily routine at home, high performance automata in industries or friendly toys for children. One way to teach a robot to fulfill complex tasks is to start with simple training exercises, combining them to form more difficult behavior. The objective of the Master’s thesis work was to study robot programming with visual input. Dynamic movement primitives (DMPs) were chosen as a tool for motion learning and generation. Assuming a movement to be a spring system influenced by an external force, making this system move, DMPs represent the motion as a set of non-linear differential equations. During the experiments the properties of DMP, such as temporal and spacial invariance, were examined. The effect of the DMP parameters, including spring coefficient, damping factor, temporal scaling, on the trajectory generated were studied.

3D visual sensing of human hand for remote operation of a robotic hand

New low cost sensors and open free libraries for 3D image processing are making important advances in robot vision applications possible, such as three-dimensional object recognition, semantic mapping, navigation and localization of robots, human detection and/or gesture recognition for human-machine interaction. In this paper, a novel method for recognizing and tracking the fingers of a human hand is presented. This method is based on point clouds from range images captured by a RGBD sensor. It works in real time and it does not require visual marks, camera calibration or previous knowledge of the environment. Moreover, it works successfully even when multiple objects appear in the scene or when the ambient light is changed. Furthermore, this method was designed to develop a human interface to control domestic or industrial devices, remotely. In this paper, the method was tested by operating a robotic hand. Firstly, the human hand was recognized and the fingers were detected. Secondly, the movement of the fingers was analysed and mapped to be imitated by a robotic hand.

3D Visual Data-Driven Spatiotemporal Deformations for Non-Rigid Object Grasping Using Robot Hands

Sensing techniques are important for solving problems of uncertainty inherent to intelligent grasping tasks. The main goal here is to present a visual sensing system based on range imaging technology for robot manipulation of non-rigid objects. Our proposal provides a suitable visual perception system of complex grasping tasks to support a robot controller when other sensor systems, such as tactile and force, are not able to obtain useful data relevant to the grasping manipulation task. In particular, a new visual approach based on RGBD data was implemented to help a robot controller carry out intelligent manipulation tasks with flexible objects. The proposed method supervises the interaction between the grasped object and the robot hand in order to avoid poor contact between the fingertips and an object when there is neither force nor pressure data. This new approach is also used to measure changes to the shape of an object’s surfaces and so allows us to find deformations caused by inappropriate pressure being applied by the hand’s fingers. Test was carried out for grasping tasks involving several flexible household objects with a multi-fingered robot hand working in real time. Our approach generates pulses from the deformation detection method and sends an event message to the robot controller when surface deformation is detected. In comparison with other methods, the obtained results reveal that our visual pipeline does not use deformations models of objects and materials, as well as the approach works well both planar and 3D household objects in real time. In addition, our method does not depend on the pose of the robot hand because the location of the reference system is computed from a recognition process of a pattern located place at the robot forearm. The presented experiments demonstrate that the proposed method accomplishes a good monitoring of grasping task with several objects and different grasping configurations in indoor environments.

Advanced orbital pipe welding

Since the introduction of automatic orbital welding in pipeline application in 1961, significant improvements have been obtained in orbital pipe welding systems. Requirement of more productive welding systems for pipeline application forces manufacturers to innovate new advanced systems and welding processes for orbital welding method. Various methods have been used to make welding process adaptive, such as visual sensing, passive visual sensing, real-time intelligent control, scan welding technique, multi laser vision sensor, thermal scanning, adaptive image processing, neural network model, machine vision, and optical sensing. Numerous studies are reviewed and discussed in this Master’s thesis and based on a wide range of experiments which already have been accomplished by different researches the vision sensor are reported to be the best choice for adaptive orbital pipe welding system. Also, in this study the most welding processes as well as the most pipe variations welded by orbital welding systems mainly for oil and gas pipeline applications are explained. The welding results show that Gas Metal Arc Welding (GMAW) and its variants like Surface Tension Transfer (STT) and modified short circuit are the most preferred processes in the welding of root pass and can be replaced to the Gas Tungsten Arc Welding (GTAW) in many applications. Furthermore, dual-tandem gas metal arc welding technique is currently considered the most efficient method in the welding of fill pass. Orbital GTAW process mostly is applied for applications ranging from single run welding of thin walled stainless tubes to multi run welding of thick walled pipes. Flux cored arc welding process is faster process with higher deposition rate and recently this process is getting more popular in pipe welding applications. Also, combination of gas metal arc welding and Nd:YAG laser has shown acceptable results in girth welding of land pipelines for oil and gas industry. This Master’s thesis can be implemented as a guideline in welding of pipes and tubes to achieve higher quality and efficiency. Also, this research can be used as a base material for future investigations to supplement present finding.

Development of a visual object tracking system using a compressive sensing framework for modeling the appearance

En el presente trabajo se aborda el problema del seguimiento de objetos, cuyo objetivo es encontrar la trayectoria de un objeto en una secuencia de video. Para ello, se ha desarrollado un método de seguimiento-por-detección que construye un modelo de apariencia en un dominio comprimido usando una nueva e innovadora técnica: “compressive sensing”. La única información necesaria es la situación del objeto a seguir en la primera imagen de la secuencia. El seguimiento de objetos es una aplicación típica del área de visión artificial con un desarrollo de bastantes años. Aun así, sigue siendo una tarea desafiante debido a varios factores: cambios de iluminación, oclusión parcial o total de los objetos y complejidad del fondo de la escena, los cuales deben ser considerados para conseguir un seguimiento robusto. Para lidiar lo más eficazmente posible con estos factores, hemos propuesto un algoritmo de tracking que entrena un clasificador Máquina Vector Soporte (“Support Vector Machine” o SVM en sus siglas en inglés) en modo online para separar los objetos del fondo de la escena. Con este fin, hemos generado nuestro modelo de apariencia por medio de un descriptor de características muy robusto que describe los objetos y el fondo devolviendo un vector de dimensiones muy altas. Por ello, se ha implementado seguidamente un paso para reducir la dimensionalidad de dichos vectores y así poder entrenar nuestro clasificador en un dominio mucho menor, al que denominamos domino comprimido. La reducción de la dimensionalidad de los vectores de características se basa en la teoría de “compressive sensing”, que dice que una señal con poca dispersión (pocos componentes distintos de cero) puede estar bien representada, e incluso puede ser reconstruida, a partir de un conjunto muy pequeño de muestras. La teoría de “compressive sensing” se ha aplicado satisfactoriamente en este trabajo y diferentes técnicas de medida y reconstrucción han sido probadas para evaluar nuestros vectores reducidos, de tal forma que se ha verificado que son capaces de preservar la información de los vectores originales. También incluimos una actualización del modelo de apariencia del objeto a seguir, mediante el reentrenamiento de nuestro clasificador en cada cuadro de la secuencia con muestras positivas y negativas, las cuales han sido obtenidas a partir de la posición predicha por el algoritmo de seguimiento en cada instante temporal. El algoritmo propuesto ha sido evaluado en distintas secuencias y comparado con otros algoritmos del estado del arte de seguimiento, para así demostrar el éxito de nuestro método.

Minimal computation structures for visual information applications based on printed electronics

In the early nineties, Mark Weiser wrote a series of seminal papers that introduced the concept of Ubiquitous Computing. According to Weiser, computers require too much attention from the user, drawing his focus from the tasks at hand. Instead of being the centre of attention, computers should be so natural that they would vanish into the human environment. Computers become not only truly pervasive but also effectively invisible and unobtrusive to the user. This requires not only for smaller, cheaper and low power consumption computers, but also for equally convenient display solutions that can be harmoniously integrated into our surroundings. With the advent of Printed Electronics, new ways to link the physical and the digital worlds became available. By combining common printing techniques such as inkjet printing with electro-optical functional inks, it is starting to be possible not only to mass-produce extremely thin, flexible and cost effective electronic circuits but also to introduce electronic functionalities into products where it was previously unavailable. Indeed, Printed Electronics is enabling the creation of novel sensing and display elements for interactive devices, free of form factor. At the same time, the rise in the availability and affordability of digital fabrication technologies, namely of 3D printers, to the average consumer is fostering a new industrial (digital) revolution and the democratisation of innovation. Nowadays, end-users are already able to custom design and manufacture on demand their own physical products, according to their own needs. In the future, they will be able to fabricate interactive digital devices with user-specific form and functionality from the comfort of their homes. This thesis explores how task-specific, low computation, interactive devices capable of presenting dynamic visual information can be created using Printed Electronics technologies, whilst following an approach based on the ideals behind Personal Fabrication. Focus is given on the use of printed electrochromic displays as a medium for delivering dynamic digital information. According to the architecture of the displays, several approaches are highlighted and categorised. Furthermore, a pictorial computation model based on extended cellular automata principles is used to programme dynamic simulation models into matrix-based electrochromic displays. Envisaged applications include the modelling of physical, chemical, biological, and environmental phenomena.

Web-based Implementation of Winter Maintenance Decision Support System Using GIS and Remote Sensing, May 2005

Winter maintenance, particularly snow removal and the stress of snow removal materials on public structures, is an enormous budgetary burden on municipalities and nongovernmental maintenance organizations in cold climates. Lately, geospatial technologies such as remote sensing, geographic information systems (GIS), and decision support tools are roviding a valuable tool for planning snow removal operations. A few researchers recently used geospatial technologies to develop winter maintenance tools. However, most of these winter maintenance tools, while having the potential to address some of these information needs, are not typically placed in the hands of planners and other interested stakeholders. Most tools are not constructed with a nontechnical user in mind and lack an easyto-use, easily understood interface. A major goal of this project was to implement a web-based Winter Maintenance Decision Support System (WMDSS) that enhances the capacity of stakeholders (city/county planners, resource managers, transportation personnel, citizens, and policy makers) to evaluate different procedures for managing snow removal assets optimally. This was accomplished by integrating geospatial analytical techniques (GIS and remote sensing), the existing snow removal asset management system, and webbased spatial decision support systems. The web-based system was implemented using the ESRI ArcIMS ActiveX Connector and related web technologies, such as Active Server Pages, JavaScript, HTML, and XML. The expert knowledge on snow removal procedures is gathered and integrated into the system in the form of encoded business rules using Visual Rule Studio. The system developed not only manages the resources but also provides expert advice to assist complex decision making, such as routing, optimal resource allocation, and monitoring live weather information. This system was developed in collaboration with Black Hawk County, IA, the city of Columbia, MO, and the Iowa Department of transportation. This product was also demonstrated for these agencies to improve the usability and applicability of the system.

Nonverbal social sensing in action: Unobtrusive recording and extracting of nonverbal behavior in social interactions illustrated with a research example

Nonverbal behavior coding is typically conducted by "hand". To remedy this time and resource intensive undertaking, we illustrate how nonverbal social sensing, defined as the automated recording and extracting of nonverbal behavior via ubiquitous social sensing platforms, can be achieved. More precisely, we show how and what kind of nonverbal cues can be extracted and to what extent automated extracted nonverbal cues can be validly obtained with an illustrative research example. In a job interview, the applicant's vocal and visual nonverbal immediacy behavior was automatically sensed and extracted. Results show that the applicant's nonverbal behavior can be validly extracted. Moreover, both visual and vocal applicant nonverbal behavior predict recruiter hiring decision, which is in line with previous findings on manually coded applicant nonverbal behavior. Finally, applicant average turn duration, tempo variation, and gazing best predict recruiter hiring decision. Results and implications of such a nonverbal social sensing for future research are discussed.

Efficient image mosaicing for multi-robot visual underwater mapping

Robotic platforms have advanced greatly in terms of their remote sensing capabilities, including obtaining optical information using cameras. Alongside these advances, visual mapping has become a very active research area, which facilitates the mapping of areas inaccessible to humans. This requires the efficient processing of data to increase the final mosaic quality and computational efficiency. In this paper, we propose an efficient image mosaicing algorithm for large area visual mapping in underwater environments using multiple underwater robots. Our method identifies overlapping image pairs in the trajectories carried out by the different robots during the topology estimation process, being this a cornerstone for efficiently mapping large areas of the seafloor. We present comparative results based on challenging real underwater datasets, which simulated multi-robot mapping

Integrated visual vocabulary in latent Dirichlet allocation-based scene classification for IKONOS image

Scene classification based on latent Dirichlet allocation (LDA) is a more general modeling method known as a bag of visual words, in which the construction of a visual vocabulary is a crucial quantization process to ensure success of the classification. A framework is developed using the following new aspects: Gaussian mixture clustering for the quantization process, the use of an integrated visual vocabulary (IVV), which is built as the union of all centroids obtained from the separate quantization process of each class, and the usage of some features, including edge orientation histogram, CIELab color moments, and gray-level co-occurrence matrix (GLCM). The experiments are conducted on IKONOS images with six semantic classes (tree, grassland, residential, commercial/industrial, road, and water). The results show that the use of an IVV increases the overall accuracy (OA) by 11 to 12% and 6% when it is implemented on the selected and all features, respectively. The selected features of CIELab color moments and GLCM provide a better OA than the implementation over CIELab color moment or GLCM as individuals. The latter increases the OA by only ∼2 to 3%. Moreover, the results show that the OA of LDA outperforms the OA of C4.5 and naive Bayes tree by ∼20%. © 2014 Society of Photo-Optical Instrumentation Engineers (SPIE) [DOI: 10.1117/1.JRS.8.083690]

Use of Information Visualization Methods Eliminating Cross Talk in Multiple Sensing Units Investigated for a Light-Addressable Potentiometric Sensor

The integration of nanostructured films containing biomolecules and silicon-based technologies is a promising direction for reaching miniaturized biosensors that exhibit high sensitivity and selectivity. A challenge, however, is to avoid cross talk among sensing units in an array with multiple sensors located on a small area. In this letter, we describe an array of 16 sensing units, of a light-addressable potentiometric sensor (LAPS), which was made with layer-by-Layer (LbL) films of a poly(amidomine) dendrimer (PAMAM) and single-walled carbon nanotubes (SWNTs), coated with a layer of the enzyme penicillinase. A visual inspection of the data from constant-current measurements with liquid samples containing distinct concentrations of penicillin, glucose, or a buffer indicated a possible cross talk between units that contained penicillinase and those that did not. With the use of multidimensional data projection techniques, normally employed in information Visualization methods, we managed to distinguish the results from the modified LAPS, even in cases where the units were adjacent to each other. Furthermore, the plots generated with the interactive document map (IDMAP) projection technique enabled the distinction of the different concentrations of penicillin, from 5 mmol L(-1) down to 0.5 mmol L(-1). Data visualization also confirmed the enhanced performance of the sensing units containing carbon nanotubes, consistent with the analysis of results for LAPS sensors. The use of visual analytics, as with projection methods, may be essential to handle a large amount of data generated in multiple sensor arrays to achieve high performance in miniaturized systems.

Sensoriamento remoto e geoprocessamento aplicado ao estudo temporal do uso da terra e na comparação entre classificação não supervisionada e análise visual

Pós-graduação em Agronomia (Energia na Agricultura) - FCA

Exploração e análise de dados coletados pelo sistema integrado de monitoramento ambiental utilizando técnicas de visual analytics

Pós-graduação em Ciências Cartográficas - FCT

Urban Expansion and its impact on urban agriculture - remote sensing based change analysis of Kizinga and Mzinga Valley - Dar Es Salaam, Tanzania

Urban agriculture is a phenomenon that can be observed world-wide, particularly in cities of devel- oping countries. It is contributing significantly to food security and food safety and has sustained livelihood of the urban and peri-urban low income dwe llers in developing countries for many years. Population increase due to rural-urban migration and natural - formal as well as informal - urbani- sation are competing with urban farming for available space and scarce water resources. A mul- titemporal and multisensoral urban change analysis over the period of 25 years (1982-2007) was performed in order to measure and visualise the urban expansion along the Kizinga and Mzinga valley in the south of Dar Es Salaam. Airphotos and VHR satellite data were analysed by using a combination of a composition of anisotropic textural measures and spectral information. The study revealed that unplanned built-up area is expanding continuously, and vegetation covers and agricultural lands decline at a fast rate. The validation showed that the overall classification accuracy varied depending on the database. The extracted built-up areas were used for visual in- terpretation mapping purposes and served as information source for another research project. The maps visualise an urban congestion and expansion of nearly 18% of the total analysed area that had taken place in the Kizinga valley between 1982 and 2007. The same development can be ob- served in the less developed and more remote Mzinga valley between 1981 and 2002. Both areas underwent fast changes where land prices still tend to go up and an influx of people both from rural and urban areas continuously increase the density with the consequence of increasing multiple land use interests.

Land use/cover change using Remote Sensing and Geographic Information Systems : Pic Macaya National Park, Haiti

A post classification change detection technique based on a hybrid classification approach (unsupervised and supervised) was applied to Landsat Thematic Mapper (TM), Landsat Enhanced Thematic Plus (ETM+), and ASTER images acquired in 1987, 2000 and 2004 respectively to map land use/cover changes in the Pic Macaya National Park in the southern region of Haiti. Each image was classified individually into six land use/cover classes: built-up, agriculture, herbaceous, open pine forest, mixed forest, and barren land using unsupervised ISODATA and maximum likelihood supervised classifiers with the aid of field collected ground truth data collected in the field. Ground truth information, collected in the field in December 2007, and including equalized stratified random points which were visual interpreted were used to assess the accuracy of the classification results. The overall accuracy of the land classification for each image was respectively: 1987 (82%), 2000 (82%), 2004 (87%). A post classification change detection technique was used to produce change images for 1987 to 2000, 1987 to 2004, and 2000 to 2004. It was found that significant changes in the land use/cover occurred over the 17- year period. The results showed increases in built up (from 10% to 17%) and herbaceous (from 5% to 14%) areas between 1987 and 2004. The increase of herbaceous was mostly caused by the abandonment of exhausted agriculture lands. At the same time, open pine forest and mixed forest areas lost (75%) and (83%) of their area to other land use/cover types. Open pine forest (from 20% to 14%) and mixed forest (from18 to 12%) were transformed into agriculture area or barren land. This study illustrated the continuing deforestation, land degradation and soil erosion in the region, which in turn is leading to decrease in vegetative cover. The study also showed the importance of Remote Sensing (RS) and Geographic Information System (GIS) technologies to estimate timely changes in the land use/cover, and to evaluate their causes in order to design an ecological based management plan for the park.