Métodes d'aprenentatge per reforç basats en la política i la seva aplicació a la robótica

Projecte de recerca elaborat a partir d’una estada al Robot Locomotion Group del Massachusetts Institute of Technology, Estats Units, entre març i agost del 2006. Es descriu la feina portada a terme en el camp de l'aprenentatge per reforç (RL), una metodologia molt utilitzada en aprenentatge artificial. En RL, un agent intenta maximitzar un valor escalar (càstig o premi) obtingut com a resultat de la seva interacció amb l'entorn. L'objectiu d'un sistema basat en RL és el de trobar una política d'actuació òptima que relaciona l'estat de l'entorn amb una acció determinada que maximitzi la suma de reforços futurs. El principal avantatge és que no utilitza cap base de dades conegudes, així que l'agent no rep informació sobre quina decisió triar, com succeeix en molts tipus d'aprenentatge, sinó que ha de triar per descobrir aquelles accions que tenen un valor més alt, sent molt adient en robòtica aplicada. Els principals desavantatges són uns temps de convergència sovint elevats i la manca de generalització quan tractem variables contínues. Principalment, el treball s’ha centrat en l'estudi de noves i més complexes metodologies basades en RL que combinessin dos tipus d'algorismes: els basats en funcions de valor i els representats únicament per una política d'actuació. Posteriorment s'analitzà la seva aplicabilitat en aplicacions robòtiques reals. En tots els estudis i les simulacions s’ha utilitzat un braç robòtic dissenyat i contruït al laboratori. El tipus de robot, anomenat Acrobot, és un banc de proves molt utilitzat en els camps de teoria de control i aprenentatge.

Object recognition for autonomous objects: comparison of two approaches

Report for the scientific sojourn at the Swiss Federal Institute of Technology Zurich, Switzerland, between September and December 2007. In order to make robots useful assistants for our everyday life, the ability to learn and recognize objects is of essential importance. However, object recognition in real scenes is one of the most challenging problems in computer vision, as it is necessary to deal with difficulties. Furthermore, in mobile robotics a new challenge is added to the list: computational complexity. In a dynamic world, information about the objects in the scene can become obsolete before it is ready to be used if the detection algorithm is not fast enough. Two recent object recognition techniques have achieved notable results: the constellation approach proposed by Lowe and the bag of words approach proposed by Nistér and Stewénius. The Lowe constellation approach is the one currently being used in the robot localization project of the COGNIRON project. This report is divided in two main sections. The first section is devoted to briefly review the currently used object recognition system, the Lowe approach, and bring to light the drawbacks found for object recognition in the context of indoor mobile robot navigation. Additionally the proposed improvements for the algorithm are described. In the second section the alternative bag of words method is reviewed, as well as several experiments conducted to evaluate its performance with our own object databases. Furthermore, some modifications to the original algorithm to make it suitable for object detection in unsegmented images are proposed.

On the fate of sexual traits under asexuality.

Environmental shifts and life-history changes may result in formerly adaptive traits becoming non-functional or maladaptive. In the absence of pleiotropy and other constraints, such traits may decay as a consequence of neutral mutation accumulation or selective processes, highlighting the importance of natural selection for adaptations. A suite of traits are expected to lose their adaptive function in asexual organisms derived from sexual ancestors, and the many independent transitions to asexuality allow for comparative studies of parallel trait maintenance versus decay. In addition, because certain traits, notably male-specific traits, are usually not exposed to selection under asexuality, their decay would have to occur as a consequence of drift. Selective processes could drive the decay of traits associated with costs, which may be the case for the majority of sexual traits expressed in females. We review the fate of male and female sexual traits in 93 animal lineages characterized by asexual reproduction, covering a broad taxon range including molluscs, arachnids, diplopods, crustaceans and eleven different hexapod orders. Many asexual lineages are still able occasionally to produce males. These asexually produced males are often largely or even fully functional, revealing that major developmental pathways can remain quiescent and functional over extended time periods. By contrast, for asexual females, there is a parallel and rapid decay of sexual traits, especially of traits related to mate attraction and location, as expected given the considerable costs often associated with the expression of these traits. The level of decay of female sexual traits, in addition to asexual females being unable to fertilize their eggs, would severely impede reversals to sexual reproduction, even in recently derived asexual lineages. More generally, the parallel maintenance versus decay of different trait types across diverse asexual lineages suggests that neutral traits display little or no decay even after extended periods under relaxed selection, while extensive decay for selected traits occurs extremely quickly. These patterns also highlight that adaptations can fix rapidly in natural populations of asexual organisms, in spite of their mode of reproduction.

ETIKETAGGER: herramienta para el etiquetaje de secuencias de imágenes

La adaptación del reconocimiento de objetos sobre la robótica móvil requiere un enfoque y nuevas aplicaciones que optimicen el entrenamiento de los robots para obtener resultados satisfactorios. Es conocido que el proceso de entrenamiento es largo y tedioso, donde la intervención humana es absolutamente necesaria para supervisar el comportamiento del robot y la dirección hacia los objetivos. Es por esta razón que se ha desarrollado una herramienta que reduce notablemente el esfuerzo humano que se debe hacer para esta supervisión, automatizando el proceso necesario para obtener una evaluación de resultados, y minimizando el tiempo que se malgasta debido a errores humanos o falta de infraestructuras.

Estudi de comportaments socials amb eixams de robots reconfigurables

Els eixams de robots distribuïts representen tot un món de possibilitats al camp de la microrobòtica, però existeixen pocs estudis que n'analitzin els comportaments socials i les interaccions entre robots autònoms distribuïts. Aquests comportaments han de permetre assolir de la manera més efectiva possible un bon resultat. Prenent com a base l'objectiu esmentat, aquest treball detalla diferents polítiques de cerca i de reconfiguració dels robots i estudia els seus comportaments per tal de determinar quins d'ells són més útils per solucionar un problema concret amb les plagues d'erugues i corcs als camps de cigroneres.

Robotic, fluoroscopic or EMG assisted pedicle screw insertion. A CT based comparative study

Introduction: In order to improve safety of pedicle screw placement several techniques have been developed. More recently robotically assisted pedicle insertion has been introduced aiming at increasing accuracy. The aim of this study was to compare this new technique with the two main pedicle insertion techniques in our unit namely fluoroscopically assisted vs EMG aided insertion. Material and methods: A total of 382 screws (78 thoracic,304 lumbar) were introduced in 64 patients (m/f = 1.37, equally distributed between insertion technique groups) by a single experienced spinal surgeon. From those, 64 (10 thoracic, 54 lumbar) were introduced in 11 patients using a miniature robotic device based on pre operative CT images under fluoroscopic control. 142 (4 thoracic, 138 lumbar) screws were introduced using lateral fluoroscopy in 27 patients while 176 (64 thoracic, 112 lumbar) screws in 26 patients were inserted using both fluoroscopy and EMG monitoring. There was no difference in the distribution of scoliotic spines between the 3 groups (n = 13). Screw position was assessed by an independent observer on CTs in axial, sagittal and coronal planes using the Rampersaud A to D classification. Data of lumbar and thoracic screws were processed separately as well as data obtained from axial, sagittal and coronal CT planes. Results: Intra- and interobserver reliability of the Rampersaud classification was moderate, (0.35 and 0.45 respectively) being the least good on axial plane. The total number of misplaced screws (C&D grades) was generally low (12 thoracic and 12 lumbar screws). Misplacement rates were same in straight and scoliotic spines. The only difference in misplacement rates was observed on axial and coronal images in the EMG assisted thoracic screw group with a higher proportion of C or D grades (p <0.05) in that group. Recorded compound muscle action potentials (CMAP) values of the inserted screws were 30.4 mA for the robot and 24.9mA for the freehand technique with a CI of 3.8 of the mean difference of 5.5 mA. Discussion: Robotic placement did improve the placement of thoracic screws but not that of lumbar screws possibly because our misplacement rates in general near that of published navigation series. Robotically assisted spine surgery might therefore enhance the safety of screw placement in particular in training settings were different users at various stages of their learning curve are involved in pedicle instrumentation.

Análisis de prestaciones de sistemas de comunicaciones inalámbricas en aplicaciones de robótica: sistema software

Una característica importante de la robótica es la comunicación entre sistema base y robot que puede establecerse de forma remota. Ello representa la base del proyecto que se describe a continuación, el cual se descompone de dos partes, una por cada miembro del proyecto: sistema software y sistema hardware. En el sistema software analizaremos las diferentes tecnologías inalámbricas (características, funcionamiento, seguridad, etc.), se realizará una comparativa de los diferentes módulos de comunicación y finalmente decidiremos aquellos que nos interesa para la implementación en Radiofrecuencia (RF) y Bluetooth. En este sistema también estudiaremos la interfaz gráfica que se utilizará, así como los programas creados en este entorno para realizar las implementaciones. En el sistema hardware trataremos de realizar el control de dos periféricos de forma independiente, un servomotor y un sonar, que nos servirán como ejemplo para analizar una posible comunicación entre varios robots y un PC. Por lo tanto, en este apartado analizaremos a fondo los diferentes componentes que harán posible tanto la comunicación, vía RF y Bluetooth, como el control de los diferentes dispositivos.

Análisis de prestaciones de sistemas de comunicaciones inalámbricas en aplicaciones de robótica: sistema hardware

Una característica importante de la robótica es la comunicación entre sistema base y robot que puede establecerse de forma remota. Ello representa la base del proyecto que se describe a continuación, el cual se descompone de dos partes, una por cada miembro del proyecto: Sistema software y sistema hardware. En el sistema software analizaremos las diferentes tecnologías inalámbricas (características, funcionamiento, seguridad, etc.), se realizará una comparativa de los diferentes módulos de comunicación y finalmente decidiremos aquellos que nos interesa para la implementación en Radiofrecuencia (RF) y Bluetooth. En este sistema también estudiaremos la interfaz gráfica que se utilizará, así como los programas creados en este entorno para realizar las implementaciones. En el sistema hardware trataremos de realizar el control de dos periféricos de forma independiente, un servomotor y un sonar, que nos servirán como ejemplo para analizar una posible comunicación entre varios robots y un PC. Por lo tanto, en este apartado analizaremos a fondo los diferentes componentes que harán posible tanto la comunicación, vía RF y Bluetooth, como el control de los diferentes dispositivos.

Implementación de movimientos suaves en robots industriales mediante splines cúbicos: aplicación a un laboratorio remoto de robótica

La solución a los problemas de disponibilidad horaria para la realización de sesiones prácticas por parte de los estudiantes se encuentra en los laboratorios remotos, que permiten a estos interactuar con los elementos instalados en los laboratorios sin necesidad de estar presentes físicamente. Este proyecto pretende crear un laboratorio remoto para la asignatura “Robótica y Automatización Industrial” impartida en la ETSE, UAB, en el cual los estudiantes puedan ejecutar trayectorias de tipo spline cúbico en un brazo robot y observar a través de vídeo en tiempo real los movimientos del robot desde cualquier lugar con conexión a Internet.

Estudi de comportaments socials d'aixams robòtics amb aplicació a la neteja d'espais no estructurats

La intel·ligència d’eixams és una branca de la intel·ligència artificial que està agafant molta força en els últims temps, especialment en el camp de la robòtica. En aquest projecte estudiarem el comportament social sorgit de les interaccions entre un nombre determinat de robots autònoms en el camp de la neteja de grans superfícies. Un cop triat un escenari i un robot que s’ajustin als requeriments del projecte, realitzarem una sèrie de simulacions a partir de diferents polítiques de cerca que ens permetran avaluar el comportament dels robots per unes condicions inicials de distribució dels robots i zones a netejar. A partir dels resultats obtinguts serem capaços de determinar quina configuració genera millors resultats.

Diseño y puesta en marcha de un laboratorio remoto de robótica

Este proyecto surge de la iniciativa de mejorar la calidad docente de las prácticas en la asignatura Robótica y Automatización Industrial impartida en la ETSE (Escola Tècnica Superior d’Enginyeria) de la UAB, mediante un sistema innovador. El objetivo es sustituir las actuales prácticas, basadas en la realización de simulaciones en entorno MATLAB para verificar las ecuaciones que gobiernan a los robots manipuladores, por un entorno de prácticas más atractivo consistente en un robot manipulador real, que podrá ser programado para la realización de tareas de PPO (Pick and Place Operation).

Policy gradient based Reinforcement Learning for real autonomous underwater cable tracking

This paper proposes a field application of a high-level reinforcement learning (RL) control system for solving the action selection problem of an autonomous robot in cable tracking task. The learning system is characterized by using a direct policy search method for learning the internal state/action mapping. Policy only algorithms may suffer from long convergence times when dealing with real robotics. In order to speed up the process, the learning phase has been carried out in a simulated environment and, in a second step, the policy has been transferred and tested successfully on a real robot. Future steps plan to continue the learning process on-line while on the real robot while performing the mentioned task. We demonstrate its feasibility with real experiments on the underwater robot ICTINEU AUV

Autonomous underwater vehicle control using reinforcement learning policy search methods

Autonomous underwater vehicles (AUV) represent a challenging control problem with complex, noisy, dynamics. Nowadays, not only the continuous scientific advances in underwater robotics but the increasing number of subsea missions and its complexity ask for an automatization of submarine processes. This paper proposes a high-level control system for solving the action selection problem of an autonomous robot. The system is characterized by the use of reinforcement learning direct policy search methods (RLDPS) for learning the internal state/action mapping of some behaviors. We demonstrate its feasibility with simulated experiments using the model of our underwater robot URIS in a target following task

A system to evaluate the accuracy of a visual mosaicking methodology

When underwater vehicles navigate close to the ocean floor, computer vision techniques can be applied to obtain motion estimates. A complete system to create visual mosaics of the seabed is described in this paper. Unfortunately, the accuracy of the constructed mosaic is difficult to evaluate. The use of a laboratory setup to obtain an accurate error measurement is proposed. The system consists on a robot arm carrying a downward looking camera. A pattern formed by a white background and a matrix of black dots uniformly distributed along the surveyed scene is used to find the exact image registration parameters. When the robot executes a trajectory (simulating the motion of a submersible), an image sequence is acquired by the camera. The estimated motion computed from the encoders of the robot is refined by detecting, to subpixel accuracy, the black dots of the image sequence, and computing the 2D projective transform which relates two consecutive images. The pattern is then substituted by a poster of the sea floor and the trajectory is executed again, acquiring the image sequence used to test the accuracy of the mosaicking system