Learning Multirobot Hose Transportation and Deployment by Distributed Round-Robin Q-Learning

Multi-Agent Reinforcement Learning (MARL) algorithms face two main difficulties: the curse of dimensionality, and environment non-stationarity due to the independent learning processes carried out by the agents concurrently. In this paper we formalize and prove the convergence of a Distributed Round Robin Q-learning (D-RR-QL) algorithm for cooperative systems. The computational complexity of this algorithm increases linearly with the number of agents. Moreover, it eliminates environment non sta tionarity by carrying a round-robin scheduling of the action selection and execution. That this learning scheme allows the implementation of Modular State-Action Vetoes (MSAV) in cooperative multi-agent systems, which speeds up learning convergence in over-constrained systems by vetoing state-action pairs which lead to undesired termination states (UTS) in the relevant state-action subspace. Each agent's local state-action value function learning is an independent process, including the MSAV policies. Coordination of locally optimal policies to obtain the global optimal joint policy is achieved by a greedy selection procedure using message passing. We show that D-RR-QL improves over state-of-the-art approaches, such as Distributed Q-Learning, Team Q-Learning and Coordinated Reinforcement Learning in a paradigmatic Linked Multi-Component Robotic System (L-MCRS) control problem: the hose transportation task. L-MCRS are over-constrained systems with many UTS induced by the interaction of the passive linking element and the active mobile robots.

Avaliação do efeito da interação solo estrutura sobre o comportamento estrutural de edificações em aço e mistas (aço concreto).

O conhecimento do tema da interação solo-estrutura permite que as edificações sejam projetadas de maneira mais realista com o comportamento físico. Há décadas atrás seria inviável um dimensionamento considerando a deformabilidade do solo de fundação, e as estruturas eram dimensionadas adotando-se as fundações como indeslocáveis, sob uma base rígida. Essa consideração conduz a respostas estruturais inadequadas, por vezes comprometendo a segurança e a estabilidade do conjunto estrutural. Atualmente, o avanço tecnológico permite a obtenção de resultados de milhões de cálculos matemáticos em questões de segundos, podendo-se obter soluções mais arrojadas e dinâmicas, facilitando o dimensionamento estrutural de forma a atender ao novo padrão de construção e arquitetura. A relevância de tal assunto motivou a análise numérica de um edifício de 4 pavimentos de estrutura mista (aço-concreto), considerando o efeito da interação solo-estrutura. As análises foram conduzidas com o programa ANSYS, de elementos finitos, substituindo-se os apoios indeslocáveis por molas discretas, lineares e elásticas, com rigidez equivalente ao solo, conforme hipótese de Winkler. Os parâmetros dos solos de fundação foram adquiridos a partir de correlações empíricas existentes na literatura e da utilização do programa computacional PLAXIS para a determinação das constantes elásticas das molas. Neste trabalho, foram comparados os resultados de reações verticais, esforços normais, momentos fletores e deslocamentos da estrutura obtidos pelo modelo clássico de projeto, que considera apoios indeslocáveis, e pelo modelo de Winkler, que considera a interação solo-estrutura. As análises foram executadas para seis diferentes tipos de solos argilosos, siltosos e arenosos. Os resultados obtidos mostraram claramente a redistribuição dos momentos fletores, esforços normais e reações verticais nos pilares com diferenças significativas para os pilares de canto e periféricos. Observou-se uma tendência de alívio dos esforços nos pilares mais solicitados, adotando a estrutura assentada em uma base rígida. As análises ressaltaram a relevância da interação solo-estrutura, com efeitos provenientes do rearranjo do solo de fundação nos elementos estruturais das edificações.

Erros pré-analíticos em medicina laboratorial: uma revisão sistemática

MENEZES, Patrick Lourenço. Erros pré-analíticos em medicina laboratorial: uma revisão sistemática. 2013. 98 f. Dissertação (Mestrado em Saúde, Medicina Laboratorial e Tecnologia Forense) - Instituto de Biologia Roberto Alcântara Gomes, Universidade do Estado do Rio de Janeiro, Rio de Janeiro, 2013. A relevância evidente dos erros pré-analíticos como problema de saúde pública fica patente tanto no dano potencial aos pacientes quanto nos custos ao sistema de saúde, ambos desnecessários e evitáveis. Alguns estudos apontam que a fase pré-analítica é a mais vulnerável a erros, sendo responsável por, aproximadamente, 60 a 90% dos erros laboratoriais em consequência da falta orientação aos pacientes sobre os procedimentos que serão realizados no laboratório clínico. Objetivos: Sistematizar as evidências científicas relacionadas aos erros pré-analíticos dos exames laboratoriais de análises clínicas. Método: Uma revisão sistemática foi realizada, buscando as bases de dados do Medical Literature Analysis and Retrieval System Online (MEDLINE), Scopus(que inclui MEDLINE e Embase), ISI Web of Knowledge, SciFinder, Literatura Latino-Americana e do Caribe em Ciências da Saúde (Lilacs) (que inclui a Scientific Electronic Library Online SciELO) e o Índice Bibliográfico Espanhol de Ciências de Saúde (IBECS), para artigos publicados entre janeiro de 1990 e junho de 2012 sobre erros de exames laboratoriais que possam ocorrer na fase pré-analítica. Os estudos foram incluídos de acordo com os seguintes exames laboratoriais: hemograma, análise bioquímica do sangue total ou do soro, exames de coagulação sanguínea,uroanálise e exames hematológicos ou bioquímicos em outros materiais e categorizados pelo tipo de erro pré-analítico e pela frequência dos incidentes. Resultados: A busca nas bases de dados bibliográficas resultou no seguinte número de artigos recuperados: 547 na MEDLINE, 229 na Scopus, 110 na ISI, 163 na SciFinder, 228 na Lilacs e 64 na IBECS, perfazendo um total de 1.341 títulos. Ao fim da revisão sistemática, obteve-se um conjunto de 83 artigos para leitura de texto completo, dos quais 14 foram incluídos na revisão. Os estudos abrangeram diferentes tipos de laboratórios, setores técnicos e origem de erros, segundo a fase do processo laboratorial. Discussão: Sete artigos demonstraram erros de pedidos médicos, com uma alta variabilidade nos valores de incidência. Os seis artigos que estudaram erros de coleta de amostra observaram redução deste desfecho. As proporções de eventos adversos relatados e os impactos clínicos variaram, levando a consequências descritas como: erros decorrentes da flebotomia, recoleta de amostras, repetições de exames, atrasos na liberação de resultados de exames e possíveis danos ao paciente. Conclusões: O laboratório deve ter instruções por escrito para cada teste, que descreva o tipo de amostra e procedimento de coleta de amostra. Meios de identificação por código de barras, sistemas robóticos e analíticos reduzem os erros pré-analíticos. A melhoria da fase pré-analítica de testes laboratoriais permanece um desafio para muitos laboratórios clínicos.

Localização colaborativa em robótica de enxame.

Diversas das possíveis aplicações da robótica de enxame demandam que cada robô seja capaz de estimar a sua posição. A informação de localização dos robôs é necessária, por exemplo, para que cada elemento do enxame possa se posicionar dentro de uma formatura de robôs pré-definida. Da mesma forma, quando os robôs atuam como sensores móveis, a informação de posição é necessária para que seja possível identificar o local dos eventos medidos. Em virtude do tamanho, custo e energia dos dispositivos, bem como limitações impostas pelo ambiente de operação, a solução mais evidente, i.e. utilizar um Sistema de Posicionamento Global (GPS), torna-se muitas vezes inviável. O método proposto neste trabalho permite que as posições absolutas de um conjunto de nós desconhecidos sejam estimadas, com base nas coordenadas de um conjunto de nós de referência e nas medidas de distância tomadas entre os nós da rede. A solução é obtida por meio de uma estratégia de processamento distribuído, onde cada nó desconhecido estima sua própria posição e ajuda os seus vizinhos a calcular as suas respectivas coordenadas. A solução conta com um novo método denominado Multi-hop Collaborative Min-Max Localization (MCMM), ora proposto com o objetivo de melhorar a qualidade da posição inicial dos nós desconhecidos em caso de falhas durante o reconhecimento dos nós de referência. O refinamento das posições é feito com base nos algoritmos de busca por retrocesso (BSA) e de otimização por enxame de partículas (PSO), cujos desempenhos são comparados. Para compor a função objetivo, é introduzido um novo método para o cálculo do fator de confiança dos nós da rede, o Fator de Confiança pela Área Min-Max (MMA-CF), o qual é comparado com o Fator de Confiança por Saltos às Referências (HTA-CF), previamente existente. Com base no método de localização proposto, foram desenvolvidos quatro algoritmos, os quais são avaliados por meio de simulações realizadas no MATLABr e experimentos conduzidos em enxames de robôs do tipo Kilobot. O desempenho dos algoritmos é avaliado em problemas com diferentes topologias, quantidades de nós e proporção de nós de referência. O desempenho dos algoritmos é também comparado com o de outros algoritmos de localização, tendo apresentado resultados 40% a 51% melhores. Os resultados das simulações e dos experimentos demonstram a eficácia do método proposto.

Impedance control reduces instability that arises from motor noise.

There is ample evidence that humans are able to control the endpoint impedance of their arms in response to active destabilizing force fields. However, such fields are uncommon in daily life. Here, we examine whether the CNS selectively controls the endpoint impedance of the arm in the absence of active force fields but in the presence of instability arising from task geometry and signal-dependent noise (SDN) in the neuromuscular system. Subjects were required to generate forces, in two orthogonal directions, onto four differently curved rigid objects simulated by a robotic manipulandum. The endpoint stiffness of the limb was estimated for each object curvature. With increasing curvature, the endpoint stiffness increased mainly parallel to the object surface and to a lesser extent in the orthogonal direction. Therefore, the orientation of the stiffness ellipses did not orient to the direction of instability. Simulations showed that the observed stiffness geometries and their pattern of change with instability are the result of a tradeoff between maximizing the mechanical stability and minimizing the destabilizing effects of SDN. Therefore, it would have been suboptimal to align the stiffness ellipse in the direction of instability. The time course of the changes in stiffness geometry suggests that modulation takes place both within and across trials. Our results show that an increase in stiffness relative to the increase in noise can be sufficient to reduce kinematic variability, thereby allowing stiffness control to improve stability in natural tasks.

Multiple grasp-specific representations of tool dynamics mediate skillful manipulation.

Skillful tool use requires knowledge of the dynamic properties of tools in order to specify the mapping between applied force and tool motion. Importantly, this mapping depends on the orientation of the tool in the hand. Here we investigate the representation of dynamics during skillful manipulation of a tool that can be grasped at different orientations. We ask whether the motor system uses a single general representation of dynamics for all grasp contexts or whether it uses multiple grasp-specific representations. Using a novel robotic interface, subjects rotated a virtual tool whose orientation relative to the hand could be varied. Subjects could immediately anticipate the force direction for each orientation of the tool based on its visual geometry, and, with experience, they learned to parameterize the force magnitude. Surprisingly, this parameterization of force magnitude showed limited generalization when the orientation of the tool changed. Had subjects parameterized a single general representation, full generalization would be expected. Thus, our results suggest that object dynamics are captured by multiple representations, each of which encodes the mapping associated with a specific grasp context. We suggest that the concept of grasp-specific representations may provide a unifying framework for interpreting previous results related to dynamics learning.

Computational fluid-structure interaction methods for simulation of inflatable aerodynamic decelerators

Inflatable aerodynamic decelerators have potential advantages for planetary re-entry in robotic and human exploration missions. It is theorized that volume-mass characteristics of these decelerators are superior to those of common supersonic/subsonic parachutes and after deployment they may suffer no instabilities at high Mach numbers. A high fidelity computational fluid-structure interaction model is employed to investigate the behavior of tension cone inflatable aeroshells at supersonic speeds up to Mach 2.0. The computational framework targets the large displacements regime encountered during the inflation of the decelerator using fast level set techniques to incorporate boundary conditions of the moving structure. The preliminary results indicate large but steady aeroshell displacement with rich dynamics, including buckling of the inflatable torus that maintains the decelerator open under normal operational conditions, owing to interactions with the turbulent wake. Copyright © 2009 by the American Institute of Aeronautics and Astronautics, Inc.

Fluid-structure interaction simulation of an inflatable aerodynamic tension-cone decelerator

Inflatable aerodynamic decelerators have potential advantages for planetary re-entry in robotic and human exploration missions. In this paper, we focus on an inflatable tension cone design that has potential advantages over other geometries. A computational fluid-structure interaction model of a tension cone is employed to investigate the behavior of the inflatable aeroshell at supersonic speeds for conditions matching recent experimental results. A parametric study is carried out to investigate the deflections of the tension cone as a function of inflation pressure of the torus at a Mach of 2.5. Comparison of the behavior of the structure, amplitude of deformations, and determined loads are reported. © 2010 by the American Institute of Aeronautics and Astronautics, Inc.

Workspace comparisons of setup configurations for human-robot interaction

In virtual assembly verification or remote maintenance tasks, bimanual haptic interfaces play a crucial role in successful task completion. This paper proposes a method for objectively comparing how well a haptic interface covers the reachable workspace of human arms. Two system configurations are analyzed for a recently introduced haptic device that is based on two DLR-KUKA light weight robots: the standard configuration, where the device is opposite the human operator, and the ergonomic configuration, where the haptic device is mounted behind the human operator. The human operator directly controls the robotic arms using handles. The analysis is performed using a representation of the robot arm workspace. The merits of restricting the comparisons to the most significant regions of the human workspace are discussed. Using this method, a greater workspace correspondence for the ergonomic configuration was shown. ©2010 IEEE.

Functional significance of stiffness in adaptation of multijoint arm movements to stable and unstable dynamics.

This study compared the mechanisms of adaptation to stable and unstable dynamics from the perspective of changes in joint mechanics. Subjects were instructed to make point to point movements in force fields generated by a robotic manipulandum which interacted with the arm in either a stable or an unstable manner. After subjects adjusted to the initial disturbing effects of the force fields they were able to produce normal straight movements to the target. In the case of the stable interaction, subjects modified the joint torques in order to appropriately compensate for the force field. No change in joint torque or endpoint force was required or observed in the case of the unstable interaction. After adaptation, the endpoint stiffness of the arm was measured by applying displacements to the hand in eight different directions midway through the movements. This was compared to the stiffness measured similarly during movements in a null force field. After adaptation, the endpoint stiffness under both the stable and unstable dynamics was modified relative to the null field. Adaptation to unstable dynamics was achieved by selective modification of endpoint stiffness in the direction of the instability. To investigate whether the change in endpoint stiffness could be accounted for by change in joint torque or endpoint force, we estimated the change in stiffness on each trial based on the change in joint torque relative to the null field. For stable dynamics the change in endpoint stiffness was accurately predicted. However, for unstable dynamics the change in endpoint stiffness could not be reproduced. In fact, the predicted endpoint stiffness was similar to that in the null force field. Thus, the change in endpoint stiffness seen after adaptation to stable dynamics was directly related to changes in net joint torque necessary to compensate for the dynamics in contrast to adaptation to unstable dynamics, where a selective change in endpoint stiffness occurred without any modification of net joint torque.

Impedance control balances stability with metabolically costly muscle activation.

Humans are able to stabilize their movements in environments with unstable dynamics by selectively modifying arm impedance independently of force and torque. We further investigated adaptation to unstable dynamics to determine whether the CNS maintains a constant overall level of stability as the instability of the environmental dynamics is varied. Subjects performed reaching movements in unstable force fields of varying strength, generated by a robotic manipulator. Although the force fields disrupted the initial movements, subjects were able to adapt to the novel dynamics and learned to produce straight trajectories. After adaptation, the endpoint stiffness of the arm was measured at the midpoint of the movement. The stiffness had been selectively modified in the direction of the instability. The stiffness in the stable direction was relatively unchanged from that measured during movements in a null force field prior to exposure to the unstable force field. This impedance modification was achieved without changes in force and torque. The overall stiffness of the arm and environment in the direction of instability was adapted to the force field strength such that it remained equivalent to that of the null force field. This suggests that the CNS attempts both to maintain a minimum level of stability and minimize energy expenditure.

Reflex contributions to the directional tuning of arm stiffness

It has been shown that during arm movement, humans selectively change the endpoint stiffness of their arm to compensate for the instability in an unstable environment. When the direction of the instability is rotated with respect to the direction of movement, it was found that humans modify the antisymmetric component of their endpoint stiffness. The antisymmetric component of stiffness arises due to reflex responses suggesting that the subjects may have tuned their reflex responses as part of the feedforward adaptive control. The goal of this study was to examine whether the CNS modulates the gain of the reflex response for selective tuning of endpoint impedance. Subjects performed reaching movements in three unstable force fields produced by a robotic manipulandum, each field differing only in the rotational component. After subjects had learned to compensate for the field, allowing them to make unperturbed movements to the target, the endpoint stiffness of the arm was estimated in the middle of the movements. At the same time electromyographic activity (EMG) of six arm muscles was recorded. Analysis of the EMG revealed differences across force fields in the reflex gain of these muscles consistent with stiffness changes. This study suggests that the CNS modulates the reflex gain as part of the adaptive feedforward command in which the endpoint impedance is selectively tuned to overcome environmental instability. © 2008 Springer-Verlag Berlin Heidelberg.

Human-humanoid interaction: Is a humanoid robot perceived as a human?

As humanoid robots become more commonplace in our society, it is important to understand the relation between humans and humanoid robots. In human face-to-face interaction, the observation of another individual performing an action facilitates the execution of a similar action, and interferes with the execution of different action. This phenomenon has been explained by the existence of shared internal representations for the execution and perception of actions, which would be automatically activated by the perception of another individual's action. In one interference experiment, null interference was reported when subjects observed a robotic arm perform the incongruent task, suggesting that this effect may be specific to interacting with other humans. This experimental paradigm, designed to investigate motor interference in human interactions, was adapted to investigate how similar the implicit perception of a humanoid robot is to a human agent. Subjects performed rhythmic arm movements while observing either a human agent or humanoid robot performing either congruent or incongruent movements. The variance of the executed movements was used as a measure of the amount of interference in the movements. Both the human and humanoid agents produced significant interference effect. These results suggest that observing the action of humanoid robot and human agent may rely on similar perceptual processes. Furthermore, the ratio of the variance in incongruent to congruent conditions varied between the human agent and humanoid robot. We speculate this ratio describes how the implicit perception of a robot is similar to that of a human, so that this paradigm could provide an objective measure of the reaction to different types of robots and be used to guide the design of humanoid robots interacting with humans. © 2004 IEEE.

Visuomotor feedback gains upregulate during the learning of novel dynamics.

At an early stage of learning novel dynamics, changes in muscle activity are mainly due to corrective feedback responses. These feedback contributions to the overall motor command are gradually reduced as feedforward control is learned. The temporary increased use of feedback could arise simply from the large errors in early learning with either unaltered gains or even slightly downregulated gains, or from an upregulation of the feedback gains when feedforward prediction is insufficient. We therefore investigated whether the sensorimotor control system alters feedback gains during adaptation to a novel force field generated by a robotic manipulandum. To probe the feedback gains throughout learning, we measured the magnitude of involuntary rapid visuomotor responses to rapid shifts in the visual location of the hand during reaching movements. We found large increases in the magnitude of the rapid visuomotor response whenever the dynamics changed: both when the force field was first presented, and when it was removed. We confirmed that these changes in feedback gain are not simply a byproduct of the change in background load, by demonstrating that this rapid visuomotor response is not load sensitive. Our results suggest that when the sensorimotor control system experiences errors, it increases the gain of the visuomotor feedback pathways to deal with the unexpected disturbances until the feedforward controller learns the appropriate dynamics. We suggest that these feedback gains are upregulated with increased uncertainty in the knowledge of the dynamics to counteract any errors or disturbances and ensure accurate and skillful movements.

Use of automatic sand pourers for loose sand models

The consistency of laboratory sand model preparation for physical testing is a fundamental criterion in representing identical geotechnical issues at prototype scale. This objective led to the development of robotic apparatus to eliminate the non-uniformity in manual pouring. Previous studies have shown consistent sand models with high relative density between 50 to 90% produced by the automatic moving-hopper sand pourer at the University of Cambridge, based primarily on a linear correlation to flow rate. However, in the case of loose samples, the influence of other parameters, particularly the drop height, becomes more apparent. In this paper, findings on the effect of flow rate and drop height are discussed in relation to the layer thickness and relative density of loose sand samples. Design charts are presented to illustrate their relationships. The effect of these factors on different sand types is also covered to extend the use of the equipment. © 2010 Taylor & Francis Group, London.