Aprendizado por Reforço com Valores deInfluência em Sistemas Multi-Agente

We propose a new paradigm for collective learning in multi-agent systems (MAS) as a solution to the problem in which several agents acting over the same environment must learn how to perform tasks, simultaneously, based on feedbacks given by each one of the other agents. We introduce the proposed paradigm in the form of a reinforcement learning algorithm, nominating it as reinforcement learning with influence values. While learning by rewards, each agent evaluates the relation between the current state and/or action executed at this state (actual believe) together with the reward obtained after all agents that are interacting perform their actions. The reward is a result of the interference of others. The agent considers the opinions of all its colleagues in order to attempt to change the values of its states and/or actions. The idea is that the system, as a whole, must reach an equilibrium, where all agents get satisfied with the obtained results. This means that the values of the state/actions pairs match the reward obtained by each agent. This dynamical way of setting the values for states and/or actions makes this new reinforcement learning paradigm the first to include, naturally, the fact that the presence of other agents in the environment turns it a dynamical model. As a direct result, we implicitly include the internal state, the actions and the rewards obtained by all the other agents in the internal state of each agent. This makes our proposal the first complete solution to the conceptual problem that rises when applying reinforcement learning in multi-agent systems, which is caused by the difference existent between the environment and agent models. With basis on the proposed model, we create the IVQ-learning algorithm that is exhaustive tested in repetitive games with two, three and four agents and in stochastic games that need cooperation and in games that need collaboration. This algorithm shows to be a good option for obtaining solutions that guarantee convergence to the Nash optimum equilibrium in cooperative problems. Experiments performed clear shows that the proposed paradigm is theoretical and experimentally superior to the traditional approaches. Yet, with the creation of this new paradigm the set of reinforcement learning applications in MAS grows up. That is, besides the possibility of applying the algorithm in traditional learning problems in MAS, as for example coordination of tasks in multi-robot systems, it is possible to apply reinforcement learning in problems that are essentially collaborative

Seguimento de pessoas em sistemas multi-câmara

Trabalho Final de Mestrado para obtenção do grau de Mestre em Engenharia de Redes de Comunicação e Multimédia

Decentralized Target Tracking based on Multi-Robot Cooperative Triangulation

Target tracking with bearing-only sensors is a challenging problem when the target moves dynamically in complex scenarios. Besides the partial observability of such sensors, they have limited field of views, occlusions can occur, etc. In those cases, cooperative approaches with multiple tracking robots are interesting, but the different sources of uncertain information need to be considered appropriately in order to achieve better estimates. Even though there exist probabilistic filters that can estimate the position of a target dealing with incertainties, bearing-only measurements bring usually additional problems with initialization and data association. In this paper, we propose a multi-robot triangulation method with a dynamic baseline that can triangulate bearing-only measurements in a probabilistic manner to produce 3D observations. This method is combined with a decentralized stochastic filter and used to tackle those initialization and data association issues. The approach is validated with simulations and field experiments where a team of aerial and ground robots with cameras track a dynamic target.

Multi-robot cooperative stereo for outdoor scenarios

13th International Conference on Autonomous Robot Systems (Robotica), 2013

Simulation Environment for Multi-Robot Cooperative 3D Target Perception

4th International Conference, SIMPAR 2014, Bergamo, Italy, October 20-23, 2014

Uncertainty based Multi-Robot Cooperative Triangulation

The paper presents a multi-robot cooperative framework to estimate the 3D position of dynamic targets, based on bearing-only vision measurements. The uncertainty of the observation provided by each robot equipped with a bearing-only vision system is effectively addressed for cooperative triangulation purposes by weighing the contribution of each monocular bearing ray in a probabilistic manner. The envisioned framework is evaluated in an outdoor scenario with a team of heterogeneous robots composed of an Unmanned Ground and Aerial Vehicle.

Optimização energética de sistemas multi-componente da zona fria do Steam Cracker

Dissertação para obtenção do Grau de Mestre em Engenharia Química e Bioquímica

Segurança no nível físico em sistemas multi-antena com diretividade na informação

Os sistemas de comunicação sem fios são sistemas de difusão por natureza. Devido a essa sua natureza, um dos problemas inerentes à mesma deve-se à segurança e ao secretismo, pois se o canal é partilhado a informação facilmente é obtida por um utilizador não autorizado, ao contrário dos sistemas de comunicação com fios. Tradicionalmente, a introdução de segurança em sistemas de comunicação, resulta na encriptação da informação, resultante de protocolos de encriptação. No entanto, a segurança através da criptografia baseia-se na premissa de que o utilizador não autorizado tem uma capacidade de processamento limitada, pois senão poderia simplesmente tentar todas as combinações possíveis e obter a chave de encriptação. Como a capacidade de processamento tem crescido exponencialmente, este tipo de sistemas tem se tornado cada vez mais complexos para não se tornarem obsoletos. A introdução de segurança na camada física torna-se então uma opção apelativa pois pode servir como um complemento, visto que os sistemas de criptografia funcionam em camadas superiores independentes da camada fisica, apresentando assim uma abordagem multi-camada em termos de segurança. Tipicamente as técnicas de segurança no nível físico podem se agrupar em 2 tipos: técnicas que se baseiam em códigos, ou técnicas que exploram variações temporais e espaciais do canal. As primeiras diminuem a eficiência espectral do sistema, e as segundas apresentam bons resultados em ambientes dinâmicos, mas em ambientes estáticos não são muito promissores. Há também a necessidade de aumentar as taxas de transmissão nos próximos sistemas de comunicação. Devido a estes requisitos, uma das tecnologias propostas para a nova geração de comunicações, é uma tecnologia baseada numa arquitectura Multiple-Input-Multiple-Output(MIMO). Esta tecnologia é promissora e consegue atingir taxas de transferências que correspondem aos requisitos propostos. Apresenta-se assim uma nova técnica de segurança no nível físico, que explora as caracteristicas físicas do sistema, como um complemento a outras medidas de segurança em camadas mais altas. Esta técnica não provoca diminuição da eficiência espectral e é independente do canal, o que tenta solucionar os problemas das restantes técnicas já existentes.

Interfaz de usuario de un sistema multi-robot de reconocimiento de superficies con Google Earth

Els sistemes multi-robot de reconeixement de superfícies es poden utilitzar tant per a l'exploració de llocs remots, de difícil accés o perillosos. Normalment, els robots no són autònoms, depenen d'operadors humans per dirigir-los. La informació que capten ha de ser processada i mostrada a l'usuari o usuària del sistema de forma intel·ligible. Un exemple d'aplicació seria el d'un sistema multirobot format per diversos helicòpters no tripulats que proporciona informació d'una àrea que ha patit algun desastre. El sistema informàtic recolliria la informació i la transmetria al coordinador de l'operatiu d'assistència de l'emergència. La idea del projecte és la de combinar la informació proporcionada pel sistema multi-robot amb la de la zona disponible a Google Earth i fer d'aquesta eina l'interfície d'usuari de l'aplicació.

Efficient image mosaicing for multi-robot visual underwater mapping

Robotic platforms have advanced greatly in terms of their remote sensing capabilities, including obtaining optical information using cameras. Alongside these advances, visual mapping has become a very active research area, which facilitates the mapping of areas inaccessible to humans. This requires the efficient processing of data to increase the final mosaic quality and computational efficiency. In this paper, we propose an efficient image mosaicing algorithm for large area visual mapping in underwater environments using multiple underwater robots. Our method identifies overlapping image pairs in the trajectories carried out by the different robots during the topology estimation process, being this a cornerstone for efficiently mapping large areas of the seafloor. We present comparative results based on challenging real underwater datasets, which simulated multi-robot mapping

Design and Development of a Remotely Operated Underwater Multi-Robot Manipulator Controller

For the scientific and commercial utilization of Ocean resources, the role of intelligent underwater robotic systems are of great importance. Scientific activities like Marine Bio-technology, Hydrographic mapping, and commercial applications like Marine mining, Ocean energy, fishing, aquaculture, cable laying and pipe lining are a few utilization of ocean resources. As most of the deep undersea exploration are beyond the reachability of divers and also as the use of operator controlled and teleoperated Remotely Operated Vehicles (ROVs) and Diver Transport Vehicles (DTVs) turn out to be highly inefficient, it is essential to have a fully automated system capable providing stable control and communication links for the unstructured undersea environment.

Multi-Robot Task Allocation for Inspection Problems with Cooperative Tasks Using Hybrid Genetic Algorithms

In dieser Dissertation werden Methoden zur optimalen Aufgabenverteilung in Multirobotersystemen (engl. Multi-Robot Task Allocation – MRTA) zur Inspektion von Industrieanlagen untersucht. MRTA umfasst die Verteilung und Ablaufplanung von Aufgaben für eine Gruppe von Robotern unter Berücksichtigung von operativen Randbedingungen mit dem Ziel, die Gesamteinsatzkosten zu minimieren. Dank zunehmendem technischen Fortschritt und sinkenden Technologiekosten ist das Interesse an mobilen Robotern für den Industrieeinsatz in den letzten Jahren stark gestiegen. Viele Arbeiten konzentrieren sich auf Probleme der Mobilität wie Selbstlokalisierung und Kartierung, aber nur wenige Arbeiten untersuchen die optimale Aufgabenverteilung. Da sich mit einer guten Aufgabenverteilung eine effizientere Planung erreichen lässt (z. B. niedrigere Kosten, kürzere Ausführungszeit), ist das Ziel dieser Arbeit die Entwicklung von Lösungsmethoden für das aus Inspektionsaufgaben mit Einzel- und Zweiroboteraufgaben folgende Such-/Optimierungsproblem. Ein neuartiger hybrider Genetischer Algorithmus wird vorgestellt, der einen teilbevölkerungbasierten Genetischen Algorithmus zur globalen Optimierung mit lokalen Suchheuristiken kombiniert. Zur Beschleunigung dieses Algorithmus werden auf die fittesten Individuen einer Generation lokale Suchoperatoren angewendet. Der vorgestellte Algorithmus verteilt die Aufgaben nicht nur einfach und legt den Ablauf fest, sondern er bildet auch temporäre Roboterverbünde für Zweiroboteraufgaben, wodurch räumliche und zeitliche Randbedingungen entstehen. Vier alternative Kodierungsstrategien werden für den vorgestellten Algorithmus entworfen: Teilaufgabenbasierte Kodierung: Hierdurch werden alle möglichen Lösungen abgedeckt, allerdings ist der Suchraum sehr groß. Aufgabenbasierte Kodierung: Zwei Möglichkeiten zur Zuweisung von Zweiroboteraufgaben wurden implementiert, um die Effizienz des Algorithmus zu steigern. Gruppierungsbasierte Kodierung: Zeitliche Randbedingungen zur Gruppierung von Aufgaben werden vorgestellt, um gute Lösungen innerhalb einer kleinen Anzahl von Generationen zu erhalten. Zwei Umsetzungsvarianten werden vorgestellt. Dekompositionsbasierte Kodierung: Drei geometrische Zerlegungen wurden entworfen, die Informationen über die räumliche Anordnung ausnutzen, um Probleme zu lösen, die Inspektionsgebiete mit rechteckigen Geometrien aufweisen. In Simulationsstudien wird die Leistungsfähigkeit der verschiedenen hybriden Genetischen Algorithmen untersucht. Dazu wurde die Inspektion von Tanklagern einer Erdölraffinerie mit einer Gruppe homogener Inspektionsroboter als Anwendungsfall gewählt. Die Simulationen zeigen, dass Kodierungsstrategien, die auf der geometrischen Zerlegung basieren, bei einer kleinen Anzahl an Generationen eine bessere Lösung finden können als die anderen untersuchten Strategien. Diese Arbeit beschäftigt sich mit Einzel- und Zweiroboteraufgaben, die entweder von einem einzelnen mobilen Roboter erledigt werden können oder die Zusammenarbeit von zwei Robotern erfordern. Eine Erweiterung des entwickelten Algorithmus zur Behandlung von Aufgaben, die mehr als zwei Roboter erfordern, ist möglich, würde aber die Komplexität der Optimierungsaufgabe deutlich vergrößern.

Heterogeneous Multi-Robot Cooperation

This report addresses the problem of achieving cooperation within small- to medium- sized teams of heterogeneous mobile robots. I describe a software architecture I have developed, called ALLIANCE, that facilitates robust, fault tolerant, reliable, and adaptive cooperative control. In addition, an extended version of ALLIANCE, called L-ALLIANCE, is described, which incorporates a dynamic parameter update mechanism that allows teams of mobile robots to improve the efficiency of their mission performance through learning. A number of experimental results of implementing these architectures on both physical and simulated mobile robot teams are described. In addition, this report presents the results of studies of a number of issues in mobile robot cooperation, including fault tolerant cooperative control, adaptive action selection, distributed control, robot awareness of team member actions, improving efficiency through learning, inter-robot communication, action recognition, and local versus global control.

Suboptimal system recovery from communication loss in a multi-robot localization scenario using EKF algorithms

This paper describes a multi-robot localization scenario where, for a period of time, the robot team loses communication with one of the robots due to system error. In this novel approach, extended Kalman filter (EKF) algorithms utilize relative measurements to localize the robots in space. These measurements are used to reliably compensate "dead-com" periods were no information can be exchanged between the members of the robot group.

Modeling and simulating a mathematical tool for multi-robot pattern transformation

The work reported in this paper is motivated by the need to investigate general methods for pattern transformation. A formal definition for pattern transformation is provided and four special cases namely, elementary and geometric transformation based on repositioning all and some agents in the pattern are introduced. The need for a mathematical tool and simulations for visualizing the behavior of a transformation method is highlighted. A mathematical method based on the Moebius transformation is proposed. The transformation method involves discretization of events for planning paths of individual robots in a pattern. Simulations on a particle physics simulator are used to validate the feasibility of the proposed method.