Aprendizado por Reforço com Valores deInfluência em Sistemas Multi-Agente

We propose a new paradigm for collective learning in multi-agent systems (MAS) as a solution to the problem in which several agents acting over the same environment must learn how to perform tasks, simultaneously, based on feedbacks given by each one of the other agents. We introduce the proposed paradigm in the form of a reinforcement learning algorithm, nominating it as reinforcement learning with influence values. While learning by rewards, each agent evaluates the relation between the current state and/or action executed at this state (actual believe) together with the reward obtained after all agents that are interacting perform their actions. The reward is a result of the interference of others. The agent considers the opinions of all its colleagues in order to attempt to change the values of its states and/or actions. The idea is that the system, as a whole, must reach an equilibrium, where all agents get satisfied with the obtained results. This means that the values of the state/actions pairs match the reward obtained by each agent. This dynamical way of setting the values for states and/or actions makes this new reinforcement learning paradigm the first to include, naturally, the fact that the presence of other agents in the environment turns it a dynamical model. As a direct result, we implicitly include the internal state, the actions and the rewards obtained by all the other agents in the internal state of each agent. This makes our proposal the first complete solution to the conceptual problem that rises when applying reinforcement learning in multi-agent systems, which is caused by the difference existent between the environment and agent models. With basis on the proposed model, we create the IVQ-learning algorithm that is exhaustive tested in repetitive games with two, three and four agents and in stochastic games that need cooperation and in games that need collaboration. This algorithm shows to be a good option for obtaining solutions that guarantee convergence to the Nash optimum equilibrium in cooperative problems. Experiments performed clear shows that the proposed paradigm is theoretical and experimentally superior to the traditional approaches. Yet, with the creation of this new paradigm the set of reinforcement learning applications in MAS grows up. That is, besides the possibility of applying the algorithm in traditional learning problems in MAS, as for example coordination of tasks in multi-robot systems, it is possible to apply reinforcement learning in problems that are essentially collaborative

Energy Management and Control for Islanded Microgrid Using Multi-Agents

This paper presents a multi-agent system for real-time operation of simulated microgrid using the Smart-Grid Test Bed at Washington State University. The multi-agent system (MAS) was developed in JADE (Java Agent DEvelopment Framework) which is a Foundation for Intelligent Physical Agents (FIPA) compliant open source multi-agent platform. The proposed operational strategy is mainly focused on using an appropriate energy management and control strategies to improve the operation of an islanded microgrid, formed by photovoltaic (PV) solar energy, batteries and resistive and rotating machines loads. The focus is on resource management and to avoid impact on loads from abrupt variations or interruption that changes the operating conditions. The management and control of the PV system is performed in JADE, while the microgrid model is simulated in RSCAD/RTDS (Real-Time Digital Simulator). Finally, the outcome of simulation studies demonstrated the feasibility of the proposed multi-agent approach for real-time operation of a microgrid.

Self-organising coordination systems

Many research fields are pushing the engineering of large-scale, mobile, and open systems towards the adoption of techniques inspired by self-organisation: pervasive computing, but also distributed artificial intelligence, multi-agent systems, social networks, peer-topeer and grid architectures exploit adaptive techniques to make global system properties emerge in spite of the unpredictability of interactions and behaviour. Such a trend is visible also in coordination models and languages, whenever a coordination infrastructure needs to cope with managing interactions in highly dynamic and unpredictable environments. As a consequence, self-organisation can be regarded as a feasible metaphor to define a radically new conceptual coordination framework. The resulting framework defines a novel coordination paradigm, called self-organising coordination, based on the idea of spreading coordination media over the network, and charge them with services to manage interactions based on local criteria, resulting in the emergence of desired and fruitful global coordination properties of the system. Features like topology, locality, time-reactiveness, and stochastic behaviour play a key role in both the definition of such a conceptual framework and the consequent development of self-organising coordination services. According to this framework, the thesis presents several self-organising coordination techniques developed during the PhD course, mainly concerning data distribution in tuplespace-based coordination systems. Some of these techniques have been also implemented in ReSpecT, a coordination language for tuple spaces, based on logic tuples and reactions to events occurring in a tuple space. In addition, the key role played by simulation and formal verification has been investigated, leading to analysing how automatic verification techniques like probabilistic model checking can be exploited in order to formally prove the emergence of desired behaviours when dealing with coordination approaches based on self-organisation. To this end, a concrete case study is presented and discussed.

Mobile TuCSoN: theoretical and technological requirements for TuCSoN's porting over Android mobile devices

Communication and coordination are two key-aspects in open distributed agent system, being both responsible for the system’s behaviour integrity. An infrastructure capable to handling these issues, like TuCSoN, should to be able to exploit modern technologies and tools provided by fast software engineering contexts. Thesis aims to demonstrate TuCSoN infrastructure’s abilities to cope new possibilities, hardware and software, offered by mobile technology. The scenarios are going to configure, are related to the distributed nature of multi-agent systems where an agent should be located and runned just on a mobile device. We deal new mobile technology frontiers concerned with smartphones using Android operating system by Google. Analysis and deployment of a distributed agent-based system so described go first to impact with quality and quantity considerations about available resources. Engineering issue at the base of our research is to use TuCSoN against to reduced memory and computing capability of a smartphone, without the loss of functionality, efficiency and integrity for the infrastructure. Thesis work is organized on two fronts simultaneously: the former is the rationalization process of the available hardware and software resources, the latter, totally orthogonal, is the adaptation and optimization process about TuCSoN architecture for an ad-hoc client side release.

BDI agents for Real Time Strategy games

While the use of distributed intelligence has been incrementally spreading in the design of a great number of intelligent systems, the field of Artificial Intelligence in Real Time Strategy games has remained mostly a centralized environment. Despite turn-based games have attained AIs of world-class level, the fast paced nature of RTS games has proven to be a significant obstacle to the quality of its AIs. Chapter 1 introduces RTS games describing their characteristics, mechanics and elements. Chapter 2 introduces Multi-Agent Systems and the use of the Beliefs-Desires-Intentions abstraction, analysing the possibilities given by self-computing properties. In Chapter 3 the current state of AI development in RTS games is analyzed highlighting the struggles of the gaming industry to produce valuable. The focus on improving multiplayer experience has impacted gravely on the quality of the AIs thus leaving them with serious flaws that impair their ability to challenge and entertain players. Chapter 4 explores different aspects of AI development for RTS, evaluating the potential strengths and weaknesses of an agent-based approach and analysing which aspects can benefit the most against centralized AIs. Chapter 5 describes a generic agent-based framework for RTS games where every game entity becomes an agent, each of which having its own knowledge and set of goals. Different aspects of the game, like economy, exploration and warfare are also analysed, and some agent-based solutions are outlined. The possible exploitation of self-computing properties to efficiently organize the agents activity is then inspected. Chapter 6 presents the design and implementation of an AI for an existing Open Source game in beta development stage: 0 a.d., an historical RTS game on ancient warfare which features a modern graphical engine and evolved mechanics. The entities in the conceptual framework are implemented in a new agent-based platform seamlessly nested inside the existing game engine, called ABot, widely described in Chapters 7, 8 and 9. Chapter 10 and 11 include the design and realization of a new agent based language useful for defining behavioural modules for the agents in ABot, paving the way for a wider spectrum of contributors. Chapter 12 concludes the work analysing the outcome of tests meant to evaluate strategies, realism and pure performance, finally drawing conclusions and future works in Chapter 13.

Multi Processor Systems On Chip with Configurable Hardware Acceleration

The evolution of the electronics embedded applications forces electronics systems designers to match their ever increasing requirements. This evolution pushes the computational power of digital signal processing systems, as well as the energy required to accomplish the computations, due to the increasing mobility of such applications. Current approaches used to match these requirements relies on the adoption of application specific signal processors. Such kind of devices exploits powerful accelerators, which are able to match both performance and energy requirements. On the other hand, the too high specificity of such accelerators often results in a lack of flexibility which affects non-recurrent engineering costs, time to market, and market volumes too. The state of the art mainly proposes two solutions to overcome these issues with the ambition of delivering reasonable performance and energy efficiency: reconfigurable computing and multi-processors computing. All of these solutions benefits from the post-fabrication programmability, that definitively results in an increased flexibility. Nevertheless, the gap between these approaches and dedicated hardware is still too high for many application domains, especially when targeting the mobile world. In this scenario, flexible and energy efficient acceleration can be achieved by merging these two computational paradigms, in order to address all the above introduced constraints. This thesis focuses on the exploration of the design and application spectrum of reconfigurable computing, exploited as application specific accelerators for multi-processors systems on chip. More specifically, it introduces a reconfigurable digital signal processor featuring a heterogeneous set of reconfigurable engines, and a homogeneous multi-core system, exploiting three different flavours of reconfigurable and mask-programmable technologies as implementation platform for applications specific accelerators. In this work, the various trade-offs concerning the utilization multi-core platforms and the different configuration technologies are explored, characterizing the design space of the proposed approach in terms of programmability, performance, energy efficiency and manufacturing costs.

Tecnologie e framework per la programmazione multi-agente

Negli ultimi anni le tecnologie informatiche sono state al centro di uno sviluppo esponenziale. Fra le incalcolabili innovazioni presentate, ha preso sempre più campo il paradigma per la programmazione ad agenti, che permette la realizzazione di sistemi software complessi, i quali, nell'informatica moderna, ricoprono un ruolo di fondamentale importanza. Questi sistemi, denominati autonomi, mostrano caratteristiche interessanti per scenari dinamici; essi infatti devono essere robusti e resistenti, in grado di adattarsi al contesto ambientale e quindi reagire a determinate modifiche che si verificano nell'ambiente, comportandosi di conseguenza. Indicano perciò la pro-attività dell'entità presa in considerazione. In questa tesi saranno spiegate queste tipologie di sistemi, introdotte le loro caratteristiche e mostrate le loro potenzialità. Tali caratteristiche permettono di responsabilizzare i soggetti, rendendo il sistema auto-organizzato, con una migliore scalabilità e modularità, riducendo quindi le elevate esigenze di calcolo. L'organizzazione di questo documento prevede i primi capitoli atti a introdurre il mondo dei sistemi autonomi, partendo dalle definizioni di autonomia e di agenti software, concludendo con i sistemi multi-agenti, allo scopo di permettere al lettore una comprensione adatta ed esaustiva. I successivi capitoli riguardano le fasi di progettazione delle entità prese in esame, le loro forme di standardizzazione e i modelli che possono adottare, tra i quali il più conosciuto, il modello BDI. Ne seguono due diverse metodologie per l'ingegneria del software orientata agli agenti. Si conclude con la presentazione dello stato dell'arte degli ambienti di sviluppo conosciuti, contenente un'esauriente introduzione ad ognuno di essi ed una visione nel mondo del lavoro del loro apporto negli applicativi in commercio. Infine la tesi terminerà con un capitolo di conclusioni e di riflessioni sui possibili aspetti futuri.

Agent-based modeling and simulation for the design of the future european Air Traffic Management system: the experience of CASSIOPEIA

The SESAR (Single European Sky ATM Research) program is an ambitious re-search and development initiative to design the future European air traffic man-agement (ATM) system. The study of the behavior of ATM systems using agent-based modeling and simulation tools can help the development of new methods to improve their performance. This paper presents an overview of existing agent-based approaches in air transportation (paying special attention to the challenges that exist for the design of future ATM systems) and, subsequently, describes a new agent-based approach that we proposed in the CASSIOPEIA project, which was developed according to the goals of the SESAR program. In our approach, we use agent models for different ATM stakeholders, and, in contrast to previous work, our solution models new collaborative decision processes for flow traffic management, it uses an intermediate level of abstraction (useful for simulations at larger scales), and was designed to be a practical tool (open and reusable) for the development of different ATM studies. It was successfully applied in three stud-ies related to the design of future ATM systems in Europe.

An agent-based approach for the design of the future european air traffic management system

This paper describes an agent-based approach for the simulation of air traffic management (ATM) in Europe that was designed to help analyze proposals for future ATM systems. This approach is able to represent new collaborative deci-sion processes for flow traffic management, it uses an intermediate level of ab-straction (useful for simulations at larger scales), and was designed to be a practi-cal tool (open and reusable) for the development of different ATM studies. It was successfully applied in three studies related to the design of future ATM systems in Europe.

Application of Agent Technology for Fault Diagnosis of Telecommunication Networks

La presente tesis doctoral contribuye al problema del diagnóstico autonómico de fallos en redes de telecomunicación. En las redes de telecomunicación actuales, las operadoras realizan tareas de diagnóstico de forma manual. Dichas operaciones deben ser llevadas a cabo por ingenieros altamente cualificados que cada vez tienen más dificultades a la hora de gestionar debidamente el crecimiento exponencial de la red tanto en tamaño, complejidad y heterogeneidad. Además, el advenimiento del Internet del Futuro hace que la demanda de sistemas que simplifiquen y automaticen la gestión de las redes de telecomunicación se haya incrementado en los últimos años. Para extraer el conocimiento necesario para desarrollar las soluciones propuestas y facilitar su adopción por los operadores de red, se propone una metodología de pruebas de aceptación para sistemas multi-agente enfocada en simplificar la comunicación entre los diferentes grupos de trabajo involucrados en todo proyecto de desarrollo software: clientes y desarrolladores. Para contribuir a la solución del problema del diagnóstico autonómico de fallos, se propone una arquitectura de agente capaz de diagnosticar fallos en redes de telecomunicación de manera autónoma. Dicha arquitectura extiende el modelo de agente Belief-Desire- Intention (BDI) con diferentes modelos de diagnóstico que gestionan las diferentes sub-tareas del proceso. La arquitectura propuesta combina diferentes técnicas de razonamiento para alcanzar su propósito gracias a un modelo estructural de la red, que usa razonamiento basado en ontologías, y un modelo causal de fallos, que usa razonamiento Bayesiano para gestionar debidamente la incertidumbre del proceso de diagnóstico. Para asegurar la adecuación de la arquitectura propuesta en situaciones de gran complejidad y heterogeneidad, se propone un marco de argumentación que permite diagnosticar a agentes que estén ejecutando en dominios federados. Para la aplicación de este marco en un sistema multi-agente, se propone un protocolo de coordinación en el que los agentes dialogan hasta alcanzar una conclusión para un caso de diagnóstico concreto. Como trabajos futuros, se consideran la extensión de la arquitectura para abordar otros problemas de gestión como el auto-descubrimiento o la auto-optimización, el uso de técnicas de reputación dentro del marco de argumentación para mejorar la extensibilidad del sistema de diagnóstico en entornos federados y la aplicación de las arquitecturas propuestas en las arquitecturas de red emergentes, como SDN, que ofrecen mayor capacidad de interacción con la red. ABSTRACT This PhD thesis contributes to the problem of autonomic fault diagnosis of telecommunication networks. Nowadays, in telecommunication networks, operators perform manual diagnosis tasks. Those operations must be carried out by high skilled network engineers which have increasing difficulties to properly manage the growing of those networks, both in size, complexity and heterogeneity. Moreover, the advent of the Future Internet makes the demand of solutions which simplifies and automates the telecommunication network management has been increased in recent years. To collect the domain knowledge required to developed the proposed solutions and to simplify its adoption by the operators, an agile testing methodology is defined for multiagent systems. This methodology is focused on the communication gap between the different work groups involved in any software development project, stakeholders and developers. To contribute to overcoming the problem of autonomic fault diagnosis, an agent architecture for fault diagnosis of telecommunication networks is defined. That architecture extends the Belief-Desire-Intention (BDI) agent model with different diagnostic models which handle the different subtasks of the process. The proposed architecture combines different reasoning techniques to achieve its objective using a structural model of the network, which uses ontology-based reasoning, and a causal model, which uses Bayesian reasoning to properly handle the uncertainty of the diagnosis process. To ensure the suitability of the proposed architecture in complex and heterogeneous environments, an argumentation framework is defined. This framework allows agents to perform fault diagnosis in federated domains. To apply this framework in a multi-agent system, a coordination protocol is defined. This protocol is used by agents to dialogue until a reliable conclusion for a specific diagnosis case is reached. Future work comprises the further extension of the agent architecture to approach other managements problems, such as self-discovery or self-optimisation; the application of reputation techniques in the argumentation framework to improve the extensibility of the diagnostic system in federated domains; and the application of the proposed agent architecture in emergent networking architectures, such as SDN, which offers new capabilities of control for the network.

Aplicações híbridas entre sistemas multiagentes e técnicas de inteligência artificial para redes inteligentes de distribuição de energia elétrica

Os smart grids representam a nova geração dos sistemas elétricos de potência, combinando avanços em computação, sistemas de comunicação, processos distribuídos e inteligência artificial para prover novas funcionalidades quanto ao acompanhamento em tempo real da demanda e do consumo de energia elétrica, gerenciamento em larga escala de geradores distribuídos, entre outras, a partir de um sistema de controle distribuído sobre a rede elétrica. Esta estrutura modifica profundamente a maneira como se realiza o planejamento e a operação de sistemas elétricos nos dias de hoje, em especial os de distribuição, e há interessantes possibilidades de pesquisa e desenvolvimento possibilitada pela busca da implementação destas funcionalidades. Com esse cenário em vista, o presente trabalho utiliza uma abordagem baseada no uso de sistemas multiagentes para simular esse tipo de sistema de distribuição de energia elétrica, considerando opções de controle distintas. A utilização da tecnologia de sistemas multiagentes para a simulação é baseada na conceituação de smart grids como um sistema distribuído, algo também realizado nesse trabalho. Para validar a proposta, foram simuladas três funcionalidades esperadas dessas redes elétricas: classificação de cargas não-lineares; gerenciamento de perfil de tensão; e reconfiguração topológica com a finalidade de reduzir as perdas elétricas. Todas as modelagens e desenvolvimentos destes estudos estão aqui relatados. Por fim, o trabalho se propõe a identificar os sistemas multiagentes como uma tecnologia a ser empregada tanto para a pesquisa, quanto para implementação dessas redes elétricas.

Arquitectura de un sistema multi-agente para el soporte de la anotación funcional

Predecir la función biológica de secuencias de Ácido Desoxirribonucleico (ADN) es unos de los mayores desafíos a los que se enfrenta la Bioinformática. Esta tarea se denomina anotación funcional y es un proceso complejo, laborioso y que requiere mucho tiempo. Dado su impacto en investigaciones y anotaciones futuras, la anotación debe ser lo más able y precisa posible. Idealmente, las secuencias deberían ser estudiadas y anotadas manualmente por un experto, garantizando así resultados precisos y de calidad. Sin embargo, la anotación manual solo es factible para pequeños conjuntos de datos o genomas de referencia. Con la llegada de las nuevas tecnologías de secuenciación, el volumen de datos ha crecido signi cativamente, haciendo aún más crítica la necesidad de implementaciones automáticas del proceso. Por su parte, la anotación automática es capaz de manejar grandes cantidades de datos y producir un análisis consistente. Otra ventaja de esta aproximación es su rapidez y bajo coste en relación a la manual. Sin embargo, sus resultados son menos precisos que los manuales y, en general, deben ser revisados ( curados ) por un experto. Aunque los procesos colaborativos de la anotación en comunidad pueden ser utilizados para reducir este cuello de botella, los esfuerzos en esta línea no han tenido hasta ahora el éxito esperado. Además, el problema de la anotación, como muchos otros en el dominio de la Bioinformática, abarca información heterogénea, distribuida y en constante evolución. Una posible aproximación para superar estos problemas consiste en cambiar el foco del proceso de los expertos individuales a su comunidad, y diseñar las herramientas de manera que faciliten la gestión del conocimiento y los recursos. Este trabajo adopta esta línea y propone MASSA (Multi-Agent System to Support functional Annotation), una arquitectura de Sistema Multi-Agente (SMA) para Soportar la Anotación funcional...

Arquitetura computacional para redes orgânicas e heterogêneas: plano de controle do sistema operacional swarm.

Computational Swarms (enxames computacionais), consistindo da integração de sensores e atuadores inteligentes no nosso mundo conectado, possibilitam uma extensão da info-esfera no mundo físico. Nós chamamos esta info-esfera extendida, cíber-física, de Swarm. Este trabalho propõe uma visão de Swarm onde dispositivos computacionais cooperam dinâmica e oportunisticamente, gerando redes orgânicas e heterogêneas. A tese apresenta uma arquitetura computacional do Plano de Controle do Sistema Operacional do Swarm, que é uma camada de software distribuída embarcada em todos os dispositivos que fazem parte do Swarm, responsável por gerenciar recursos, definindo atores, como descrever e utilizar serviços e recursos (como divulgá-los e descobrí-los, como realizar transações, adaptações de conteúdos e cooperação multiagentes). O projeto da arquitetura foi iniciado com uma revisão da caracterização do conceito de Swarm, revisitando a definição de termos e estabelecendo uma terminologia para ser utilizada. Requisitos e desafios foram identificados e uma visão operacional foi proposta. Esta visão operacional foi exercitada com casos de uso e os elementos arquiteturais foram extraídos dela e organizados em uma arquitetura. A arquitetura foi testada com os casos de uso, gerando revisões do sistema. Cada um dos elementos arquiteturais requereram revisões do estado da arte. Uma prova de conceito do Plano de Controle foi implementada e uma demonstração foi proposta e implementada. A demonstração selecionada foi o Smart Jukebox, que exercita os aspectos distribuídos e a dinamicidade do sistema proposto. Este trabalho apresenta a visão do Swarm computacional e apresenta uma plataforma aplicável na prática. A evolução desta arquitetura pode ser a base de uma rede global, heterogênea e orgânica de redes de dispositivos computacionais alavancando a integração de sistemas cíber-físicos na núvem permitindo a cooperação de sistemas escaláveis e flexíveis, interoperando para alcançar objetivos comuns.

Multi-threaded Communicating Agents in Qu-Prolog

In this tutorial paper we summarise the key features of the multi-threaded Qu-Prolog language for implementing multi-threaded communicating agent applications. Internal threads of an agent communicate using the shared dynamic database used as a generalisation of Linda tuple store. Threads in different agents, perhaps on different hosts, communicate using either a thread-to-thread store and forward communication system, or by a publish and subscribe mechanism in which messages are routed to their destinations based on content test subscriptions. We illustrate the features using an auction house application. This is fully distributed with multiple auctioneers and bidders which participate in simultaneous auctions. The application makes essential use of the three forms of inter-thread communication of Qu-Prolog. The agent bidding behaviour is specified graphically as a finite state automaton and its implementation is essentially the execution of its state transition function. The paper assumes familiarity with Prolog and the basic concepts of multi-agent systems.