Experience generalization for multi-agent reinforcement learning

On-line learning methods have been applied successfully in multi-agent systems to achieve coordination among agents. Learning in multi-agent systems implies in a non-stationary scenario perceived by the agents, since the behavior of other agents may change as they simultaneously learn how to improve their actions. Non-stationary scenarios can be modeled as Markov Games, which can be solved using the Minimax-Q algorithm a combination of Q-learning (a Reinforcement Learning (RL) algorithm which directly learns an optimal control policy) and the Minimax algorithm. However, finding optimal control policies using any RL algorithm (Q-learning and Minimax-Q included) can be very time consuming. Trying to improve the learning time of Q-learning, we considered the QS-algorithm. in which a single experience can update more than a single action value by using a spreading function. In this paper, we contribute a Minimax-QS algorithm which combines the Minimax-Q algorithm and the QS-algorithm. We conduct a series of empirical evaluation of the algorithm in a simplified simulator of the soccer domain. We show that even using a very simple domain-dependent spreading function, the performance of the learning algorithm can be improved.

Multi-Agent Framework for Spatial Load Forecasting

A multi-agent framework for spatial electric load forecasting, especially suited to simulate the different dynamics involved on distribution systems, is presented. The service zone is divided into several sub-zones, each subzone is considered as an independent agent identified with a corresponding load level, and their relationships with the neighbor zones are represented as development probabilities. With this setting, different kind of agents can be developed to simulate the growth pattern of the loads in distribution systems. This paper presents two different kinds of agents to simulate different situations, presenting some promissory results.

A multi-agent system with a percolation approach to simulate the driving pattern of Plug-In ELectric Vehicles

A multi-agent system with a percolation approach to simulate the driving pattern of Plug-In Electric Vehicle (PEV), especially suited to simulate the PEVs behavior on any distribution systems, is presented. This tool intends to complement information about the driving patterns database on systems where that kind of information is not available. So, this paper aims to provide a framework that is able to work with any kind of technology and load generated of PEVs. The service zone is divided into several sub-zones, each subzone is considered as an independent agent identified with corresponding load level, and their relationships with the neighboring zones are represented as network probabilities. A percolation approach is used to characterize the autonomy of the battery of the PVEs to move through the city. The methodology is tested with data from a mid-size city real distribution system. The result shows the sub-area where the battery of PEVs will need to be recharge and gives the planners of distribution systems the necessary input for a medium to long term network planning in a smart grid environment. © 2012 IEEE.

Multi-Agent Simulation of Urban Social Dynamics for Spatial Load Forecasting

Coordenação de Aperfeiçoamento de Pessoal de Nível Superior (CAPES)

A multi-agent architecture for intelligent data analysis

The present study introduces a multi-agent architecture designed for doing automation process of data integration and intelligent data analysis. Different from other approaches the multi-agent architecture was designed using a multi-agent based methodology. Tropos, an agent based methodology was used for design. Based on the proposed architecture, we describe a Web based application where the agents are responsible to analyse petroleum well drilling data to identify possible abnormalities occurrence. The intelligent data analysis methods used was the Neural Network.

Spatial electric load forecasting using a local movement approach

An agent based model for spatial electric load forecasting using a local movement approach for the spatiotemporal allocation of the new loads in the service zone is presented. The density of electrical load for each of the major consumer classes in each sub-zone is used as the current state of the agents. The spatial growth is simulated with a walking agent who starts his path in one of the activity centers of the city and goes to the limits of the city following a radial path depending on the different load levels. A series of update rules are established to simulate the S growth behavior and the complementarity between classes. The results are presented in future load density maps. The tests in a real system from a mid-size city show a high rate of success when compared with other techniques. The most important features of this methodology are the need for few data and the simplicity of the algorithm, allowing for future scalability. © 2009 IEEE.

Spatial load forecasting using a demand propagation approach

A method for spatial electric load forecasting using multi-agent systems, especially suited to simulate the local effect of special loads in distribution systems is presented. The method based on multi-agent systems uses two kinds of agents: reactive and proactive. The reactive agents represent each sub-zone in the service zone, characterizing each one with their corresponding load level, represented in a real number, and their relationships with other sub-zones represented in development probabilities. The proactive agent carry the new load expected to be allocated because of the new special load, this agent distribute the new load in a propagation pattern. The results are presented with maps of future expected load levels in the service zone. The method is tested with data from a mid-size city real distribution system, simulating the effect of a load with attraction and repulsion attributes. The method presents good results and performance. © 2011 IEEE.

Aplicações de sistemas multiagentes na previsão espacial de demanda elétrica em sistemas de distribuição

Pós-graduação em Engenharia Elétrica - FEIS

Arquitetura de um sistema multiagente autônomo para supervisão e controle

This paper presents a multi-agent architecture that was designed to develop processes supervision and control systems, with the main objective to automate tasks that are repetitive and stressful, and error prone when performed by humans. A set of agents were identified, based on the study of a number of applications found in the literature, that use the approach of multi-agent systems for data integration and process monitoring to faults detection and diagnosis, these agents are used as basis of the proposed multi-agent architecture. A prototype system for the analysis of abnormalities during oil wells drilling was developed.

Multi-choice mixed integer goal programming optimization for real problems in a sugar and ethanol milling company

Goal Programming (GP) is an important analytical approach devised to solve many realworld problems. The first GP model is known as Weighted Goal Programming (WGP). However, Multi-Choice Aspirations Level (MCAL) problems cannot be solved by current GP techniques. In this paper, we propose a Multi-Choice Mixed Integer Goal Programming model (MCMI-GP) for the aggregate production planning of a Brazilian sugar and ethanol milling company. The MC-MIGP model was based on traditional selection and process methods for the design of lots, representing the production system of sugar, alcohol, molasses and derivatives. The research covers decisions on the agricultural and cutting stages, sugarcane loading and transportation by suppliers and, especially, energy cogeneration decisions; that is, the choice of production process, including storage stages and distribution. The MCMIGP allows decision-makers to set multiple aspiration levels for their problems in which the more/higher, the better and the less/lower, the better in the aspiration levels are addressed. An application of the proposed model for real problems in a Brazilian sugar and ethanol mill was conducted; producing interesting results that are herein reported and commented upon. Also, it was made a comparison between MCMI GP and WGP models using these real cases. © 2013 Elsevier Inc.

VAr planning using genetic algorithm and linear programming

A combined methodology consisting of successive linear programming (SLP) and a simple genetic algorithm (SGA) solves the reactive planning problem. The problem is divided into operating and planning subproblems; the operating subproblem, which is a nonlinear, ill-conditioned and nonconvex problem, consists of determining the voltage control and the adjustment of reactive sources. The planning subproblem consists of obtaining the optimal reactive source expansion considering operational, economical and physical characteristics of the system. SLP solves the optimal reactive dispatch problem related to real variables, while SGA is used to determine the necessary adjustments of both the binary and discrete variables existing in the modelling problem. Once the set of candidate busbars has been defined, the program implemented gives the location and size of the reactive sources needed, if any, to maintain the operating and security constraints.

Hardware implementation of type-2 programmable fuzzifier

The design of full programmable type-2 membership function circuit is presented in this paper. This circuit is used to implement the fuzzifier block of Type-2 Fuzzy Logic Controller chip. In this paper the type-2 fuzzy set was obtained by blurring the width of the type-1 fuzzy set. This circuit allows programming the height and the shape of the membership function. It operates in current mode, with supply voltage of 3.3V. The simulation results of interval type-2 membership function circuit have been done in CMOS 0.35μm technology using Mentor Graphics software. © 2011 IEEE.

Multiobjective short-term planning of electric power distribution systems using NSGA-II

This paper presents a mixed integer nonlinear programming multiobjective model for short-term planning of distribution networks that considers in an integrated manner the following planning activities: allocation of capacitor banks; voltage regulators; the cable replacement of branches and feeders. The objective functions considered in the proposed model are: to minimize operational and investment costs and minimize the voltage deviations in the the network buses, subject to a set of technical and operational constraints. A multiobjective genetic algorithm based on a Non-Dominated Sorting Genetic Algorithm (NSGA-II) is proposed to solve this model. The proposed mathematical model and solution methodology is validated testing a medium voltage distribution system with 135 buses. © 2013 Brazilian Society for Automatics - SBA.

Transaction scheduling using dynamic conflict avoidance

Software transaction memory (STM) systems have been used as an approach to improve performance, by allowing the concurrent execution of atomic blocks. However, under high-contention workloads, STM-based systems can considerably degrade performance, as transaction conflict rate increases. Contention management policies have been used as a way to select which transaction to abort when a conflict occurs. In general, contention managers are not capable of avoiding conflicts, as they can only select which transaction to abort and the moment it should restart. Since contention managers act only after a conflict is detected, it becomes harder to effectively increase transaction throughput. More proactive approaches have emerged, aiming at predicting when a transaction is likely to abort, postponing its execution. Nevertheless, most of the proposed proactive techniques are limited, as they do not replace the doomed transaction by another or, when they do, they rely on the operating system for that, having little or no control on which transaction to run. This article proposes LUTS, a lightweight user-level transaction scheduler. Unlike other techniques, LUTS provides the means for selecting another transaction to run in parallel, thus improving system throughput. We discuss LUTS design and propose a dynamic conflict-avoidance heuristic built around its scheduling capabilities. Experimental results, conducted with the STAMP and STMBench7 benchmark suites, running on TinySTM and SwissTM, show how our conflict-avoidance heuristic can effectively improve STM performance on high contention applications. © 2012 Springer Science+Business Media, LLC.

Sustainable urban energy planning: The case study of a tropical city

The urbanization of modern societies has imposed to the planners and decision-makers a more precise attention to facts not considered before. Several aspects, such as the energy availability and the deleterious effect of pollution on the populations, must be considered in the policy decisions of cities urbanization. The current paradigm presents centralized power stations supplying a city, and a combination of technologies may compose the energy mix of a country, such as thermal power plants, hydroelectric plants, wind systems and solar-based systems, with their corresponding emission pattern. A goal programming multi-objective optimization model is presented for the electric expansion analysis of a tropical city, and also a case study for the city of Guaratinguetá, Brazil, considering a particular wind and solar radiation patterns established according to actual data and modeled via the time series analysis method. Scenarios are proposed and the results of single environmental objective, single economic objective and goal programming multi-objective modeling are discussed. The consequences of each dispatch decision, which considers pollutant emission exportation to the neighborhood or the need of supplementing electricity by purchasing it from the public electric power grid, are discussed. The results revealed energetic dispatch for the alternatives studied and the optimum environmental and economic solution was obtained. © 2012 Elsevier Ltd.