Intelligent decision making in electricity markets: simulated annealing Q-Learning

Electricity markets are complex environments, involving a large number of different entities, playing in a dynamic scene to obtain the best advantages and profits. MASCEM is a multi-agent electricity market simulator to model market players and simulate their operation in the market. Market players are entities with specific characteristics and objectives, making their decisions and interacting with other players. MASCEM is integrated with ALBidS, a system that provides several dynamic strategies for agents’ behavior. This paper presents a method that aims at enhancing ALBidS competence in endowing market players with adequate strategic bidding capabilities, allowing them to obtain the higher possible gains out of the market. This method uses a reinforcement learning algorithm to learn from experience how to choose the best from a set of possible actions. These actions are defined accordingly to the most probable points of bidding success. With the purpose of accelerating the convergence process, a simulated annealing based algorithm is included.

Multi-agent simulation of continental, regional, and micro electricity markets

Renewable based power generation has significantly increased over the last years. However, this process has evolved separately from electricity markets, leading to an inadequacy of the present market models to cope with huge quantities of renewable energy resources, and to take full advantage of the presently existing and the increasing envisaged renewable based and distributed energy resources. This paper proposes the modelling of electricity markets at several levels (continental, regional and micro), taking into account the specific characteristics of the players and resources involved in each level and ensuring that the proposed models accommodate adequate business models able to support the contribution of all the resources in the system, from the largest to the smaller ones. The proposed market models are integrated in MASCEM (Multi- Agent Simulator of Competitive Electricity Markets), using the multi agent approach advantages for overcoming the current inadequacy and significant limitations of the presently existing electricity market simulators to deal with the complex electricity market models that must be adopted.

A multi-agent system for the support of producer coalition formation in electricity markets

All over the world Distributed Generation is seen as a valuable help to get cleaner and more efficient electricity. To get negotiation power and advantages of scale economy, distributed producers can be aggregated giving place to a new concept: the Virtual Power Producer. Virtual Power Producers are multitechnology and multi-site heterogeneous entities. Virtual Power Producers should adopt organization and management methodologies so that they can make Distributed Generation a really profitable activity, able to participate in the market. In this paper we address the development of a multi-agent market simulator – MASCEM – able to study alternative coalitions of distributed producers in order to identify promising Virtual Power Producers in an electricity market.

Electricity Markets Simulation: MASCEM Contributions to the Challenging Reality

Electricity Markets are not only a new reality but an evolving one as the involved players and rules change at a relatively high rate. Multi-agent simulation combined with Artificial Intelligence techniques may result in sophisticated tools very helpful under this context. Some simulation tools have already been developed, some of them very interesting. However, at the present state it is important to go a step forward in Electricity Markets simulators as this is crucial for facing changes in Power Systems. This paper explains the context and needs of electricity market simulation, describing the most important characteristics of available simulators. We present our work concerning MASCEM simulator, presenting its features as well as the improvements being made to accomplish the change and challenging reality of Electricity Markets.

Metalearning in ALBidS: A Strategic Bidding System for electricity markets

Metalearning is a subfield of machine learning with special pro-pensity for dynamic and complex environments, from which it is difficult to extract predictable knowledge. The field of study of this work is the electricity market, which due to the restructuring that recently took place, became an especially complex and unpredictable environment, involving a large number of different entities, playing in a dynamic scene to obtain the best advantages and profits. This paper presents the development of a metalearner, applied to the decision support of electricity markets’ negotia-tion entities. The proposed metalearner takes advantage on several learning algorithms implemented in ALBidS, an adaptive learning system that pro-vides decision support to electricity markets’ participating players. Using the outputs of each different strategy as inputs, the metalearner creates its own output, considering each strategy with a different weight, depending on its individual quality of performance. The results of the proposed meth-od are studied and analyzed using MASCEM - a multi-agent electricity market simulator that models market players and simulates their operation in the market. This simulator provides the chance to test the metalearner in scenarios based on real electricity market´s data.

Virtual power producers market strategies

This paper presents MASCEM - a multi-agent based electricity market simulator. MASCEM uses game theory, machine learning techniques, scenario analysis and optimisation techniques to model market agents and to provide them with decision-support. This paper mainly focus on the MASCEM ability to provide the means to model and simulate Virtual Power Producers (VPP). VPPs are represented as a coalition of agents, with specific characteristics and goals. The paper detail some of the most important aspects considered in VPP formation and in the aggregation of new producers and includes a case study.

Multiagent system for adaptive strategy formulation in electricity markets

Electricity markets are complex environments with very particular characteristics. MASCEM is a market simulator developed to allow deep studies of the interactions between the players that take part in the electricity market negotiations. This paper presents a new proposal for the definition of MASCEM players’ strategies to negotiate in the market. The proposed methodology is multiagent based, using reinforcement learning algorithms to provide players with the capabilities to perceive the changes in the environment, while adapting their bids formulation according to their needs, using a set of different techniques that are at their disposal.

VPP´s multi-level negotiation in smart grids and competitive electricity markets

The increase of distributed generation (DG) has brought about new challenges in electrical networks electricity markets and in DG units operation and management. Several approaches are being developed to manage the emerging potential of DG, such as Virtual Power Players (VPPs), which aggregate DG plants; and Smart Grids, an approach that views generation and associated loads as a subsystem. This paper presents a multi-level negotiation mechanism for Smart Grids optimal operation and negotiation in the electricity markets, considering the advantages of VPPs’ management. The proposed methodology is implemented and tested in MASCEM – a multiagent electricity market simulator, developed to allow deep studies of the interactions between the players that take part in the electricity market negotiations.

Cost dependent strategy for electricity markets bidding based on adaptive reinforcement learning

Electricity markets are complex environments, involving a large number of different entities, playing in a dynamic scene to obtain the best advantages and profits. MASCEM is a multi-agent electricity market simulator to model market players and simulate their operation in the market. Market players are entities with specific characteristics and objectives, making their decisions and interacting with other players. MASCEM provides several dynamic strategies for agents’ behavior. This paper presents a method that aims to provide market players with strategic bidding capabilities, allowing them to obtain the higher possible gains out of the market. This method uses a reinforcement learning algorithm to learn from experience how to choose the best from a set of possible bids. These bids are defined accordingly to the cost function that each producer presents.

A new approach for multi-agent coalition formation and management in the scope of electricity markets

This paper presents a new methodology for the creation and management of coalitions in Electricity Markets. This approach is tested using the multi-agent market simulator MASCEM, taking advantage of its ability to provide the means to model and simulate VPP (Virtual Power Producers). VPPs are represented as coalitions of agents, with the capability of negotiating both in the market, and internally, with their members, in order to combine and manage their individual specific characteristics and goals, with the strategy and objectives of the VPP itself. The new features include the development of particular individual facilitators to manage the communications amongst the members of each coalition independently from the rest of the simulation, and also the mechanisms for the classification of the agents that are candidates to join the coalition. In addition, a global study on the results of the Iberian Electricity Market is performed, to compare and analyze different approaches for defining consistent and adequate strategies to integrate into the agents of MASCEM. This, combined with the application of learning and prediction techniques provide the agents with the ability to learn and adapt themselves, by adjusting their actions to the continued evolving states of the world they are playing in.

Renewable-based generation integration in electricity markets with virtual power producers

The development of renewable energy sources and Distributed Generation (DG) of electricity is of main importance in the way towards a sustainable development. However, the management, in large scale, of these technologies is complicated because of the intermittency of primary resources (wind, sunshine, etc.) and small scale of some plants. The aggregation of DG plants gives place to a new concept: the Virtual Power Producer (VPP). VPPs can reinforce the importance of these generation technologies making them valuable in electricity markets. VPPs can ensure a secure, environmentally friendly generation and optimal management of heat, electricity and cold as well as optimal operation and maintenance of electrical equipment, including the sale of electricity in the energy market. For attaining these goals, there are important issues to deal with, such as reserve management strategies, strategies for bids formulation, the producers’ remuneration, and the producers’ characterization for coalition formation. This chapter presents the most important concepts related with renewable-based generation integration in electricity markets, using VPP paradigm. The presented case studies make use of two main computer applications:ViProd and MASCEM. ViProd simulates VPP operation, including the management of plants in operation. MASCEM is a multi-agent based electricity market simulator that supports the inclusion of VPPs in the players set.

Bid definition method for electricity markets based on an adaptive multiagent system

Electricity markets are complex environments with very particular characteristics. MASCEM is a market simulator developed to allow deep studies of the interactions between the players that take part in the electricity market negotiations. This paper presents a new proposal for the definition of MASCEM players’ strategies to negotiate in the market. The proposed methodology is multiagent based, using reinforcement learning algorithms to provide players with the capabilities to perceive the changes in the environment, while adapting their bids formulation according to their needs, using a set of different techniques that are at their disposal. Each agent has the knowledge about a different method for defining a strategy for playing in the market, the main agent chooses the best among all those, and provides it to the market player that requests, to be used in the market. This paper also presents a methodology to manage the efficiency/effectiveness balance of this method, to guarantee that the degradation of the simulator processing times takes the correct measure.

Strategic bidding methodology for electricity markets using adaptive learning

The very particular characteristics of electricity markets, require deep studies of the interactions between the involved players. MASCEM is a market simulator developed to allow studying electricity market negotiations. This paper presents a new proposal for the definition of MASCEM players’ strategies to negotiate in the market. The proposed methodology is implemented as a multiagent system, using reinforcement learning algorithms to provide players with the capabilities to perceive the changes in the environment, while adapting their bids formulation according to their needs, using a set of different techniques that are at their disposal. This paper also presents a methodology to define players’ models based on the historic of their past actions, interpreting how their choices are affected by past experience, and competition.

Multi-agent negotiation for coalition formation and management in electricity markets

Negotiation is a fundamental tool for reaching understandings that allow each involved party to gain an advantage for themselves by the end of the process. In recent years, with the increasing of compe-titiveness in most sectors, negotiation procedures become present in practically all of them. One particular environment in which the competitiveness has been increasing exponentially is the electricity markets sector. This work is directed to the study of electricity markets’ partici-pating entities interaction, namely in what concerns the formation, management and operation of aggregating entities – Virtual Power Players (VPPs). VPPs are responsible for managing coalitions of market players with small market negotiating influence, which take strategic advantage in entering such aggregations, to increase their negotiating power. This chapter presents a negotiation methodology for the creation and management of coalitions in Electricity Markets. This approach is tested using MASCEM, taking advantage of its ability to provide the means to model and simulate VPPs. VPPs are represented as coalitions of agents, with the capability of negotiating both in the market, and internally, with their members, in order to combine and manage their individual specific characteristics and goals, with the strategy and objectives of the VPP itself.

Virtual power producers simulation - negotiating renewable distributed generation in competitive electricity markets

Sustainable development concerns made renewable energy sources to be increasingly used for electricity distributed generation. However, this is mainly due to incentives or mandatory targets determined by energy policies as in European Union. Assuring a sustainable future requires distributed generation to be able to participate in competitive electricity markets. To get more negotiation power in the market and to get advantages of scale economy, distributed generators can be aggregated giving place to a new concept: the Virtual Power Producer (VPP). VPPs are multi-technology and multisite heterogeneous entities that should adopt organization and management methodologies so that they can make distributed generation a really profitable activity, able to participate in the market. This paper presents ViProd, a simulation tool that allows simulating VPPs operation, in the context of MASCEM, a multi-agent based eletricity market simulator.