Reinforcement Learning Approaches to Power System Scheduling

One major component of power system operation is generation scheduling. The objective of the work is to develop efficient control strategies to the power scheduling problems through Reinforcement Learning approaches. The three important active power scheduling problems are Unit Commitment, Economic Dispatch and Automatic Generation Control. Numerical solution methods proposed for solution of power scheduling are insufficient in handling large and complex systems. Soft Computing methods like Simulated Annealing, Evolutionary Programming etc., are efficient in handling complex cost functions, but find limitation in handling stochastic data existing in a practical system. Also the learning steps are to be repeated for each load demand which increases the computation time.Reinforcement Learning (RL) is a method of learning through interactions with environment. The main advantage of this approach is it does not require a precise mathematical formulation. It can learn either by interacting with the environment or interacting with a simulation model. Several optimization and control problems have been solved through Reinforcement Learning approach. The application of Reinforcement Learning in the field of Power system has been a few. The objective is to introduce and extend Reinforcement Learning approaches for the active power scheduling problems in an implementable manner. The main objectives can be enumerated as:(i) Evolve Reinforcement Learning based solutions to the Unit Commitment Problem.(ii) Find suitable solution strategies through Reinforcement Learning approach for Economic Dispatch. (iii) Extend the Reinforcement Learning solution to Automatic Generation Control with a different perspective. (iv) Check the suitability of the scheduling solutions to one of the existing power systems.First part of the thesis is concerned with the Reinforcement Learning approach to Unit Commitment problem. Unit Commitment Problem is formulated as a multi stage decision process. Q learning solution is developed to obtain the optimwn commitment schedule. Method of state aggregation is used to formulate an efficient solution considering the minimwn up time I down time constraints. The performance of the algorithms are evaluated for different systems and compared with other stochastic methods like Genetic Algorithm.Second stage of the work is concerned with solving Economic Dispatch problem. A simple and straight forward decision making strategy is first proposed in the Learning Automata algorithm. Then to solve the scheduling task of systems with large number of generating units, the problem is formulated as a multi stage decision making task. The solution obtained is extended in order to incorporate the transmission losses in the system. To make the Reinforcement Learning solution more efficient and to handle continuous state space, a fimction approximation strategy is proposed. The performance of the developed algorithms are tested for several standard test cases. Proposed method is compared with other recent methods like Partition Approach Algorithm, Simulated Annealing etc.As the final step of implementing the active power control loops in power system, Automatic Generation Control is also taken into consideration.Reinforcement Learning has already been applied to solve Automatic Generation Control loop. The RL solution is extended to take up the approach of common frequency for all the interconnected areas, more similar to practical systems. Performance of the RL controller is also compared with that of the conventional integral controller.In order to prove the suitability of the proposed methods to practical systems, second plant ofNeyveli Thennal Power Station (NTPS IT) is taken for case study. The perfonnance of the Reinforcement Learning solution is found to be better than the other existing methods, which provide the promising step towards RL based control schemes for practical power industry.Reinforcement Learning is applied to solve the scheduling problems in the power industry and found to give satisfactory perfonnance. Proposed solution provides a scope for getting more profit as the economic schedule is obtained instantaneously. Since Reinforcement Learning method can take the stochastic cost data obtained time to time from a plant, it gives an implementable method. As a further step, with suitable methods to interface with on line data, economic scheduling can be achieved instantaneously in a generation control center. Also power scheduling of systems with different sources such as hydro, thermal etc. can be looked into and Reinforcement Learning solutions can be achieved.

A Reinforcement Learning Algorithm to Economic Dispatch Considering Transmission Losses

Reinforcement Learning (RL) refers to a class of learning algorithms in which learning system learns which action to take in different situations by using a scalar evaluation received from the environment on performing an action. RL has been successfully applied to many multi stage decision making problem (MDP) where in each stage the learning systems decides which action has to be taken. Economic Dispatch (ED) problem is an important scheduling problem in power systems, which decides the amount of generation to be allocated to each generating unit so that the total cost of generation is minimized without violating system constraints. In this paper we formulate economic dispatch problem as a multi stage decision making problem. In this paper, we also develop RL based algorithm to solve the ED problem. The performance of our algorithm is compared with other recent methods. The main advantage of our method is it can learn the schedule for all possible demands simultaneously.

Implications of the North Atlantic Oscillation for a UK–Norway renewable power system

UK wind-power capacity is increasing and new transmission links are proposed with Norway, where hydropower dominates the electricity mix. Weather affects both these renewable resources and the demand for electricity. The dominant large-scale pattern of Euro-Atlantic atmospheric variability is the North Atlantic Oscillation (NAO), associated with positive correlations in wind, temperature and precipitation over northern Europe. The NAO's effect on wind-power and demand in the UK and Norway is examined, focussing on March when Norwegian hydropower reserves are low and the combined power system might be most susceptible to atmospheric variations. The NCEP/NCAR meteorological reanalysis dataset (1948–2010) is used to drive simple models for demand and wind-power, and ‘demand-net-wind’ (DNW) is estimated for positive, neutral and negative NAO states. Cold, calm conditions in NAO− cause increased demand and decreased wind-power compared to other NAO states. Under a 2020 wind-power capacity scenario, the increase in DNW in NAO− relative to NAO neutral is equivalent to nearly 25% of the present-day average rate of March Norwegian hydropower usage. As the NAO varies on long timescales (months to decades), and there is potentially some skill in monthly predictions, we argue that it is important to understand its impact on European power systems.

Optimal reactive power flow via the modified barrier Lagrangian function approach

Conselho Nacional de Desenvolvimento Científico e Tecnológico (CNPq)

Transient stability analysis of electric energy systems via a fuzzy ART-ARTMAP neural network

This work presents a methodology to analyze transient stability (first oscillation) of electric energy systems, using a neural network based on ART architecture (adaptive resonance theory), named fuzzy ART-ARTMAP neural network for real time applications. The security margin is used as a stability analysis criterion, considering three-phase short circuit faults with a transmission line outage. The neural network operation consists of two fundamental phases: the training and the analysis. The training phase needs a great quantity of processing for the realization, while the analysis phase is effectuated almost without computation effort. This is, therefore the principal purpose to use neural networks for solving complex problems that need fast solutions, as the applications in real time. The ART neural networks have as primordial characteristics the plasticity and the stability, which are essential qualities to the training execution and to an efficient analysis. The fuzzy ART-ARTMAP neural network is proposed seeking a superior performance, in terms of precision and speed, when compared to conventional ARTMAP, and much more when compared to the neural networks that use the training by backpropagation algorithm, which is a benchmark in neural network area. (c) 2005 Elsevier B.V. All rights reserved.

Interior point algorithm for linear programming used in transmission network synthesis

This article presents a well-known interior point method (IPM) used to solve problems of linear programming that appear as sub-problems in the solution of the long-term transmission network expansion planning problem. The linear programming problem appears when the transportation model is used, and when there is the intention to solve the planning problem using a constructive heuristic algorithm (CHA), ora branch-and-bound algorithm. This paper shows the application of the IPM in a CHA. A good performance of the IPM was obtained, and then it can be used as tool inside algorithm, used to solve the planning problem. Illustrative tests are shown, using electrical systems known in the specialized literature. (C) 2005 Elsevier B.V. All rights reserved.

Alternative parameters for the continuation power flow method

The conventional Newton's method has been considered inadequate to obtain the maximum loading point (MLP) of power systems. It is due to the Jacobian matrix singularity at this point. However, the MLP can be efficiently computed through parameterization techniques of continuation methods. This paper presents and tests new parameterization schemes, namely the total power losses (real and reactive), the power at the slack bus (real or reactive), the reactive power at generation buses, the reactive power at shunts (capacitor or reactor), the transmission lines power losses (real and reactive), and transmission lines power (real and reactive). Besides their clear physical meaning, which makes easier the development and application of continuation methods for power systems analysis, the main advantage of some of the proposed parameters is that its not necessary to change the parameter in the vicinity of the MLP. Studies on the new parameterization schemes performed on the IEEE 118 buses system show that the ill-conditioning problems at and near the MLP are eliminated. So, the characteristics of the conventional Newton's method are not only preserved but also improved. (C) 2003 Elsevier B.V. B.V. All rights reserved.

An Improved Parameterization Technique for the Continuation Power Flow

Continuation methods have been long used in P-V curve tracing due to their efficiency in the resolution of ill-conditioned cases, with close to singular Jacobian matrices, such as the maximum loading point of power systems. Several parameterization techniques have been proposed to avoid matrix singularity and successfully solve those cases. This paper presents a simple geometric parameterization technique to overcome the singularity of the Jacobian matrix by the addition of a line equations located at the plane determined by a bus voltage magnitude and the loading factor. This technique enlarges the set of voltage variables that can be used to whole P-V curve tracing, without ill-conditioning problems and no need of parameter changes. Simulation results, obtained for large realistic Brazilian and American power systems, show that the robustness and efficiency of the conventional power flow are not only preserved but also improved.

Pole placement by coordinated tuning of Power System Stabilizers and FACTS-POD stabilizers

This work presents the application of the Decentralized Modal Control method for pole placement in multimachine power systems utilizing FACTS (Flexible AC Transmission Systems), STATCOM (Static Synchronous Compensator) and UPFC (Unified Power Flow Controller) devices. For this, these devices are equipped with supplementary damping controllers, denominated POD ( Power Oscillation Damping), achieving a coordinated project with local controllers (Power System Stabilizers - PSS). Comparative analysis on the function of damping of the FACTS, STATCOM and UPFC is performed using the New England System that has 10 generators, 39 buses and 46 transmission lines. (c) 2011 Elsevier Ltd. All rights reserved.

Reactive Control for Transmission Overload Relief Based on Sensitivity Analysis and Cooperative Game Theory

Coordenação de Aperfeiçoamento de Pessoal de Nível Superior (CAPES)

Efficient method for AC transmission network expansion planning

A combinatorial mathematical model in tandem with a metaheuristic technique for solving transmission network expansion planning (TNEP) using an AC model associated with reactive power planning (RPP) is presented in this paper. AC-TNEP is handled through a prior DC model while additional lines as well as VAr-plants are used as reinforcements to cope with real network requirements. The solution of the reinforcement stage can be obtained by assuming all reactive demands are supplied locally to achieve a solution for AC-TNEP and by neglecting the local reactive sources, a reactive power planning (RPP) will be managed to find the minimum required reactive power sources. Binary GA as well as a real genetic algorithm (RCA) are employed as metaheuristic optimization techniques for solving this combinatorial TNEP as well as the RPP problem. High quality results related with lower investment costs through case studies on test systems show the usefulness of the proposal when working directly with the AC model in transmission network expansion planning, instead of relaxed models. (C) 2010 Elsevier B.V. All rights reserved.

A Strategy to Solve the Multistage Transmission Expansion Planning Problem

Conselho Nacional de Desenvolvimento Científico e Tecnológico (CNPq)

Proposal of a Transmission Line Model Based on Lumped Elements: An Analytic Solution

Coordenação de Aperfeiçoamento de Pessoal de Nível Superior (CAPES)

A Bilevel Approach to Transmission Expansion Planning Within a Market Environment

We present a bilevel model for transmission expansion planning within a market environment, where producers and consumers trade freely electric energy through a pool. The target of the transmission planner, modeled through the upper-level problem, is to minimize network investment cost while facilitating energy trading. This upper-level problem is constrained by a collection of lower-level market clearing problems representing pool trading, and whose individual objective functions correspond to social welfare. Using the duality theory the proposed bilevel model is recast as a mixed-integer linear programming problem, which is solvable using branch-and-cut solvers. Detailed results from an illustrative example and a case study are presented and discussed. Finally, some relevant conclusions are drawn.

Single real transformation matrices applied to double three-phase transmission lines

Single real transformation matrices are tested as phase-mode transformation matrices of typical symmetrical systems with double three-phase and two parallel double three-phase transmission lines. These single real transformation matrices are achieved from eigenvector matrices of the mentioned systems and they are based on Clarke's matrix. Using linear combinations of the Clarke's matrix elements, the techniques applied to the single three-phase lines are extended to systems with 6 or 12 phase conductors. For transposed double three-phase lines, phase Z and Y matrices are changed into diagonal matrices in mode domain. Considering non-transposed cases of double three-phase lines, the results are not exact and the error analyses are performed using the exact eigenvalues. In case of two parallel double three-phase lines, the exact single real transformation matrix has not been obtained yet. Searching for this exact matrix, the analyses are based on a single homopolar reference. For all analyses in this paper, the homopolar mode is used as the only homopolar reference for all phase conductors of the studied system. (C) 2008 Elsevier B.V. All rights reserved.