Reinforcement Learning approaches to Economic Dispatch problem

This paper presents Reinforcement Learning (RL) approaches to Economic Dispatch problem. In this paper, formulation of Economic Dispatch as a multi stage decision making problem is carried out, then two variants of RL algorithms are presented. A third algorithm which takes into consideration the transmission losses is also explained. Efficiency and flexibility of the proposed algorithms are demonstrated through different representative systems: a three generator system with given generation cost table, IEEE 30 bus system with quadratic cost functions, 10 generator system having piecewise quadratic cost functions and a 20 generator system considering transmission losses. A comparison of the computation times of different algorithms is also carried out.

A Reinforcement Learning Algorithm to Economic Dispatch Considering Transmission Losses

Reinforcement Learning (RL) refers to a class of learning algorithms in which learning system learns which action to take in different situations by using a scalar evaluation received from the environment on performing an action. RL has been successfully applied to many multi stage decision making problem (MDP) where in each stage the learning systems decides which action has to be taken. Economic Dispatch (ED) problem is an important scheduling problem in power systems, which decides the amount of generation to be allocated to each generating unit so that the total cost of generation is minimized without violating system constraints. In this paper we formulate economic dispatch problem as a multi stage decision making problem. In this paper, we also develop RL based algorithm to solve the ED problem. The performance of our algorithm is compared with other recent methods. The main advantage of our method is it can learn the schedule for all possible demands simultaneously.

Reinforcement Learning Approaches to Power System Scheduling

One major component of power system operation is generation scheduling. The objective of the work is to develop efficient control strategies to the power scheduling problems through Reinforcement Learning approaches. The three important active power scheduling problems are Unit Commitment, Economic Dispatch and Automatic Generation Control. Numerical solution methods proposed for solution of power scheduling are insufficient in handling large and complex systems. Soft Computing methods like Simulated Annealing, Evolutionary Programming etc., are efficient in handling complex cost functions, but find limitation in handling stochastic data existing in a practical system. Also the learning steps are to be repeated for each load demand which increases the computation time.Reinforcement Learning (RL) is a method of learning through interactions with environment. The main advantage of this approach is it does not require a precise mathematical formulation. It can learn either by interacting with the environment or interacting with a simulation model. Several optimization and control problems have been solved through Reinforcement Learning approach. The application of Reinforcement Learning in the field of Power system has been a few. The objective is to introduce and extend Reinforcement Learning approaches for the active power scheduling problems in an implementable manner. The main objectives can be enumerated as:(i) Evolve Reinforcement Learning based solutions to the Unit Commitment Problem.(ii) Find suitable solution strategies through Reinforcement Learning approach for Economic Dispatch. (iii) Extend the Reinforcement Learning solution to Automatic Generation Control with a different perspective. (iv) Check the suitability of the scheduling solutions to one of the existing power systems.First part of the thesis is concerned with the Reinforcement Learning approach to Unit Commitment problem. Unit Commitment Problem is formulated as a multi stage decision process. Q learning solution is developed to obtain the optimwn commitment schedule. Method of state aggregation is used to formulate an efficient solution considering the minimwn up time I down time constraints. The performance of the algorithms are evaluated for different systems and compared with other stochastic methods like Genetic Algorithm.Second stage of the work is concerned with solving Economic Dispatch problem. A simple and straight forward decision making strategy is first proposed in the Learning Automata algorithm. Then to solve the scheduling task of systems with large number of generating units, the problem is formulated as a multi stage decision making task. The solution obtained is extended in order to incorporate the transmission losses in the system. To make the Reinforcement Learning solution more efficient and to handle continuous state space, a fimction approximation strategy is proposed. The performance of the developed algorithms are tested for several standard test cases. Proposed method is compared with other recent methods like Partition Approach Algorithm, Simulated Annealing etc.As the final step of implementing the active power control loops in power system, Automatic Generation Control is also taken into consideration.Reinforcement Learning has already been applied to solve Automatic Generation Control loop. The RL solution is extended to take up the approach of common frequency for all the interconnected areas, more similar to practical systems. Performance of the RL controller is also compared with that of the conventional integral controller.In order to prove the suitability of the proposed methods to practical systems, second plant ofNeyveli Thennal Power Station (NTPS IT) is taken for case study. The perfonnance of the Reinforcement Learning solution is found to be better than the other existing methods, which provide the promising step towards RL based control schemes for practical power industry.Reinforcement Learning is applied to solve the scheduling problems in the power industry and found to give satisfactory perfonnance. Proposed solution provides a scope for getting more profit as the economic schedule is obtained instantaneously. Since Reinforcement Learning method can take the stochastic cost data obtained time to time from a plant, it gives an implementable method. As a further step, with suitable methods to interface with on line data, economic scheduling can be achieved instantaneously in a generation control center. Also power scheduling of systems with different sources such as hydro, thermal etc. can be looked into and Reinforcement Learning solutions can be achieved.

A Reinforcement Learning Approach to Economic Dispatch using Neural Networks

This paper presents a Reinforcement Learning (RL) approach to economic dispatch (ED) using Radial Basis Function neural network. We formulate the ED as an N stage decision making problem. We propose a novel architecture to store Qvalues and present a learning algorithm to learn the weights of the neural network. Even though many stochastic search techniques like simulated annealing, genetic algorithm and evolutionary programming have been applied to ED, they require searching for the optimal solution for each load demand. Also they find limitation in handling stochastic cost functions. In our approach once we learn the Q-values, we can find the dispatch for any load demand. We have recently proposed a RL approach to ED. In that approach, we could find only the optimum dispatch for a set of specified discrete values of power demand. The performance of the proposed algorithm is validated by taking IEEE 6 bus system, considering transmission losses

Differential induction, isolation, physicochemical and molecular characterization of temperate phages of environmental Vibrio cholerae as evidence of phage mediated Horizontal Gene Transfer

The resurgence of the enteric pathogen Vibrio cholerae, the causative organism of epidemic cholera, remains a major health problem in many developing countries like India. The southern Indian state of Kerala is endemic to cholera. The outbreaks of cholera follow a seasonal pattern in regions of endemicity. Marine aquaculture settings and mangrove environments of Kerala serve as reservoirs for V. cholerae. The non-O1/non-O139 environmental isolates of V. cholerae with incomplete ‘virulence casette’ are to be dealt with caution as they constitute a major reservoir of diverse virulence genes in the marine environment and play a crucial role in pathogenicity and horizontal gene transfer. The genes coding cholera toxin are borne on, and can be infectiously transmitted by CTXΦ, a filamentous lysogenic vibriophages. Temperate phages can provide crucial virulence and fitness factors affecting cell metabolism, bacterial adhesion, colonization, immunity, antibiotic resistance and serum resistance. The present study was an attempt to screen the marine environments like aquafarms and mangroves of coastal areas of Alappuzha and Cochin, Kerala for the presence of lysogenic V. cholerae, to study their pathogenicity and also gene transfer potential. Phenotypic and molecular methods were used for identification of isolates as V. cholerae. The thirty one isolates which were Gram negative, oxidase positive, fermentative, with or without gas production on MOF media and which showed yellow coloured colonies on TCBS (Thiosulfate Citrate Bile salt Sucrose) agar were segregated as vibrios. Twenty two environmental V. cholerae strains of both O1 and non- O1/non-O139 serogroups on induction with mitomycin C showed the presence of lysogenic phages. They produced characteristic turbid plaques in double agar overlay assay using the indicator strain V. cholerae El Tor MAK 757. PCR based molecular typing with primers targeting specific conserved sequences in the bacterial genome, demonstrated genetic diversity among these lysogen containing non-O1 V. cholerae . Polymerase chain reaction was also employed as a rapid screening method to verify the presence of 9 virulence genes namely, ctxA, ctxB, ace, hlyA, toxR, zot,tcpA, ninT and nanH, using gene specific primers. The presence of tcpA gene in ALPVC3 was alarming, as it indicates the possibility of an epidemic by accepting the cholera. Differential induction studies used ΦALPVC3, ΦALPVC11, ΦALPVC12 and ΦEKM14, underlining the possibility of prophage induction in natural ecosystems, due to abiotic factors like antibiotics, pollutants, temperature and UV. The efficiency of induction of prophages varied considerably in response to the different induction agents. The growth curve of lysogenic V. cholerae used in the study drastically varied in the presence of strong prophage inducers like antibiotics and UV. Bacterial cell lysis was directly proportional to increase in phage number due to induction. Morphological characterization of vibriophages by Transmission Electron Microscopy revealed hexagonal heads for all the four phages. Vibriophage ΦALPVC3 exhibited isometric and contractile tails characteristic of family Myoviridae, while phages ΦALPVC11 and ΦALPVC12 demonstrated the typical hexagonal head and non-contractile tail of family Siphoviridae. ΦEKM14, the podophage was distinguished by short non-contractile tail and icosahedral head. This work demonstrated that environmental parameters can influence the viability and cell adsorption rates of V. cholerae phages. Adsorption studies showed 100% adsorption of ΦALPVC3 ΦALPVC11, ΦALPVC12 and ΦEKM14 after 25, 30, 40 and 35 minutes respectively. Exposure to high temperatures ranging from 50ºC to 100ºC drastically reduced phage viability. The optimum concentration of NaCl required for survival of vibriophages except ΦEKM14 was 0.5 M and that for ΦEKM14 was 1M NaCl. Survival of phage particles was maximum at pH 7-8. V. cholerae is assumed to have existed long before their human host and so the pathogenic clones may have evolved from aquatic forms which later colonized the human intestine by progressive acquisition of genes. This is supported by the fact that the vast majority of V. cholerae strains are still part of the natural aquatic environment. CTXΦ has played a critical role in the evolution of the pathogenicity of V. cholerae as it can transmit the ctxAB gene. The unusual transformation of V. cholerae strains associated with epidemics and the emergence of V. cholera O139 demonstrates the evolutionary success of the organism in attaining greater fitness. Genetic changes in pathogenic V. cholerae constitute a natural process for developing immunity within an endemically infected population. The alternative hosts and lysogenic environmental V. cholerae strains may potentially act as cofactors in promoting cholera phage ‘‘blooms’’ within aquatic environments, thereby influencing transmission of phage sensitive, pathogenic V. cholerae strains by aquatic vehicles. Differential induction of the phages is a clear indication of the impact of environmental pollution and global changes on phage induction. The development of molecular biology techniques offered an accessible gateway for investigating the molecular events leading to genetic diversity in the marine environment. Using nucleic acids as targets, the methods of fingerprinting like ERIC PCR and BOX PCR, revealed that the marine environment harbours potentially pathogenic group of bacteria with genetic diversity. The distribution of virulence associated genes in the environmental isolates of V. cholerae provides tangible material for further investigation. Nucleotide and protein sequence analysis alongwith protein structure prediction aids in better understanding of the variation inalleles of same gene in different ecological niche and its impact on the protein structure for attaining greater fitness of pathogens. The evidences of the co-evolution of virulence genes in toxigenic V. cholerae O1 from different lineages of environmental non-O1 strains is alarming. Transduction studies would indicate that the phenomenon of acquisition of these virulence genes by lateral gene transfer, although rare, is not quite uncommon amongst non-O1/non-O139 V. cholerae and it has a key role in diversification. All these considerations justify the need for an integrated approach towards the development of an effective surveillance system to monitor evolution of V. cholerae strains with epidemic potential. Results presented in this study, if considered together with the mechanism proposed as above, would strongly suggest that the bacteriophage also intervenes as a variable in shaping the cholera bacterium, which cannot be ignored and hinting at imminent future epidemics.