A tutorial survey of reinforcement learning

This paper gives a compact, self-contained tutorial survey of reinforcement learning, a tool that is increasingly finding application in the development of intelligent dynamic systems. Research on reinforcement learning during the past decade has led to the development of a variety of useful algorithms. This paper surveys the literature and presents the algorithms in a cohesive framework.

Sensing of delaminations in composite laminates using embedded magnetostrictive particle layers

This is an exploratory study to illustrate the feasibility of detecting delamination type of damage in polymeric laminates with one layer of magnetostrictive particles. One such beam encircled with excitation and sensing coils is used for this study. The change in stress gradient of the magnetostrictive layer in the vicinity of delamination shows up as a change in induced voltage in the sensing coil, and therefore provides a means to sense the presence of delamination. Recognizing the constitutive behavior of the Terfenol-D material is highly nonlinear, analytical expressions for the constitutive relations are developed by using curve fitting techniques to the experimental data. Analytical expressions that relate the applied excitation field with the stress and magnetic flux densities induced in the magnetostrictive layer are developed. Numerical methods are used to find the relative change in the induced voltage in the sensing coil due to the presence of delamination. A typical example of unidirectional laminate, with embedded delaminations, is used for the simulation purposes. This exploratory study illustrates that the open-circuit voltage induced in the sensing coil changes significantly (as large of 68 millivolts) with the occurrence of delamination. This feature can be exploited for device off-line inspection techniques and/or linking monitoring procedures for practical applications.

FE analysis of piezo-laminate composites under thermal loads

A shear flexible 4-noded finite element formulation, having five mechanical degrees of freedom per node, is presented for modeling the dynamic as well as the static thermal response of laminated composites containing distributed piezoelectric layers. This element has been developed to have one electrical degree of freedom per piezoelectric layer. The mass, stiffness and thermo-electro-mechanical coupling effects on the actuator and sensor layers have been considered. Numerical studies have been conducted to investigate both the sensory and active responses on piezoelectric composite beam and plate structures. It is. concluded that both the thermal and pyroelectric effects are important and need to be considered in the precision distributed control of intelligent structures.

Reinforcement learning with average cost for adaptive control of traffic lights at intersections

We propose for the first time two reinforcement learning algorithms with function approximation for average cost adaptive control of traffic lights. One of these algorithms is a version of Q-learning with function approximation while the other is a policy gradient actor-critic algorithm that incorporates multi-timescale stochastic approximation. We show performance comparisons on various network settings of these algorithms with a range of fixed timing algorithms, as well as a Q-learning algorithm with full state representation that we also implement. We observe that whereas (as expected) on a two-junction corridor, the full state representation algorithm shows the best results, this algorithm is not implementable on larger road networks. The algorithm PG-AC-TLC that we propose is seen to show the best overall performance.

Evaluation of Suitable Locations for Generation Expansion in Restructured Power Systems: A Novel Concept of T-index Thukaram

In developing countries, a high rate of growth in the demand for electric energy is felt, and so the addition of new generating units becomes inevitable. In deregulated power systems, private generating stations are encouraged to add new generations. Some of the factors considered while placing a new generating unit are: availability of esources, ease of transmitting power, distance from the load centre, etc. Finding the most appropriate locations for generation expansion can be done by running repeated power flows and carrying system studies like analyzing the voltage profile, voltage stability, loss analysis, etc. In this paper a new methodology is proposed which will mainly consider the existing network topology. A concept of T-index is introduced in this paper, which considers the electrical distances between generator and load nodes. This index is used for ranking the most significant new generation expansion locations and also indicates the amount of permissible generations that can be installed at these new locations. This concept facilitates for the medium and long term planning of power generation expansions within the available transmission corridors. Studies carried out on an EHV equivalent 10-bus system and IEEE 30 bus systems are presented for illustration purposes.

Electromechanical dynamics and optimization of pectoral fin-based ionic polymer-metal composite underwater propulsor

Ionic polymer-metal composites are soft artificial muscle-like bending actuators, which can work efficiently in wet environments such as water. Therefore, there is significant motivation for research on the development and design analysis of ionic polymer-metal composite based biomimetic underwater propulsion systems. Among aquatic animals, fishes are efficient swimmers with advantages such as high maneuverability, high cruising speed, noiseless propulsion, and efficient stabilization. Fish swimming mechanisms provide biomimetic inspiration for underwater propulsor design. Fish locomotion can be broadly classified into body and/or caudal fin propulsion and median and/or paired pectoral fin propulsion. In this article, the paired pectoral fin-based oscillatory propulsion using ionic polymer-metal composite for aquatic propulsor applications is studied. Beam theory and the concept of hydrodynamic function are used to describe the interaction between the beam and water. Furthermore, a quasi-steady blade element model that accounts for unsteady phenomena such as added mass effects, dynamic stall, and the cumulative Wagner effect is used to obtain hydrodynamic performance of the ionic polymer-metal composite propulsor. Dynamic characteristics of ionic polymer-metal composite fin are analyzed using numerical simulations. It is shown that the use of optimization methods can lead to significant improvement in performance of the ionic polymer-metal composite fin.

Arithmetic coding based lossless compression schemes for power system steady state operational data

Introduction of processor based instruments in power systems is resulting in the rapid growth of the measured data volume. The present practice in most of the utilities is to store only some of the important data in a retrievable fashion for a limited period. Subsequently even this data is either deleted or stored in some back up devices. The investigations presented here explore the application of lossless data compression techniques for the purpose of archiving all the operational data - so that they can be put to more effective use. Four arithmetic coding methods suitably modified for handling power system steady state operational data are proposed here. The performance of the proposed methods are evaluated using actual data pertaining to the Southern Regional Grid of India. (C) 2012 Elsevier Ltd. All rights reserved.

Design of S bend waveguides for delay line applications in integrated optics

A novel scheme for generation of phase using optical delay lines is proposed. The design of the optical components in the circuit which includes the S bend waveguides and straight waveguide couplers are very important for integrated optics. Beam propagation Method and MatLab is employed for the design.

State estimation in power systems using linear model infinity norm-based trust region approach

State estimation is one of the most important functions in an energy control centre. An computationally efficient state estimator which is free from numerical instability/ill-conditioning is essential for security assessment of electric power grid. Whereas approaches to successfully overcome the numerical ill-conditioning issues have been proposed, an efficient algorithm for addressing the convergence issues in the presence of topological errors is yet to be evolved. Trust region (TR) methods have been successfully employed to overcome the divergence problem to certain extent. In this study, case studies are presented where the conventional algorithms including the existing TR methods would fail to converge. A linearised model-based TR method for successfully overcoming the convergence issues is proposed. On the computational front, unlike the existing TR methods for state estimation which employ quadratic models, the proposed linear model-based estimator is computationally efficient because the model minimiser can be computed in a single step. The model minimiser at each step is computed by minimising the linearised model in the presence of TR and measurement mismatch constraints. The infinity norm is used to define the geometry of the TR. Measurement mismatch constraints are employed to improve the accuracy. The proposed algorithm is compared with the quadratic model-based TR algorithm with case studies on the IEEE 30-bus system, 205-bus and 514-bus equivalent systems of part of Indian grid.

Design and Evaluation of a Robust Optical Beam-Interruption-Based Vehicle Classifier System

This paper presents the design and development of a novel optical vehicle classifier system, which is based on interruption of laser beams, that is suitable for use in places with poor transportation infrastructure. The system can estimate the speed, axle count, wheelbase, tire diameter, and the lane of motion of a vehicle. The design of the system eliminates the need for careful optical alignment, whereas the proposed estimation strategies render the estimates insensitive to angular mounting errors and to unevenness of the road. Strategies to estimate vehicular parameters are described along with the optimization of the geometry of the system to minimize estimation errors due to quantization. The system is subsequently fabricated, and the proposed features of the system are experimentally demonstrated. The relative errors in the estimation of velocity and tire diameter are shown to be within 0.5% and to change by less than 17% for angular mounting errors up to 30 degrees. In the field, the classifier demonstrates accuracy better than 97.5% and 94%, respectively, in the estimation of the wheelbase and lane of motion and can classify vehicles with an average accuracy of over 89.5%.

Time-Optimal Convergence to a Rectilinear Path in the Presence of Wind

This paper considers the problem of determining the time-optimal path of a fixed-wing Miniature Air Vehicle (MAV), in the presence of wind. The MAV, which is subject to a bounded turn rate, is required to eventually converge to a straight line starting from a known initial position and orientation. Earlier work in the literature uses Pontryagin's Minimum Principle (PMP) to solve this problem only for the no-wind case. In contrast, the present work uses a geometric approach to solve the problem completely in the presence of wind. In addition, it also shows how PMP can be used to partially solve the problem. Using a 6-DOF model of a MAV the generated optimal path is tracked by an autopilot consisting of proportional-integral-derivative (PID) controllers. The simulation results show the path generation and tracking for cases with steady and time-varying wind. Some issues on real-time path planning are also addressed.

Optimal Trajectory Generation for Convergence to a Rectilinear Path

This paper presents a strategy to determine the shortest path of a fixed-wing Miniature Air Vehicle (MAV), constrained by a bounded turning rate, to eventually fly along a given straight line, starting from an arbitrary but known initial position and orientation. Unlike the work available in the literature that solves the problem using the Pontryagin's Minimum Principle (PMP) the trajectory generation algorithm presented here considers a geometrical approach which is intuitive and easy to understand. This also computes the explicit solution for the length of the optimal path as a function of the initial configuration. Further, using a 6-DOF model of a MAV the generated optimal path is tracked by an autopilot consisting of proportional-integral-derivative (PID) controllers. The simulation results show the path generation and tracking for different cases.

Rollover-Preventive Force Synthesis at Active Suspensions in a Vehicle Performing a Severe Maneuver With Wheels Lifted Off

Among the intelligent safety technologies for road vehicles, active suspensions controlled by embedded computing elements for preventing rollover have received a lot of attention. The existing models for synthesizing and allocating forces in such suspensions are conservatively based on the constraints that are valid until no wheels lift off the ground. However, the fault tolerance of the rollover-preventive systems can be enhanced if the smart/active suspensions can intervene in the more severe situation in which the wheels have just lifted off the ground. The difficulty in computing control in the last situation is that the vehicle dynamics then passes into the regime that yields a model involving disjunctive constraints on the dynamics. Simulation of dynamics with disjunctive constraints in this context becomes necessary to estimate, synthesize, and allocate the intended hardware realizable forces in an active suspension. In this paper, we give an algorithm for the previously mentioned problem by solving it as a disjunctive dynamic optimization problem. Based on this, we synthesize and allocate the roll-stabilizing time-dependent active suspension forces in terms of sensor output data. We show that the forces obtained from disjunctive dynamics are comparable with existing force allocations and, hence, are possibly realizable in the existing hardware framework toward enhancing the safety and fault tolerance.

Benchmarking sustainability of Indian electricity system: An indicator approach

India needs to significantly increase its electricity consumption levels, in a sustainable manner, if it has to ensure rapid economic development, a goal that remains the most potent tool for delivering adaptation capacity to its poor who will suffer the worst consequences of climate change. Resource/supply constraints faced by conventional energy sources, techno-economic constraints faced by renewable energy sources, and the bounds imposed by climate change on fossil fuel use are likely to undermine India's quest for having a robust electricity system that can effectively contribute to achieving accelerated, sustainable and inclusive economic growth. One possible way out could be transitioning into a sustainable electricity system, which is a trade-off solution having taken into account the economic, social and environmental concerns. As a first step toward understanding this transition, we contribute an indicator based hierarchical multidimensional framework as an analytical tool for sustainability assessment of electricity systems, and validate it for India's national electricity system. We evaluate Indian electricity system using this framework by comparing it with a hypothetical benchmark sustainable electrical system, which was created using best indicator values realized across national electricity systems in the world. This framework, we believe, can be used to examine the social, economic and environmental implications of the current Indian electricity system as well as setting targets for future development. The analysis with the indicator framework provides a deeper understanding of the system, identify and quantify the prevailing sustainability gaps and generate specific targets for interventions. We use this framework to compute national electricity system sustainability index (NESSI) for India. (C) 2014 Elsevier Ltd. All rights reserved.