An actor-critic algorithm with function approximation for discounted cost constrained Markov decision processes

We develop in this article the first actor-critic reinforcement learning algorithm with function approximation for a problem of control under multiple inequality constraints. We consider the infinite horizon discounted cost framework in which both the objective and the constraint functions are suitable expected policy-dependent discounted sums of certain sample path functions. We apply the Lagrange multiplier method to handle the inequality constraints. Our algorithm makes use of multi-timescale stochastic approximation and incorporates a temporal difference (TD) critic and an actor that makes a gradient search in the space of policy parameters using efficient simultaneous perturbation stochastic approximation (SPSA) gradient estimates. We prove the asymptotic almost sure convergence of our algorithm to a locally optimal policy. (C) 2010 Elsevier B.V. All rights reserved.

Chance constrained uncertain classification via robust optimization

This paper studies the problem of constructing robust classifiers when the training is plagued with uncertainty. The problem is posed as a Chance-Constrained Program (CCP) which ensures that the uncertain data points are classified correctly with high probability. Unfortunately such a CCP turns out to be intractable. The key novelty is in employing Bernstein bounding schemes to relax the CCP as a convex second order cone program whose solution is guaranteed to satisfy the probabilistic constraint. Prior to this work, only the Chebyshev based relaxations were exploited in learning algorithms. Bernstein bounds employ richer partial information and hence can be far less conservative than Chebyshev bounds. Due to this efficient modeling of uncertainty, the resulting classifiers achieve higher classification margins and hence better generalization. Methodologies for classifying uncertain test data points and error measures for evaluating classifiers robust to uncertain data are discussed. Experimental results on synthetic and real-world datasets show that the proposed classifiers are better equipped to handle data uncertainty and outperform state-of-the-art in many cases.

Constrained optimization of heat exchangers

Optimizing a shell and tube heat exchanger for a given duty is an important and relatively difficult task. There is a need for a simple, general and reliable method for realizing this task. The authors present here one such method for optimizing single phase shell-and-tube heat exchangers with given geometric and thermohydraulic constraints. They discuss the problem in detail. Then they introduce a basic algorithm for optimizing the exchanger. This algorithm is based on data from an earlier study of a large collection of feasible designs generated for different process specifications. The algorithm ensures a near-optimal design satisfying the given heat duty and geometric constraints. The authors also provide several sub-algorithms to satisfy imposed velocity limitations. They illustrate how useful these sub-algorithms are with several examples where the exchanger weight is minimized.

Finite element simulations of ductile rupture in a constrained metal foil

A numerical study of the ductile rupture in a metal foil constrained between two stiff ceramic blocks is performed. The finite element analysis is carried out under the conditions of mode I, plane strain, small-scale yielding. The rate-independent version of the Gurson model that accounts for the ductile failure mechanisms of microvoid nucleation, growth and coalescence is employed to represent the behavior of the metal foil. Different distributions of void nucleating sites in the metal foil are considered for triggering the initiation of discrete voids. The results clearly show that far-field triaxiality-induced cavitation is the dominant failure mode when the spacing of the void nucleating sites is large. On the contrary, void coalescence near the notch tip is found to be the operative failure mechanism when closely spaced void nucleating sites are considered.

A QFT-based robust SVC controller for improving the dynamic stability of power systems

A new design technique for an SVC-based power system damping controller has been proposed. The controller attempts to place all plant poles within a specified region on the s-plane to guarantee the desired closed loop performance. The use of Horowitz's quantitative feedback theory (QFT) permits the design of a 'fixed gain controller' that maintains its performance in spite of large variations in the plant parameters during its normal course of operation. The required controller parameters are arrived at by solving an optimization problem that incorporates the control specifications. The performance of this robust controller has been evaluated on a single machine infinite bus system equipped with a mid point SVC, and the results are shown to be consistent with the expected performance of the stabilizer. (C) 1998 Elsevier Science S.A. All rights reserved.

A Rotor Flux Estimation During Zero and Active Vector Periods Using Current Error Space Vector From a Hysteresis Controller for a Sensorless Vector Control of IM Drive

This paper proposes a sensorless vector control scheme for general-purpose induction motor drives using the current error space phasor-based hysteresis controller. In this paper, a new technique for sensorless operation is developed to estimate rotor voltage and hence rotor flux position using the stator current error during zero-voltage space vectors. It gives a comparable performance with the vector control drive using sensors especially at a very low speed of operation (less than 1 Hz). Since no voltage sensing is made, the dead-time effect and loss of accuracy in voltage sensing at low speed are avoided here, with the inherent advantages of the current error space phasor-based hysteresis controller. However, appropriate device on-state drops are compensated to achieve a steady-state operation up to less than 1 Hz. Moreover, using a parabolic boundary for current error, the switching frequency of the inverter can be maintained constant for the entire operating speed range. Simple sigma L-s estimation is proposed, and the parameter sensitivity of the control scheme to changes in stator resistance, R-s is also investigated in this paper. Extensive experimental results are shown at speeds less than 1 Hz to verify the proposed concept. The same control scheme is further extended from less than 1 Hz to rated 50 Hz six-step operation of the inverter. Here, the magnetic saturation is ignored in the control scheme.

Modeling and control of unified power flow controller for transient stability

FACTS controllers are emerging as viable and economic solutions to the problems of large interconnected ne networks, which can endanger the system security. These devices are characterized by their fast response, absence of inertia, and minimum maintenance requirements. Thyristor controlled equipment like Thyristor Controlled Series Capacitor (TCSC), Static Var Compensator (SVC), Thyristor Controlled Phase angle Regulator (TCPR) etc. which involve passive elements result in devices of large sizes with substantial cost and significant labour for installation. An all solid-state device using GTOs leads to reduction in equipment size and has improved performance. The Unified Power Flow Controller (UPFC) is a versatile controller which can be used to control the active and reactive power in the Line independently. The concept of UPFC makes it possible to handle practically all power flow control and transmission line compensation problems, using solid-state controllers, which provide functional flexibility, generally not attainable by conventional thyristor controlled systems. In this paper, we present the development of a control scheme for the series injected voltage of the UPFC to damp the power oscillations and improve transient stability in a power system. (C) 1998 Elsevier Science Ltd. All rights reserved.

Robust controller for an autonomous small refrigeration unit

The authors describe the constructional features of a controller for operating an autonomous refrigeration unit powered by a field of photovoltaic panels and backed up by a generator set. The controller enables three voltage levels of operation of an inverter to meet the start, run and off cycle conditions of the refrigerator compressor. The algorithm considers several input and output parameters and status signals from each subsystem of the unit to deduce a control strategy. Such units find application for storage of vaccines and life-saving medicines requiring uninterrupted refrigeration, in medical shops, rural health centres, veterinary laboratories etc.

Synthesis of nonlinear controller to recover an unstable aircraft from poststall regime

Dynamics of the aircraft configuration considered in this paper show a unique characteristic in that there are no stable attractors in the entire high angle-of-attack flight envelope. As a result, once the aircraft has departed from the normal flight regime, no standard technique can be applied to recover the aircraft. In this paper, using feedback linearization technique, a nonlinear controller is designed at high angles of attack, which is engaged after the aircraft departs from normal flight regime. This controller stabilizes the aircraft into a stable spin. Then a set of synthetic pilot inputs is applied to cause an automatic transition from the spin equilibrium to low angles of attack where the second controller is connected. This controller is a normal gain-scheduled controller designed to have a large domain of attraction at low angles of attack. It traps the aircraft into a low angle-of-attack level flight. This entire concept of recovery has been verified using six-degrees-of-freedom nonlinear simulation. Feedback linearization technique used to design a controller ensures internal stability only if the nonlinear plant has stable zero dynamics. Because zero dynamics depend on the selection of outputs, a new method of choosing outputs is described to obtain a plant that has stable zero dynamics. Certain important aspects pertaining to the implementation of a feedback linearization-based controller are also discussed.

A current error space vector based hysteresis controller with constant switching frequency and simple online boundary computation for VSI fed IM drive

This paper proposes a simple current error space vector based hysteresis controller for two-level inverter fed Induction Motor (IM) drives. This proposed hysteresis controller retains all advantages of conventional current error space vector based hysteresis controllers like fast dynamic response, simple to implement, adjacent voltage vector switching etc. The additional advantage of this proposed hysteresis controller is that it gives a phase voltage frequency spectrum exactly similar to that of a constant switching frequency space vector pulse width modulated (SVPWM) inverter. In this proposed hysteresis controller the boundary is computed online using estimated stator voltages along alpha and beta axes thus completely eliminating look up tables used for obtaining parabolic hysteresis boundary proposed in. The estimation of stator voltage is carried out using current errors along alpha and beta axes and steady state model of induction motor. The proposed scheme is simple and capable of taking inverter upto six step mode operation, if demanded by drive system. The proposed hysteresis controller based inverter fed drive scheme is simulated extensively using SIMULINK toolbox of MATLAB for steady state and transient performance. The experimental verification for steady state performance of the proposed scheme is carried out on a 3.7kW IM.

Delay constrained optimal relay placement for planned wireless sensor networks

In this paper, we study the problem of wireless sensor network design by deploying a minimum number of additional relay nodes (to minimize network design cost) at a subset of given potential relay locationsin order to convey the data from already existing sensor nodes (hereafter called source nodes) to a Base Station within a certain specified mean delay bound. We formulate this problem in two different ways, and show that the problem is NP-Hard. For a problem in which the number of existing sensor nodes and potential relay locations is n, we propose an O(n) approximation algorithm of polynomial time complexity. Results show that the algorithm performs efficiently (in over 90% of the tested scenarios, it gave solutions that were either optimal or exceeding optimal just by one relay) in various randomly generated network scenarios.

Throughput Maximization of Delay-Constrained Traffic in Wireless Energy Harvesting Sensors

A wireless Energy Harvesting Sensor (EHS) needs to send data packets arriving in its queue over a fading channel at maximum possible throughput while ensuring acceptable packet delays. At the same time, it needs to ensure that energy neutrality is satisfied, i.e., the average energy drawn from a battery should equal the amount of energy deposited in it minus the energy lost due to the inefficiency of the battery. In this work, a framework is developed under which a system designer can optimize the performance of the EHS node using power control based on the current channel state information, when the EHS node employs a single modulation and coding scheme and the channel is Rayleigh fading. Optimal system parameters for throughput optimal, delay optimal and delay-constrained throughput optimal policies that ensure energy neutrality are derived. It is seen that a throughput optimal (maximal) policy is packet delay-unbounded and an average delay optimal (minimal) policy achieves negligibly small throughput. Finally, the influence of the harvested energy profile on the performance of the EHS is illustrated through the example of solar energy harvesting.