An Online Actor-Critic Algorithm with Function Approximation for Constrained Markov Decision Processes

We develop an online actor-critic reinforcement learning algorithm with function approximation for a problem of control under inequality constraints. We consider the long-run average cost Markov decision process (MDP) framework in which both the objective and the constraint functions are suitable policy-dependent long-run averages of certain sample path functions. The Lagrange multiplier method is used to handle the inequality constraints. We prove the asymptotic almost sure convergence of our algorithm to a locally optimal solution. We also provide the results of numerical experiments on a problem of routing in a multi-stage queueing network with constraints on long-run average queue lengths. We observe that our algorithm exhibits good performance on this setting and converges to a feasible point.

Online Computation of Hysteresis Boundary for Constant Switching Frequency Current-Error Space-Vector-Based Hysteresis Controller for VSI-Fed IM Drives

This paper proposes a current-error space-vector-based hysteresis controller with online computation of boundary for two-level inverter-fed induction motor (IM) drives. The proposed hysteresis controller has got all advantages of conventional current-error space-vector-based hysteresis controllers like quick transient response, simplicity, adjacent voltage vector switching, etc. Major advantage of the proposed controller-based voltage-source-inverters-fed drive is that phase voltage frequency spectrum produced is exactly similar to that of a constant switching frequency space-vector pulsewidth modulated (SVPWM) inverter. In this proposed hysteresis controller, stator voltages along alpha- and beta-axes are estimated during zero and active voltage vector periods using current errors along alpha- and beta-axes and steady-state model of IM. Online computation of hysteresis boundary is carried out using estimated stator voltages in the proposed hysteresis controller. The proposed scheme is simple and capable of taking inverter upto six-step-mode operation, if demanded by drive system. The proposed hysteresis-controller-based inverter-fed drive scheme is experimentally verified. The steady state and transient performance of the proposed scheme is extensively tested. The experimental results are giving constant frequency spectrum for phase voltage similar to that of constant frequency SVPWM inverter-fed drive.

Optimal Control of Markov Processes with Age-Dependent Transition Rates

We study optimal control of Markov processes with age-dependent transition rates. The control policy is chosen continuously over time based on the state of the process and its age. We study infinite horizon discounted cost and infinite horizon average cost problems. Our approach is via the construction of an equivalent semi-Markov decision process. We characterise the value function and optimal controls for both discounted and average cost cases.

Markov chain modeling of evolution of strains in reinforced concrete flexural beams

From the analysis of experimentally observed variations in surface strains with loading in reinforced concrete beams, it is noted that there is a need to consider the evolution of strains (with loading) as a stochastic process. Use of Markov Chains for modeling stochastic evolution of strains with loading in reinforced concrete flexural beams is studied in this paper. A simple, yet practically useful, bi-level homogeneous Gaussian Markov Chain (BLHGMC) model is proposed for determining the state of strain in reinforced concrete beams. The BLHGMC model will be useful for predicting behavior/response of reinforced concrete beams leading to more rational design.

Unusual Salt-Induced Color Modulation through Aggregation-Induced Emission Switching of a Bis-cationic Phenylenedivinylene-Based pi Hydrogelator

The synthesis, hydrogelation, and aggregation-induced emission switching of the phenylenedivinylene bis-N-octyl pyridinium salt is described. Hydrogelation occurs as a consequence of pi-stacking, van der Waals, and electrostatic interactions that lead to a high gel melting temperature and significant mechanical properties at a very low weight percentage of the gelator. A morphology transition from fiber-to-coil-to-tube was observed depending on the concentration of the gelator. Variation in the added salt type, salt concentrations, or temperature profoundly influenced the order of aggregation of the gelator molecules in aqueous solution. Formation of a novel chromophore assembly in this way leads to an aggregation-induced switch of the emission colors. The emission color switches from sky blue to white to orange depending upon the extent of aggregation through mere addition of external inorganic salts. Remarkably, the salt effect on the assembly of such cationic phenylenedivinylenes in water follow the behavior predicted from the well-known Hofmeister effects. Mechanistic insights for these aggregation processes were obtained through the counterion exchange studies. The aggregation-induced emission switching that leads to a room-temperature white-light emission from a single chromophore in a single solvent (water) is highly promising for optoelectronic applications.

Methodology for Pavement Design Reliability and Back Analysis Using Markov Chain Monte Carlo Simulation

Given the increasing cost of designing and building new highway pavements, reliability analysis has become vital to ensure that a given pavement performs as expected in the field. Recognizing the importance of failure analysis to safety, reliability, performance, and economy, back analysis has been employed in various engineering applications to evaluate the inherent uncertainties of the design and analysis. The probabilistic back analysis method formulated on Bayes' theorem and solved using the Markov chain Monte Carlo simulation method with a Metropolis-Hastings algorithm has proved to be highly efficient to address this issue. It is also quite flexible and is applicable to any type of prior information. In this paper, this method has been used to back-analyze the parameters that influence the pavement life and to consider the uncertainty of the mechanistic-empirical pavement design model. The load-induced pavement structural responses (e.g., stresses, strains, and deflections) used to predict the pavement life are estimated using the response surface methodology model developed based on the results of linear elastic analysis. The failure criteria adopted for the analysis were based on the factor of safety (FOS), and the study was carried out for different sample sizes and jumping distributions to estimate the most robust posterior statistics. From the posterior statistics of the case considered, it was observed that after approximately 150 million standard axle load repetitions, the mean values of the pavement properties decrease as expected, with a significant decrease in the values of the elastic moduli of the expected layers. An analysis of the posterior statistics indicated that the parameters that contribute significantly to the pavement failure were the moduli of the base and surface layer, which is consistent with the findings from other studies. After the back analysis, the base modulus parameters show a significant decrease of 15.8% and the surface layer modulus a decrease of 3.12% in the mean value. The usefulness of the back analysis methodology is further highlighted by estimating the design parameters for specified values of the factor of safety. The analysis revealed that for the pavement section considered, a reliability of 89% and 94% can be achieved by adopting FOS values of 1.5 and 2, respectively. The methodology proposed can therefore be effectively used to identify the parameters that are critical to pavement failure in the design of pavements for specified levels of reliability. DOI: 10.1061/(ASCE)TE.1943-5436.0000455. (C) 2013 American Society of Civil Engineers.

Switching of a Sixteen Electrode Array for Wireless EIT System Using a RF-Based 8-Bit Digital Data Transmission Technique

Surface electrode switching of 16-electrode wireless EIT is studied using a Radio Frequency (RF) based digital data transmission technique operating with 8 channel encoder/decoder ICs. An electrode switching module is developed the analog multiplexers and switched with 8-bit parallel digital data transferred by transmitter/receiver module developed with radio frequency technology. 8-bit parallel digital data collected from the receiver module are converted to 16-bit digital data by using binary adder circuits and then used for switching the electrodes in opposite current injection protocol. 8-bit parallel digital data are generated using NI USB 6251 DAQ card in LabVIEW software and sent to the transmission module which transmits the digital data bits to the receiver end. Receiver module supplies the parallel digital bits to the binary adder circuits and adder circuit outputs are fed to the multiplexers of the electrode switching module for surface electrode switching. 1 mA, 50 kHz sinusoidal constant current is injected at the phantom boundary using opposite current injection protocol. The boundary potentials developed at the voltage electrodes are measured and studied to assess the wireless data transmission.

Nearly constant switching frequency hysteresis current controller for general n-level inverter fed induction motor drive

A current-error space phasor based hysteresis controller with nearly constant switching frequency is proposed for a general n-level voltage source inverter fed three-phase induction motor drive. Like voltage-controlled space vector PWM (SVPWM), the proposed controller can precisely detect sub-sector changes and for switching it selects only the nearest switching voltage vectors using the information of the estimated fundamental stator voltages along α and β axes. It provides smooth transition between voltage levels, including operation in over modulation region. Due to adjacent switching amongst the nearest switching vectors forming a triangular sub-sector, in which tip of the fundamental stator voltage vector of the machine lies, switching loss is reduced while keeping the current-error space phasor within the varying parabolic boundary. Appropriate dimension and orientation of this parabolic boundary ensures similar switching frequency spectrum like constant switching frequency SVPWM-based induction motor (IM) drive. Inherent advantages of multi-level inverter and space phasor based current hysteresis controller are retained. The proposed controller is simulated as well as implemented on a 5-level inverter fed 7.5 kW open-end winding IM drive.

A Space-Vector-Based Hysteresis Current Controller for a General n-Level Inverter-Fed Drive With Nearly Constant Switching Frequency Control

A current-error space-vector-based hysteresis current controller for a general n-level voltage-source inverter (VSI)-fed three-phase induction motor (IM) drive is proposed here, with control of the switching frequency variation for the full linear modulation range. The proposed current controller monitors the space-vector-based current error of an n-level VSI-fed IM to keep the current error within a parabolic boundary, using the information of the current triangular sector in which the tip of the reference vector lies. Information of the reference voltage vector is estimated using the measured current-error space vectors, along the alpha- and beta-axes. Appropriate dimension and orientation of this parabolic boundary ensure a switching frequency spectrum similar to that of a constant-switching-frequency voltage-controlled space vector pulsewidth modulation (PWM) (SVPWM)-based IM drive. Like SVPWM for multilevel inverters, the proposed controller selects inverter switching vectors, forming a triangular sector in which the tip of the reference vector stays, for the hysteresis PWM control. The sector in the n-level inverter space vector diagram, in which the tip of the fundamental stator voltage stays, is precisely detected, using the sampled reference space vector estimated from the instantaneous current-error space vectors. The proposed controller retains all the advantages of a conventional hysteresis controller such as fast current control, with smooth transition to the overmodulation region. The proposed controller is implemented on a five-level VSI-fed 7.5-kW IM drive.

Non-Stationary Semi-Markov Decision Processes on a Finite Horizon

We introduce and study a class of non-stationary semi-Markov decision processes on a finite horizon. By constructing an equivalent Markov decision process, we establish the existence of a piecewise open loop relaxed control which is optimal for the finite horizon problem.

Parametric study of the behavior of double toggle switching mechanisms

The paper identified and characterized a special multi-degree of freedom toggle behavior, called double toggle, observed in a typical MCCB switching mechanism. For an idealized system, the condition of toggle sequence is derived geometrically. The existing tools available in a multi-body dynamics package are used for exploring the dynamic behavior of such systems parametrically. The double toggle mechanism is found to make the system insensitive to the operator's behavior; however, the system is vulnerable under extreme usage. The linkage kinematics and stopper locations are found to have dominant role on the behavior of the system. It is revealed that the operating time is immune to the inertial property of the input link and sensitive to that of the output link. Novel designs exploiting this observation, in terms of spring and toggle placements, to enhance switching performance have also been reported in the paper. Detailed study revealed that strategic placement of the spring helps in selective alteration of system performance. Thus, the study establishes the critical importance of the kinematic design of MCCB over the dynamic parameters. (C) 2013 Elsevier Ltd. All rights reserved.

Electrical switching studies on amorphous Ge‐Te‐Sn thin films

Electrical switching studies on amorphous Ge17Te83−xSnx thin films (1 ≤ x ≤ 4) has been done to find their suitability for Phase Change Memory application; Bulk ingots in glassy form are prepared using conventional melt quenching technique and the thin films are coated using flash evaporation technique. Samples are found to exhibit memory type of electrical switching behavior. The switching voltages of Ge17Te83−xSnx thin films have been found to decrease with increase in Sn concentration. The comparatively lower switching voltages of Ge17Te83−xSnx samples, make them suitable candidates for phase change memory applications.

Measurement of IGBT switching characteristics and loss using coaxial current transformer

Device switching times and switching energy losses are required over a wide range of practical working conditions for successful design of insulated gate bipolar transistor (IGBT) based power converters. This paper presents a cost-effective experimental setup using a co-axial current transformer for measurement of IGBT switching characteristics and switching energy loss. Measurements are carried out on a 50A, 1200V IGBT (SKM50GB123D) for different values of gate resistance, device current and junction temperature. These measurements augment the technical data available in the device datasheet.Short circuit transients are also investigated experimentally under hard switched fault as well as fault under load conditions.

Experimental investigation on switching characteristics of IGBTs for traction application

In traction application, inverters need to have high reliability on account of wide variation in operating conditions, extreme ambient conditions, thermal cycling and varying DC link voltage. Hence it is important to have a good knowledge of switching characteristics of the devices used. The focus of this paper is to investigate and compare switching characteristics and losses of IGBT modules for traction application. Dependence of device transition times and switching energy losses on dc link voltage, device current and operating temperature is studied experimentally.

Control of switching surges due to energization of 765kV UHV transmission line

Reduction of switching surge over voltages allows an economic design of UHV transmission system with reduced insulation. The various means of switching surge over voltage control with pre-insertion resistors/closing resistors, shunt re-actors and controlled switching are illustrated. The switching surge over voltages during the energization of series compensated line are compared with uncompensated line. An Electromagnetic transients program has been developed for studying the effect of various means of control of switching transients during 765kV UHV transmission line energization. This paper presents the studies carried out on switching surges control in 765kV UHV transmission line energization.