A novel Q-learning algorithm with function approximation for constrained Markov decision processes

We present a novel multi-timescale Q-learning algorithm for average cost control in a Markov decision process subject to multiple inequality constraints. We formulate a relaxed version of this problem through the Lagrange multiplier method. Our algorithm is different from Q-learning in that it updates two parameters - a Q-value parameter and a policy parameter. The Q-value parameter is updated on a slower time scale as compared to the policy parameter. Whereas Q-learning with function approximation can diverge in some cases, our algorithm is seen to be convergent as a result of the aforementioned timescale separation. We show the results of experiments on a problem of constrained routing in a multistage queueing network. Our algorithm is seen to exhibit good performance and the various inequality constraints are seen to be satisfied upon convergence of the algorithm.

Multi-resistance states in the electrical switching behavior of amorphous Si15Te75Ge10 thin films: Possibility of multi-bit storage

Electrical switching studies on amorphous Si15Te75Ge10 thin film devices reveal the existence of two distinct, stable low-resistance, SET states, achieved by varying the electrical input to the device. The multiple resistance levels can be attributed to multi-stage crystallization, as observed from temperature dependant resistance studies. The devices are tested for their ability to be RESET with minimal resistance degradation; further, they exhibit a minimal drift in the SET resistance value even after several months of switching. (c) 2013 Elsevier B.V. All rights reserved.

pH Induced switching in hydrogel coated fiber bragg grating sensor

In this paper we report a novel hydrogel functionalized optical Fiber Bragg Grating (FBG) sensor based on chemo-mechanical-optical sensing, and demonstrate its specific application in pH activated process monitoring. The sensing mechanism is based on the stress due to ion diffusion and polymer phase transition which produce strain in the FBG. This results in shift in the Bragg wavelength which is detected by an interrogator system. A simple dip coating method to coat a thin layer of hydrogel on the FBG has been established. The gel consists of sodium alginate and calcium chloride. Gel formation is observed in real-time by continuously monitoring the Bragg wavelength shift. We have demonstrated pH sensing in the range of pH of 2 to 10. Another interesting phenomenon is observed by swelling and deswelling of FBG functionalized with hydrogel by a sequence of alternate dipping between acidic and base solutions. It is observed that the Bragg wavelength undergoes reversible and repeatable pH dependent switching.

A Markov Decision Theoretic Approach to Pilot Allocation and Receive Antenna Selection

This paper considers antenna selection (AS) at a receiver equipped with multiple antenna elements but only a single radio frequency chain for packet reception. As information about the channel state is acquired using training symbols (pilots), the receiver makes its AS decisions based on noisy channel estimates. Additional information that can be exploited for AS includes the time-correlation of the wireless channel and the results of the link-layer error checks upon receiving the data packets. In this scenario, the task of the receiver is to sequentially select (a) the pilot symbol allocation, i.e., how to distribute the available pilot symbols among the antenna elements, for channel estimation on each of the receive antennas; and (b) the antenna to be used for data packet reception. The goal is to maximize the expected throughput, based on the past history of allocation and selection decisions, and the corresponding noisy channel estimates and error check results. Since the channel state is only partially observed through the noisy pilots and the error checks, the joint problem of pilot allocation and AS is modeled as a partially observed Markov decision process (POMDP). The solution to the POMDP yields the policy that maximizes the long-term expected throughput. Using the Finite State Markov Chain (FSMC) model for the wireless channel, the performance of the POMDP solution is compared with that of other existing schemes, and it is illustrated through numerical evaluation that the POMDP solution significantly outperforms them.

Electrical switching in amorphous Si-Te-Ge thin films: Impact of input energy on crystallization process and switching parameters

Electrical switching studies on amorphous Si15Te74Ge11 thin film devices show interesting changes in the switching behavior with changes in the input energy supplied; the input energy determines the extent of crystallization in the active volume, which is reflected in the value of SET resistances. This in turn, determines the trend exhibited by switching voltage (V-t) for different input conditions. The results obtained are analyzed on the basis of the amount of Joule heat generated, which determines the temperature of the active volume. Depending on the final temperature, devices are rendered either in the intermediate state with a resistance of 5*10(2) Omega or the ON state with a resistance of 5*10(1) Omega. (C) 2013 Elsevier B.V. All rights reserved.

A nearly constant switching frequency current hysteresis controller with generalized parabolic boundaries using 12-sided polygonal voltage space vectors for IM drives

In this paper, a current hysteresis controller with parabolic boundaries for a 12-sided polygonal voltage space vector inverter fed induction motor (IM) drive is proposed. Parabolic boundaries with generalized vector selection logic, valid for all sectors and rotational direction, is used in the proposed controller. The current error space phasor boundary is obtained by first studying the drive scheme with space vector based PWM (SVPWM) controller. Four parabolas are used to approximate this current error space phasor boundary. The system is then run with space phasor based hysteresis PWM controller by limiting the current error space vector (CESV) within the parabolic boundary. The proposed controller has simple controller implementation, nearly constant switching frequency, extended modulation range and fast dynamic response with smooth transition to the over modulation region.

A Medium-Voltage Inverter-Fed IM Drive Using Multilevel 12-Sided Polygonal Vectors, With Nearly Constant Switching Frequency Current Hysteresis Controller

In this paper, a current error space vector (CESV)-based hysteresis current controller for a multilevel 12-sided voltage space vector-based inverter-fed induction motor (IM) drive is proposed. The proposed controller gives a nearly constant switching frequency operation throughout different speeds in the linear modulation region. It achieves the elimination of 6n +/- 1, n = odd harmonics from the phase voltages and currents in the entire modulation range, with an increase in the linear modulation range. It also exhibits fast dynamic behavior under different transient conditions and has a simple controller implementation. Nearly constant switching frequency is obtained by matching the steady-state CESV boundaries of the proposed controller with that of a constant switching frequency SVPWM-based drive. In the proposed controller, the CESV reference boundaries are computed online, using the switching dwell time and voltage error vector of each applied vector. These quantities are calculated from estimated sampled reference phase voltages. Vector change is decided by projecting the actual current error along the computed hysteresis space vector boundary of the presently applied vector. The estimated reference phase voltages are found from the stator current error ripple and the parameters of the IM.

Switching and release dynamics of an electrostatically actuated MEMS switch under the influence of squeeze-film damping

This paper reports analytical modeling, simulation and experimental validation for switching and release times of an electrostatically actuated micromachined switch. Presented work is an extension of our earlier work [1] that analytically argued, and numerically and experimentally demonstrated, why pull-in time is larger that pull-up time when the actuation voltage is less than twice of the pull-in voltage. In this paper, switching dynamics is investigated under the influence of squeeze-film damping. Tests were performed on SOI (silicon-on-insulator) based parallel beams structures.Typical voltage requirement for actuation is in the range of 10-30 V. All the experiments were performed in normal atmospheric pressure. Measurement results confirm that the quality factor Q has appreciable effect on the release time compared to the switching time. The quality factor Q is extracted from the response measurement and compared with the ANSYS simulation result. In addition, the dynamic pull-in effect has also been studied and reported in this paper. A contribution of this work includes the effect of various phenomena such as squeeze-film damping, dynamic pull-in, and frequency pull-in effects on the switching dynamics of a MEMS switch.

Design of double toggle switching mechanisms

Toggle mechanisms are ubiquitous in electrical switches. However, literature for their mechanical design is scarce. This paper defines and classifies the toggle phenomena observed during switching. The concept of double toggle introduced in this paper enables a systematic screening of kinematic structure for the suitability in high performance switches. Seven structural and three kinematic criteria are identified for this purpose. It is also demonstrated that each such feasible kinematic structure lends itself to multiple physical embodiments. Therefore, the theory and procedure presented in this work can be used for design of numerous kinematically distinct mechanisms. One representative mechanical embodiment for a novel double toggle switch, including mass and geometric shape of links has been included in the paper. The switching behavior of the design is validated using Pro/Mechanism (TM). (C) 2013 Elsevier Ltd. All rights reserved.

Comparative Evaluation of Switching and Average Models of a DC-DC Boost Converter for Real-Time Simulation

This paper presents a comparative evaluation of the average and switching models of a dc-dc boost converter from the point of view of real-time simulation. Both the models are used to simulate the converter in real-time on a Field Programmable Gate Array (FPGA) platform. The converter is considered to function over a wide range of operating conditions, and could do transition between continuous conduction mode (CCM) and discontinuous conduction mode (DCM). While the average model is known to be computationally efficient from the perspective of off-line simulation, the same is shown here to consume more logical resources than the switching model for real-time simulation of the dc-dc converter. Further, evaluation of the boundary condition between CCM and DCM is found to be the main reason for the increased consumption of resources by the average model.

Electrical switching behavior of amorphous Ge15Te85 Si--x(x) thin films with phase change memory applications

Amorphous Ge15Te85-xSix thin film switching devices (1 <= x <= 6) have been deposited in sandwich geometry, on glass substrates with aluminum electrodes, by flash evaporation technique. These devices exhibit memory type electrical switching, like bulk Ge15Te85-xSix glasses. However, unlike the bulk glasses, a-Ge15Te85-xSix films exhibit a smooth electrical switching behavior. The electrical switching fields of a-Ge15Te85-xSix thin film samples are also comparable with other chalcogenide samples used in memory applications. The switching fields of a-Ge15Te85-xSix films have been found to increase with increasing Si concentration. Also, the optical band gap of a-Ge15Te85-xSix films is found to increase with Si content. The observed results have been understood on the basis of increase in network connectivity and rigidity with Si addition. (C) 2013 Elsevier Ltd. All rights reserved.

Nearly Constant Switching Frequency Hysteresis Current Controller with Fast Online Computation of Boundary for a 2-Level Induction Motor Drive

A nearly constant switching frequency current hysteresis controller for a 2-level inverter fed induction motor drive is proposed in this paper: The salient features of this controller are fast dynamics for the current, inherent protection against overloads and less switching frequency variation. The large variation of switching frequency as in the conventional hysteresis controller is avoided by defining a current-error boundary which is obtained from the current-error trajectory of the standard space vector PWM. The current-error boundary is computed at every sampling interval based on the induction machine parameters and from the estimated fundamental stator voltage. The stator currents are always monitored and when the current-error exceeds the boundary, voltage space vector is switched to reduce the current-error. The proposed boundary computation algorithm is applicable in linear and over-modulation region and it is simple to implement in any standard digital signal processor: Detailed experimental verification is done using a 7.5 kW induction motor and the results are given to show the performance of the drive at various operating conditions and validate the proposed advantages.

Analytical Closed-Form Expressions for Harmonic Distortion Corresponding to Novel Switching Sequences for Neutral-Point-Clamped Inverters

Analytical closed-form expressions for harmonic distortion factors corresponding to various pulsewidth modulation (PWM) techniques for a two-level inverter have been reported in the literature. This paper derives such analytical closed-form expressions, pertaining to centered space-vector PWM (CSVPWM) and eight different advanced bus-clamping PWM (ABCPWM) schemes, for a three-level neutral-point-clamped (NPC) inverter. These ABCPWM schemes switch each phase at twice the nominal switching frequency in certain intervals of the line cycle while clamping each phase to one of the dc terminals over certain other intervals. The harmonic spectra of the output voltages, corresponding to the eight ABCPWM schemes, are studied and compared experimentally with that of CSVPWM over the entire modulation range. The measured values of weighted total harmonic distortion (WTHD) of the line voltage V-WTHD are used to validate the analytical closed-form expressions derived. The analytical expressions, pertaining to two of the ABCPWM methods, are also validated by measuring the total harmonic distortion (THD) in the line current I-THD on a 2.2-kW constant volts-per-hertz induction motor drive.

Electrical switching, SET-RESET, and Raman scattering studies on Ge15Te80-xIn5Agx glasses

Bulk Ge15Te85-xIn5Agx glasses are shown to exhibit electrical switching with switching/threshold voltages in the range of 70-120V for a sample thickness of 0.3 mm. Further, the samples exhibit threshold or memory behavior depending on the ON state current. The compositional studies confirm the presence of an intermediate phase in the range 8 <= x <= 16, revealed earlier by thermal studies. Further, SET-RESET studies have been performed by these glasses using a triangular pulse of 6 mA amplitude (for SET) and 21 mA amplitude (for RESET). Raman studies of the samples after the SET and RESET operations reveal that the SET state is a crystalline phase which is obtained by thermal annealing and the RESET state is the glassy state, similar to the as-quenched samples. It is interesting to note that the samples in the intermediate phase, especially compositions at x = 10, 12, and 14 withstand more set-reset cycles. This indicates compositions in the intermediate phase are better suited for phase change memory applications. (C) 2014 AIP Publishing LLC.

Minimum switching loss pulse width modulation for reduced power conversion loss in reactive power compensators

Advanced bus-clamping switching sequences, which employ an active vector twice in a subcycle, are used to reduce line current distortion and switching loss in a space vector modulated voltage source converter. This study evaluates minimum switching loss pulse width modulation (MSLPWM), which is a combination of such sequences, for static reactive power compensator (STATCOM) application. It is shown that MSLPWM results in a significant reduction in device loss over conventional space vector pulse width modulation. Experimental verification is presented at different power levels of up to 150 kVA.