An actor-critic algorithm with function approximation for discounted cost constrained Markov decision processes

We develop in this article the first actor-critic reinforcement learning algorithm with function approximation for a problem of control under multiple inequality constraints. We consider the infinite horizon discounted cost framework in which both the objective and the constraint functions are suitable expected policy-dependent discounted sums of certain sample path functions. We apply the Lagrange multiplier method to handle the inequality constraints. Our algorithm makes use of multi-timescale stochastic approximation and incorporates a temporal difference (TD) critic and an actor that makes a gradient search in the space of policy parameters using efficient simultaneous perturbation stochastic approximation (SPSA) gradient estimates. We prove the asymptotic almost sure convergence of our algorithm to a locally optimal policy. (C) 2010 Elsevier B.V. All rights reserved.

Integral treatment for the representation of thermodynamic properties in multicomponent systems using interaction parameters

The partial thermodynamic functions of the solvent component of a ternary system have been deduced in terms of the interaction parameters by integration of several series which emerge from the Maclaurin infinite series based on the integral property of the system and subjected to appropriate boundary conditions. The series integration shows that the resulting partial functions are suitable for interpreting the thermodynamic properties of the system and are independent of compositional paths. In the present analysis, the higher order terms of these series are found to make insignificant contributions.

A QFT-based robust SVC controller for improving the dynamic stability of power systems

A new design technique for an SVC-based power system damping controller has been proposed. The controller attempts to place all plant poles within a specified region on the s-plane to guarantee the desired closed loop performance. The use of Horowitz's quantitative feedback theory (QFT) permits the design of a 'fixed gain controller' that maintains its performance in spite of large variations in the plant parameters during its normal course of operation. The required controller parameters are arrived at by solving an optimization problem that incorporates the control specifications. The performance of this robust controller has been evaluated on a single machine infinite bus system equipped with a mid point SVC, and the results are shown to be consistent with the expected performance of the stabilizer. (C) 1998 Elsevier Science S.A. All rights reserved.

On the optimality of exhaustive service policies in multiclass queueing systems with modulated arrivals and switchovers

Consider a single-server multiclass queueing system with K classes where the individual queues are fed by K-correlated interrupted Poisson streams generated in the states of a K-state stationary modulating Markov chain. The service times for all the classes are drawn independently from the same distribution. There is a setup time (and/or a setup cost) incurred whenever the server switches from one queue to another. It is required to minimize the sum of discounted inventory and setup costs over an infinite horizon. We provide sufficient conditions under which exhaustive service policies are optimal. We then present some simulation results for a two-class queueing system to show that exhaustive, threshold policies outperform non-exhaustive policies.

Representation of thermodynamic properties of quaternary systems using interaction parameters

Integral excess free energy of a quaternary system has been expressed in terms of the MacLaurin infinite series. The series is subjected to appropriate boundary conditions and each of the derivatives correlated to the corresponding interaction coefficients. The derivation of the partial functions involves extensive summation of various infinite series pertaining to the first order and quaternary parameters to remove any truncational error. The thermodynamic consistency of the derived partials has been established based on the Gibbs-Duhem relations. The equations are used to interpret the thermodynamic properties of the Fe-Cr-Ni-N system.

Efficient algorithms for learning kernels from multiple similarity matrices with general convex loss functions

In this paper we consider the problem of learning an n × n kernel matrix from m(1) similarity matrices under general convex loss. Past research have extensively studied the m = 1 case and have derived several algorithms which require sophisticated techniques like ACCP, SOCP, etc. The existing algorithms do not apply if one uses arbitrary losses and often can not handle m > 1 case. We present several provably convergent iterative algorithms, where each iteration requires either an SVM or a Multiple Kernel Learning (MKL) solver for m > 1 case. One of the major contributions of the paper is to extend the well knownMirror Descent(MD) framework to handle Cartesian product of psd matrices. This novel extension leads to an algorithm, called EMKL, which solves the problem in O(m2 log n 2) iterations; in each iteration one solves an MKL involving m kernels and m eigen-decomposition of n × n matrices. By suitably defining a restriction on the objective function, a faster version of EMKL is proposed, called REKL,which avoids the eigen-decomposition. An alternative to both EMKL and REKL is also suggested which requires only an SVMsolver. Experimental results on real world protein data set involving several similarity matrices illustrate the efficacy of the proposed algorithms.

A Nonlinear Voltage Regulator With One Tunable Parameter for Multimachine Power Systems

This paper proposes a nonlinear voltage regulator with one tunable parameter for multimachine power systems. Based on output feedback linearization, this regulator can achieve simultaneous voltage regulation and small-signal performance objectives. Conventionally output feedback linearization has been used for voltage regulator design by taking infinite bus voltage as reference. Unfortunately, this controller has poor small-signal performance and cannot be applied to multimachine systems without the estimation of the equivalent external reactance seen from the generator. This paper proposes a voltage regulator design by redefining the rotor angle at each generator with respect to the secondary voltage of the step-up transformer as reference instead of a common synchronously rotating reference frame. Using synchronizing and damping torques analysis, we show that the proposed voltage regulator achieves simultaneous voltage regulation and damping performance over a range of system and operating conditions by controlling the relative angle between the generator internal voltage angle delta and the secondary voltage of the step up transformer. The performance of the proposed voltage regulator is evaluated on a single machine infinite bus system and two widely used multimachine test systems.

Precoder Designs for MIMO Broadcast Channels with Imperfect CSI

[1] D. Tse and P. Viswanath, Fundamentals of Wireless Communication.Cambridge University Press, 2006. [2] H. Bolcskei, D. Gesbert, C. B. Papadias, and A.-J. van der Veen, Spacetime Wireless Systems: From Array Processing to MIMO Communications.Cambridge University Press, 2006. [3] Q. H. Spencer, C. B. Peel, A. L. Swindlehurst, and M. Haardt, “An introduction to the multiuser MIMO downlink,” IEEE Commun. Mag.,vol. 42, pp. 60–67, Oct. 2004. [4] K. Kusume, M. Joham,W. Utschick, and G. Bauch, “Efficient tomlinsonharashima precoding for spatial multiplexing on flat MIMO channel,”in Proc. IEEE ICC’2005, May 2005, pp. 2021–2025. [5] R. Fischer, C. Windpassinger, A. Lampe, and J. Huber, “MIMO precoding for decentralized receivers,” in Proc. IEEE ISIT’2002, 2002, p.496. [6] M. Schubert and H. Boche, “Iterative multiuser uplink and downlink beamforming under SINR constraints,” IEEE Trans. Signal Process.,vol. 53, pp. 2324–2334, Jul. 2005. [7] ——, “Solution of multiuser downlink beamforming problem with individual SINR constraints,” IEEE Trans. Veh. Technol., vol. 53, pp.18–28, Jan. 2004. [8] A. Wiesel, Y. C. Eldar, and Shamai, “Linear precoder via conic optimization for fixed MIMO receivers,” IEEE Trans. Signal Process., vol. 52,pp. 161–176, Jan. 2006. [9] N. Jindal, “MIMO broadcast channels with finite rate feed-back,” in Proc. IEEE GLOBECOM’2005, Nov. 2005. [10] R. Hunger, F. Dietrich, M. Joham, and W. Utschick, “Robust transmit zero-forcing filters,” in Proc. ITG Workshop on Smart Antennas, Munich,Mar. 2004, pp. 130–137. [11] M. B. Shenouda and T. N. Davidson, “Linear matrix inequality formulations of robust QoS precoding for broadcast channels,” in Proc.CCECE’2007, Apr. 2007, pp. 324–328. [12] M. Payaro, A. Pascual-Iserte, and M. A. Lagunas, “Robust power allocation designs for multiuser and multiantenna downlink communication systems through convex optimization,” IEEE J. Sel. Areas Commun.,vol. 25, pp. 1392–1401, Sep. 2007. [13] M. Biguesh, S. Shahbazpanahi, and A. B. Gershman, “Robust downlink power control in wireless cellular systems,” EURASIP Jl. Wireless Commun. Networking, vol. 2, pp. 261–272, 2004. [14] B. Bandemer, M. Haardt, and S. Visuri, “Liner MMSE multi-user MIMO downlink precoding for users with multple antennas,” in Proc.PIMRC’06, Sep. 2006, pp. 1–5. [15] J. Zhang, Y. Wu, S. Zhou, and J. Wang, “Joint linear transmitter and receiver design for the downlink of multiuser MIMO systems,” IEEE Commun. Lett., vol. 9, pp. 991–993, Nov. 2005. [16] S. Shi, M. Schubert, and H. Boche, “Downlink MMSE transceiver optimization for multiuser MIMO systems: Duality and sum-mse minimization,”IEEE Trans. Signal Process., vol. 55, pp. 5436–5446, Nov.2007. [17] A. Mezghani, M. Joham, R. Hunger, and W. Utschick, “Transceiver design for multi-user MIMO systems,” in Proc. WSA 2006, Mar. 2006. [18] R. Doostnejad, T. J. Lim, and E. Sousa, “Joint precoding and beamforming design for the downlink in a multiuser MIMO system,” in Proc.WiMob’2005, Aug. 2005, pp. 153–159. [19] N. Vucic, H. Boche, and S. Shi, “Robust transceiver optimization in downlink multiuser MIMO systems with channel uncertainty,” in Proc.IEEE ICC’2008, Beijing, China, May 2008. [20] A. Ben-Tal and A. Nemirovsky, “Selected topics in robust optimization,”Math. Program., vol. 112, pp. 125–158, Feb. 2007. [21] D. Bertsimas and M. Sim, “Tractable approximations to robust conic optimization problems,” Math. Program., vol. 107, pp. 5–36, Jun. 2006. [22] P. Ubaidulla and A. Chockalingam, “Robust Transceiver Design for Multiuser MIMO Downlink,” in Proc. IEEE Globecom’2008, New Orleans, USA, Dec. 2008, to appear. [23] S. Boyd and L. Vandenberghe, Convex Optimization. Cambridge University Press, 2004. [24] G. H. Golub and C. F. V. Loan, Matrix Computations. The John Hopkins University Press, 1996.

Pricing a Shared Access Link for Fair and Efficient Operation with Variable User Data Rates

The literature on pricing implicitly assumes an "infinite data" model, in which sources can sustain any data rate indefinitely. We assume a more realistic "finite data" model, in which sources occasionally run out of data; this leads to variable user data rates. Further, we assume that users have contracts with the service provider, specifying the rates at which they can inject traffic into the network. Our objective is to study how prices can be set such that a single link can be shared efficiently and fairly among users in a dynamically changing scenario where a subset of users occasionally has little data to send. User preferences are modelled by concave increasing utility functions. Further, we introduce two additional elements: a convex increasing disutility function and a convex increasing multiplicative congestion-penally function. The disutility function takes the shortfall (contracted rate minus present rate) as its argument, and essentially encourages users to send traffic at their contracted rates, while the congestion-penalty function discourages heavy users from sending excess data when the link is congested. We obtain simple necessary and sufficient conditions on prices for fair and efficient link sharing; moreover, we show that a single price for all users achieves this. We illustrate the ideas using a simple experiment.

Decentralized linear quadratic power system stabilizers for multi-machine power systems

Linear quadratic stabilizers are well-known for their superior control capabilities when compared to the conventional lead-lag power system stabilizers. However, they have not seen much of practical importance as the state variables are generally not measurable; especially the generator rotor angle measurement is not available in most of the power plants. Full state feedback controllers require feedback of other machine states in a multi-machine power system and necessitate block diagonal structure constraints for decentralized implementation. This paper investigates the design of Linear Quadratic Power System Stabilizers using a recently proposed modified Heffron-Phillip's model. This model is derived by taking the secondary bus voltage of the step-up transformer as reference instead of the infinite bus. The state variables of this model can be obtained by local measurements. This model allows a coordinated linear quadratic control design in multi machine systems. The performance of the proposed controller has been evaluated on two widely used multi-machine power systems, 4 generator 10 bus and 10 generator 39 bus systems. It has been observed that the performance of the proposed controller is superior to that of the conventional Power System Stabilizers (PSS) over a wide range of operating and system conditions.

Infinite push: a new support vector ranking algorithm that directly optimizes accuracy at the absolute top of the list

Ranking problems have become increasingly important in machine learning and data mining in recent years, with applications ranging from information retrieval and recommender systems to computational biology and drug discovery. In this paper, we describe a new ranking algorithm that directly maximizes the number of relevant objects retrieved at the absolute top of the list. The algorithm is a support vector style algorithm, but due to the different objective, it no longer leads to a quadratic programming problem. Instead, the dual optimization problem involves l1, ∞ constraints; we solve this dual problem using the recent l1, ∞ projection method of Quattoni et al (2009). Our algorithm can be viewed as an l∞-norm extreme of the lp-norm based algorithm of Rudin (2009) (albeit in a support vector setting rather than a boosting setting); thus we refer to the algorithm as the ‘Infinite Push’. Experiments on real-world data sets confirm the algorithm’s focus on accuracy at the absolute top of the list.

Violation of entropic Leggett-Garg inequality in nuclear spins

We report an experimental study of recently formulated entropic Leggett-Garg inequality (ELGI) by Usha Devi et al. Phys. Rev. A 87, 052103 (2013)]. This inequality places a bound on the statistical measurement outcomes of dynamical observables describing a macrorealistic system. Such a bound is not necessarily obeyed by quantum systems, and therefore provides an important way to distinguish quantumness from classical behavior. Here we study ELGI using a two-qubit nuclear magnetic resonance system. To perform the noninvasive measurements required for the ELGI study, we prepare the system qubit in a maximally mixed state as well as use the ``ideal negative result measurement'' procedure with the help of an ancilla qubit. The experimental results show a clear violation of ELGI by over four standard deviations. These results agree with the predictions of quantum theory. The violation of ELGI is attributed to the fact that certain joint probabilities are not legitimate in the quantum scenario, in the sense they do not reproduce all the marginal probabilities. Using a three-qubit system, we also demonstrate that three-time joint probabilities do not reproduce certain two-time marginal probabilities.

Electric current-induced mass flow in very thin infinite metallic films

This paper reports on the mass transport behavior of infinitely extended, continuous, and very thin metallic films under the influence of electric current. Application of direct current of high densities (> 10(8) A/m(2)) results in visible melting of thin film at only one of the electrodes, and the melt then flows towards the other electrode in a circularly symmetric fashion forming a microscale ring pattern. For the two tested thin film systems, namely Cr and Al, of thicknesses ranging from 4 to 20 nm, the above directional flow consistently occurred from cathode to anode and anode to cathode, respectively. Furthermore, application of alternating electric current results in flow of the liquid material from both the electrodes. The dependence of critical flow behavior parameters, such as flow direction, flow velocity, and evolution of the ring diameter, are experimentally determined. Analytical models based on the principles of electromigration in liquid-phase materials are developed to explain the experimental observations.

Power and delay optimal policies for wireless systems

In this paper we consider a single discrete time queue with infinite buffer. The channel may experience fading. The transmission rate is a linear function of power used for transmission. In this scenario we explicitly obtain power control policies which minimize mean power and/or mean delay. There may also be peak power constraint.

CANDO and the infinite drug discovery frontier

The Computational Analysis of Novel Drug Opportunities (CANDO) platform (http://protinfo.org/cando) uses similarity of compound-proteome interaction signatures to infer homology of compound/drug behavior. We constructed interaction signatures for 3733 human ingestible compounds covering 48,278 protein structures mapping to 2030 indications based on basic science methodologies to predict and analyze protein structure, function, and interactions developed by us and others. Our signature comparison and ranking approach yielded benchmarking accuracies of 12-25% for 1439 indications with at least two approved compounds. We prospectively validated 49/82 `high value' predictions from nine studies covering seven indications, with comparable or better activity to existing drugs, which serve as novel repurposed therapeutics. Our approach may be generalized to compounds beyond those approved by the FDA, and can also consider mutations in protein structures to enable personalization. Our platform provides a holistic multiscale modeling framework of complex atomic, molecular, and physiological systems with broader applications in medicine and engineering.