A tutorial survey of reinforcement learning

This paper gives a compact, self-contained tutorial survey of reinforcement learning, a tool that is increasingly finding application in the development of intelligent dynamic systems. Research on reinforcement learning during the past decade has led to the development of a variety of useful algorithms. This paper surveys the literature and presents the algorithms in a cohesive framework.

A reinforcement learning approach to automatic generation control

This paper formulates the automatic generation control (AGC) problem as a stochastic multistage decision problem. A strategy for solving this new AGC problem formulation is presented by using a reinforcement learning (RL) approach This method of obtaining an AGC controller does not depend on any knowledge of the system model and more importantly it admits considerable flexibility in defining the control objective. Two specific RL based AGC algorithms are presented. The first algorithm uses the traditional control objective of limiting area control error (ACE) excursions, where as, in the second algorithm, the controller can restore the load-generation balance by only monitoring deviation in tie line flows and system frequency and it does not need to know or estimate the composite ACE signal as is done by all current approaches. The effectiveness and versatility of the approaches has been demonstrated using a two area AGC model. (C) 2002 Elsevier Science B.V. All rights reserved.

An analog MVUE for a wireless sensor network

An analog minimum-variance unbiased estimator(MVUE) over an asymmetric wireless sensor network is studied.Minimisation of variance is cast into a constrained non-convex optimisation problem. An explicit algorithm that solves the problem is provided. The solution is obtained by decomposing the original problem into a finite number of convex optimisation problems with explicit solutions. These solutions are then juxtaposed together by exploiting further structure in the objective function.

Throughput analysis of primary and secondary networks in a shared IEEE 802.11 system

In this paper, we analyze the coexistence of a primary and a secondary (cognitive) network when both networks use the IEEE 802.11 based distributed coordination function for medium access control. Specifically, we consider the problem of channel capture by a secondary network that uses spectrum sensing to determine the availability of the channel, and its impact on the primary throughput. We integrate the notion of transmission slots in Bianchi's Markov model with the physical time slots, to derive the transmission probability of the secondary network as a function of its scan duration. This is used to obtain analytical expressions for the throughput achievable by the primary and secondary networks. Our analysis considers both saturated and unsaturated networks. By performing a numerical search, the secondary network parameters are selected to maximize its throughput for a given level of protection of the primary network throughput. The theoretical expressions are validated using extensive simulations carried out in the Network Simulator 2. Our results provide critical insights into the performance and robustness of different schemes for medium access by the secondary network. In particular, we find that the channel captures by the secondary network does not significantly impact the primary throughput, and that simply increasing the secondary contention window size is only marginally inferior to silent-period based methods in terms of its throughput performance.

DMT of parallel-path and layered networks under the half-duplex constraint

In this paper, we study the diversity-multiplexing-gain tradeoff (DMT) of wireless relay networks under the half-duplex constraint. It is often unclear what penalty if any, is imposed by the half-duplex constraint on the DMT of such networks. We study two classes of networks; the first class, called KPP(I) networks, is the class of networks with the relays organized in K parallel paths between the source and the destination. While we assume that there is no direct source-destination path, the K relaying paths can interfere with each other. The second class, termed as layered networks, is comprised of relays organized in layers, where links exist only between adjacent layers. We present a communication scheme based on static schedules and amplify-and-forward relaying for these networks. We also show that for KPP(I) networks with K >= 3, the proposed schemes can achieve full-duplex DMT performance, thus demonstrating that there is no performance hit on the DMT due to the half-duplex constraint. We also show that, for layered networks, a linear DMT of d(max)(1 - r)(+) between the maximum diversity d(max) and the maximum MG, r(max) = 1 is achievable. We adapt existing DMT optimal coding schemes to these networks, thus specifying the end-to-end communication strategy explicitly.

Feature Search in the Grassmanian in Online Reinforcement Learning

We consider the problem of finding the best features for value function approximation in reinforcement learning and develop an online algorithm to optimize the mean square Bellman error objective. For any given feature value, our algorithm performs gradient search in the parameter space via a residual gradient scheme and, on a slower timescale, also performs gradient search in the Grassman manifold of features. We present a proof of convergence of our algorithm. We show empirical results using our algorithm as well as a similar algorithm that uses temporal difference learning in place of the residual gradient scheme for the faster timescale updates.

Maximizing transported data in a wireless sensor network: how much can the network transport before partition?

We consider a scenario where the communication nodes in a sensor network have limited energy, and the objective is to maximize the aggregate bits transported from sources to respective destinations before network partition due to node deaths. This performance metric is novel, and captures the useful information that a network can provide over its lifetime. The optimization problem that results from our approach is nonlinear; however, we show that it can be converted to a Multicommodity Flow (MCF) problem that yields the optimal value of the metric. Subsequently, we compare the performance of a practical routing strategy, based on Node Disjoint Paths (NDPs), with the ideal corresponding to the MCF formulation. Our results indicate that the performance of NDP-based routing is within 7.5% of the optimal.

A power-scalable RF CMOS receiver for 2.4 GHz Wireless Sensor Network applications

A power scalable receiver architecture is presented for low data rate Wireless Sensor Network (WSN) applications in 130nm RF-CMOS technology. Power scalable receiver is motivated by the ability to leverage lower run-time performance requirement to save power. The proposed receiver is able to switch power settings based on available signal and interference levels while maintaining requisite BER. The Low-IF receiver consists of Variable Noise and Linearity LNA, IQ Mixers, VGA, Variable Order Complex Bandpass Filter and Variable Gain and Bandwidth Amplifier (VGBWA) capable of driving variable sampling rate ADC. Various blocks have independent power scaling controls depending on their noise, gain and interference rejection (IR) requirements. The receiver is designed for constant envelope QPSK-type modulation with 2.4GHz RF input, 3MHz IF and 2MHz bandwidth. The chip operates at 1V Vdd with current scalable from 4.5mA to 1.3mA and chip area of 0.65mm2.

Secure Communication in Amplify-and-Forward Networks with Multiple Eavesdroppers: Decoding with SNR Thresholds

The problem of secure unicast communication over a two hop Amplify-and-Forward wireless relay network with multiple eavesdroppers is considered. Assuming that a receiver (destination or eavesdropper) can decode a message only if the received SNR is above a predefined threshold, we consider this problem in two scenarios. In the first scenario, we maximize the SNR at the legitimate destination, subject to the condition that the received SNR at each eavesdropper is below the target threshold. Due to the non-convex nature of the objective function and eavesdroppers' constraints, we transform variables and obtain a quadratically constrained quadratic program (QCQP) with convex constraints, which can be solved efficiently. When the constraints are not convex, we consider a semidefinite relaxation (SDR) to obtain computationally efficient approximate solution. In the second scenario, we minimize the total power consumed by all relay nodes, subject to the condition that the received SNR at the legitimate destination is above the threshold and at every eavesdropper, it is below the corresponding threshold. We propose a semidefinite relaxation of the problem in this scenario and also provide an analytical lower bound.

Structure, energetics and diffusion properties of isomers of trimethyl benzene in beta zeolite: Uptake and Monte Carlo simulation study

A Monte Carlo study along with experimental uptake measurements of 1,2,3-trimethyl benzene, 1,2,4-trimethyl benzene and 1,3,5-trimethyl benzene (TMB) in beta zeolite is reported. The TraPPE potential has been employed for hydrocarbon interaction and harmonic potential of Demontis for modeling framework of the zeolite. Structure, energetics and dynamics of TMB in zeolite beta from Monte Carlo runs reveal interesting information about the diameter, properties of these isomers on confinement. Of the three isomers, 135TMB is supposed to have the largest diameter. It is seen TraPPE with Demontis potential predicts a restricted motion of 135TMB in the channels of zeolite beta.Experimentally, 135TMB has the highest transport diffusivity whereas MID results suggest this has the lowest self diffusivity. (C) 2009 Elsevier Inc. Ail rights reserved.

Allostery and conformational free energy changes in human tryptophanyl-tRNA synthetase from essential dynamics and structure networks

The interdependence of the concept of allostery and enzymatic catalysis, and they being guided by conformational mobility is gaining increased prominence. However, to gain a molecular level understanding of llostery and hence of enzymatic catalysis, it is of utter importance that the networks of amino acids participating in allostery be deciphered. Our lab has been exploring the methods of network analysis combined with molecular dynamics simulations to understand allostery at molecular level. Earlier we had outlined methods to obtain communication paths and then to map the rigid/flexible regions of proteins through network parameters like the shortest correlated paths, cliques, and communities. In this article, we advance the methodology to estimate the conformational populations in terms of cliques/communities formed by interactions including the side-chains and then to compute the ligand-induced population shift. Finally, we obtain the free-energy landscape of the protein in equilibrium, characterizing the free-energy minima accessed by the protein complexes. We have chosen human tryptophanyl-tRNA synthetase (hTrpRS), a protein esponsible for charging tryptophan to its cognate tRNA during protein biosynthesis for this investigation. This is a multidomain protein exhibiting excellent allosteric communication. Our approach has provided valuable structural as well as functional insights into the protein. The methodology adopted here is highly generalized to illuminate the linkage between protein structure networks and conformational mobility involved in the allosteric mechanism in any protein with known structure.

Coordination polymers and hybrid networks of different dimensionalities formed by metal sulfites

In our effort to explore the use of the sulfite ion to design hybrid and open-framework materials, we have been able to prepare, under hydrothermal conditions, zero-dimensional [Zn(C12H8N2)(SO3)]center dot 2H(2)O, I (a = 7.5737(5) angstrom, b = 10.3969(6) angstrom, c = 10.3986(6) angstrom, alpha = 64.172(1)degrees, beta = 69.395(1)degrees, gamma = 79.333(1)degrees, Z = 2, and space group P (1) over bar), one-dimensional [Zn-2(C12H8N2)(SO3)(2)(H2O)], II (a = 8.0247(3) angstrom, b = 9.4962(3) angstrom, c = 10.2740(2) A, alpha = 81.070(1)degrees, beta = 80.438(1)degrees, gamma = 75.66(5)degrees, Z = 2, and space group P (1) over bar), two-dimensional [Zn-2(C10H8N2)(SO3)(2)]center dot H2O, III (a = 16.6062(1) angstrom, b = 4.7935(1) angstrom, c = 19.2721(5) angstrom, beta = 100.674(2)degrees, Z = 4, and space group C2/c), and three-dimensional [Zn-4(C6H12N2)(SO3)(4)(H2O)(4)], IV (a = 11.0793(3) angstrom, c = 8.8246(3) angstrom, Z = 2, and space group P42nm), of which the last three are coordination polymers. A hybrid open-framework sulfite-sulfate of the composition [C2H10N2][Nd(SO3)(SO4)(H2O)](2), V (a = 9.0880(3) angstrom, b = 6.9429(2) angstrom, c = 13.0805(5) A, beta = 91.551(2)degrees, Z = 2, and space group P2(1)/c), with a layered structure containing metal-oxygen-metal bonds has also been described.

The GAF Domain of the cGMP-Binding, cGMP-Specific Phosphodiesterase (PDE5) Is a Sensor and a Sink for cGMP

We describe here a novel sensor for cGMP based on the GAF domain of the cGMP-binding, cGMP-specific phosphodiesterase 5 (PDE5) using bioluminescence resonance energy transfer (BRET). The wild type GAFa domain, capable of binding cGMP with high affinity, and a mutant (GAFaF163A) unable to bind cGMP were cloned as fusions between GFP and Rluc for BRET2 assays. BRET2 ratios of the wild type GAFa fusion protein, but not GAFaF163A, increased in the presence of cGMP but not cAMP. Higher basal BRET2 ratios were observed in cells expressing the wild type GAFa domain than in cells expressing GAFaF163A. This was correlated with elevated basal intracellular levels of cGMP, indicating that the GAF domain could act as a sink for cGMP. The tandem GAF domains in full length PDE5 could also sequester cGMP when the catalytic activity of PDE5 was inhibited. Therefore, these results describe a cGMP sensor utilizing BRET2 technology and experimentally demonstrate the reservoir of cGMP that can be present in cells that express cGMP-binding GAF domain-containing proteins. PDE5 is the target for the anti-impotence drug sildenafil citrate; therefore, this GAF-BRET2 sensor could be used for the identification of novel compounds that inhibit cGMP binding to the GAF domain, thereby regulating PDE5 catalytic activity.