A novel Q-learning algorithm with function approximation for constrained Markov decision processes

We present a novel multi-timescale Q-learning algorithm for average cost control in a Markov decision process subject to multiple inequality constraints. We formulate a relaxed version of this problem through the Lagrange multiplier method. Our algorithm is different from Q-learning in that it updates two parameters - a Q-value parameter and a policy parameter. The Q-value parameter is updated on a slower time scale as compared to the policy parameter. Whereas Q-learning with function approximation can diverge in some cases, our algorithm is seen to be convergent as a result of the aforementioned timescale separation. We show the results of experiments on a problem of constrained routing in a multistage queueing network. Our algorithm is seen to exhibit good performance and the various inequality constraints are seen to be satisfied upon convergence of the algorithm.

Effective Q Criterion for Disk Stability in an External Potential

The standard Q criterion (with Q > 1) describes the stability against local, axisymmetric perturbations in a disk supported by rotation and random motion. Most astrophysical disks, however, are under the influence of an external gravitational potential, which can significantly affect their stability. A typical example is a galactic disk embedded in a dark matter halo. Here, we do a linear perturbation analysis for a disk in an external potential and obtain a generalized dispersion relation and the effective stability criterion. An external potential, such as that due to the dark matter halo concentric with the disk, contributes to the unperturbed rotational field and significantly increases its stability. We obtain the values for the effective Q parameter for the Milky Way and for a low surface brightness galaxy, UGC 7321. We find that in each case the stellar disk by itself is barely stable and it is the dark matter halo that stabilizes the disk against local, axisymmetric gravitational instabilities. Thus, the dark matter halo is necessary to ensure local disk stability. This result has been largely missed so far because in practice the Q parameter for a galactic disk is obtained using the observed rotational field that already includes the effect of the halo

EFFECTIVE Q CRITERION FOR DISK STABILITY IN AN EXTERNAL POTENTIAL

standard Q criterion (with Q > 1) describes the stability against local, axisymmetric perturbations in a disk supported by rotation and random motion. Most astrophysical disks, however, are under the influence of an external gravitational potential, which can significantly affect their stability. A typical example is a galactic disk embedded in a dark matter halo. Here, we do a linear perturbation analysis for a disk in an external potential and obtain a generalized dispersion relation and the effective stability criterion. An external potential, such as that due to the dark matter halo concentric with the disk, contributes to the unperturbed rotational field and significantly increases its stability. We obtain the values for the effective Q parameter for the Milky Way and for a low surface brightness galaxy, UGC 7321. We find that in each case the stellar disk by itself is barely stable and it is the dark matter halo that stabilizes the disk against local, axisymmetric gravitational instabilities. Thus, the dark matter halo is necessary to ensure local disk stability. This result has been largely missed so far because in practice the Q parameter for a galactic disk is obtained using the observed rotational field that already includes the effect of the halo.

Smoothed Functional Algorithms for Stochastic Optimization Using q-Gaussian Distributions

Smoothed functional (SF) schemes for gradient estimation are known to be efficient in stochastic optimization algorithms, especially when the objective is to improve the performance of a stochastic system However, the performance of these methods depends on several parameters, such as the choice of a suitable smoothing kernel. Different kernels have been studied in the literature, which include Gaussian, Cauchy, and uniform distributions, among others. This article studies a new class of kernels based on the q-Gaussian distribution, which has gained popularity in statistical physics over the last decade. Though the importance of this family of distributions is attributed to its ability to generalize the Gaussian distribution, we observe that this class encompasses almost all existing smoothing kernels. This motivates us to study SF schemes for gradient estimation using the q-Gaussian distribution. Using the derived gradient estimates, we propose two-timescale algorithms for optimization of a stochastic objective function in a constrained setting with a projected gradient search approach. We prove the convergence of our algorithms to the set of stationary points of an associated ODE. We also demonstrate their performance numerically through simulations on a queuing model.

Two timescale convergent Q-learning for sleep-scheduling in wireless sensor networks

In this paper, we consider an intrusion detection application for Wireless Sensor Networks. We study the problem of scheduling the sleep times of the individual sensors, where the objective is to maximize the network lifetime while keeping the tracking error to a minimum. We formulate this problem as a partially-observable Markov decision process (POMDP) with continuous stateaction spaces, in a manner similar to Fuemmeler and Veeravalli (IEEE Trans Signal Process 56(5), 2091-2101, 2008). However, unlike their formulation, we consider infinite horizon discounted and average cost objectives as performance criteria. For each criterion, we propose a convergent on-policy Q-learning algorithm that operates on two timescales, while employing function approximation. Feature-based representations and function approximation is necessary to handle the curse of dimensionality associated with the underlying POMDP. Our proposed algorithm incorporates a policy gradient update using a one-simulation simultaneous perturbation stochastic approximation estimate on the faster timescale, while the Q-value parameter (arising from a linear function approximation architecture for the Q-values) is updated in an on-policy temporal difference algorithm-like fashion on the slower timescale. The feature selection scheme employed in each of our algorithms manages the energy and tracking components in a manner that assists the search for the optimal sleep-scheduling policy. For the sake of comparison, in both discounted and average settings, we also develop a function approximation analogue of the Q-learning algorithm. This algorithm, unlike the two-timescale variant, does not possess theoretical convergence guarantees. Finally, we also adapt our algorithms to include a stochastic iterative estimation scheme for the intruder's mobility model and this is useful in settings where the latter is not known. Our simulation results on a synthetic 2-dimensional network setting suggest that our algorithms result in better tracking accuracy at the cost of only a few additional sensors, in comparison to a recent prior work.

Newton-based stochastic optimization using q-Gaussian smoothed functional algorithms

We present the first q-Gaussian smoothed functional (SF) estimator of the Hessian and the first Newton-based stochastic optimization algorithm that estimates both the Hessian and the gradient of the objective function using q-Gaussian perturbations. Our algorithm requires only two system simulations (regardless of the parameter dimension) and estimates both the gradient and the Hessian at each update epoch using these. We also present a proof of convergence of the proposed algorithm. In a related recent work (Ghoshdastidar, Dukkipati, & Bhatnagar, 2014), we presented gradient SF algorithms based on the q-Gaussian perturbations. Our work extends prior work on SF algorithms by generalizing the class of perturbation distributions as most distributions reported in the literature for which SF algorithms are known to work turn out to be special cases of the q-Gaussian distribution. Besides studying the convergence properties of our algorithm analytically, we also show the results of numerical simulations on a model of a queuing network, that illustrate the significance of the proposed method. In particular, we observe that our algorithm performs better in most cases, over a wide range of q-values, in comparison to Newton SF algorithms with the Gaussian and Cauchy perturbations, as well as the gradient q-Gaussian SF algorithms. (C) 2014 Elsevier Ltd. All rights reserved.

Graded IR Filters: Distinguishing Between Single and Multipoint NO2 center dot center dot center dot I Halogen Bonded Supramolecular Synthons (P, Q, and R Synthons)

The NO2 center dot center dot center dot I supramolecular synthon is a halogen bonded recognition pattern that is present in the crystal structures of many compounds that contain these functional groups. These synthons have been previously distinguished as P, Q, and R types using topological and geometrical criteria. A five step IR spectroscopic sequence is proposed here to distinguish between these synthon types in solid samples. Sets of known compounds that contain the P, Q, and R synthons are first taken to develop IR spectroscopic identifiers for them. The identifiers are then used to create graded IR filters that sieve the synthons. These filters contain signatures of the individual NO2 center dot center dot center dot I synthons and may be applied to distinguish between P, Q, and R synthon varieties. They are also useful to identify synthons that are of a borderline character, synthons in disordered structures wherein the crystal structure in itself is not sufficient to distinguish synthon types, and in the identification of the NO2 center dot center dot center dot I synthons in compounds with unknown crystal structures. This study establishes clear differences for the three different geometries P, Q, and Rand in the chemical differences in the intermolecular interactions contained in the synthons. Our IR method can be conveniently employed when single crystals are not readily available also in high throughput analysis. It is possible that such identification may also be adopted as an input for crystal structure prediction analysis of compounds with unknown crystal structures.

Transverse single spin asymmetry in e plus p up arrow -> e plus J/psi plus X and Q(2) evolution of Sivers function-II

We present estimates of single spin asymmetry in the electroproduction of J/psi taking into account the transverse momentum-dependent (TMD) evolution of the gluon Sivers function. We estimate single spin asymmetry for JLab, HERMES, COMPASS and eRHIC energies using the color evaporation model of J/psi. We have calculated the asymmetry using recent parameters extracted by Echevarria et al. using the Collins-Soper-Sterman approach to TMD evolution. These recent TMD evolution fits are based on the evolution kernel in which the perturbative part is resummed up to next-to-leading logarithmic accuracy. We have also estimated the asymmetry by using parameters which had been obtained by a fit by Anselmino et al., using both an exact numerical and an approximate analytical solution of the TMD evolution equations. We find that the variation among the different estimates obtained using TMD evolution is much smaller than between these on one hand and the estimates obtained using DGLAP evolution on the other. Even though the use of TMD evolution causes an overall reduction in asymmetries compared to the ones obtained without it, they remain sizable. Overall, upon use of TMD evolution, predictions for asymmetries stabilize.

Investigation of schemes for incorporating generator Q limits in the fast decoupled load flow method

Fast Decoupled Load Flow (FDLF) is a very popular and widely used power flow analysis method because of its simplicity and efficiency. Even though the basic FDLF algorithm is well investigated, the same is not true in the case of additional schemes/modifications required to obtain adjusted load flow solutions using the FDLF method. Handling generator Q limits is one such important feature needed in any practical load flow method. This paper presents a comprehensive investigation of two classes of schemes intended to handle this aspect i.e. the bus type switching scheme and the sensitivity scheme. We propose two new sensitivity based schemes and assess their performance in comparison with the existing schemes. In addition, a new scheme to avoid the possibility of anomalous solutions encountered while using the conventional schemes is also proposed and evaluated. Results from extensive simulation studies are provided to highlight the strengths and weaknesses of these existing and proposed schemes, especially from the point of view of reliability.

A Variant of the Hadwiger-Debrunner (p, q)-Problem in the Plane

Let X be a convex curve in the plane (say, the unit circle), and let be a family of planar convex bodies such that every two of them meet at a point of X. Then has a transversal of size at most . Suppose instead that only satisfies the following ``(p, 2)-condition'': Among every p elements of , there are two that meet at a common point of X. Then has a transversal of size . For comparison, the best known bound for the Hadwiger-Debrunner (p, q)-problem in the plane, with , is . Our result generalizes appropriately for if is, for example, the moment curve.

Quark and gluon distribution functions in a viscous quark-gluon plasma medium and dilepton production via q(q)over-bar annihilation

Viscous modifications to the thermal distributions of quark-antiquarks and gluons have been studied in a quasiparticle description of the quark-gluon-plasma medium created in relativistic heavy-ion collision experiments. The model is described in terms of quasipartons that encode the hot QCD medium effects in their respective effective fugacities. Both shear and bulk viscosities have been taken in to account in the analysis, and the modifications to thermal distributions have been obtained by modifying the energy-momentum tensor in view of the nontrivial dispersion relations for the gluons and quarks. The interactions encoded in the equation of state induce significant modifications to the thermal distributions. As an implication, the dilepton production rate in the q (q) over bar annihilation process has been investigated. The equation of state is found to have a significant impact on the dilepton production rate along with the viscosities.

Binary excitation of a high-Q bulk acoustic microresonator by actuation polarity inversion

We present a technique for independently exciting two resonant modes of vibration in a single-crystal silicon bulk mode microresonator using the same electrode configuration through control of the polarity of the DC actuation voltage. Applications of this technique may include built-in temperature compensation by the simultaneous selective excitation of two closely spaced modes that may have different temperature coefficients of resonant frequency. The technique is simple and requires minimum circuit overhead for implementation. The technique is implemented on square plate resonators with quality factors as high as 3.06 × 106. Copyright © 2008 by ASME.

Anchor limited q in flexural mode resonators

This paper reports a preliminary examination of the effect of anchor geometry design on the quality factor of flexural mode resonators operating in vacuum using both FE simulation and measurements of resonator frequency response. Three types of structures have been considered in this study: an elliptical mode ring, a double ended tuning fork, and a doubly-clamped beam. We consider the relative distribution of strain energies in both the resonant structure and the connecting stem, which is indicative of the measured quality factor. The measured quality factors of the different structures are compared against each other, based on which suggestions are proposed for optimizing the anchor limited quality factor (Q) in flexural mode micromechanical resonators. ©2008 IEEE.