Two timescale convergent Q-learning for sleep-scheduling in wireless sensor networks

In this paper, we consider an intrusion detection application for Wireless Sensor Networks. We study the problem of scheduling the sleep times of the individual sensors, where the objective is to maximize the network lifetime while keeping the tracking error to a minimum. We formulate this problem as a partially-observable Markov decision process (POMDP) with continuous stateaction spaces, in a manner similar to Fuemmeler and Veeravalli (IEEE Trans Signal Process 56(5), 2091-2101, 2008). However, unlike their formulation, we consider infinite horizon discounted and average cost objectives as performance criteria. For each criterion, we propose a convergent on-policy Q-learning algorithm that operates on two timescales, while employing function approximation. Feature-based representations and function approximation is necessary to handle the curse of dimensionality associated with the underlying POMDP. Our proposed algorithm incorporates a policy gradient update using a one-simulation simultaneous perturbation stochastic approximation estimate on the faster timescale, while the Q-value parameter (arising from a linear function approximation architecture for the Q-values) is updated in an on-policy temporal difference algorithm-like fashion on the slower timescale. The feature selection scheme employed in each of our algorithms manages the energy and tracking components in a manner that assists the search for the optimal sleep-scheduling policy. For the sake of comparison, in both discounted and average settings, we also develop a function approximation analogue of the Q-learning algorithm. This algorithm, unlike the two-timescale variant, does not possess theoretical convergence guarantees. Finally, we also adapt our algorithms to include a stochastic iterative estimation scheme for the intruder's mobility model and this is useful in settings where the latter is not known. Our simulation results on a synthetic 2-dimensional network setting suggest that our algorithms result in better tracking accuracy at the cost of only a few additional sensors, in comparison to a recent prior work.

Logarithmic corrections to extremal black hole entropy in N=2, 4 and 8 supergravity

We compute the logarithmic correction to black hole entropy about exponentially suppressed saddle points of the Quantum Entropy Function corresponding to Z(N) orbifolds of the near horizon geometry of the extremal black hole under study. By carefully accounting for zero mode contributions we show that the logarithmic contributions for quarter-BPS black holes in N = 4 supergravity and one-eighth BPS black holes in N = 8 supergravity perfectly match with the prediction from the microstate counting. We also find that the logarithmic contribution for half-BPS black holes in N = 2 supergravity depends non-trivially on the Z(N) orbifold. Our analysis draws heavily on the results we had previously obtained for heat kernel coefficients on Z(N) orbifolds of spheres and hyperboloids in arXiv:1311.6286 and we also propose a generalization of the Plancherel formula to Z(N) orbifolds of hyperboloids to an expression involving the Harish-Chandra character of sl (2, R), a result which is of possible mathematical interest.

Optimal control of grinding mill circuit using model predictive static programming: A new nonlinear MPC paradigm

The recently developed reference-command tracking version of model predictive static programming (MPSP) is successfully applied to a single-stage closed grinding mill circuit. MPSP is an innovative optimal control technique that combines the philosophies of model predictive control (MPC) and approximate dynamic programming. The performance of the proposed MPSP control technique, which can be viewed as a `new paradigm' under the nonlinear MPC philosophy, is compared to the performance of a standard nonlinear MPC technique applied to the same plant for the same conditions. Results show that the MPSP control technique is more than capable of tracking the desired set-point in the presence of model-plant mismatch, disturbances and measurement noise. The performance of MPSP and nonlinear MPC compare very well, with definite advantages offered by MPSP. The computational speed of MPSP is increased through a sequence of innovations such as the conversion of the dynamic optimization problem to a low-dimensional static optimization problem, the recursive computation of sensitivity matrices and using a closed form expression to update the control. To alleviate the burden on the optimization procedure in standard MPC, the control horizon is normally restricted. However, in the MPSP technique the control horizon is extended to the prediction horizon with a minor increase in the computational time. Furthermore, the MPSP technique generally takes only a couple of iterations to converge, even when input constraints are applied. Therefore, MPSP can be regarded as a potential candidate for online applications of the nonlinear MPC philosophy to real-world industrial process plants. (C) 2014 Elsevier Ltd. All rights reserved.

Heat kernels on the AdS(2) cone and logarithmic corrections to extremal black hole entropy

We develop new techniques to efficiently evaluate heat kernel coefficients for the Laplacian in the short-time expansion on spheres and hyperboloids with conical singularities. We then apply these techniques to explicitly compute the logarithmic contribution to black hole entropy from an N = 4 vector multiplet about a Z(N) orbifold of the near-horizon geometry of quarter-BPS black holes in N = 4 supergravity. We find that this vanishes, matching perfectly with the prediction from the microstate counting. We also discuss possible generalisations of our heat kernel results to higher-spin fields over ZN orbifolds of higher-dimensional spheres and hyperboloids.

Energy Sharing for Multiple Sensor Nodes With Finite Buffers

We consider the problem of finding optimal energy sharing policies that maximize the network performance of a system comprising of multiple sensor nodes and a single energy harvesting (EH) source. Sensor nodes periodically sense the random field and generate data, which is stored in the corresponding data queues. The EH source harnesses energy from ambient energy sources and the generated energy is stored in an energy buffer. Sensor nodes receive energy for data transmission from the EH source. The EH source has to efficiently share the stored energy among the nodes to minimize the long-run average delay in data transmission. We formulate the problem of energy sharing between the nodes in the framework of average cost infinite-horizon Markov decision processes (MDPs). We develop efficient energy sharing algorithms, namely Q-learning algorithm with exploration mechanisms based on the epsilon-greedy method as well as upper confidence bound (UCB). We extend these algorithms by incorporating state and action space aggregation to tackle state-action space explosion in the MDP. We also develop a cross entropy based method that incorporates policy parameterization to find near optimal energy sharing policies. Through simulations, we show that our algorithms yield energy sharing policies that outperform the heuristic greedy method.

Regulation of Growth, Cell Shape, Cell Division, and Gene Expression by Second Messengers (p)ppGpp and Cyclic Di-GMP in Mycobacterium smegmatis

The alarmone (p)ppGpp regulates transcription, translation, replication, virulence, lipid synthesis, antibiotic sensitivity, biofilm formation, and other functions in bacteria. Signaling nucleotide cyclic di-GMP (c-di-GMP) regulates biofilm formation, motility, virulence, the cell cycle, and other functions. In Mycobacterium smegmatis, both (p) ppGpp and c-di-GMP are synthesized and degraded by bifunctional proteins Rel(Msm) and DcpA, encoded by rel(Msm) and dcpA genes, respectively. We have previously shown that the Delta rel(Msm) and Delta dcpA knockout strains are antibiotic resistant and defective in biofilm formation, show altered cell surface properties, and have reduced levels of glycopeptidolipids and polar lipids in their cell wall (K. R. Gupta, S. Kasetty, and D. Chatterji, Appl Environ Microbiol 81:2571-2578, 2015, http://dx.doi.org/10.1128/AEM.03999-14). In this work, we have explored the phenotypes that are affected by both (p) ppGpp and c-di-GMP in mycobacteria. We have shown that both (p) ppGpp and c-di-GMP are needed to maintain the proper growth rate under stress conditions such as carbon deprivation and cold shock. Scanning electron microscopy showed that low levels of these second messengers result in elongated cells, while high levels reduce the cell length and embed the cells in a biofilm-like matrix. Fluorescence microscopy revealed that the elongated Delta rel(Msm) and Delta dcpA cells are multinucleate, while transmission electron microscopy showed that the elongated cells are multiseptate. Gene expression analysis also showed that genes belonging to functional categories such as virulence, detoxification, lipid metabolism, and cell-wall-related processes were differentially expressed. Our results suggests that both (p) ppGpp and c-di-GMP affect some common phenotypes in M. smegmatis, thus raising a possibility of cross talk between these two second messengers in mycobacteria. IMPORTANCE Our work has expanded the horizon of (p) ppGpp and c-di-GMP signaling in Gram-positive bacteria. We have come across a novel observation that M. smegmatis needs (p) ppGpp and c-di-GMP for cold tolerance. We had previously shown that the Delta rel(Msm) and Delta dcpA strains are defective in biofilm formation. In this work, the overproduction of (p) ppGpp and c-di-GMP encased M. smegmatis in a biofilm-like matrix, which shows that both (p) ppGpp and c-di-GMP are needed for biofilm formation. The regulation of cell length and cell division by (p) ppGpp was known in mycobacteria, but our work shows that c-di-GMP also affects the cell size and cell division in mycobacteria. This is perhaps the first report of c-di-GMP regulating cell division in mycobacteria.