Adaptive Sleep-Wake Control using Reinforcement Learning in Sensor Networks

The aim in this paper is to allocate the `sleep time' of the individual sensors in an intrusion detection application so that the energy consumption from the sensors is reduced, while keeping the tracking error to a minimum. We propose two novel reinforcement learning (RL) based algorithms that attempt to minimize a certain long-run average cost objective. Both our algorithms incorporate feature-based representations to handle the curse of dimensionality associated with the underlying partially-observable Markov decision process (POMDP). Further, the feature selection scheme used in our algorithms intelligently manages the energy cost and tracking cost factors, which in turn assists the search for the optimal sleeping policy. We also extend these algorithms to a setting where the intruder's mobility model is not known by incorporating a stochastic iterative scheme for estimating the mobility model. The simulation results on a synthetic 2-d network setting are encouraging.

Timer-Based Distributed Node Selection Scheme Exploiting Power Control and Capture

Opportunistic selection in multi-node wireless systems improves system performance by selecting the ``best'' node and by using it for data transmission. In these systems, each node has a real-valued local metric, which is a measure of its ability to improve system performance. Our goal is to identify the best node, which has the largest metric. We propose, analyze, and optimize a new distributed, yet simple, node selection scheme that combines the timer scheme with power control. In it, each node sets a timer and transmit power level as a function of its metric. The power control is designed such that the best node is captured even if. other nodes simultaneously transmit with it. We develop several structural properties about the optimal metric-to-timer-and-power mapping, which maximizes the probability of selecting the best node. These significantly reduce the computational complexity of finding the optimal mapping and yield valuable insights about it. We show that the proposed scheme is scalable and significantly outperforms the conventional timer scheme. We investigate the effect of. and the number of receive power levels. Furthermore, we find that the practical peak power constraint has a negligible impact on the performance of the scheme.

Optimal power allocation for protective jamming in wireless networks: A flow based model

We address the problem of passive eavesdroppers in multi-hop wireless networks using the technique of friendly jamming. The network is assumed to employ Decode and Forward (DF) relaying. Assuming the availability of perfect channel state information (CSI) of legitimate nodes and eavesdroppers, we consider a scheduling and power allocation (PA) problem for a multiple-source multiple-sink scenario so that eavesdroppers are jammed, and source-destination throughput targets are met while minimizing the overall transmitted power. We propose activation sets (AS-es) for scheduling, and formulate an optimization problem for PA. Several methods for finding AS-es are discussed and compared. We present an approximate linear program for the original nonlinear, non-convex PA optimization problem, and argue that under certain conditions, both the formulations produce identical results. In the absence of eavesdroppers' CSI, we utilize the notion of Vulnerability Region (VR), and formulate an optimization problem with the objective of minimizing the VR. Our results show that the proposed solution can achieve power-efficient operation while defeating eavesdroppers and achieving desired source-destination throughputs simultaneously. (C) 2015 Elsevier B.V. All rights reserved.

Optimal Power Allocation Policies for Low Powered Wireless Communication Systems

We consider optimal average power allocation policies in a wireless channel in the presence of individual delay constraints on the transmitted packets. Power is consumed in transmission of data only. We consider the case when the power used in transmission is a linear function of the data transmitted. The transmission channel may experience multipath fading. We have developed a computationally efficient online algorithm, when there is same hard delay constraint for all packets. Later on, we generalize it to the case when there are multiple real time streams with different hard deadline constraints. Our algorithm uses linear programming and has very low complexity.

Alternative Formulation of the Discrete Linear Quadratic Regulator (DLQR)

In this paper, an alternative apriori and aposteriori formulation has been derived for the discrete linear quadratic regulator (DLQR) in a manner analogous to that used in the discrete Kalman filter. It has been shown that the formulation seamlessly fits into the available formulation of the DLQR and the equivalent terms in the existing formulation and the proposed formulation have been identified. Thereafter, the significance of this alternative formulation has been interpreted in terms of the sensitivity of the controller performances to any changes in the states or to changes in the control inputs. The implications of this alternative formulation to adaptive controller tuning have also been discussed.

Multi-agent Reinforcement Learning for Traffic Signal Control

Optimal control of traffic lights at junctions or traffic signal control (TSC) is essential for reducing the average delay experienced by the road users amidst the rapid increase in the usage of vehicles. In this paper, we formulate the TSC problem as a discounted cost Markov decision process (MDP) and apply multi-agent reinforcement learning (MARL) algorithms to obtain dynamic TSC policies. We model each traffic signal junction as an independent agent. An agent decides the signal duration of its phases in a round-robin (RR) manner using multi-agent Q-learning with either is an element of-greedy or UCB 3] based exploration strategies. It updates its Q-factors based on the cost feedback signal received from its neighbouring agents. This feedback signal can be easily constructed and is shown to be effective in minimizing the average delay of the vehicles in the network. We show through simulations over VISSIM that our algorithms perform significantly better than both the standard fixed signal timing (FST) algorithm and the saturation balancing (SAT) algorithm 15] over two real road networks.

Taming Control Divergence in GPUs through Control Flow Linearization

Branch divergence is a very commonly occurring performance problem in GPGPU in which the execution of diverging branches is serialized to execute only one control flow path at a time. Existing hardware mechanism to reconverge threads using a stack causes duplicate execution of code for unstructured control flow graphs. Also the stack mechanism cannot effectively utilize the available parallelism among diverging branches. Further, the amount of nested divergence allowed is also limited by depth of the branch divergence stack. In this paper we propose a simple and elegant transformation to handle all of the above mentioned problems. The transformation converts an unstructured CFG to a structured CFG without duplicating user code. It incurs only a linear increase in the number of basic blocks and also the number of instructions. Our solution linearizes the CFG using a predicate variable. This mechanism reconverges the divergent threads as early as possible. It also reduces the depth of the reconvergence stack. The available parallelism in nested branches can be effectively extracted by scheduling the basic blocks to reduce the effect of stalls due to memory accesses. It can also increase execution efficiency of nested loops with different trip counts for different threads. We implemented the proposed transformation at PTX level using the Ocelot compiler infrastructure. We evaluated the technique using various benchmarks to show that it can be effective in handling the performance problem due to divergence in unstructured CFGs.

Broadcast Control Mechanism for Positional Consensus in Multiagent Systems

In this paper, a strategy for controlling a group of agents to achieve positional consensus is presented. The problem is constrained by the requirement that every agent must be given the same control input through a broadcast communication mechanism. Although the control command is computed using state information in a global framework, the control input is implemented by the agents in a local coordinate frame. We propose a novel linear programming (LP) formulation that is computationally less intensive than earlier proposed methods. Moreover, a random perturbation input in the control command that helps the agents to come close to each other even for a large number of agents, which was not possible with an existing strategy in the literature, is introduced. The method is extended to achieve positional consensus at a prespecified location. The effectiveness of the approach is illustrated through simulation results. A comparison between the LP approach and the existing second-order cone programming-based approach is also presented. The algorithm was successfully implemented on a robotic platform with three robots.

Risk-Sensitive Ergodic Control of Continuous Time Markov Processes With Denumerable State Space

In this article, we study risk-sensitive control problem with controlled continuous time Markov chain state dynamics. Using multiplicative dynamic programming principle along with the atomic structure of the state dynamics, we prove the existence and a characterization of optimal risk-sensitive control under geometric ergodicity of the state dynamics along with a smallness condition on the running cost.

Flexible implementation of power system corrective topology control

This paper proposes a novel decision making framework for optimal transmission switching satisfying the AC feasibility, stability and circuit breaker (CB) reliability requirements needed for practical implementation. The proposed framework can be employed as a corrective tool in day to day operation planning scenarios in response to potential contingencies. The switching options are determined using an efficient heuristic algorithm based on DC optimal power flow, and are presented in a multi-branch tree structure. Then, the AC feasibility and stability checks are conducted and the CB condition monitoring data are employed to perform a CB reliability and line availability assessment. Ultimately, the operator will be offered multiple AC feasible and stable switching options with associated benefits. The operator can use this information, other operating conditions not explicitly considered in the optimization, and his/her own experience to implement the best and most reliable switching action(s). The effectiveness of the proposed approach is validated on the IEEE-118 bus test system. (C) 2015 Elsevier B.V. All rights reserved.

Impact of lifetime control on the threshold of quantum dot lasers

Despite significant improvements in their properties as emitters, colloidal quantum dots have not had much success in emerging as suitable materials for laser applications. Gain in most colloidal systems is short lived, and needs to compete with biexcitonic decay. This has necessitated the use of short pulsed lasers to pump quantum dots to thresholds needed for amplified spontaneous emission or lasing. Continuous wave pumping of gain that is possible in some inorganic phosphors has therefore remained a very distant possibility for quantum dots. Here, we demonstrate that trilayer heterostructures could provide optimal conditions for demonstration of continuous wave lasing in colloidal materials. The design considerations for these materials are discussed in terms of a kinetic model. The electronic structure of the proposed dot architectures is modeled within effective mass theory.

Use of polydispersity index as control parameter to study melting/freezing of Lennard-Jones system: Comparison among predictions of bifurcation theory with Lindemann criterion, inherent structure analysis and Hansen-Verlet rule

Using polydispersity index as an additional order parameter we investigate freezing/melting transition of Lennard-Jones polydisperse systems (with Gaussian polydispersity in size), especially to gain insight into the origin of the terminal polydispersity. The average inherent structure (IS) energy and root mean square displacement (RMSD) of the solid before melting both exhibit quite similar polydispersity dependence including a discontinuity at solid-liquid transition point. Lindemann ratio, obtained from RMSD, is found to be dependent on temperature. At a given number density, there exists a value of polydispersity index (delta (P)) above which no crystalline solid is stable. This transition value of polydispersity(termed as transition polydispersity, delta (P) ) is found to depend strongly on temperature, a feature missed in hard sphere model systems. Additionally, for a particular temperature when number density is increased, delta (P) shifts to higher values. This temperature and number density dependent value of delta (P) saturates surprisingly to a value which is found to be nearly the same for all temperatures, known as terminal polydispersity (delta (TP)). This value (delta (TP) similar to 0.11) is in excellent agreement with the experimental value of 0.12, but differs from hard sphere transition where this limiting value is only 0.048. Terminal polydispersity (delta (TP)) thus has a quasiuniversal character. Interestingly, the bifurcation diagram obtained from non-linear integral equation theories of freezing seems to provide an explanation of the existence of unique terminal polydispersity in polydisperse systems. Global bond orientational order parameter is calculated to obtain further insights into mechanism for melting.

Selection of Optimal Time-to-go in Generalized Vector Explicit Guidance

A new method of selection of time-to-go (t(go)) for Generalized Vector Explicit Guidance (GENEX) law have been proposed in this paper. t(go) is known to be an important parameter in the control and cost function of GENEX guidance law. In this paper the formulation has been done to find an optimal value of t(go) that minimizes the performance cost. Mechanization of GENEX with this optimal t(go) reduces the lateral acceleration demand and consequently increases the range of the interceptor. This new formulation of computing t(go) comes in closed form and thus it can be implemented onboard. This new formulation is applied in the terminal phase of an surface-to-air interceptor for an angle constrained engagement. Results generated by simulation justify the use of optimal t(go).

Fuel-optimal G-MPSP Guidance for Powered Descent Phase of Soft Lunar Landing

A fuel optimal nonlinear sub-optimal guidance scheme is presented in this paper for soft landing of a lunar craft during the powered descent phase. The recently developed Generalized Model Predictive Static Programming (G-MPSP) is used to compute the required magnitude and angle of the thrust vector. Both terminal position and velocity vector are imposed as hard constraints, which ensures high position accuracy and facilitates initiation of vertical descent at the end of the powered descent phase. A key feature of the G-MPSP algorithm is that it converts the nonlinear dynamic programming problem into a low-dimensional static optimization problem (of the same dimension as the output vector). The control history update is done in closed form after computing a time-varying weighting matrix through a backward integration process. This feature makes the algorithm computationally efficient, which makes it suitable for on-board applications. The effectiveness of the proposed guidance algorithm is demonstrated through promising simulation results.

Optimal Guidance for Accurate Lunar Soft Landing with Minimum Fuel Consumption using Model Predictive Static Programming

In this paper the soft lunar landing with minimum fuel expenditure is formulated as a nonlinear optimal guidance problem. The realization of pinpoint soft landing with terminal velocity and position constraints is achieved using Model Predictive Static Programming (MPSP). The high accuracy of the terminal conditions is ensured as the formulation of the MPSP inherently poses final conditions as a set of hard constraints. The computational efficiency and fast convergence make the MPSP preferable for fixed final time onboard optimal guidance algorithm. It has also been observed that the minimum fuel requirement strongly depends on the choice of the final time (a critical point that is not given due importance in many literature). Hence, to optimally select the final time, a neural network is used to learn the mapping between various initial conditions in the domain of interest and the corresponding optimal flight time. To generate the training data set, the optimal final time is computed offline using a gradient based optimization technique. The effectiveness of the proposed method is demonstrated with rigorous simulation results.