Two timescale convergent Q-learning for sleep-scheduling in wireless sensor networks

In this paper, we consider an intrusion detection application for Wireless Sensor Networks. We study the problem of scheduling the sleep times of the individual sensors, where the objective is to maximize the network lifetime while keeping the tracking error to a minimum. We formulate this problem as a partially-observable Markov decision process (POMDP) with continuous stateaction spaces, in a manner similar to Fuemmeler and Veeravalli (IEEE Trans Signal Process 56(5), 2091-2101, 2008). However, unlike their formulation, we consider infinite horizon discounted and average cost objectives as performance criteria. For each criterion, we propose a convergent on-policy Q-learning algorithm that operates on two timescales, while employing function approximation. Feature-based representations and function approximation is necessary to handle the curse of dimensionality associated with the underlying POMDP. Our proposed algorithm incorporates a policy gradient update using a one-simulation simultaneous perturbation stochastic approximation estimate on the faster timescale, while the Q-value parameter (arising from a linear function approximation architecture for the Q-values) is updated in an on-policy temporal difference algorithm-like fashion on the slower timescale. The feature selection scheme employed in each of our algorithms manages the energy and tracking components in a manner that assists the search for the optimal sleep-scheduling policy. For the sake of comparison, in both discounted and average settings, we also develop a function approximation analogue of the Q-learning algorithm. This algorithm, unlike the two-timescale variant, does not possess theoretical convergence guarantees. Finally, we also adapt our algorithms to include a stochastic iterative estimation scheme for the intruder's mobility model and this is useful in settings where the latter is not known. Our simulation results on a synthetic 2-dimensional network setting suggest that our algorithms result in better tracking accuracy at the cost of only a few additional sensors, in comparison to a recent prior work.

Energy Sharing for Multiple Sensor Nodes With Finite Buffers

We consider the problem of finding optimal energy sharing policies that maximize the network performance of a system comprising of multiple sensor nodes and a single energy harvesting (EH) source. Sensor nodes periodically sense the random field and generate data, which is stored in the corresponding data queues. The EH source harnesses energy from ambient energy sources and the generated energy is stored in an energy buffer. Sensor nodes receive energy for data transmission from the EH source. The EH source has to efficiently share the stored energy among the nodes to minimize the long-run average delay in data transmission. We formulate the problem of energy sharing between the nodes in the framework of average cost infinite-horizon Markov decision processes (MDPs). We develop efficient energy sharing algorithms, namely Q-learning algorithm with exploration mechanisms based on the epsilon-greedy method as well as upper confidence bound (UCB). We extend these algorithms by incorporating state and action space aggregation to tackle state-action space explosion in the MDP. We also develop a cross entropy based method that incorporates policy parameterization to find near optimal energy sharing policies. Through simulations, we show that our algorithms yield energy sharing policies that outperform the heuristic greedy method.

A unified approach to the solution of plane problems of Magneto-elasticity with special reference to a hole in a thin infinite conducting plate

Under certain specific assumption it has been observed that the basic equations of magneto-elasticity in the case of plane deformation lead to a biharmonic equation, as in the case of the classical plane theory of elasticity. The method of solving boundary value problems has been properly modified and a unified approach in solving such problems has been suggested with special reference to problems relating thin infinite plates with a hole. Closed form expressions have been obtained for the stresses due to a uniform magnetic field present in the plane of deformation of a thin infinite conducting plate with a circular hole, the plate being deformed by a tension acting parallel to the direction of the magnetic field.

The effect of surface tension in porous wave maker problems

Using a mixed-type Fourier transform of a general form in the case of water of infinite depth and the method of eigenfunction expansion in the case of water of finite depth, several boundary-value problems involving the propagation and scattering of time harmonic surface water waves by vertical porous walls have been fully investigated, taking into account the effect of surface tension also. Known results are recovered either directly or as particular cases of the general problems under consideration.

Boundary value problems for third-order nonlinear ordinary differential equations

In this paper, we describe how to analyze boundary value problems for third-order nonlinear ordinary differential equations over an infinite interval. Several physical problems of interest are governed by such systems. The seminumerical schemes described here offer some advantages over solutions obtained by using traditional methods such as finite differences, shooting method, etc. These techniques also reveal the analytic structure of the solution function. For illustrative purposes, several physical problems, mainly drawn from fluid mechanics, are considered; they clearly demonstrate the efficiency of the techniques presented here.

Boundary value problems for third-order nonlinear ordinary differential equations

In this paper, we describe how to analyze boundary value problems for third-order nonlinear ordinary differential equations over an infinite interval. Several physical problems of interest are governed by such systems. The seminumerical schemes described here offer some advantages over solutions obtained by using traditional methods such as finite differences, shooting method, etc. These techniques also reveal the analytic structure of the solution function. For illustrative purposes, several physical problems, mainly drawn from fluid mechanics, are considered; they clearly demonstrate the efficiency of the techniques presented here.

A reinforcement learning based algorithm for finite horizon Markov decision processes

We develop a simulation based algorithm for finite horizon Markov decision processes with finite state and finite action space. Illustrative numerical experiments with the proposed algorithm are shown for problems in flow control of communication networks and capacity switching in semiconductor fabrication.

Theoretical and experimental investigation of framed structure-layered soil interaction problems

The plane stress solution for the interaction analysis of a framed structure, with a foundation beam, resting on a layered soil has been studied using both theoretical and photoelastic methods. The theoretical analysis has been done by using a combined analytical and finite element method. In this, the analytical solution has been used for the semi-infinite layered medium and finite element method for the framed structure. The experimental investigation has been carried out using two-dimensional photoelasticity in which modelling of the layered semi-infinite plane and a method to obtain contact pressure distribution have been discussed. The theoretical and experimental results in respect of contact pressure distribution between the foundation beam and layered soil medium, the fibre stresses in the foundation beam and framed structure have been compared. These results have also been compared with theoretical results obtained by idealizing the layered semi-infinite plane as (a) a Winkler model and (b) an equivalent homogeneous semi-infinite medium

Spin distributions in antiferromagnetically coupled Mn2+Cu2+ systems: from the pair to the infinite chain

Probably the most informative description of the ground slate of a magnetic molecular species is provided by the spin density map. Such a map may be experimentally obtained from polarized neutron diffraction (PND) data or theoretically calculated using quantum chemical approaches. Density functional theory (DFT) methods have been proved to be well-adapted for this. Spin distributions in one-dimensional compounds may also be computed using the density matrix renormalization group (DMRG) formalism. These three approaches, PND, DFT, and DMRG, have been utilized to obtain new insights on the ground state of two antiferromagnetically coupled Mn2+Cu2+ compounds, namely [Mn(Me-6-[14]ane-N-4)Cu(oxpn)](CF3SO3)(2) and MnCu(pba)(H2O)(3) . 2H(2)O, with Me-6-[14]ane-N-4 = (+/-)-5,7,7,12,14,14-hexamethyl-1,4,8,11-tetraazacyclotetradecane, oxpn = N,N'-bis(3-aminopropyl)oxamido and pba = 1,3-propylenebis(oxamato). Three problems in particular have been investigated: the spin distribution in the mononuclear precursors [Cu(oxpn)] and [Cu(pba)](2-), the spin density maps in the two Mn2+Cu2+ compounds, and the evolution of the spin distributions on the Mn2+ and Cu2+ sites when passing from a pair to a one-dimensional ferrimagnet.

An inverse problem for the Helmholtz equation involving two semi-infinite fluids

An analytical method is developed for solving an inverse problem for Helmholtz's equation associated with two semi-infinite incompressible fluids of different variable refractive indices, separated by a plane interface. The unknowns of the inverse problem are: (i) the refractive indices of the two fluids, (ii) the ratio of the densities of the two fluids, and (iii) the strength of an acoustic source assumed to be situated at the interface of the two fluids. These are determined from the pressure on the interface produced by the acoustic source. The effect of the surface tension force at the interface is taken into account in this paper. The application of the proposed analytical method to solve the inverse problem is also illustrated with several examples. In particular, exact solutions of two direct problems are first derived using standard classical methods which are then used in our proposed inverse method to recover the unknowns of the corresponding inverse problems. The results are found to be in excellent agreement.

An Actor-Critic Algorithm for Finite Horizon Markov Decision Processes

We develop a simulation based algorithm for finite horizon Markov decision processes with finite state and finite action space. Illustrative numerical experiments with the proposed algorithm are shown for problems in flow control of communication networks and capacity switching in semiconductor fabrication.

On porous wave maker problems

The problem of generation of surface water waves at tile interface of two immiscible liquids by a onesided porous wave maker is studied in both the cases of water of infinite as well as finite depth by suitable application of the generalisation of Havelock's expansion theorem. The solution of the the problem of reflection of water waves due to a fixed porous wall is derived as a particular case.

Infinite push: a new support vector ranking algorithm that directly optimizes accuracy at the absolute top of the list

Ranking problems have become increasingly important in machine learning and data mining in recent years, with applications ranging from information retrieval and recommender systems to computational biology and drug discovery. In this paper, we describe a new ranking algorithm that directly maximizes the number of relevant objects retrieved at the absolute top of the list. The algorithm is a support vector style algorithm, but due to the different objective, it no longer leads to a quadratic programming problem. Instead, the dual optimization problem involves l1, ∞ constraints; we solve this dual problem using the recent l1, ∞ projection method of Quattoni et al (2009). Our algorithm can be viewed as an l∞-norm extreme of the lp-norm based algorithm of Rudin (2009) (albeit in a support vector setting rather than a boosting setting); thus we refer to the algorithm as the ‘Infinite Push’. Experiments on real-world data sets confirm the algorithm’s focus on accuracy at the absolute top of the list.

Optimality Conditions and Duality for Semi-Infinite Mathematical Programming Problem with Equilibrium Constraints

This article considers a semi-infinite mathematical programming problem with equilibrium constraints (SIMPEC) defined as a semi-infinite mathematical programming problem with complementarity constraints. We establish necessary and sufficient optimality conditions for the (SIMPEC). We also formulate Wolfe- and Mond-Weir-type dual models for (SIMPEC) and establish weak, strong and strict converse duality theorems for (SIMPEC) and the corresponding dual problems under invexity assumptions.

Static analysis of infinite laminated cylindrical shell panel

This paper presents an approximate three-dimensional elasticity solution for an infinitely long, cross-ply laminated circular cylindrical shell panel with simply supported boundary conditions, subjected to an arbitrary discontinuous transverse loading. The solution is based on the principal assumption that the ratio of the thickness of the lamina to its middle surface radius is negligible compared to unity. The validity of this assumption and the range of application of this approximate solution have been established through a comparison with an exact solution. Results of classical and first-order shear deformation shell theories have been compared with the results of the present solution to bring out the accuracy of these theories. It is also shown that for very shallow shell panels the definition of a thin shell should be based on the ratio of thickness to chord width rather than the ratio of thickness to mean radius.