Two timescale convergent Q-learning for sleep-scheduling in wireless sensor networks

In this paper, we consider an intrusion detection application for Wireless Sensor Networks. We study the problem of scheduling the sleep times of the individual sensors, where the objective is to maximize the network lifetime while keeping the tracking error to a minimum. We formulate this problem as a partially-observable Markov decision process (POMDP) with continuous stateaction spaces, in a manner similar to Fuemmeler and Veeravalli (IEEE Trans Signal Process 56(5), 2091-2101, 2008). However, unlike their formulation, we consider infinite horizon discounted and average cost objectives as performance criteria. For each criterion, we propose a convergent on-policy Q-learning algorithm that operates on two timescales, while employing function approximation. Feature-based representations and function approximation is necessary to handle the curse of dimensionality associated with the underlying POMDP. Our proposed algorithm incorporates a policy gradient update using a one-simulation simultaneous perturbation stochastic approximation estimate on the faster timescale, while the Q-value parameter (arising from a linear function approximation architecture for the Q-values) is updated in an on-policy temporal difference algorithm-like fashion on the slower timescale. The feature selection scheme employed in each of our algorithms manages the energy and tracking components in a manner that assists the search for the optimal sleep-scheduling policy. For the sake of comparison, in both discounted and average settings, we also develop a function approximation analogue of the Q-learning algorithm. This algorithm, unlike the two-timescale variant, does not possess theoretical convergence guarantees. Finally, we also adapt our algorithms to include a stochastic iterative estimation scheme for the intruder's mobility model and this is useful in settings where the latter is not known. Our simulation results on a synthetic 2-dimensional network setting suggest that our algorithms result in better tracking accuracy at the cost of only a few additional sensors, in comparison to a recent prior work.

Energy Sharing for Multiple Sensor Nodes With Finite Buffers

We consider the problem of finding optimal energy sharing policies that maximize the network performance of a system comprising of multiple sensor nodes and a single energy harvesting (EH) source. Sensor nodes periodically sense the random field and generate data, which is stored in the corresponding data queues. The EH source harnesses energy from ambient energy sources and the generated energy is stored in an energy buffer. Sensor nodes receive energy for data transmission from the EH source. The EH source has to efficiently share the stored energy among the nodes to minimize the long-run average delay in data transmission. We formulate the problem of energy sharing between the nodes in the framework of average cost infinite-horizon Markov decision processes (MDPs). We develop efficient energy sharing algorithms, namely Q-learning algorithm with exploration mechanisms based on the epsilon-greedy method as well as upper confidence bound (UCB). We extend these algorithms by incorporating state and action space aggregation to tackle state-action space explosion in the MDP. We also develop a cross entropy based method that incorporates policy parameterization to find near optimal energy sharing policies. Through simulations, we show that our algorithms yield energy sharing policies that outperform the heuristic greedy method.

Static analysis of infinite laminated cylindrical shell panel

This paper presents an approximate three-dimensional elasticity solution for an infinitely long, cross-ply laminated circular cylindrical shell panel with simply supported boundary conditions, subjected to an arbitrary discontinuous transverse loading. The solution is based on the principal assumption that the ratio of the thickness of the lamina to its middle surface radius is negligible compared to unity. The validity of this assumption and the range of application of this approximate solution have been established through a comparison with an exact solution. Results of classical and first-order shear deformation shell theories have been compared with the results of the present solution to bring out the accuracy of these theories. It is also shown that for very shallow shell panels the definition of a thin shell should be based on the ratio of thickness to chord width rather than the ratio of thickness to mean radius.

Unsteady Compressible Boundary-Layer Flow On The Stagnation Line Of An Infinite Swept Cylinder

Numerical solutions of flow and heat transfer process on the unsteady flow of a compressible viscous fluid with variable gas properties in the vicinity of the stagnation line of an infinite swept cylinder are presented. Results are given for the case where the unsteady temperature field is produced by (i) a sudden change in the wall temperature (enthalpy) as the impulsive motion is started and (ii) a sudden change in the free-stream velocity. Solutions for the simultaneous development of the thermal and momentum boundary layers are obtained by using quasilinearization technique with an implicit finite difference scheme. Attention is given to the transient phenomenon from the initial flow to the final steady-state distribution. Results are presented for the skin friction and heat transfer coefficients as well as for the velocity and enthalpy profiles. The effects of wail enthalpy parameter, sweep parameter, fluid properties and transpiration cooling on the heat transfer and skin friction are considered.

Diffraction Of Elastic Waves By Two Parallel Rigid Strips Embedded In An Infinite Orthotropic Medium

The elastodynamic response of a pair of parallel rigid strips embedded in an infinite orthotropic medium due to elastic waves incident normally on the strips has been investigated. The mixed boundary value problem has been solved by the Integral Equation method. The normal stress and the vertical displacement have been derived in closed form. Numerical values of stress intensity factors at inner and outer edges of the strips and vertical displacement at points in the plane of the strips for several orthotropic materials have been calculated and plotted graphically to show the effect of material orthotropy.

Infinite Dimensional Slow Modulations In A Delayed Model For Orthogonal Cutting Vibrations

We apply the method of multiple scales (MMS) to a well known model of regenerative cutting vibrations in the large delay regime. By ``large'' we mean the delay is much larger than the time scale of typical cutting tool oscillations. The MMS upto second order for such systems has been developed recently, and is applied here to study tool dynamics in the large delay regime. The second order analysis is found to be much more accurate than first order analysis. Numerical integration of the MMS slow flow is much faster than for the original equation, yet shows excellent accuracy. The main advantage of the present analysis is that infinite dimensional dynamics is retained in the slow flow, while the more usual center manifold reduction gives a planar phase space. Lower-dimensional dynamical features, such as Hopf bifurcations and families of periodic solutions, are also captured by the MMS. Finally, the strong sensitivity of the dynamics to small changes in parameter values is seen clearly.

Time dependent rotational flow of a viscous fluid over an infinite porous disk with a magnetic field

Both the semi-similar and self-similar flows due to a viscous fluid rotating with time dependent angular velocity over a porous disk of large radius at rest with or without a magnetic field are investigated. For the self-similar case the resulting equations for the suction and no mass transfer cases are solved numerically by quasilinearization method whereas for the semi-similar case and injection in the self-similar case an implicit finite difference method with Newton's linearization is employed. For rapid deceleration of fluid and for moderate suction in the case of self-similar flow there exists a layer of fluid, close to the disk surface where the sense of rotation is opposite to that of the fluid rotating far away. The velocity profiles in the absence of magnetic field are found to be oscillatory except for suction. For the accelerating freestream, (semi-similar flow) the effect of time is to reduce the amplitude of the oscillations of the velocity components. On the other hand the effect of time for the oscillating case is just the opposite.

Incompressible micropolar fluid flow over a semi-infinite plate

Micropolar fluid flow over a semi-infinite flat plate has been described by using the parabolic co-ordinates and the method of series truncation in order to study the flow for low to large Reynolds numbers. These co-ordinates permit to study the flow regime at the leading edge. Numerical results have been presented for different Reynolds numbers. Results show a reduction in skin friction.

Unsteady flow of a dilute suspension in a semi-infinite contracting or expanding pipe

In the present paper an exact similar solution of the Navier-Stokes equation for unsteady flow of a dilute suspension in a semi-infinite contracting or expanding circular pipe is presented. The effects of the Schmidt number (Sc), Reynolds number (|ε|), the volume fraction (α) and the relaxation time (τ) of the particulate phase on the flow characteristics are examined. The presence of the solid particles has been observed to influence the flow behaviour significantly. These solutions are valid down to the state of a completely collapsed pipe, since the nonlinearity is retained fully. The results may help understanding the flow near the heart and certain forced contractions or expansions of valved veins.

Transversely isotropic infinite cylindrical shell subjected to a radial axisymmetric line load

The problem of an infinite transversely isotropic circular cylindrical shell subjected to an axisymmetric radial external line load is investigated using elasticity theory, classical shell theory and shear deformation theory. The results obtained by these methods are compared for two ratios of inner to outer shell radius and for varying degrees of anisotropy. Some typical results are given here to show the effect of anisotropy and the thickness of the shell on the distribution of stresses and displacements.

Heat transfer with solid liquid phase change initiating at a point in a semi infinite mold

Short-time analytical solutions of temperature and moving boundary in two-dimensional two-phase freezing due to a cold spot are presented in this paper. The melt occupies a semi-infinite region. Although the method of solution is valid for various other types of boundary conditions, the results in this paper are given only for the prescribed flux boundary conditions which could be space and time dependent. The freezing front propagations along the interior of the melt region exhibit well known behaviours but the propagations along the surface are of new type. The freezing front always depends on material parameters. Several interesting results can be obtained as particular cases of the general results.

Portfolio Optimization in a Semi-Markov Modulated Market

We address a portfolio optimization problem in a semi-Markov modulated market. We study both the terminal expected utility optimization on finite time horizon and the risk-sensitive portfolio optimization on finite and infinite time horizon. We obtain optimal portfolios in relevant cases. A numerical procedure is also developed to compute the optimal expected terminal utility for finite horizon problem.

Unsteady flow of a dilute suspension in a semi-infinite contracting or expanding pipe

In the present paper an exact similar solution of the Navier-Stokes equation for unsteady flow of a dilute suspension in a semi-infinite contracting or expanding circular pipe is presented. The effects of the Schmidt number (Sc), Reynolds number (|ε|), the volume fraction (α) and the relaxation time (τ) of the particulate phase on the flow characteristics are examined. The presence of the solid particles has been observed to influence the flow behaviour significantly. These solutions are valid down to the state of a completely collapsed pipe, since the nonlinearity is retained fully. The results may help understanding the flow near the heart and certain forced contractions or expansions of valved veins.