An actor-critic algorithm with function approximation for discounted cost constrained Markov decision processes

We develop in this article the first actor-critic reinforcement learning algorithm with function approximation for a problem of control under multiple inequality constraints. We consider the infinite horizon discounted cost framework in which both the objective and the constraint functions are suitable expected policy-dependent discounted sums of certain sample path functions. We apply the Lagrange multiplier method to handle the inequality constraints. Our algorithm makes use of multi-timescale stochastic approximation and incorporates a temporal difference (TD) critic and an actor that makes a gradient search in the space of policy parameters using efficient simultaneous perturbation stochastic approximation (SPSA) gradient estimates. We prove the asymptotic almost sure convergence of our algorithm to a locally optimal policy. (C) 2010 Elsevier B.V. All rights reserved.

Dynamic scheduling in manufacturing systems using brownian approximations

Recently, Brownian networks have emerged as an effective stochastic model to approximate multiclass queueing networks with dynamic scheduling capability, under conditions of balanced heavy loading. This paper is a tutorial introduction to dynamic scheduling in manufacturing systems using Brownian networks. The article starts with motivational examples. It then provides a review of relevant weak convergence concepts, followed by a description of the limiting behaviour of queueing systems under heavy traffic. The Brownian approximation procedure is discussed in detail and generic case studies are provided to illustrate the procedure and demonstrate its effectiveness. This paper places emphasis only on the results and aspires to provide the reader with an up-to-date understanding of dynamic scheduling based on Brownian approximations.

Free vibration analysis Of stochastic strings

The free vibration of strings with randomly varying mass and stiffness is considered. The joint probability density functions of the eigenvalues and eigenfunctions are characterized in terms of the solution of a pair of stochastic non-linear initial value problems. Analytical solutions of these equations based on the method of stochastic averaging are obtained. The effects of the mean and autocorrelation of the mass process are included in the analysis. Numerical results for the marginal probability density functions of eigenvalues and eigenfunctions are obtained and are found to compare well with Monte Carlo simulation results. The random eigenvalues, when normalized with respect to their corresponding deterministic values, are observed to tend to become first order stochastically stationary with respect to the mode count.

Convergence of Power Control in a Random Channel Environment

In this paper, we study the Foschini Miljanic algorithm, which was originally proposed in a static channel environment. We investigate the algorithm in a random channel environment, study its convergence properties and apply the Gerschgorin theorem to derive sufficient conditions for the convergence of the algorithm. We apply the Foschini and Miljanic algorithm to cellular networks and derive sufficient conditions for the convergence of the algorithm in distribution and validate the results with simulations. In cellular networks, the conditions which ensure convergence in distribution can be easily verified.

Precipitation in small systems-I. stochastic analysis

Precipitation in small droplets involving emulsions, microemulsions or vesicles is important for Producing multicomponent ceramics and nanoparticles. Because of the random nature of nucleation and the small number of particles in a droplet, the use of a deterministic population balance equation for predicting the number density of particles may lead to erroneous results even for evaluating the mean behavior of such systems. A comparison between the predictions made through stochastic simulation and deterministic population balance involving small droplets has been made for two simple systems, one involving crystallization and the other a single-component precipitation. The two approaches have been found to yield quite different results under a variety of conditions. Contrary to expectation, the smallness of the population alone does not cause these deviations. Thus, if fluctuation in supersaturation is negligible, the population balance and simulation predictions concur. However, for large fluctuations in supersaturation, the predictions differ significantly, indicating the need to take the stochastic nature of the phenomenon into account. This paper describes the stochastic treatment of populations, which involves a sequence of so-called product density equations and forms an appropriate framework for handling small systems.

Monte-Carlo estimation of time-dependent statistical characteristics of random dynamical systems

The problem of estimating the time-dependent statistical characteristics of a random dynamical system is studied under two different settings. In the first, the system dynamics is governed by a differential equation parameterized by a random parameter, while in the second, this is governed by a differential equation with an underlying parameter sequence characterized by a continuous time Markov chain. We propose, for the first time in the literature, stochastic approximation algorithms for estimating various time-dependent process characteristics of the system. In particular, we provide efficient estimators for quantities such as the mean, variance and distribution of the process at any given time as well as the joint distribution and the autocorrelation coefficient at different times. A novel aspect of our approach is that we assume that information on the parameter model (i.e., its distribution in the first case and transition probabilities of the Markov chain in the second) is not available in either case. This is unlike most other work in the literature that assumes availability of such information. Also, most of the prior work in the literature is geared towards analyzing the steady-state system behavior of the random dynamical system while our focus is on analyzing the time-dependent statistical characteristics which are in general difficult to obtain. We prove the almost sure convergence of our stochastic approximation scheme in each case to the true value of the quantity being estimated. We provide a general class of strongly consistent estimators for the aforementioned statistical quantities with regular sample average estimators being a specific instance of these. We also present an application of the proposed scheme on a widely used model in population biology. Numerical experiments in this framework show that the time-dependent process characteristics as obtained using our algorithm in each case exhibit excellent agreement with exact results. (C) 2010 Elsevier Inc. All rights reserved.

Stability of stochastic Leipholz column with stochastic loading

The Leipholz column which is having the Young modulus and mass per unit length as stochastic processes and also the distributed tangential follower load behaving stochastically is considered. The non self-adjoint differential equation and boundary conditions are considered to have random field coefficients. The standard perturbation method is employed. The non self-adjoint operators are used within the regularity domain. Full covariance structure of the free vibration eigenvalues and critical loads is derived in terms of second order properties of input random fields characterizing the system parameter fluctuations. The mean value of critical load is calculated using the averaged problem and the corresponding eigenvalue statistics are sought. Through the frequency equation a transformation is done to yield load parameter statistics. A numerical study incorporating commonly observed correlation models is reported which illustrates the full potentials of the derived expressions.

Stochastic differential games: Occupation measure based approach

A new approach based on occupation measures is introduced for studying stochastic differential games. For two-person zero-sum games, the existence of values and optimal strategies for both players is established for various payoff criteria. ForN-person games, the existence of equilibria in Markov strategies is established for various cases.

Stochastic fluctuations of the solar dynamo

Attempts in the past to model the irregularities of the solar cycle (such as the Maunder minimum) were based on studies of the nonlinear feedback of magnetic fields on the dynamo source terms. Since the alpha-coefficient is obtained by averaging over the turbulence, it is expected to have stochastic fluctuations, and we show that these fluctuations can explain the irregularities of the solar cycle in a more satisfactory way. We solve the dynamo equations in a slab with a single mode, taking the alpha-coefficient to be constant in space but fluctuating stochastically in time with some given amplitude and given correlation time. The same level of percentile fluctuations (about 10 %) produces no effect on an alpha-omega dynamo, but makes an alpha-2 dynamo completely chaotic. The level of irregularities in an alpha-2-omega dynamo qualitatively agrees with the solar behavior, reinforcing the conclusion of Choudhuri (1990a) that the solar dynamo is of the alpha-2-omega-type. The irregularities are found to increase on increasing either the amplitude or the correlation time of the stochastic fluctuations. The alpha-quenching mechanism tends to make the system stable against the irregularities and hence it is inferred that the alpha-quenching should not be too strong so that the irregularities are not completely suppressed. We also present a simple-minded analysis to understand why the stochastic fluctuations in the alpha-omega, alpha-2-omega and alpha-2 regimes have such different outcomes.

Stochastic modelling of flexible manufacturing systems

Mathematical modelling plays a vital role in the design, planning and operation of flexible manufacturing systems (FMSs). In this paper, attention is focused on stochastic modelling of FMSs using Markov chains, queueing networks, and stochastic Petri nets. We bring out the role of these modelling tools in FMS performance evaluation through several illustrative examples and provide a critical comparative evaluation. We also include a discussion on the modelling of deadlocks which constitute an important source of performance degradation in fully automated FMSs.

Variability of sif and cod of stochastic structural systems

Stochastic structural systems having a stochastic distribution of material properties and stochastic external loadings in space are analysed when a crack of deterministic size is present. The material properties and external loadings are considered to constitute independent, two-dimensional, univariate, real, homogeneous stochastic fields. The stochastic fields are characterized by their means, variances, autocorrelation functions or the equivalent power spectral density functions, and scale fluctuations. The Young's modulus and Poisson's ratio are treated to be stochastic quantities. The external loading is treated to be a stochastic field in space. The energy release rate is derived using the method of virtual crack extension. The deterministic relationship is derived to represent the sensitivities of energy release rate with respect to both virtual crack extension and real system parameter fluctuations. Taylor series expansion is used and truncation is made to the first order. This leads to the determination of second-order properties of the output quantities to the first order. Using the linear perturbations about the mean values of the output quantities, the statistical information about the energy release rates, SIF and crack opening displacements are obtained. Both plane stress and plane strain cases are considered. The general expressions for the SIF in all the three fracture modes are derived and a more detailed analysis is conducted for a mode I situation. A numerical example is given.

Improving the convergence rate of the inverse iteration method

Solution of generalized eigenproblem, K phi = lambda M phi, by the classical inverse iteration method exhibits slow convergence for some eigenproblems. In this paper, a modified inverse iteration algorithm is presented for improving the convergence rate. At every iteration, an optimal linear combination of the latest and the preceding iteration vectors is used as the input vector for the next iteration. The effectiveness of the proposed algorithm is demonstrated for three typical eigenproblems, i.e. eigenproblems with distinct, close and repeated eigenvalues. The algorithm yields 29, 96 and 23% savings in computational time, respectively, for these problems. The algorithm is simple and easy to implement, and this renders the algorithm even more attractive.

Vibration and Stability of Fluid Conveying Pipes with Stochastic Parameters

Flexible cantilever pipes conveying fluids with high velocity are analysed for their dynamic response and stability behaviour. The Young's modulus and mass per unit length of the pipe material have a stochastic distribution. The stochastic fields, that model the fluctuations of Young's modulus and mass density are characterized through their respective means, variances and autocorrelation functions or their equivalent power spectral density functions. The stochastic non self-adjoint partial differential equation is solved for the moments of characteristic values, by treating the point fluctuations to be stochastic perturbations. The second-order statistics of vibration frequencies and mode shapes are obtained. The critical flow velocity is-first evaluated using the averaged eigenvalue equation. Through the eigenvalue equation, the statistics of vibration frequencies are transformed to yield critical flow velocity statistics. Expressions for the bounds of eigenvalues are obtained, which in turn yield the corresponding bounds for critical flow velocities.

Stochastic Aeroelastic Analysis of Composite Helicopter Rotor

The effect of uncertainty in composite material properties on the aeroelastic response, vibratory loads, and stability of a hingeless helicopter rotor is investigated. The uncertainty impact on rotating natural frequencies of the blade is studied with Monte Carlo simulations and first-order reliability methods. The stochastic aeroelastic analyses in hover and forward flight are carried out with Monte Carlo simulations. The flap, lag, and torsion responses show considerable scatter from their baseline values, and the uncertainty impact varies with the azimuth angle. Furthermore, the blade response shows finite probability of resonance-type conditions caused by modal frequencies approaching multiples of the rotor speed. The 4/rev vibratory forces show large deviations from their baseline values. The lag mode damping shows considerable scatter due to uncertain material properties with an almost 40% probability of instability in hover.