The actor-critic algorithm as multi-time-scale stochastic approximation

The actor-critic algorithm of Barto and others for simulation-based optimization of Markov decision processes is cast as a two time Scale stochastic approximation. Convergence analysis, approximation issues and an example are studied.

Optimal control of a stochastic hybrid system with discounted cost

We address the optimal control problem of a very general stochastic hybrid system with both autonomous and impulsive jumps. The planning horizon is infinite and we use the discounted-cost criterion for performance evaluation. Under certain assumptions, we show the existence of an optimal control. We then derive the quasivariational inequalities satisfied by the value function and establish well-posedness. Finally, we prove the usual verification theorem of dynamic programming.

Process-kinetics in facing targets sputtering of multi-component oxide thin films

The kinetics of the processes in facing targets sputtering of multicomponent oxide films is presented. The novel configuration of the process exhibits an enhanced ionization efficiency. Discharge diagnostics performed using optical emission spectroscopy revealed strong dependence of plasma parameters on process conditions. Numerical simulation based on thermalization and diffusion of sputtered atoms has been performed to estimate the transport efficiency in off-axis mode. Composition, structure and epitaxial quality of YBa2Cu3O7-x films prepared was found to be strongly dependent on atomic flux ratios (of Cu/Y and Ba/Y) arriving at the substrate, resputtering effect and phase stability of YBa2Cu3O7-x These studies have been shown to be useful in understanding the complex processes that occur in sputtering of multicomponent films. (C) 1999 Elsevier Science S.A. All rights reserved.

Stochastic dynamics modeling of the protein sequence length distribution in genomes: implications for microbial evolution

In this paper, we report an analysis of the protein sequence length distribution for 13 bacteria, four archaea and one eukaryote whose genomes have been completely sequenced, The frequency distribution of protein sequence length for all the 18 organisms are remarkably similar, independent of genome size and can be described in terms of a lognormal probability distribution function. A simple stochastic model based on multiplicative processes has been proposed to explain the sequence length distribution. The stochastic model supports the random-origin hypothesis of protein sequences in genomes. Distributions of large proteins deviate from the overall lognormal behavior. Their cumulative distribution follows a power-law analogous to Pareto's law used to describe the income distribution of the wealthy. The protein sequence length distribution in genomes of organisms has important implications for microbial evolution and applications. (C) 1999 Elsevier Science B.V. All rights reserved.

2-Halooxyethylene ethers of cholesterol as novel single component, room temperature cholesteric LC materials

A series of 2-haloethoxyethyl cholesteryl ethers has been synthesized. Each material shows attractive liquid-crystalline properties as revealed by differential scanning calorimetry, polarizing microscopy, and temperature-dependence of selective reflection characteristic of the cholesteric mesophase. These are interesting examples of simple, nonpolymeric, single component systems that show the cholesteric mesophase at room temperature.

Providing quality of service via opportunistic splitting using stochastic approximation

We consider the problem of wireless channel allocation to multiple users. A slot is given to a user with a highest metric (e.g., channel gain) in that slot. The scheduler may not know the channel states of all the users at the beginning of each slot. In this scenario opportunistic splitting is an attractive solution. However this algorithm requires that the metrics of different users form independent, identically distributed (iid) sequences with same distribution and that their distribution and number be known to the scheduler. This limits the usefulness of opportunistic splitting. In this paper we develop a parametric version of this algorithm. The optimal parameters of the algorithm are learnt online through a stochastic approximation scheme. Our algorithm does not require the metrics of different users to have the same distribution. The statistics of these metrics and the number of users can be unknown and also vary with time. Each metric sequence can be Markov. We prove the convergence of the algorithm and show its utility by scheduling the channel to maximize its throughput while satisfying some fairness and/or quality of service constraints.

Opportunistic splitting for scheduling via stochastic approximation

We consider the problem of scheduling a wireless channel among multiple users. A slot is given to a user with a highest metric (e.g., channel gain) in that slot. The scheduler may not know the channel states of all the users at the beginning of each slot. In this scenario opportunistic splitting is an attractive solution. However this algorithm requires that the metrics of different users form independent, identically distributed (iid) sequences with same distribution and that their distribution and number be known to the scheduler. This limits the usefulness of opportunistic splitting. In this paper we develop a parametric version of this algorithm. The optimal parameters of the algorithm are learnt online through a stochastic approximation scheme. Our algorithm does not require the metrics of different users to have the same distribution. The statistics of these metrics and the number of users can be unknown and also vary with time. We prove the convergence of the algorithm and show its utility by scheduling the channel to maximize its throughput while satisfying some fairness and/or quality of service constraints.

Common mode DC bus filter for Active Front-End converter

Active Front-End (AFE) converter operation produces electrically noisy DC bus on common mode basis. This results in higher ground current as compared to three phase diode bridge rectifier. Filter topologies for DC bus have to deal problems with switching frequency and harmonic currents. The proposed filter approach reduces common mode voltage and circulates third harmonic current within the system, resulting in minimal ground current injection. The filtering technique, its constrains and design to attenuate common mode voltage and eliminate lower order harmonics injection to ground is discussed. The experimental results for operation of the converter with both SPWM and CSVPWM are presented.

Nonlinear Dynamics of Beams with Stochastic Parameter Variations

The aim of this paper is to investigate the steady state response of beams under the action of random support motions. The study is of relevance in the context of earthquake response of extended land based structures such as pipelines and long span bridges, and, secondary systems such as piping networks in nuclear power plant installations. The following complicating features are accounted for in the response analysis: (a) differential support motions: this is characterized in terms of cross power spectral density functions associated with distinct support motions, (b) nonlinear support conditions, and (c) stochastically inhomogeneous stiffness and mass variations of the beam structure; questions on non-Gaussian models for these variations are considered. The method of stochastic finite elements is combined with equivalent linearization technique and Monte Carlo simulations to obtain response moments.

The Borkar-Meyn theorem for asynchronous stochastic approximations

In this paper, we give a generalization of a result by Borkar and Meyn (2000) 1], on the stability and convergence of synchronous-update stochastic approximation algorithms, to the case of asynchronous stochastic approximations with delays. We then describe an interesting application of the result to asynchronous distributed temporal difference (TD) learning with function approximation and delays. (C) 2011 Elsevier B.V. All rights reserved.

Modeling of nonreacting and reacting turbulent spray jets using a fully stochastic separated flow approach

Numerical modeling of several turbulent nonreacting and reacting spray jets is carried out using a fully stochastic separated flow (FSSF) approach. As is widely used, the carrier-phase is considered in an Eulerian framework, while the dispersed phase is tracked in a Lagrangian framework following the stochastic separated flow (SSF) model. Various interactions between the two phases are taken into account by means of two-way coupling. Spray evaporation is described using a thermal model with an infinite conductivity in the liquid phase. The gas-phase turbulence terms are closed using the k-epsilon model. A novel mixture fraction based approach is used to stochastically model the fluctuating temperature and composition in the gas phase and these are then used to refine the estimates of the heat and mass transfer rates between the droplets and the surrounding gas-phase. In classical SSF (CSSF) methods, stochastic fluctuations of only the gas-phase velocity are modeled. Successful implementation of the FSSF approach to turbulent nonreacting and reacting spray jets is demonstrated. Results are compared against experimental measurements as well as with predictions using the CSSF approach for both nonreacting and reacting spray jets. The FSSF approach shows little difference from the CSSF predictions for nonreacting spray jets but differences are significant for reacting spray jets. In general, the FSSF approach gives good predictions of the flame length and structure but further improvements in modeling may be needed to improve the accuracy of some details of the Predictions. (C) 2011 The Combustion Institute. Published by Elsevier Inc. All rights reserved.