Multiscale stochastic approximation for parametric optimization of hidden Markov models

A two-time scale stochastic approximation algorithm is proposed for simulation-based parametric optimization of hidden Markov models, as an alternative to the traditional approaches to ''infinitesimal perturbation analysis.'' Its convergence is analyzed, and a queueing example is presented.

The actor-critic algorithm as multi-time-scale stochastic approximation

The actor-critic algorithm of Barto and others for simulation-based optimization of Markov decision processes is cast as a two time Scale stochastic approximation. Convergence analysis, approximation issues and an example are studied.

Optimal control of a stochastic hybrid system with discounted cost

We address the optimal control problem of a very general stochastic hybrid system with both autonomous and impulsive jumps. The planning horizon is infinite and we use the discounted-cost criterion for performance evaluation. Under certain assumptions, we show the existence of an optimal control. We then derive the quasivariational inequalities satisfied by the value function and establish well-posedness. Finally, we prove the usual verification theorem of dynamic programming.

Stochastic dynamics modeling of the protein sequence length distribution in genomes: implications for microbial evolution

In this paper, we report an analysis of the protein sequence length distribution for 13 bacteria, four archaea and one eukaryote whose genomes have been completely sequenced, The frequency distribution of protein sequence length for all the 18 organisms are remarkably similar, independent of genome size and can be described in terms of a lognormal probability distribution function. A simple stochastic model based on multiplicative processes has been proposed to explain the sequence length distribution. The stochastic model supports the random-origin hypothesis of protein sequences in genomes. Distributions of large proteins deviate from the overall lognormal behavior. Their cumulative distribution follows a power-law analogous to Pareto's law used to describe the income distribution of the wealthy. The protein sequence length distribution in genomes of organisms has important implications for microbial evolution and applications. (C) 1999 Elsevier Science B.V. All rights reserved.

Providing quality of service via opportunistic splitting using stochastic approximation

We consider the problem of wireless channel allocation to multiple users. A slot is given to a user with a highest metric (e.g., channel gain) in that slot. The scheduler may not know the channel states of all the users at the beginning of each slot. In this scenario opportunistic splitting is an attractive solution. However this algorithm requires that the metrics of different users form independent, identically distributed (iid) sequences with same distribution and that their distribution and number be known to the scheduler. This limits the usefulness of opportunistic splitting. In this paper we develop a parametric version of this algorithm. The optimal parameters of the algorithm are learnt online through a stochastic approximation scheme. Our algorithm does not require the metrics of different users to have the same distribution. The statistics of these metrics and the number of users can be unknown and also vary with time. Each metric sequence can be Markov. We prove the convergence of the algorithm and show its utility by scheduling the channel to maximize its throughput while satisfying some fairness and/or quality of service constraints.

Opportunistic splitting for scheduling via stochastic approximation

We consider the problem of scheduling a wireless channel among multiple users. A slot is given to a user with a highest metric (e.g., channel gain) in that slot. The scheduler may not know the channel states of all the users at the beginning of each slot. In this scenario opportunistic splitting is an attractive solution. However this algorithm requires that the metrics of different users form independent, identically distributed (iid) sequences with same distribution and that their distribution and number be known to the scheduler. This limits the usefulness of opportunistic splitting. In this paper we develop a parametric version of this algorithm. The optimal parameters of the algorithm are learnt online through a stochastic approximation scheme. Our algorithm does not require the metrics of different users to have the same distribution. The statistics of these metrics and the number of users can be unknown and also vary with time. We prove the convergence of the algorithm and show its utility by scheduling the channel to maximize its throughput while satisfying some fairness and/or quality of service constraints.

Nonlinear Dynamics of Beams with Stochastic Parameter Variations

The aim of this paper is to investigate the steady state response of beams under the action of random support motions. The study is of relevance in the context of earthquake response of extended land based structures such as pipelines and long span bridges, and, secondary systems such as piping networks in nuclear power plant installations. The following complicating features are accounted for in the response analysis: (a) differential support motions: this is characterized in terms of cross power spectral density functions associated with distinct support motions, (b) nonlinear support conditions, and (c) stochastically inhomogeneous stiffness and mass variations of the beam structure; questions on non-Gaussian models for these variations are considered. The method of stochastic finite elements is combined with equivalent linearization technique and Monte Carlo simulations to obtain response moments.

The Borkar-Meyn theorem for asynchronous stochastic approximations

In this paper, we give a generalization of a result by Borkar and Meyn (2000) 1], on the stability and convergence of synchronous-update stochastic approximation algorithms, to the case of asynchronous stochastic approximations with delays. We then describe an interesting application of the result to asynchronous distributed temporal difference (TD) learning with function approximation and delays. (C) 2011 Elsevier B.V. All rights reserved.

Modeling of nonreacting and reacting turbulent spray jets using a fully stochastic separated flow approach

Numerical modeling of several turbulent nonreacting and reacting spray jets is carried out using a fully stochastic separated flow (FSSF) approach. As is widely used, the carrier-phase is considered in an Eulerian framework, while the dispersed phase is tracked in a Lagrangian framework following the stochastic separated flow (SSF) model. Various interactions between the two phases are taken into account by means of two-way coupling. Spray evaporation is described using a thermal model with an infinite conductivity in the liquid phase. The gas-phase turbulence terms are closed using the k-epsilon model. A novel mixture fraction based approach is used to stochastically model the fluctuating temperature and composition in the gas phase and these are then used to refine the estimates of the heat and mass transfer rates between the droplets and the surrounding gas-phase. In classical SSF (CSSF) methods, stochastic fluctuations of only the gas-phase velocity are modeled. Successful implementation of the FSSF approach to turbulent nonreacting and reacting spray jets is demonstrated. Results are compared against experimental measurements as well as with predictions using the CSSF approach for both nonreacting and reacting spray jets. The FSSF approach shows little difference from the CSSF predictions for nonreacting spray jets but differences are significant for reacting spray jets. In general, the FSSF approach gives good predictions of the flame length and structure but further improvements in modeling may be needed to improve the accuracy of some details of the Predictions. (C) 2011 The Combustion Institute. Published by Elsevier Inc. All rights reserved.

Doubly spectral stochastic finite element method (DSSFEM) for random field problems

Uncertainties in complex dynamic systems play an important role in the prediction of a dynamic response in the mid- and high-frequency ranges. For distributed parameter systems, parametric uncertainties can be represented by random fields leading to stochastic partial differential equations. Over the past two decades, the spectral stochastic finite-element method has been developed to discretize the random fields and solve such problems. On the other hand, for deterministic distributed parameter linear dynamic systems, the spectral finite-element method has been developed to efficiently solve the problem in the frequency domain. In spite of the fact that both approaches use spectral decomposition (one for the random fields and the other for the dynamic displacement fields), very little overlap between them has been reported in literature. In this paper, these two spectral techniques are unified with the aim that the unified approach would outperform any of the spectral methods considered on their own. An exponential autocorrelation function for the random fields, a frequency-dependent stochastic element stiffness, and mass matrices are derived for the axial and bending vibration of rods. Closed-form exact expressions are derived by using the Karhunen-Loève expansion. Numerical examples are given to illustrate the unified spectral approach.

Stochastic Algorithms for Discrete Parameter Simulation Optimization

We present two efficient discrete parameter simulation optimization (DPSO) algorithms for the long-run average cost objective. One of these algorithms uses the smoothed functional approximation (SFA) procedure, while the other is based on simultaneous perturbation stochastic approximation (SPSA). The use of SFA for DPSO had not been proposed previously in the literature. Further, both algorithms adopt an interesting technique of random projections that we present here for the first time. We give a proof of convergence of our algorithms. Next, we present detailed numerical experiments on a problem of admission control with dependent service times. We consider two different settings involving parameter sets that have moderate and large sizes, respectively. On the first setting, we also show performance comparisons with the well-studied optimal computing budget allocation (OCBA) algorithm and also the equal allocation algorithm. Note to Practitioners-Even though SPSA and SFA have been devised in the literature for continuous optimization problems, our results indicate that they can be powerful techniques even when they are adapted to discrete optimization settings. OCBA is widely recognized as one of the most powerful methods for discrete optimization when the parameter sets are of small or moderate size. On a setting involving a parameter set of size 100, we observe that when the computing budget is small, both SPSA and OCBA show similar performance and are better in comparison to SFA, however, as the computing budget is increased, SPSA and SFA show better performance than OCBA. Both our algorithms also show good performance when the parameter set has a size of 10(8). SFA is seen to show the best overall performance. Unlike most other DPSO algorithms in the literature, an advantage with our algorithms is that they are easily implementable regardless of the size of the parameter sets and show good performance in both scenarios.