The actor-critic algorithm as multi-time-scale stochastic approximation

The actor-critic algorithm of Barto and others for simulation-based optimization of Markov decision processes is cast as a two time Scale stochastic approximation. Convergence analysis, approximation issues and an example are studied.

Stochastic dynamics modeling of the protein sequence length distribution in genomes: implications for microbial evolution

In this paper, we report an analysis of the protein sequence length distribution for 13 bacteria, four archaea and one eukaryote whose genomes have been completely sequenced, The frequency distribution of protein sequence length for all the 18 organisms are remarkably similar, independent of genome size and can be described in terms of a lognormal probability distribution function. A simple stochastic model based on multiplicative processes has been proposed to explain the sequence length distribution. The stochastic model supports the random-origin hypothesis of protein sequences in genomes. Distributions of large proteins deviate from the overall lognormal behavior. Their cumulative distribution follows a power-law analogous to Pareto's law used to describe the income distribution of the wealthy. The protein sequence length distribution in genomes of organisms has important implications for microbial evolution and applications. (C) 1999 Elsevier Science B.V. All rights reserved.

Non-Stationary Semi-Markov Decision Processes on a Finite Horizon

We introduce and study a class of non-stationary semi-Markov decision processes on a finite horizon. By constructing an equivalent Markov decision process, we establish the existence of a piecewise open loop relaxed control which is optimal for the finite horizon problem.

Confinement induced stochastic sensing of charged coronene and perylene aggregates in alpha-hemolysin nanochannels

Biological nanopores provide optimum dimensions and an optimal environment to study early aggregation kinetics of charged polyaromatic molecules in the nano-confined regime. It is expected that probing early stages of nucleation will enable us to design a strategy for supramolecular assembly and biocrystallization processes. Specifically, we have studied translocation dynamics of coronene and perylene based salts, through the alpha-hemolysin (alpha-HL) protein nanopore. The characteristic blocking events in the time-series signal are a function of concentration and bias voltage. We argue that different blocking events arise due to different aggregation processes as captured by all atomistic molecular dynamics (MD) simulations. These confinement induced aggregations of polyaromatic chromophores during the different stages of translocation are correlated with the spatial symmetry and charge distribution of the molecules.

Risk-Sensitive Ergodic Control of Continuous Time Markov Processes With Denumerable State Space

In this article, we study risk-sensitive control problem with controlled continuous time Markov chain state dynamics. Using multiplicative dynamic programming principle along with the atomic structure of the state dynamics, we prove the existence and a characterization of optimal risk-sensitive control under geometric ergodicity of the state dynamics along with a smallness condition on the running cost.

Stochastic differential games with multiple modes

We have studied two person stochastic differential games with multiple modes. For the zero-sum game we have established the existence of optimal strategies for both players. For the nonzero-sum case we have proved the existence of a Nash equilibrium.

Study of the processes occurring when sputtering YBa2Cu3O7-x in pure oxygen

The sputter deposition of YBa2Cu3O7-x in a de-diode was performed in pure oxygen medium and an optical spectroscopic study of the resultant discharge revealed strong emissions from both metal atoms and oxygen ions. Emission intensities were studied in pressure range from 0.5 to 3 mbar, with substrate temperatures from 150 to 850 degrees C. Raising the substrate temperature to 850 degrees C increased the number of positive ions and excited neutral atoms. Raising the pressure decreased the emission intensities of excited neutral and ionic species. The results have been compared with those obtained from Langmuir probe measurements. The rise in emission intensities of excited neutrals and ions with temperature suggested the possibility of chemically enhanced physical sputtering of YBa2Cu3O7-x. The effect of process conditions on film composition and quality is also discussed.

Fluctuations in protein synthesis from a single RNA template: Stochastic kinetics of ribosomes

Proteins are polymerized by cyclic machines called ribosomes, which use their messenger RNA (mRNA) track also as the corresponding template, and the process is called translation. We explore, in depth and detail, the stochastic nature of the translation. We compute various distributions associated with the translation process; one of them-namely, the dwell time distribution-has been measured in recent single-ribosome experiments. The form of the distribution, which fits best with our simulation data, is consistent with that extracted from the experimental data. For our computations, we use a model that captures both the mechanochemistry of each individual ribosome and their steric interactions. We also demonstrate the effects of the sequence inhomogeneities of real genes on the fluctuations and noise in translation. Finally, inspired by recent advances in the experimental techniques of manipulating single ribosomes, we make theoretical predictions on the force-velocity relation for individual ribosomes. In principle, all our predictions can be tested by carrying out in vitro experiments.

Intermittent Dynamics, Stochastic Resonance and Dynamical Heterogeneity in Supercooled Liquid Water

Here we find through computer simulations and theoretical analysis that the low temperature thermodynamic anomalies of liquid water arises from the intermittent fluctuation between its high density and low density forms, consisting largely of 5-coordinated and 4-coordinated water molecules, respectively. The fluctuations exhibit strong dynamic heterogeneity (defined by the four point time correlation function), accompanied by a divergence like growth of the dynamic correlation length, of the type encountered in fragile supercooled liquids. The intermittency has been explained by invoking a two state model often employed to understand stochastic resonance, with the relevant periodic perturbation provided here by the fluctuation of the total volume of the system.

Influence of friction during forming processes-a study using a numerical simulation technique

Friction has an important influence in metal forming operations, as it contributes to the success or otherwise of the process. In the present investigation, the effect of friction on metal forming was studied by simulating compression tests on cylindrical Al-Mg alloy using the finite element method (FEM) technique. Three kinds of compression tests were considered wherein a constant coefficient of friction was employed at the upper die-work-piece interface. However, the coefficient of friction between the lower die-work-piece interfaces was varied in the tests. The simulation results showed that a difference in metal flow occurs near the interfaces owing to the differences in the coefficient of friction. It was concluded that the variations in the coefficient of friction between the dies and the work-piece directly affect the stress distribution and shape of the work-piece, having implications on the microstructure of the material being processed.

Thermal Degradation Processes in Poly(xylylene sulfides) Investigated by Comparative Direct Pyrolysis MS and Flash Pyrolysis GC/MS Experiments

The thermal degradation processes of two sulfur polymers, poly(xylylene sulfide) (PXM) and poly(xylylene disulfide) (PXD), were investigated in parallel by direct pyrolysis mass spectrometry (DPMS) and flash pyrolysis GC/MS (Py-GC/MS). Thermogravimetric data showed that these polymers decompose with two separate steps in the temperature ranges of 250-280 and 600-650 degrees C, leaving a high amount of residue (about 50% at 800 degrees C). The pyrolysis products detected by DPMS in the first degradation step of PXM and PXD were terminated by three types of end groups, -CH3, -CH2SH, and -CH=S, originating from thermal cleavage reactions involving a series of homolytic chain scissions followed by hydrogen transfer reactions, generating several oligomers containing some intact xylylene sulfide repeating units. The presence of pyrolysis compounds containing some stilbene-like units in the first degradation step has also been observed. Their formation has been accounted for with a parallel cleavage involving the elimination of H2S from the PXM main chains. These unsaturated units can undergo cross-linking at higher temperatures, producing the high amount of char residue observed. The thermal degradation compounds detected by DPMS in the second decomposition step at about 600-650 degrees C were constituted of condensed aromatic molecules containing dihydrofenanthrene and fenanthrene units. These compounds might be generated from the polymer chains containing stilbene units, by isomerization and dehydrogenation reactions. The pyrolysis products obtained in the Py-GC/MS of PXM and PXD at 610 degrees C are almost identical. The relative abundance in the pyrolysate and the spectral properties of the main pyrolysis products were found to be in generally good agreement with those obtained by DPMS. Polycyclic aromatic hydrocarbons (PAHs) were also detected by Py-GC/MS but in minor amounts with respect to DPMS. This apparent discrepancy was due to the simultaneous detection of PAHs together with all pyrolysis products in the Py-GC/MS, whereas in DPMS they were detected in the second thermal degradation step without the greatest part of pyrolysis compounds generated in the first degradation step. The results obtained by DPMS and PSI-GC/MS experiments showed complementary data for the degradation of PXM and PXD and, therefore, allowed the unequivocal formulation of the thermal degradation mechanism for these sulfur-containing polymers.

Precipitation In Small Systems--II. Mean Field Equations More Effective Than Population Balance

Part I (Manjunath et al., 1994, Chem. Engng Sci. 49, 1451-1463) of this paper showed that the random particle numbers and size distributions in precipitation processes in very small drops obtained by stochastic simulation techniques deviate substantially from the predictions of conventional population balance. The foregoing problem is considered in this paper in terms of a mean field approximation obtained by applying a first-order closure to an unclosed set of mean field equations presented in Part I. The mean field approximation consists of two mutually coupled partial differential equations featuring (i) the probability distribution for residual supersaturation and (ii) the mean number density of particles for each size and supersaturation from which all average properties and fluctuations can be calculated. The mean field equations have been solved by finite difference methods for (i) crystallization and (ii) precipitation of a metal hydroxide both occurring in a single drop of specified initial supersaturation. The results for the average number of particles, average residual supersaturation, the average size distribution, and fluctuations about the average values have been compared with those obtained by stochastic simulation techniques and by population balance. This comparison shows that the mean field predictions are substantially superior to those of population balance as judged by the close proximity of results from the former to those from stochastic simulations. The agreement is excellent for broad initial supersaturations at short times but deteriorates progressively at larger times. For steep initial supersaturation distributions, predictions of the mean field theory are not satisfactory thus calling for higher-order approximations. The merit of the mean field approximation over stochastic simulation lies in its potential to reduce expensive computation times involved in simulation. More effective computational techniques could not only enhance this advantage of the mean field approximation but also make it possible to use higher-order approximations eliminating the constraints under which the stochastic dynamics of the process can be predicted accurately.

Response And Stability Of A Stochastic Beam-Column Using Stochastic Fem

A beam-column resting on continuous Winkler foundation and discrete elastic supports is considered. The beam-column is of variable cross-section and the variation of sectional properties along the axis of the beam-column is deterministic. Young's modulus, mass per unit length and distributed axial loadings of the beam-column have a stochastic distribution. The foundation stiffness coefficient of the Winkler model, the stiffnesses of discrete elastic supports, stiffnesses of end springs and the end thrust, are all considered as random parameters. The material property fluctuations and distributed axial loadings are considered to constitute independent, one-dimension uni-variate homogeneous real stochastic fields in space. The foundation stiffness coefficient, stiffnesses of the discrete elastic supports, stiffnesses of end springs and the end thrust are considered to constitute independent random variables. Static response, free vibration and stability behaviour of the beam-column are studied. Hamilton's principle is used to formulate the problem using stochastic FEM. Sensitivity vectors of the response and stability parameters are evaluated. Using these statistics of free vibration frequencies, mode shapes, buckling parameters, etc., are evaluated. A numerical example is given.

Performance modelling of a fault-tolerant real-time multiprocessor using stochastic Petri nets

The fault-tolerant multiprocessor (ftmp) is a bus-based multiprocessor architecture with real-time and fault- tolerance features and is used in critical aerospace applications. A preliminary performance evaluation is of crucial importance in the design of such systems. In this paper, we review stochastic Petri nets (spn) and developspn-based performance models forftmp. These performance models enable efficient computation of important performance measures such as processing power, bus contention, bus utilization, and waiting times.

Timing the Emergence of Resistance to Anti-HIV Drugs with Large Genetic Barriers

New antiretroviral drugs that offer large genetic barriers to resistance, such as the recently approved inhibitors of HIV-1 protease, tipranavir and darunavir, present promising weapons to avert the failure of current therapies for HIV infection. Optimal treatment strategies with the new drugs, however, are yet to be established. A key limitation is the poor understanding of the process by which HIV surmounts large genetic barriers to resistance. Extant models of HIV dynamics are predicated on the predominance of deterministic forces underlying the emergence of resistant genomes. In contrast, stochastic forces may dominate, especially when the genetic barrier is large, and delay the emergence of resistant genomes. We develop a mathematical model of HIV dynamics under the influence of an antiretroviral drug to predict the waiting time for the emergence of genomes that carry the requisite mutations to overcome the genetic barrier of the drug. We apply our model to describe the development of resistance to tipranavir in in vitro serial passage experiments. Model predictions of the times of emergence of different mutant genomes with increasing resistance to tipranavir are in quantitative agreement with experiments, indicating that our model captures the dynamics of the development of resistance to antiretroviral drugs accurately. Further, model predictions provide insights into the influence of underlying evolutionary processes such as recombination on the development of resistance, and suggest guidelines for drug design: drugs that offer large genetic barriers to resistance with resistance sites tightly localized on the viral genome and exhibiting positive epistatic interactions maximally inhibit the emergence of resistant genomes.