Efficient methods for stochastic optimal control

The Hamilton Jacobi Bellman (HJB) equation is central to stochastic optimal control (SOC) theory, yielding the optimal solution to general problems specified by known dynamics and a specified cost functional. Given the assumption of quadratic cost on the control input, it is well known that the HJB reduces to a particular partial differential equation (PDE). While powerful, this reduction is not commonly used as the PDE is of second order, is nonlinear, and examples exist where the problem may not have a solution in a classical sense. Furthermore, each state of the system appears as another dimension of the PDE, giving rise to the curse of dimensionality. Since the number of degrees of freedom required to solve the optimal control problem grows exponentially with dimension, the problem becomes intractable for systems with all but modest dimension.

In the last decade researchers have found that under certain, fairly non-restrictive structural assumptions, the HJB may be transformed into a linear PDE, with an interesting analogue in the discretized domain of Markov Decision Processes (MDP). The work presented in this thesis uses the linearity of this particular form of the HJB PDE to push the computational boundaries of stochastic optimal control.

This is done by crafting together previously disjoint lines of research in computation. The first of these is the use of Sum of Squares (SOS) techniques for synthesis of control policies. A candidate polynomial with variable coefficients is proposed as the solution to the stochastic optimal control problem. An SOS relaxation is then taken to the partial differential constraints, leading to a hierarchy of semidefinite relaxations with improving sub-optimality gap. The resulting approximate solutions are shown to be guaranteed over- and under-approximations for the optimal value function. It is shown that these results extend to arbitrary parabolic and elliptic PDEs, yielding a novel method for Uncertainty Quantification (UQ) of systems governed by partial differential constraints. Domain decomposition techniques are also made available, allowing for such problems to be solved via parallelization and low-order polynomials.

The optimization-based SOS technique is then contrasted with the Separated Representation (SR) approach from the applied mathematics community. The technique allows for systems of equations to be solved through a low-rank decomposition that results in algorithms that scale linearly with dimensionality. Its application in stochastic optimal control allows for previously uncomputable problems to be solved quickly, scaling to such complex systems as the Quadcopter and VTOL aircraft. This technique may be combined with the SOS approach, yielding not only a numerical technique, but also an analytical one that allows for entirely new classes of systems to be studied and for stability properties to be guaranteed.

The analysis of the linear HJB is completed by the study of its implications in application. It is shown that the HJB and a popular technique in robotics, the use of navigation functions, sit on opposite ends of a spectrum of optimization problems, upon which tradeoffs may be made in problem complexity. Analytical solutions to the HJB in these settings are available in simplified domains, yielding guidance towards optimality for approximation schemes. Finally, the use of HJB equations in temporal multi-task planning problems is investigated. It is demonstrated that such problems are reducible to a sequence of SOC problems linked via boundary conditions. The linearity of the PDE allows us to pre-compute control policy primitives and then compose them, at essentially zero cost, to satisfy a complex temporal logic specification.

Robustness and dissipation of mitogen-activated protein kinases signal transduction network: Underlying funneled landscape against stochastic fluctuations

We uncovered the underlying energy landscape of the mitogen-activated protein kinases signal transduction cellular network by exploring the statistical natures of the Brownian dynamical trajectories. We introduce a dimensionless quantity: The robustness ratio of energy gap versus local roughness to measure the global topography of the underlying landscape. A high robustness ratio implies funneled landscape. The landscape is quite robust against environmental fluctuations and variants of the intrinsic chemical reaction rates.

Moment decay rates of solutions of stochastic differential equations

The objective of this paper is to investigate the p-ίh moment asymptotic stability decay rates for certain finite-dimensional Itό stochastic differential equations. Motivated by some practical examples, the point of our analysis is a special consideration of general decay speeds, which contain as a special case the usual exponential or polynomial type one, to meet various situations. Sufficient conditions for stochastic differential equations (with variable delays or not) are obtained to ensure their asymptotic properties. Several examples are studied to illustrate our theory.

Stability of a Seabird Population in the Presence of an Introduced Predator

We hypothesized that although large populations may appear able to withstand predation and disturbance, added stochasticity in population growth rate (λ) increases the risk of dramatic population declines. Approximately half of the Aleutian Islands' population of Least Auklets (Aethia pusilla) breed at one large colony at Kiska Island in the presence of introduced Norway rats (Rattus norvegicus) whose population erupts periodically. We evaluated two management plans, do nothing or eradicate rats, for this colony, and performed stochastic elasticity analysis to focus future research and management. Our results indicated that Least Auklets breeding at Kiska Island had the lowest absolute value of growth rate and more variable λ's (neither statistically significant) during 2001-2010, when compared with rat-free colonies at Buldir and Kasatochi islands. We found variability in the annual proportional change in population size among islands with Kiska Island having the fastest rate of decline, 78% over 20 years. Under the assumption that the eradication of rats would result in vital rates similar to those observed at rat-free Buldir and Kasatochi islands, we found the projected population decline decreased from 78% to 24% over 20 years. Overall, eradicating rats at Kiska Island is not likely to increase Least Auklet vital rates, but will decrease the amount of variation in λ, resulting in a significantly slower rate of population decline. We recommend the eradication of rats from Kiska Island to decrease the probability of dramatic population declines and ensure the future persistence of this important colony.

A Numerical method for the semilinear stochastic transport equation

Trabalho apresentado no XXXV CNMAC, Natal-RN, 2014.

Cooperative RNA Polymerase Molecules Behavior on a Stochastic Sequence-Dependent Model for Transcription Elongation

The transcription process is crucial to life and the enzyme RNA polymerase (RNAP) is the major component of the transcription machinery. The development of single-molecule techniques, such as magnetic and optical tweezers, atomic-force microscopy and single-molecule fluorescence, increased our understanding of the transcription process and complements traditional biochemical studies. Based on these studies, theoretical models have been proposed to explain and predict the kinetics of the RNAP during the polymerization, highlighting the results achieved by models based on the thermodynamic stability of the transcription elongation complex. However, experiments showed that if more than one RNAP initiates from the same promoter, the transcription behavior slightly changes and new phenomenona are observed. We proposed and implemented a theoretical model that considers collisions between RNAPs and predicts their cooperative behavior during multi-round transcription generalizing the Bai et al. stochastic sequence-dependent model. In our approach, collisions between elongating enzymes modify their transcription rate values. We performed the simulations in Mathematica® and compared the results of the single and the multiple-molecule transcription with experimental results and other theoretical models. Our multi-round approach can recover several expected behaviors, showing that the transcription process for the studied sequences can be accelerated up to 48% when collisions are allowed: the dwell times on pause sites are reduced as well as the distance that the RNAPs backtracked from backtracking sites. © 2013 Costa et al.

Stochastic rank correlation: a robust merit function for 2D/3D registration of image data obtained at different energies

In this article, the authors evaluate a merit function for 2D/3D registration called stochastic rank correlation (SRC). SRC is characterized by the fact that differences in image intensity do not influence the registration result; it therefore combines the numerical advantages of cross correlation (CC)-type merit functions with the flexibility of mutual-information-type merit functions. The basic idea is that registration is achieved on a random subset of the image, which allows for an efficient computation of Spearman's rank correlation coefficient. This measure is, by nature, invariant to monotonic intensity transforms in the images under comparison, which renders it an ideal solution for intramodal images acquired at different energy levels as encountered in intrafractional kV imaging in image-guided radiotherapy. Initial evaluation was undertaken using a 2D/3D registration reference image dataset of a cadaver spine. Even with no radiometric calibration, SRC shows a significant improvement in robustness and stability compared to CC. Pattern intensity, another merit function that was evaluated for comparison, gave rather poor results due to its limited convergence range. The time required for SRC with 5% image content compares well to the other merit functions; increasing the image content does not significantly influence the algorithm accuracy. The authors conclude that SRC is a promising measure for 2D/3D registration in IGRT and image-guided therapy in general.

Electromyographic activity of back muscles during stochastic whole body vibration

OBJECTIVES: Stochastic resonance whole body vibrations (SR-WBV) may reduce and prevent musculoskeletal problems (MSP). The aim of this study was to evaluate how activities of the lumbar erector spinae (ES) and of the ascending and descending trapezius (TA, TD) change in upright standing position during SR-WBV. METHODS: Nineteen female subjects completed 12 series of 10 seconds of SR-WBV at six different frequencies (2, 4, 6, 8, 10, 12Hz) and two types of "noise"-applications. An assessment at rest had been executed beforehand. Muscle activities were measured with EMG and normalized to the maximum voluntary contraction (MVC%). For statistical testing a three-factorial analysis of variation (ANOVA) was applied. RESULTS: The maximum activity of the respective muscles was 14.5 MVC% for the ES, 4.6 MVC% for the TA (12Hz with "noise" both), and 7.4 MVC% for the TD (10Hz without "noise"). Furthermore, all muscles varied significantly at 6Hz and above (p⋜0.047) compared to the situation at rest. No significant differences were found at SR-WBV with or without "noise". CONCLUSIONS: In general, muscle activity during SR-WBV is reasonably low and comparable to core strength stability exercises, sensorimotor training and "abdominal hollowing" in water. SR-WBV may be a therapeutic option for the relief of MSP.

Stationary Markovian Equilibrium in Overlapping Generation Models with Stochastic Nonclassical Production

This paper provides new sufficient conditions for the existence, computation via successive approximations, and stability of Markovian equilibrium decision processes for a large class of OLG models with stochastic nonclassical production. Our notion of stability is existence of stationary Markovian equilibrium. With a nonclassical production, our economies encompass a large class of OLG models with public policy, valued fiat money, production externalities, and Markov shocks to production. Our approach combines aspects of both topological and order theoretic fixed point theory, and provides the basis of globally stable numerical iteration procedures for computing extremal Markovian equilibrium objects. In addition to new theoretical results on existence and computation, we provide some monotone comparative statics results on the space of economies.

Implicit stochastic Runge-Kutta methods for stochastic differential equations

In this paper we construct implicit stochastic Runge-Kutta (SRK) methods for solving stochastic differential equations of Stratonovich type. Instead of using the increment of a Wiener process, modified random variables are used. We give convergence conditions of the SRK methods with these modified random variables. In particular, the truncated random variable is used. We present a two-stage stiffly accurate diagonal implicit SRK (SADISRK2) method with strong order 1.0 which has better numerical behaviour than extant methods. We also construct a five-stage diagonal implicit SRK method and a six-stage stiffly accurate diagonal implicit SRK method with strong order 1.5. The mean-square and asymptotic stability properties of the trapezoidal method and the SADISRK2 method are analysed and compared with an explicit method and a semi-implicit method. Numerical results are reported for confirming convergence properties and for comparing the numerical behaviour of these methods.

Adaptive stepsize based on control theory for stochastic differential equations

The numerical solution of stochastic differential equations (SDEs) has been focussed recently on the development of numerical methods with good stability and order properties. These numerical implementations have been made with fixed stepsize, but there are many situations when a fixed stepsize is not appropriate. In the numerical solution of ordinary differential equations, much work has been carried out on developing robust implementation techniques using variable stepsize. It has been necessary, in the deterministic case, to consider the best choice for an initial stepsize, as well as developing effective strategies for stepsize control-the same, of course, must be carried out in the stochastic case. In this paper, proportional integral (PI) control is applied to a variable stepsize implementation of an embedded pair of stochastic Runge-Kutta methods used to obtain numerical solutions of nonstiff SDEs. For stiff SDEs, the embedded pair of the balanced Milstein and balanced implicit method is implemented in variable stepsize mode using a predictive controller for the stepsize change. The extension of these stepsize controllers from a digital filter theory point of view via PI with derivative (PID) control will also be implemented. The implementations show the improvement in efficiency that can be attained when using these control theory approaches compared with the regular stepsize change strategy. (C) 2004 Elsevier B.V. All rights reserved.

Numerical methods for strong solutions of stochastic differential equations: an overview

This paper gives a review of recent progress in the design of numerical methods for computing the trajectories (sample paths) of solutions to stochastic differential equations. We give a brief survey of the area focusing on a number of application areas where approximations to strong solutions are important, with a particular focus on computational biology applications, and give the necessary analytical tools for understanding some of the important concepts associated with stochastic processes. We present the stochastic Taylor series expansion as the fundamental mechanism for constructing effective numerical methods, give general results that relate local and global order of convergence and mention the Magnus expansion as a mechanism for designing methods that preserve the underlying structure of the problem. We also present various classes of explicit and implicit methods for strong solutions, based on the underlying structure of the problem. Finally, we discuss implementation issues relating to maintaining the Brownian path, efficient simulation of stochastic integrals and variable-step-size implementations based on various types of control.

Magnetohydrodynamic stability of stochastically driven accretion flows

We investigate the evolution of magnetohydrodynamic (or hydromagnetic as coined by Chandrasekhar) perturbations in the presence of stochastic noise in rotating shear flows. The particular emphasis is the flows whose angular velocity decreases but specific angular momentum increases with increasing radial coordinate. Such flows, however, are Rayleigh stable but must be turbulent in order to explain astrophysical observed data and, hence, reveal a mismatch between the linear theory and observations and experiments. The mismatch seems to have been resolved, at least in certain regimes, in the presence of a weak magnetic field, revealing magnetorotational instability. The present work explores the effects of stochastic noise on such magnetohydrodynamic flows, in order to resolve the above mismatch generically for the hot flows. We essentially concentrate on a small section of such a flow which is nothing but a plane shear flow supplemented by the Coriolis effect, mimicking a small section of an astrophysical accretion disk around a compact object. It is found that such stochastically driven flows exhibit large temporal and spatial autocorrelations and cross-correlations of perturbation and, hence, large energy dissipations of perturbation, which generate instability. Interestingly, autocorrelations and cross-correlations appear independent of background angular velocity profiles, which are Rayleigh stable, indicating their universality. This work initiates our attempt to understand the evolution of three-dimensional hydromagnetic perturbations in rotating shear flows in the presence of stochastic noise. © 2013 American Physical Society.