Efficient methods for stochastic optimal control

The Hamilton Jacobi Bellman (HJB) equation is central to stochastic optimal control (SOC) theory, yielding the optimal solution to general problems specified by known dynamics and a specified cost functional. Given the assumption of quadratic cost on the control input, it is well known that the HJB reduces to a particular partial differential equation (PDE). While powerful, this reduction is not commonly used as the PDE is of second order, is nonlinear, and examples exist where the problem may not have a solution in a classical sense. Furthermore, each state of the system appears as another dimension of the PDE, giving rise to the curse of dimensionality. Since the number of degrees of freedom required to solve the optimal control problem grows exponentially with dimension, the problem becomes intractable for systems with all but modest dimension.

In the last decade researchers have found that under certain, fairly non-restrictive structural assumptions, the HJB may be transformed into a linear PDE, with an interesting analogue in the discretized domain of Markov Decision Processes (MDP). The work presented in this thesis uses the linearity of this particular form of the HJB PDE to push the computational boundaries of stochastic optimal control.

This is done by crafting together previously disjoint lines of research in computation. The first of these is the use of Sum of Squares (SOS) techniques for synthesis of control policies. A candidate polynomial with variable coefficients is proposed as the solution to the stochastic optimal control problem. An SOS relaxation is then taken to the partial differential constraints, leading to a hierarchy of semidefinite relaxations with improving sub-optimality gap. The resulting approximate solutions are shown to be guaranteed over- and under-approximations for the optimal value function. It is shown that these results extend to arbitrary parabolic and elliptic PDEs, yielding a novel method for Uncertainty Quantification (UQ) of systems governed by partial differential constraints. Domain decomposition techniques are also made available, allowing for such problems to be solved via parallelization and low-order polynomials.

The optimization-based SOS technique is then contrasted with the Separated Representation (SR) approach from the applied mathematics community. The technique allows for systems of equations to be solved through a low-rank decomposition that results in algorithms that scale linearly with dimensionality. Its application in stochastic optimal control allows for previously uncomputable problems to be solved quickly, scaling to such complex systems as the Quadcopter and VTOL aircraft. This technique may be combined with the SOS approach, yielding not only a numerical technique, but also an analytical one that allows for entirely new classes of systems to be studied and for stability properties to be guaranteed.

The analysis of the linear HJB is completed by the study of its implications in application. It is shown that the HJB and a popular technique in robotics, the use of navigation functions, sit on opposite ends of a spectrum of optimization problems, upon which tradeoffs may be made in problem complexity. Analytical solutions to the HJB in these settings are available in simplified domains, yielding guidance towards optimality for approximation schemes. Finally, the use of HJB equations in temporal multi-task planning problems is investigated. It is demonstrated that such problems are reducible to a sequence of SOC problems linked via boundary conditions. The linearity of the PDE allows us to pre-compute control policy primitives and then compose them, at essentially zero cost, to satisfy a complex temporal logic specification.

Manual de mecánica teórica

Índice: - Formulación Lagrangiana. - Formulación Hamiltoniana. - Ecuación de Hamilton-Jacobi. - Teoría de Perturbaciones. - Sistemas Continuos. - Mecánica y Geometría Diferencia

Teoria de controle ótimo com aplicações a sistemas biológicos

Neste trabalho apresentamos as etapas para a utilização do método da Programação Dinâmica, ou Princípio de Otimização de Bellman, para aplicações de controle ótimo. Investigamos a noção de funções de controle de Lyapunov (FCL) e sua relação com a estabilidade de sistemas autônomos com controle. Uma função de controle de Lyapunov deverá satisfazer a equação de Hamilton-Jacobi-Bellman (H-J-B). Usando esse fato, se uma função de controle de Lyapunov é conhecida, será então possível determinar a lei de realimentação ótima; isto é, a lei de controle que torna o sistema globalmente assintóticamente controlável a um estado de equilíbrio. Como aplicação, apresentamos uma modelagem matemática adequada a um problema de controle ótimo de certos sistemas biológicos. Este trabalho conta também com um breve histórico sobre o desenvolvimento da Teoria de Controle de forma a ilustrar a importância, o progresso e a aplicação das técnicas de controle em diferentes áreas ao longo do tempo.

High-order DES and zonal LES of a labyrinth seal

Hybrid numerical large eddy simulation (NLES) and detached eddy simulation (DES) methods are assessed on a labyrinth seal geometry. A high sixth order discretization scheme is used and is validated using a test case of a two dimensional vortex. The hybrid approach adopts a new blending function and along with DES is initially validated using a simple cavity flow. The NLES method is also validated outside of RANS zones. It is found that there is very little resolved turbulence in the cavity for the DES simulation. For the labyrinth seal calculations the DES approach is problematic giving virtually no resolved turbulence content. It is seen that over the tooth tips the extent of the LES region is small and is likely to be a strong contributor to excessive flow damping in these regions. On the other hand the zonal Hamilton-Jacobi approach did not suffer from this trait. In both cases the meshes used are considered to be hybrid RANS-LES adequate. Fortunately (or perhaps unfortunately) the DES profiles are in agreement with the time mean experimental measurements. It is concluded that for an inexperienced CFD practitioner this could have wider implications particularly if transient results such as unsteady loading are desired. Copyright © 2012 by the American Institute of Aeronautics and Astronautics, Inc.

High-order detached eddy simulation, zonal LES and URANS of cavity and labyrinth seal flows

Hybrid numerical large eddy simulation (NLES), detached eddy simulation (DES) and URANS methods are assessed on a cavity and a labyrinth seal geometry. A high sixth-order discretization scheme is used and is validated using the test case of a two-dimensional vortex. The hybrid approach adopts a new blending function. For the URANS simulations, the flow within the cavity remains steady, and the results show significant variation between models. Surprisingly, low levels of resolved turbulence are observed in the cavity for the DES simulation, and the cavity shear layer remains two dimensional. The hybrid RANS-NLES approach does not suffer from this trait.For the labyrinth seal, both the URANS and DES approaches give low levels of resolved turbulence. The zonal Hamilton-Jacobi approach on the other had given significantly more resolved content. Both DES and hybrid RANS-NLES give good agreement with the experimentally measured velocity profiles. Again, there is significant variation between the URANS models, and swirl velocities are overpredicted. © 2013 John Wiley & Sons, Ltd.

Differential equation based length scales to improve DES and RANS simulations

Surprisingly expensive to compute wall distances are still used in a range of key turbulence and peripheral physics models. Potentially economical, accuracy improving differential equation based distance algorithms are considered. These involve elliptic Poisson and hyperbolic natured Eikonal equation approaches. Numerical issues relating to non-orthogonal curvilinear grid solution of the latter are addressed. Eikonal extension to a Hamilton-Jacobi (HJ) equation is discussed. Use of this extension to improve turbulence model accuracy and, along with the Eikonal, enhance Detached Eddy Simulation (DES) techniques is considered. Application of the distance approaches is studied for various geometries. These include a plane channel flow with a wire at the centre, a wing-flap system, a jet with co-flow and a supersonic double-delta configuration. Although less accurate than the Eikonal, Poisson method based flow solutions are extremely close to those using a search procedure. For a moving grid case the Poisson method is found especially efficient. Results show the Eikonal equation can be solved on highly stretched, non-orthogonal, curvilinear grids. A key accuracy aspect is that metrics must be upwinded in the propagating front direction. The HJ equation is found to have qualitative turbulence model improving properties. © 2003 by P. G. Tucker.

Robust Dynamic Pricing Over Infinite Horizon in The Presence of Model Uncertainty( 2* Journal)

This paper studies a problem of dynamic pricing faced by a retailer with limited inventory, uncertain about the demand rate model, aiming to maximize expected discounted revenue over an infinite time horizon. The retailer doubts his demand model which is generated by historical data and views it as an approximation. Uncertainty in the demand rate model is represented by a notion of generalized relative entropy process, and the robust pricing problem is formulated as a two-player zero-sum stochastic differential game. The pricing policy is obtained through the Hamilton-Jacobi-Isaacs (HJI) equation. The existence and uniqueness of the solution of the HJI equation is shown and a verification theorem is proved to show that the solution of the HJI equation is indeed the value function of the pricing problem. The results are illustrated by an example with exponential nominal demand rate.

Exact and numerical solutions for diffusio-reaction equations with convection term

Esta dissertação estuda essencialmente dois problemas: (A) uma classe de equações unidimensionais de reacção-difusão-convecção em meios não uniformes (dependentes do espaço), e (B) um problema elíptico não-linear e paramétrico ligado a fenómenos de capilaridade. A Análise de Perturbação Singular e a dinâmica de Hamilton-Jacobi são utilizadas na obtenção de expressões assimptóticas para a solução (com comportamento de frente) e para a sua velocidade de propagação. Os seguintes três métodos de decomposição, Adomian Decomposition Method (ADM), Decomposition Method based on Infinite Products (DIP), e New Iterative Method (NIM), são apresentados e brevemente comparados. Adicionalmente, condições suficientes para a convergência da solução em série, obtida pelo ADM, e uma aplicação a um problema da Telecomunicações por Fibras Ópticas, envolvendo EDOs não-lineares designadas equações de Raman, são discutidas. Um ponto de vista mais abrangente que unifica os métodos de decomposição referidos é também apresentado. Para subclasses desta EDP são obtidas soluções numa forma explícita, para diferentes tipos de dados e usando uma variante do método de simetrias de Bluman-Cole. Usando Teoria de Pontos Críticos (o teorema usualmente designado mountain pass) e técnicas de truncatura, prova-se a existência de duas soluções não triviais (uma positiva e uma negativa) para o problema elíptico não-linear e paramétrico (B). A existência de uma terceira solução não trivial é demonstrada usando Grupos Críticos e Teoria de Morse.

Modélisation asymptotique pour la simulation aux grandes échelles de la combustion turbulente prémélangée

Thèse diffusée initialement dans le cadre d'un projet pilote des Presses de l'Université de Montréal/Centre d'édition numérique UdeM (1997-2008) avec l'autorisation de l'auteur.

Dynamic programming for stochastic target problems, Viscosity solutions and hedging in markets with Portfolio constraints and large investors

The purpose of this expository arti le is to present a self- ontained overview of some results on the hara terization of the optimal value fun tion of a sto hasti target problem as (dis ontinuous) vis osity solution of a ertain dynami programming PDE and its appli ation to the problem of hedging ontingent laims in the presen e of portfolio onstraints and large investors

Ecuaciones diferenciales estocásticas con condición final y soluciones de viscosidad de EDPS semilineales de segundo orden

El objetivo de este documento es recopilar algunos resultados clasicos sobre existencia y unicidad ´ de soluciones de ecuaciones diferenciales estocasticas (EDEs) con condici ´ on final (en ingl ´ es´ Backward stochastic differential equations) con particular enfasis en el caso de coeficientes mon ´ otonos, y su cone- ´ xion con soluciones de viscosidad de sistemas de ecuaciones diferenciales parciales (EDPs) parab ´ olicas ´ y el´ıpticas semilineales de segundo orden.

First-order actions: A new view

We analyse systems described by first-order actions using the Hamilton-Jacobi (HJ) formalism for singular systems. In this study we verify that generalized brackets appear in a natural way in HJ approach, showing us the existence of a symplectic structure in the phase space of this formalism.

Chaos control of a nonlinear oscillator with shape memory alloy using an optimal linear control: Part I: Ideal energy source

Coordenação de Aperfeiçoamento de Pessoal de Nível Superior (CAPES)

Optimal pest control problem in population dynamics

One of the main goals of the pest control is to maintain the density of the pest population in the equilibrium level below economic damages. For reaching this goal, the optimal pest control problem was divided in two parts. In the first part, the two optimal control functions were considered. These functions move the ecosystem pest-natural enemy at an equilibrium state below the economic injury level. In the second part, the one optimal control function stabilizes the ecosystem in this level, minimizing the functional that characterizes quadratic deviations of this level. The first problem was resolved through the application of the Maximum Principle of Pontryagin. The Dynamic Programming was used for the resolution of the second optimal pest control problem.

Mathematical modeling and control of population systems: Applications in biological pest control

The aim of this paper is to apply methods from optimal control theory, and from the theory of dynamic systems to the mathematical modeling of biological pest control. The linear feedback control problem for nonlinear systems has been formulated in order to obtain the optimal pest control strategy only through the introduction of natural enemies. Asymptotic stability of the closed-loop nonlinear Kolmogorov system is guaranteed by means of a Lyapunov function which can clearly be seen to be the solution of the Hamilton-Jacobi-Bellman equation, thus guaranteeing both stability and optimality. Numerical simulations for three possible scenarios of biological pest control based on the Lotka-Volterra models are provided to show the effectiveness of this method. (c) 2007 Elsevier B.V. All rights reserved.