Efficient methods for stochastic optimal control

The Hamilton Jacobi Bellman (HJB) equation is central to stochastic optimal control (SOC) theory, yielding the optimal solution to general problems specified by known dynamics and a specified cost functional. Given the assumption of quadratic cost on the control input, it is well known that the HJB reduces to a particular partial differential equation (PDE). While powerful, this reduction is not commonly used as the PDE is of second order, is nonlinear, and examples exist where the problem may not have a solution in a classical sense. Furthermore, each state of the system appears as another dimension of the PDE, giving rise to the curse of dimensionality. Since the number of degrees of freedom required to solve the optimal control problem grows exponentially with dimension, the problem becomes intractable for systems with all but modest dimension.

In the last decade researchers have found that under certain, fairly non-restrictive structural assumptions, the HJB may be transformed into a linear PDE, with an interesting analogue in the discretized domain of Markov Decision Processes (MDP). The work presented in this thesis uses the linearity of this particular form of the HJB PDE to push the computational boundaries of stochastic optimal control.

This is done by crafting together previously disjoint lines of research in computation. The first of these is the use of Sum of Squares (SOS) techniques for synthesis of control policies. A candidate polynomial with variable coefficients is proposed as the solution to the stochastic optimal control problem. An SOS relaxation is then taken to the partial differential constraints, leading to a hierarchy of semidefinite relaxations with improving sub-optimality gap. The resulting approximate solutions are shown to be guaranteed over- and under-approximations for the optimal value function. It is shown that these results extend to arbitrary parabolic and elliptic PDEs, yielding a novel method for Uncertainty Quantification (UQ) of systems governed by partial differential constraints. Domain decomposition techniques are also made available, allowing for such problems to be solved via parallelization and low-order polynomials.

The optimization-based SOS technique is then contrasted with the Separated Representation (SR) approach from the applied mathematics community. The technique allows for systems of equations to be solved through a low-rank decomposition that results in algorithms that scale linearly with dimensionality. Its application in stochastic optimal control allows for previously uncomputable problems to be solved quickly, scaling to such complex systems as the Quadcopter and VTOL aircraft. This technique may be combined with the SOS approach, yielding not only a numerical technique, but also an analytical one that allows for entirely new classes of systems to be studied and for stability properties to be guaranteed.

The analysis of the linear HJB is completed by the study of its implications in application. It is shown that the HJB and a popular technique in robotics, the use of navigation functions, sit on opposite ends of a spectrum of optimization problems, upon which tradeoffs may be made in problem complexity. Analytical solutions to the HJB in these settings are available in simplified domains, yielding guidance towards optimality for approximation schemes. Finally, the use of HJB equations in temporal multi-task planning problems is investigated. It is demonstrated that such problems are reducible to a sequence of SOC problems linked via boundary conditions. The linearity of the PDE allows us to pre-compute control policy primitives and then compose them, at essentially zero cost, to satisfy a complex temporal logic specification.

Representing measures on the Royden boundary for solutions of Δu=Pu on a Riemannian manifold

Consider the Royden compactification R* of a Riemannian n-manifold R, Γ = R*\R its Royden boundary, Δ its harmonic boundary and the elliptic differential equation Δu = Pu, P ≥ 0 on R. A regular Borel measure m^P can be constructed on Γ with support equal to the closure of Δ^P = {q ϵ Δ : q has a neighborhood U in R* with _U^ʃ_ᴖR^{P ˂ ∞} }. Every enegy-finite solution to u (i.e. E(u) = D(u) + ^ʃ_Ru²P ˂ ∞, where D(u) is the Dirichlet integral of u) can be represented by u(z) = ^ʃ_Γu(q)K(z,q)dm^P(q) where K(z,q) is a continuous function on ^Rx Γ . A _P^~_E-function is a nonnegative solution which is the infimum of a downward directed family of energy-finite solutions. A nonzero _P^~_E-function is called _P^~_E-minimal if it is a constant multiple of every nonzero _P^~_E-function dominated by it. THEOREM. There exists a _P^~_E-minimal function if and only if there exists a point in q ϵ Γ such that m^P(q) > 0. THEOREM. For q ϵ Δ^P , m^P(q) > 0 if and only if m⁰(q) > 0 .

Eigenvalue problems associated with Korn's inequalities in the theory of elasticity

Interest in the possible applications of a priori inequalities in linear elasticity theory motivated the present investigation. Korn's inequality under various side conditions is considered, with emphasis on the Korn's constant. In the "second case" of Korn's inequality, a variational approach leads to an eigenvalue problem; it is shown that, for simply-connected two-dimensional regions, the problem of determining the spectrum of this eigenvalue problem is equivalent to finding the values of Poisson's ratio for which the displacement boundary-value problem of linear homogeneous isotropic elastostatics has a non-unique solution.

Previous work on the uniqueness and non-uniqueness issue for the latter problem is examined and the results applied to the spectrum of the Korn eigenvalue problem. In this way, further information on the Korn constant for general regions is obtained.

A generalization of the "main case" of Korn's inequality is introduced and the associated eigenvalue problem is a gain related to the displacement boundary-value problem of linear elastostatics in two dimensions.

Equivalent differential equations for nonlinear dynamical systems

A technique for obtaining approximate periodic solutions to nonlinear ordinary differential equations is investigated. The approach is based on defining an equivalent differential equation whose exact periodic solution is known. Emphasis is placed on the mathematical justification of the approach. The relationship between the differential equation error and the solution error is investigated, and, under certain conditions, bounds are obtained on the latter. The technique employed is to consider the equation governing the exact solution error as a two point boundary value problem. Among other things, the analysis indicates that if an exact periodic solution to the original system exists, it is always possible to bound the error by selecting an appropriate equivalent system.

Three equivalence criteria for minimizing the differential equation error are compared, namely, minimum mean square error, minimum mean absolute value error, and minimum maximum absolute value error. The problem is analyzed by way of example, and it is concluded that, on the average, the minimum mean square error is the most appropriate criterion to use.

A comparison is made between the use of linear and cubic auxiliary systems for obtaining approximate solutions. In the examples considered, the cubic system provides noticeable improvement over the linear system in describing periodic response.

A comparison of the present approach to some of the more classical techniques is included. It is shown that certain of the standard approaches where a solution form is assumed can yield erroneous qualitative results.

Theoretical investigation of turbulent boundary layer over a wavy surface

The important features of the two-dimensional incompressible turbulent flow over a wavy surface of wavelength comparable with the boundary layer thickness are analyzed.

A turbulent field method using model equation for turbulent shear stress similar to the scheme of Bradshaw, Ferriss and Atwell (1967) is employed with suitable modification to cover the viscous sublayer. The governing differential equations are linearized based on the small but finite amplitude to wavelength ratio. An orthogonal wavy coordinate system, accurate to the second order in the amplitude ratio, is adopted to avoid the severe restriction to the validity of linearization due to the large mean velocity gradient near the wall. Analytic solution up to the second order is obtained by using the method of matched-asymptotic-expansion based on the large Reynolds number and hence the small skin friction coefficient.

In the outer part of the layer, the perturbed flow is practically "inviscid." Solutions for the velocity, Reynolds stress and also the wall pressure distributions agree well with the experimental measurement. In the wall region where the perturbed Reynolds stress plays an important role in the process of momentum transport, only a qualitative agreement is obtained. The results also show that the nonlinear second-order effect is negligible for amplitude ratio of 0.03. The discrepancies in the detailed structure of the velocity, shear stress, and skin friction distributions near the wall suggest modifications to the model are required to describe the present problem.

Stability of parametrically excited differential equations

Sufficient stability criteria for classes of parametrically excited differential equations are developed and applied to example problems of a dynamical nature.

Stability requirements are presented in terms of 1) the modulus of the amplitude of the parametric terms, 2) the modulus of the integral of the parametric terms and 3) the modulus of the derivative of the parametric terms.

The methods employed to show stability are Liapunov’s Direct Method and the Gronwall Lemma. The type of stability is generally referred to as asymptotic stability in the sense of Liapunov.

The results indicate that if the equation of the system with the parametric terms set equal to zero exhibits stability and possesses bounded operators, then the system will be stable under sufficiently small modulus of the parametric terms or sufficiently small modulus of the integral of the parametric terms (high frequency). On the other hand, if the equation of the system exhibits individual stability for all values that the parameter assumes in the time interval, then the actual system will be stable under sufficiently small modulus of the derivative of the parametric terms (slowly varying).

Particle fluid mechanics in shear flows, acoustic waves, and shock waves

Three different categories of flow problems of a fluid containing small particles are being considered here. They are: (i) a fluid containing small, non-reacting particles (Parts I and II); (ii) a fluid containing reacting particles (Parts III and IV); and (iii) a fluid containing particles of two distinct sizes with collisions between two groups of particles (Part V).

Part I

A numerical solution is obtained for a fluid containing small particles flowing over an infinite disc rotating at a constant angular velocity. It is a boundary layer type flow, and the boundary layer thickness for the mixture is estimated. For large Reynolds number, the solution suggests the boundary layer approximation of a fluid-particle mixture by assuming W = W_p. The error introduced is consistent with the Prandtl’s boundary layer approximation. Outside the boundary layer, the flow field has to satisfy the “inviscid equation” in which the viscous stress terms are absent while the drag force between the particle cloud and the fluid is still important. Increase of particle concentration reduces the boundary layer thickness and the amount of mixture being transported outwardly is reduced. A new parameter, β = 1/Ω τ_v, is introduced which is also proportional to μ. The secondary flow of the particle cloud depends very much on β. For small values of β, the particle cloud velocity attains its maximum value on the surface of the disc, and for infinitely large values of β, both the radial and axial particle velocity components vanish on the surface of the disc.

Part II

The “inviscid” equation for a gas-particle mixture is linearized to describe the flow over a wavy wall. Corresponding to the Prandtl-Glauert equation for pure gas, a fourth order partial differential equation in terms of the velocity potential ϕ is obtained for the mixture. The solution is obtained for the flow over a periodic wavy wall. For equilibrium flows where λ_v and λ_T approach zero and frozen flows in which λ_v and λ_T become infinitely large, the flow problem is basically similar to that obtained by Ackeret for a pure gas. For finite values of λ_v and λ_T, all quantities except v are not in phase with the wavy wall. Thus the drag coefficient C_D is present even in the subsonic case, and similarly, all quantities decay exponentially for supersonic flows. The phase shift and the attenuation factor increase for increasing particle concentration.

Part III

Using the boundary layer approximation, the initial development of the combustion zone between the laminar mixing of two parallel streams of oxidizing agent and small, solid, combustible particles suspended in an inert gas is investigated. For the special case when the two streams are moving at the same speed, a Green’s function exists for the differential equations describing first order gas temperature and oxidizer concentration. Solutions in terms of error functions and exponential integrals are obtained. Reactions occur within a relatively thin region of the order of λ_D. Thus, it seems advantageous in the general study of two-dimensional laminar flame problems to introduce a chemical boundary layer of thickness λ_D within which reactions take place. Outside this chemical boundary layer, the flow field corresponds to the ordinary fluid dynamics without chemical reaction.

Part IV

The shock wave structure in a condensing medium of small liquid droplets suspended in a homogeneous gas-vapor mixture consists of the conventional compressive wave followed by a relaxation region in which the particle cloud and gas mixture attain momentum and thermal equilibrium. Immediately following the compressive wave, the partial pressure corresponding to the vapor concentration in the gas mixture is higher than the vapor pressure of the liquid droplets and condensation sets in. Farther downstream of the shock, evaporation appears when the particle temperature is raised by the hot surrounding gas mixture. The thickness of the condensation region depends very much on the latent heat. For relatively high latent heat, the condensation zone is small compared with Ʌ_D.

For solid particles suspended initially in an inert gas, the relaxation zone immediately following the compression wave consists of a region where the particle temperature is first being raised to its melting point. When the particles are totally melted as the particle temperature is further increased, evaporation of the particles also plays a role.

The equilibrium condition downstream of the shock can be calculated and is independent of the model of the particle-gas mixture interaction.

Part V

For a gas containing particles of two distinct sizes and satisfying certain conditions, momentum transfer due to collisions between the two groups of particles can be taken into consideration using the classical elastic spherical ball model. Both in the relatively simple problem of normal shock wave and the perturbation solutions for the nozzle flow, the transfer of momentum due to collisions which decreases the velocity difference between the two groups of particles is clearly demonstrated. The difference in temperature as compared with the collisionless case is quite negligible.

On the Cesari fixed point method in a Banach space

In a paper published in 1961, L. Cesari [1] introduces a method which extends certain earlier existence theorems of Cesari and Hale ([2] to [6]) for perturbation problems to strictly nonlinear problems. Various authors ([1], [7] to [15]) have now applied this method to nonlinear ordinary and partial differential equations. The basic idea of the method is to use the contraction principle to reduce an infinite-dimensional fixed point problem to a finite-dimensional problem which may be attacked using the methods of fixed point indexes.

The following is my formulation of the Cesari fixed point method:

Let B be a Banach space and let S be a finite-dimensional linear subspace of B. Let P be a projection of B onto S and suppose Г≤B such that pГ is compact and such that for every x in PГ, P^-1x∩Г is closed. Let W be a continuous mapping from Г into B. The Cesari method gives sufficient conditions for the existence of a fixed point of W in Г.

Let I denote the identity mapping in B. Clearly y = Wy for some y in Г if and only if both of the following conditions hold:

(i) Py = PWy.

(ii) y = (P + (I - P)W)y.

Definition. The Cesari fixed paint method applies to (Г, W, P) if and only if the following three conditions are satisfied:

(1) For each x in PГ, P + (I - P)W is a contraction from P^-1x∩Г into itself. Let y(x) be that element (uniqueness follows from the contraction principle) of P^-1x∩Г which satisfies the equation y(x) = Py(x) + (I-P)Wy(x).

(2) The function y just defined is continuous from PГ into B.

(3) There are no fixed points of PWy on the boundary of PГ, so that the (finite- dimensional) fixed point index i(PWy, int PГ) is defined.

Definition. If the Cesari fixed point method applies to (Г, W, P) then define i(Г, W, P) to be the index i(PWy, int PГ).

The three theorems of this thesis can now be easily stated.

Theorem 1 (Cesari). If i(Г, W, P) is defined and i(Г, W, P) ≠0, then there is a fixed point of W in Г.

Theorem 2. Let the Cesari fixed point method apply to both (Г, W, P₁) and (Г, W, P₂). Assume that P₂P₁=P₁P₂=P₁ and assume that either of the following two conditions holds:

(1) For every b in B and every z in the range of P₂, we have that ‖b=P₂b‖ ≤ ‖b-z‖

(2)P₂Г is convex.

Then i(Г, W, P₁) = i(Г, W, P₂).

Theorem 3. If Ω is a bounded open set and W is a compact operator defined on Ω so that the (infinite-dimensional) Leray-Schauder index i_LS(W, Ω) is defined, and if the Cesari fixed point method applies to (Ω, W, P), then i(Ω, W, P) = i_LS(W, Ω).

Theorems 2 and 3 are proved using mainly a homotopy theorem and a reduction theorem for the finite-dimensional and the Leray-Schauder indexes. These and other properties of indexes will be listed before the theorem in which they are used.

Surface effects in simple molecular systems

This thesis examines two problems concerned with surface effects in simple molecular systems. The first is the problem associated with the interaction of a fluid with a solid boundary, and the second originates from the interaction of a liquid with its own vapor.

For a fluid in contact with a solid wall, two sets of integro-differential equations, involving the molecular distribution functions of the system, are derived. One of these is a particular form of the well-known Bogolyubov-Born-Green-Kirkwood-Yvon equations. For the second set, the derivation, in contrast with the formulation of the B.B.G.K.Y. hierarchy, is independent of the pair-potential assumption. The density of the fluid, expressed as a power series in the uniform fluid density, is obtained by solving these equations under the requirement that the wall be ideal.

The liquid-vapor interface is analyzed with the aid of equations that describe the density and pair-correlation function. These equations are simplified and then solved by employing the superposition and the low vapor density approximations. The solutions are substituted into formulas for the surface energy and surface tension, and numerical results are obtained for selected systems. Finally, the liquid-vapor system near the critical point is examined by means of the lowest order B.B.G.K.Y. equation.