Efficient methods for stochastic optimal control

The Hamilton Jacobi Bellman (HJB) equation is central to stochastic optimal control (SOC) theory, yielding the optimal solution to general problems specified by known dynamics and a specified cost functional. Given the assumption of quadratic cost on the control input, it is well known that the HJB reduces to a particular partial differential equation (PDE). While powerful, this reduction is not commonly used as the PDE is of second order, is nonlinear, and examples exist where the problem may not have a solution in a classical sense. Furthermore, each state of the system appears as another dimension of the PDE, giving rise to the curse of dimensionality. Since the number of degrees of freedom required to solve the optimal control problem grows exponentially with dimension, the problem becomes intractable for systems with all but modest dimension.

In the last decade researchers have found that under certain, fairly non-restrictive structural assumptions, the HJB may be transformed into a linear PDE, with an interesting analogue in the discretized domain of Markov Decision Processes (MDP). The work presented in this thesis uses the linearity of this particular form of the HJB PDE to push the computational boundaries of stochastic optimal control.

This is done by crafting together previously disjoint lines of research in computation. The first of these is the use of Sum of Squares (SOS) techniques for synthesis of control policies. A candidate polynomial with variable coefficients is proposed as the solution to the stochastic optimal control problem. An SOS relaxation is then taken to the partial differential constraints, leading to a hierarchy of semidefinite relaxations with improving sub-optimality gap. The resulting approximate solutions are shown to be guaranteed over- and under-approximations for the optimal value function. It is shown that these results extend to arbitrary parabolic and elliptic PDEs, yielding a novel method for Uncertainty Quantification (UQ) of systems governed by partial differential constraints. Domain decomposition techniques are also made available, allowing for such problems to be solved via parallelization and low-order polynomials.

The optimization-based SOS technique is then contrasted with the Separated Representation (SR) approach from the applied mathematics community. The technique allows for systems of equations to be solved through a low-rank decomposition that results in algorithms that scale linearly with dimensionality. Its application in stochastic optimal control allows for previously uncomputable problems to be solved quickly, scaling to such complex systems as the Quadcopter and VTOL aircraft. This technique may be combined with the SOS approach, yielding not only a numerical technique, but also an analytical one that allows for entirely new classes of systems to be studied and for stability properties to be guaranteed.

The analysis of the linear HJB is completed by the study of its implications in application. It is shown that the HJB and a popular technique in robotics, the use of navigation functions, sit on opposite ends of a spectrum of optimization problems, upon which tradeoffs may be made in problem complexity. Analytical solutions to the HJB in these settings are available in simplified domains, yielding guidance towards optimality for approximation schemes. Finally, the use of HJB equations in temporal multi-task planning problems is investigated. It is demonstrated that such problems are reducible to a sequence of SOC problems linked via boundary conditions. The linearity of the PDE allows us to pre-compute control policy primitives and then compose them, at essentially zero cost, to satisfy a complex temporal logic specification.

Range of validity of the method of averaging

Sufficient conditions are derived for the validity of approximate periodic solutions of a class of second order ordinary nonlinear differential equations. An approximate solution is defined to be valid if an exact solution exists in a neighborhood of the approximation.

Two classes of validity criteria are developed. Existence is obtained using the contraction mapping principle in one case, and the Schauder-Leray fixed point theorem in the other. Both classes of validity criteria make use of symmetry properties of periodic functions, and both classes yield an upper bound on a norm of the difference between the approximate and exact solution. This bound is used in a procedure which establishes sufficient stability conditions for the approximated solution.

Application to a system with piecewise linear restoring force (bilinear system) reveals that the approximate solution obtained by the method of averaging is valid away from regions where the response exhibits vertical tangents. A narrow instability region is obtained near one-half the natural frequency of the equivalent linear system. Sufficient conditions for the validity of resonant solutions are also derived, and two term harmonic balance approximate solutions which exhibit ultraharmonic and subharmonic resonances are studied.

Spectral density of first order Piecewise linear systems excited by white noise

The Fokker-Planck (FP) equation is used to develop a general method for finding the spectral density for a class of randomly excited first order systems. This class consists of systems satisfying stochastic differential equations of form ẋ + f(x) = m/Ʃ/j = 1 h_j(x)n_j(t) where f and the h_j are piecewise linear functions (not necessarily continuous), and the n_j are stationary Gaussian white noise. For such systems, it is shown how the Laplace-transformed FP equation can be solved for the transformed transition probability density. By manipulation of the FP equation and its adjoint, a formula is derived for the transformed autocorrelation function in terms of the transformed transition density. From this, the spectral density is readily obtained. The method generalizes that of Caughey and Dienes, J. Appl. Phys., 32.11.

This method is applied to 4 subclasses: (1) m = 1, h₁ = const. (forcing function excitation); (2) m = 1, h₁ = f (parametric excitation); (3) m = 2, h₁ = const., h₂ = f, n₁ and n₂ correlated; (4) the same, uncorrelated. Many special cases, especially in subclass (1), are worked through to obtain explicit formulas for the spectral density, most of which have not been obtained before. Some results are graphed.

Dealing with parametrically excited first order systems leads to two complications. There is some controversy concerning the form of the FP equation involved (see Gray and Caughey, J. Math. Phys., 44.3); and the conditions which apply at irregular points, where the second order coefficient of the FP equation vanishes, are not obvious but require use of the mathematical theory of diffusion processes developed by Feller and others. These points are discussed in the first chapter, relevant results from various sources being summarized and applied. Also discussed is the steady-state density (the limit of the transition density as t → ∞).

Approximate analytical description for fundamental-mode fields of graded-index fibers: Beyond the Gaussian approximation

An approximate analytical description for fundamental-mode fields of graded-index fibers is explicitly presented by use of the power-series expansion method, the maximum-value condition at the fiber axis, the decay properties of fundamental-mode fields at large distance from the fiber axis, and the approximate modal parameters U obtained from the Gaussian approximation. This analytical description is much more accurate than the Gaussian approximation and at the same time keep the simplicity of the latter. As two special examples, we present the approximate analytical formulas for the fundamental-mode fields of a step profile fiber and a Gaussian profile fiber, and we find that they are both highly accurate in the single-mode range by comparing them with the corresponding exact solutions.

Corrections to the paraxial approximation of an arbitrary free-propagation beam

A relatively simple transform from an arbitrary solution of the paraxial wave equation to the corresponding exact solution of the Helmholtz wave equation is derived in the condition that the evanescent waves are ignored and is used to study the corrections to the paraxial approximation of an arbitrary free-propagation beam. Specifically, the general lowest-order correction field is given in a very simple form and is proved to be exactly consistent with the perturbation method developed by Lax et nl. [Phys. Rev. A 11, 1365 (1975)]. Some special examples, such as the lowest-order correction to the paraxial approximation of a fundamental Gaussian beam whose waist plane has a parallel shin from the z = 0 plane, are presented. (C) 1998 Optical Society of America.

Corrections to the paraxial approximation solutions in transversely nonuniform refractive-index media

The coupled differential recurrence equations for the corrections to the paraxial approximation solutions in transversely nonuniform refractive-index media are established in terms of the perturbation method. All the corrections (including the longitudinal field corrections) to the paraxial approximation solutions are presented in the weak-guidance approximation. As a concrete application, the first-order longitudinal field correction and the second-order transverse field correction to the paraxial approximation of a Gaussian beam propagating in a transversely quadratic refractive index medium are analytically investigated. (C) 1999 Optical Society of America [S0740-3232(99)00310-5].

Focusing properties of concentric piecewise cylindrical vector beam

The focusing properties of a concentric piecewise cylindrical vector beam is investigated theoretically in this paper. The beam consists of three portions with different and changeable phase retardation and polarization. Numerical simulations show that the evolution of the focal shape is very considerable by changing the radius and polarization rotation angle of each portion of the vector beam. And some interesting focal spots may occur, such as two- or three-peak focus, dark hollow focus, ring focus, and two-ring-peak focus. Corresponding gradient force patterns are also computed, and novel trap patterns, including cup shell shape trap with one trap at its each side along axis, rectangle shell shape trap with one trap at its each side, dumbbell optical trap, spherical shell optical trap, may occur, which shows that the concentric piecewise cylindrical vector beam can be used to construct controllable optical tweezers. (c) 2006 Elsevier GmbH. All rights reserved.