Efficient methods for stochastic optimal control

The Hamilton Jacobi Bellman (HJB) equation is central to stochastic optimal control (SOC) theory, yielding the optimal solution to general problems specified by known dynamics and a specified cost functional. Given the assumption of quadratic cost on the control input, it is well known that the HJB reduces to a particular partial differential equation (PDE). While powerful, this reduction is not commonly used as the PDE is of second order, is nonlinear, and examples exist where the problem may not have a solution in a classical sense. Furthermore, each state of the system appears as another dimension of the PDE, giving rise to the curse of dimensionality. Since the number of degrees of freedom required to solve the optimal control problem grows exponentially with dimension, the problem becomes intractable for systems with all but modest dimension.

In the last decade researchers have found that under certain, fairly non-restrictive structural assumptions, the HJB may be transformed into a linear PDE, with an interesting analogue in the discretized domain of Markov Decision Processes (MDP). The work presented in this thesis uses the linearity of this particular form of the HJB PDE to push the computational boundaries of stochastic optimal control.

This is done by crafting together previously disjoint lines of research in computation. The first of these is the use of Sum of Squares (SOS) techniques for synthesis of control policies. A candidate polynomial with variable coefficients is proposed as the solution to the stochastic optimal control problem. An SOS relaxation is then taken to the partial differential constraints, leading to a hierarchy of semidefinite relaxations with improving sub-optimality gap. The resulting approximate solutions are shown to be guaranteed over- and under-approximations for the optimal value function. It is shown that these results extend to arbitrary parabolic and elliptic PDEs, yielding a novel method for Uncertainty Quantification (UQ) of systems governed by partial differential constraints. Domain decomposition techniques are also made available, allowing for such problems to be solved via parallelization and low-order polynomials.

The optimization-based SOS technique is then contrasted with the Separated Representation (SR) approach from the applied mathematics community. The technique allows for systems of equations to be solved through a low-rank decomposition that results in algorithms that scale linearly with dimensionality. Its application in stochastic optimal control allows for previously uncomputable problems to be solved quickly, scaling to such complex systems as the Quadcopter and VTOL aircraft. This technique may be combined with the SOS approach, yielding not only a numerical technique, but also an analytical one that allows for entirely new classes of systems to be studied and for stability properties to be guaranteed.

The analysis of the linear HJB is completed by the study of its implications in application. It is shown that the HJB and a popular technique in robotics, the use of navigation functions, sit on opposite ends of a spectrum of optimization problems, upon which tradeoffs may be made in problem complexity. Analytical solutions to the HJB in these settings are available in simplified domains, yielding guidance towards optimality for approximation schemes. Finally, the use of HJB equations in temporal multi-task planning problems is investigated. It is demonstrated that such problems are reducible to a sequence of SOC problems linked via boundary conditions. The linearity of the PDE allows us to pre-compute control policy primitives and then compose them, at essentially zero cost, to satisfy a complex temporal logic specification.

High pressure states in condensed matter: I. High pressure behavior of the iron-sulphur system with applications to the earth's core. II. Empirical equation of state for organic compounds at high pressures

Part I:

The earth's core is generally accepted to be composed primarily of iron, with an admixture of other elements. Because the outer core is observed not to transmit shear waves at seismic frequencies, it is known to be liquid or primarily liquid. A new equation of state is presented for liquid iron, in the form of parameters for the 4th order Birch-Murnaghan and Mie-Grüneisen equations of state. The parameters were constrained by a set of values for numerous properties compiled from the literature. A detailed theoretical model is used to constrain the P-T behavior of the heat capacity, based on recent advances in the understanding of the interatomic potentials for transition metals. At the reference pressure of 10⁵ Pa and temperature of 1811 K (the normal melting point of Fe), the parameters are: ρ = 7037 kg/m³, K_S0 = 110 GPa, K_S' = 4.53, K_S" = -.0337 GPa-1, and γ = 2.8, with γ α ρ^-1.17. Comparison of the properties predicted by this model with the earth model PREM indicates that the outer core is 8 to 10 % less dense than pure liquid Fe at the same conditions. The inner core is also found to be 3 to 5% less dense than pure liquid Fe, supporting the idea of a partially molten inner core. The density deficit of the outer core implies that the elements dissolved in the liquid Fe are predominantly of lower atomic weight than Fe. Of the candidate light elements favored by researchers, only sulfur readily dissolves into Fe at low pressure, which means that this element was almost certainly concentrated in the core at early times. New melting data are presented for FeS and FeS₂ which indicate that the FeS₂ is the S-hearing liquidus solid phase at inner core pressures. Consideration of the requirement that the inner core boundary be observable by seismological means and the freezing behavior of solutions leads to the possibility that the outer core may contain a significant fraction of solid material. It is found that convection in the outer core is not hindered if the solid particles are entrained in the fluid flow. This model for a core of Fe and S admits temperatures in the range 3450K to 4200K at the top of the core. An all liquid Fe-S outer core would require a temperature of about 4900 K at the top of the core.

Part II.

The abundance of uses for organic compounds in the modern world results in many applications in which these materials are subjected to high pressures. This leads to the desire to be able to describe the behavior of these materials under such conditions. Unfortunately, the number of compounds is much greater than the number of experimental data available for many of the important properties. In the past, one approach that has worked well is the calculation of appropriate properties by summing the contributions from the organic functional groups making up molecules of the compounds in question. A new set of group contributions for the molar volume, volume thermal expansivity, heat capacity, and the Rao function is presented for functional groups containing C, H, and O. This set is, in most cases, limited in application to low molecular liquids. A new technique for the calculation of the pressure derivative of the bulk modulus is also presented. Comparison with data indicates that the presented technique works very well for most low molecular hydrocarbon liquids and somewhat less well for oxygen-bearing compounds. A similar comparison of previous results for polymers indicates that the existing tabulations of group contributions for this class of materials is in need of revision. There is also evidence that the Rao function contributions for polymers and low molecular compounds are somewhat different.

Generalizations and extensions of the Fokker-Planck-Kolmogorov equations

The problem of determining probability density functions of general transformations of random processes is considered in this thesis. A method of solution is developed in which partial differential equations satisfied by the unknown density function are derived. These partial differential equations are interpreted as generalized forms of the classical Fokker-Planck-Kolmogorov equations and are shown to imply the classical equations for certain classes of Markov processes. Extensions of the generalized equations which overcome degeneracy occurring in the steady-state case are also obtained.

The equations of Darling and Siegert are derived as special cases of the generalized equations thereby providing unity to two previously existing theories. A technique for treating non-Markov processes by studying closely related Markov processes is proposed and is seen to yield the Darling and Siegert equations directly from the classical Fokker-Planck-Kolmogorov equations.

As illustrations of their applicability, the generalized Fokker-Planck-Kolmogorov equations are presented for certain joint probability density functions associated with the linear filter. These equations are solved for the density of the output of an arbitrary linear filter excited by Markov Gaussian noise and for the density of the output of an RC filter excited by the Poisson square wave. This latter density is also found by using the extensions of the generalized equations mentioned above. Finally, some new approaches for finding the output probability density function of an RC filter-limiter-RC filter system driven by white Gaussian noise are included. The results in this case exhibit the data required for complete solution and clearly illustrate some of the mathematical difficulties inherent to the use of the generalized equations.

A variational framework for spectral discretization of the density matrix in Kohn Sham density functional theory

Kohn-Sham density functional theory (KSDFT) is currently the main work-horse of quantum mechanical calculations in physics, chemistry, and materials science. From a mechanical engineering perspective, we are interested in studying the role of defects in the mechanical properties in materials. In real materials, defects are typically found at very small concentrations e.g., vacancies occur at parts per million, dislocation density in metals ranges from $10^{10} m^{-2}$ to $10^{15} m^{-2}$, and grain sizes vary from nanometers to micrometers in polycrystalline materials, etc. In order to model materials at realistic defect concentrations using DFT, we would need to work with system sizes beyond millions of atoms. Due to the cubic-scaling computational cost with respect to the number of atoms in conventional DFT implementations, such system sizes are unreachable. Since the early 1990s, there has been a huge interest in developing DFT implementations that have linear-scaling computational cost. A promising approach to achieving linear-scaling cost is to approximate the density matrix in KSDFT. The focus of this thesis is to provide a firm mathematical framework to study the convergence of these approximations. We reformulate the Kohn-Sham density functional theory as a nested variational problem in the density matrix, the electrostatic potential, and a field dual to the electron density. The corresponding functional is linear in the density matrix and thus amenable to spectral representation. Based on this reformulation, we introduce a new approximation scheme, called spectral binning, which does not require smoothing of the occupancy function and thus applies at arbitrarily low temperatures. We proof convergence of the approximate solutions with respect to spectral binning and with respect to an additional spatial discretization of the domain. For a standard one-dimensional benchmark problem, we present numerical experiments for which spectral binning exhibits excellent convergence characteristics and outperforms other linear-scaling methods.

Synthesis, Oxidation and Photophysics of Perfluoroborated Tetrakis(pyrophosphito)diplatinate (II) and Density Functional Theory (DFT) Study of Electrochemical CO2 Reduction by Mn Catalysts

In the first part of this thesis (Chapters I and II), the synthesis, characterization, reactivity and photophysics of per(difluoroborated) tetrakis(pyrophosphito)diplatinate(II) (Pt(POPBF₂)) are discussed. Pt(POP-BF₂) was obtained by reaction of [Pt₂(POP)₄]^4- with neat boron trifluoride diethyl etherate (BF₃·Et₂O). While Pt(POP-BF₂) and [Pt₂(POP)₄]^4- have similar structures and absorption spectra, they differ in significant ways. Firstly, as discussed in Chapter I, the former is less susceptible to oxidation, as evidenced by the reversibility of its oxidation by I2. Secondly, while the first excited triplet states (T₁) of both Pt(POP-BF₂) and [Pt₂(POP)₄]^4- exhibit long lifetimes (ca. 0.01 ms at room temperature) and substantial zero-field splitting (40 cm^-1), Pt(POP-BF₂) also has a remarkably long-lived (1.6 ns at room temperature) singlet excited state (S₁), indicating slow intersystem crossing (ISC). Fluorescence lifetime and quantum yield (QY) of Pt(POP-BF₂) were measured over a range of temperatures, providing insight into the slow ISC process. The remarkable spectroscopic and photophysical properties of Pt(POP-BF₂), both in solution and as a microcrystalline powder, form the theme of Chapter II.

In the second part of the thesis (Chapters III and IV), the electrochemical reduction of CO₂ to CO by [(L)Mn(CO)₃]^- catalysts is investigated using density functional theory (DFT). As discussed in Chapter III, the turnover frequency (TOF)-limiting step is the dehydroxylation of [(bpy)Mn(CO)₃(CO₂H)]^0/- (bpy = bipyridine) by trifluoroethanol (TFEH) to form [(bpy)Mn(CO)₄]^+/0. Because the dehydroxylation of [(bpy)Mn(CO)₃(CO₂H)]^- is faster, maximum TOF (TOF_max) is achieved at potentials sufficient to completely reduce [(bpy)Mn(CO)₃(CO₂H)]⁰ to [(bpy)Mn(CO)₃(CO₂H)]^-. Substitution of bipyridine with bipyrimidine reduces the overpotential needed, but at the expense of TOF_max. In Chapter IV, the decoration of the bipyrimidine ligand with a pendant alcohol is discussed as a strategy to increase CO₂ reduction activity. Our calculations predict that the pendant alcohol acts in concert with an external TFEH molecule, the latter acidifying the former, resulting in a ~ 80,000-fold improvement in the rate of TOF-limiting dehydroxylation of [(L)Mn(CO)₃(CO₂H)]^-.

An interesting strategy for the co-upgrading of light olefins and alkanes into heavier alkanes is the subject of Appendix B. The proposed scheme involves dimerization of the light olefin, operating in tandem with transfer hydrogenation between the olefin dimer and the light alkane. The work presented therein involved a Ta olefin dimerization catalyst and a silica-supported Ir transfer hydrogenation catalyst. Olefin dimer was formed under reaction conditions; however, this did not undergo transfer hydrogenation with the light alkane. A significant challenge is that the Ta catalyst selectively produces highly branched dimers, which are unable to undergo transfer hydrogenation.

Equivalent differential equations for nonlinear dynamical systems

A technique for obtaining approximate periodic solutions to nonlinear ordinary differential equations is investigated. The approach is based on defining an equivalent differential equation whose exact periodic solution is known. Emphasis is placed on the mathematical justification of the approach. The relationship between the differential equation error and the solution error is investigated, and, under certain conditions, bounds are obtained on the latter. The technique employed is to consider the equation governing the exact solution error as a two point boundary value problem. Among other things, the analysis indicates that if an exact periodic solution to the original system exists, it is always possible to bound the error by selecting an appropriate equivalent system.

Three equivalence criteria for minimizing the differential equation error are compared, namely, minimum mean square error, minimum mean absolute value error, and minimum maximum absolute value error. The problem is analyzed by way of example, and it is concluded that, on the average, the minimum mean square error is the most appropriate criterion to use.

A comparison is made between the use of linear and cubic auxiliary systems for obtaining approximate solutions. In the examples considered, the cubic system provides noticeable improvement over the linear system in describing periodic response.

A comparison of the present approach to some of the more classical techniques is included. It is shown that certain of the standard approaches where a solution form is assumed can yield erroneous qualitative results.

Stochastic optimal control

H. J. Kushner has obtained the differential equation satisfied by the optimal feedback control law for a stochastic control system in which the plant dynamics and observations are perturbed by independent additive Gaussian white noise processes. However, the differentiation includes the first and second functional derivatives and, except for a restricted set of systems, is too complex to solve with present techniques.

This investigation studies the optimal control law for the open loop system and incorporates it in a sub-optimal feedback control law. This suboptimal control law's performance is at least as good as that of the optimal control function and satisfies a differential equation involving only the first functional derivative. The solution of this equation is equivalent to solving two two-point boundary valued integro-partial differential equations. An approximate solution has advantages over the conventional approximate solution of Kushner's equation.

As a result of this study, well known results of deterministic optimal control are deduced from the analysis of optimal open loop control.

I. Numerical solution of exact pair equations. II. Numerical solution of first-order pair equations

Part I

Numerical solutions to the S-limit equations for the helium ground state and excited triplet state and the hydride ion ground state are obtained with the second and fourth difference approximations. The results for the ground states are superior to previously reported values. The coupled equations resulting from the partial wave expansion of the exact helium atom wavefunction were solved giving accurate S-, P-, D-, F-, and G-limits. The G-limit is -2.90351 a.u. compared to the exact value of the energy of -2.90372 a.u.

Part II

The pair functions which determine the exact first-order wavefunction for the ground state of the three-electron atom are found with the matrix finite difference method. The second- and third-order energies for the (1s1s)¹S, (1s2s)³S, and (1s2s)¹S states of the two-electron atom are presented along with contour and perspective plots of the pair functions. The total energy for the three-electron atom with a nuclear charge Z is found to be E(Z) = -1.125•Z² +1.022805•Z-0.408138-0.025515•(1/Z)+O(1/Z²)a.u.

A theory of strong and weak scintillations with applications to astrophysics

The propagation of waves in an extended, irregular medium is studied under the "quasi-optics" and the "Markov random process" approximations. Under these assumptions, a Fokker-Planck equation satisfied by the characteristic functional of the random wave field is derived. A complete set of the moment equations with different transverse coordinates and different wavenumbers is then obtained from the characteristic functional. The derivation does not require Gaussian statistics of the random medium and the result can be applied to the time-dependent problem. We then solve the moment equations for the phase correlation function, angular broadening, temporal pulse smearing, intensity correlation function, and the probability distribution of the random waves. The necessary and sufficient conditions for strong scintillation are also given.

We also consider the problem of diffraction of waves by a random, phase-changing screen. The intensity correlation function is solved in the whole Fresnel diffraction region and the temporal pulse broadening function is derived rigorously from the wave equation.

The method of smooth perturbations is applied to interplanetary scintillations. We formulate and calculate the effects of the solar-wind velocity fluctuations on the observed intensity power spectrum and on the ratio of the observed "pattern" velocity and the true velocity of the solar wind in the three-dimensional spherical model. The r.m.s. solar-wind velocity fluctuations are found to be ~200 km/sec in the region about 20 solar radii from the Sun.

We then interpret the observed interstellar scintillation data using the theories derived under the Markov approximation, which are also valid for the strong scintillation. We find that the Kolmogorov power-law spectrum with an outer scale of 10 to 100 pc fits the scintillation data and that the ambient averaged electron density in the interstellar medium is about 0.025 cm^-3. It is also found that there exists a region of strong electron density fluctuation with thickness ~10 pc and mean electron density ~7 cm^-3 between the PSR 0833-45 pulsar and the earth.

Stability of parametrically excited differential equations

Sufficient stability criteria for classes of parametrically excited differential equations are developed and applied to example problems of a dynamical nature.

Stability requirements are presented in terms of 1) the modulus of the amplitude of the parametric terms, 2) the modulus of the integral of the parametric terms and 3) the modulus of the derivative of the parametric terms.

The methods employed to show stability are Liapunov’s Direct Method and the Gronwall Lemma. The type of stability is generally referred to as asymptotic stability in the sense of Liapunov.

The results indicate that if the equation of the system with the parametric terms set equal to zero exhibits stability and possesses bounded operators, then the system will be stable under sufficiently small modulus of the parametric terms or sufficiently small modulus of the integral of the parametric terms (high frequency). On the other hand, if the equation of the system exhibits individual stability for all values that the parameter assumes in the time interval, then the actual system will be stable under sufficiently small modulus of the derivative of the parametric terms (slowly varying).