Formal methods for control synthesis in partially observed environments : application to autonomous robotic manipulation

Modern robots are increasingly expected to function in uncertain and dynamically challenging environments, often in proximity with humans. In addition, wide scale adoption of robots requires on-the-fly adaptability of software for diverse application. These requirements strongly suggest the need to adopt formal representations of high level goals and safety specifications, especially as temporal logic formulas. This approach allows for the use of formal verification techniques for controller synthesis that can give guarantees for safety and performance. Robots operating in unstructured environments also face limited sensing capability. Correctly inferring a robot's progress toward high level goal can be challenging.

This thesis develops new algorithms for synthesizing discrete controllers in partially known environments under specifications represented as linear temporal logic (LTL) formulas. It is inspired by recent developments in finite abstraction techniques for hybrid systems and motion planning problems. The robot and its environment is assumed to have a finite abstraction as a Partially Observable Markov Decision Process (POMDP), which is a powerful model class capable of representing a wide variety of problems. However, synthesizing controllers that satisfy LTL goals over POMDPs is a challenging problem which has received only limited attention.

This thesis proposes tractable, approximate algorithms for the control synthesis problem using Finite State Controllers (FSCs). The use of FSCs to control finite POMDPs allows for the closed system to be analyzed as finite global Markov chain. The thesis explicitly shows how transient and steady state behavior of the global Markov chains can be related to two different criteria with respect to satisfaction of LTL formulas. First, the maximization of the probability of LTL satisfaction is related to an optimization problem over a parametrization of the FSC. Analytic computation of gradients are derived which allows the use of first order optimization techniques.

The second criterion encourages rapid and frequent visits to a restricted set of states over infinite executions. It is formulated as a constrained optimization problem with a discounted long term reward objective by the novel utilization of a fundamental equation for Markov chains - the Poisson equation. A new constrained policy iteration technique is proposed to solve the resulting dynamic program, which also provides a way to escape local maxima.

The algorithms proposed in the thesis are applied to the task planning and execution challenges faced during the DARPA Autonomous Robotic Manipulation - Software challenge.

Stationary absolute distributions for chains of infinite order

Let {Ƶ_n}^∞_{n = -∞} be a stochastic process with state space S₁ = {0, 1, …, D – 1}. Such a process is called a chain of infinite order. The transitions of the chain are described by the functions

Q_i(i⁽⁰⁾) = Ƥ(Ƶ_n = i | Ƶ_{n - 1} = i ⁽⁰⁾₁, Ƶ_{n - 2} = i ⁽⁰⁾₂, …) (i ɛ S₁), where i⁽⁰⁾ = (i⁽⁰⁾₁, i⁽⁰⁾₂, …) ranges over infinite sequences from S₁. If i⁽ⁿ⁾ = (i⁽ⁿ⁾₁, i⁽ⁿ⁾₂, …) for n = 1, 2,…, then i⁽ⁿ⁾ → i⁽⁰⁾ means that for each k, i⁽ⁿ⁾_k = i⁽⁰⁾_k for all n sufficiently large.

Given functions Q_i(i⁽⁰⁾) such that

(i) 0 ≤ Q_i(i⁽⁰) ≤ ξ ˂ 1

(ii)D – 1/Ʃ/i = 0 Q_i(i⁽⁰⁾) Ξ 1

(iii) Q_i(i⁽ⁿ⁾) → Q_i(i⁽⁰⁾) whenever i⁽ⁿ⁾ → i⁽⁰⁾,

we prove the existence of a stationary chain of infinite order {Ƶ_n} whose transitions are given by

Ƥ (Ƶ_n = i | Ƶ_{n - 1}, Ƶ_{n - 2}, …) = Q_i(Ƶ_{n - 1}, Ƶ_{n - 2}, …)

With probability 1. The method also yields stationary chains {Ƶ_n} for which (iii) does not hold but whose transition probabilities are, in a sense, “locally Markovian.” These and similar results extend a paper by T.E. Harris [Pac. J. Math., 5 (1955), 707-724].

Included is a new proof of the existence and uniqueness of a stationary absolute distribution for an Nth order Markov chain in which all transitions are possible. This proof allows us to achieve our main results without the use of limit theorem techniques.

Chains of Non-regular de Branges Spaces

We consider canonical systems with singular left endpoints, and discuss the concept of a scalar spectral measure and the corresponding generalized Fourier transform associated with a canonical system with a singular left endpoint. We use the framework of de Branges’ theory of Hilbert spaces of entire functions to study the correspondence between chains of non-regular de Branges spaces, canonical systems with singular left endpoints, and spectral measures.

We find sufficient integrability conditions on a Hamiltonian H which ensure the existence of a chain of de Branges functions in the first generalized Pólya class with Hamiltonian H. This result generalizes de Branges’ Theorem 41, which showed the sufficiency of stronger integrability conditions on H for the existence of a chain in the Pólya class. We show the conditions that de Branges came up with are also necessary. In the case of Krein’s strings, namely when the Hamiltonian is diagonal, we show our proposed conditions are also necessary.

We also investigate the asymptotic conditions on chains of de Branges functions as t approaches its left endpoint. We show there is a one-to-one correspondence between chains of de Branges functions satisfying certain asymptotic conditions and chains in the Pólya class. In the case of Krein’s strings, we also establish the one-to-one correspondence between chains satisfying certain asymptotic conditions and chains in the generalized Pólya class.

Force chains, friction, and flow: Behavior of granular media across length scales

We study the behavior of granular materials at three length scales. At the smallest length scale, the grain-scale, we study inter-particle forces and "force chains". Inter-particle forces are the natural building blocks of constitutive laws for granular materials. Force chains are a key signature of the heterogeneity of granular systems. Despite their fundamental importance for calibrating grain-scale numerical models and elucidating constitutive laws, inter-particle forces have not been fully quantified in natural granular materials. We present a numerical force inference technique for determining inter-particle forces from experimental data and apply the technique to two-dimensional and three-dimensional systems under quasi-static and dynamic load. These experiments validate the technique and provide insight into the quasi-static and dynamic behavior of granular materials.

At a larger length scale, the mesoscale, we study the emergent frictional behavior of a collection of grains. Properties of granular materials at this intermediate scale are crucial inputs for macro-scale continuum models. We derive friction laws for granular materials at the mesoscale by applying averaging techniques to grain-scale quantities. These laws portray the nature of steady-state frictional strength as a competition between steady-state dilation and grain-scale dissipation rates. The laws also directly link the rate of dilation to the non-steady-state frictional strength.

At the macro-scale, we investigate continuum modeling techniques capable of simulating the distinct solid-like, liquid-like, and gas-like behaviors exhibited by granular materials in a single computational domain. We propose a Smoothed Particle Hydrodynamics (SPH) approach for granular materials with a viscoplastic constitutive law. The constitutive law uses a rate-dependent and dilation-dependent friction law. We provide a theoretical basis for a dilation-dependent friction law using similar analysis to that performed at the mesoscale. We provide several qualitative and quantitative validations of the technique and discuss ongoing work aiming to couple the granular flow with gas and fluid flows.