Efficient methods for stochastic optimal control

The Hamilton Jacobi Bellman (HJB) equation is central to stochastic optimal control (SOC) theory, yielding the optimal solution to general problems specified by known dynamics and a specified cost functional. Given the assumption of quadratic cost on the control input, it is well known that the HJB reduces to a particular partial differential equation (PDE). While powerful, this reduction is not commonly used as the PDE is of second order, is nonlinear, and examples exist where the problem may not have a solution in a classical sense. Furthermore, each state of the system appears as another dimension of the PDE, giving rise to the curse of dimensionality. Since the number of degrees of freedom required to solve the optimal control problem grows exponentially with dimension, the problem becomes intractable for systems with all but modest dimension.

In the last decade researchers have found that under certain, fairly non-restrictive structural assumptions, the HJB may be transformed into a linear PDE, with an interesting analogue in the discretized domain of Markov Decision Processes (MDP). The work presented in this thesis uses the linearity of this particular form of the HJB PDE to push the computational boundaries of stochastic optimal control.

This is done by crafting together previously disjoint lines of research in computation. The first of these is the use of Sum of Squares (SOS) techniques for synthesis of control policies. A candidate polynomial with variable coefficients is proposed as the solution to the stochastic optimal control problem. An SOS relaxation is then taken to the partial differential constraints, leading to a hierarchy of semidefinite relaxations with improving sub-optimality gap. The resulting approximate solutions are shown to be guaranteed over- and under-approximations for the optimal value function. It is shown that these results extend to arbitrary parabolic and elliptic PDEs, yielding a novel method for Uncertainty Quantification (UQ) of systems governed by partial differential constraints. Domain decomposition techniques are also made available, allowing for such problems to be solved via parallelization and low-order polynomials.

The optimization-based SOS technique is then contrasted with the Separated Representation (SR) approach from the applied mathematics community. The technique allows for systems of equations to be solved through a low-rank decomposition that results in algorithms that scale linearly with dimensionality. Its application in stochastic optimal control allows for previously uncomputable problems to be solved quickly, scaling to such complex systems as the Quadcopter and VTOL aircraft. This technique may be combined with the SOS approach, yielding not only a numerical technique, but also an analytical one that allows for entirely new classes of systems to be studied and for stability properties to be guaranteed.

The analysis of the linear HJB is completed by the study of its implications in application. It is shown that the HJB and a popular technique in robotics, the use of navigation functions, sit on opposite ends of a spectrum of optimization problems, upon which tradeoffs may be made in problem complexity. Analytical solutions to the HJB in these settings are available in simplified domains, yielding guidance towards optimality for approximation schemes. Finally, the use of HJB equations in temporal multi-task planning problems is investigated. It is demonstrated that such problems are reducible to a sequence of SOC problems linked via boundary conditions. The linearity of the PDE allows us to pre-compute control policy primitives and then compose them, at essentially zero cost, to satisfy a complex temporal logic specification.

On switched control design of linear time-invariant systems with polytopic uncertainties

This paper proposes a new switched control design method for some classes of linear time-invariant systems with polytopic uncertainties. This method uses a quadratic Lyapunov function to design the feedback controller gains based on linear matrix inequalities (LMIs). The controller gain is chosen by a switching law that returns the smallest value of the time derivative of the Lyapunov function. The proposed methodology offers less conservative alternative than the well-known controller for uncertain systems with only one state feedback gain. The control design of a magnetic levitator illustrates the procedure. © 2013 Wallysonn A. de Souza et al.

Energy decay for the linear Klein-Gordon equation and boundary control

Coordenação de Aperfeiçoamento de Pessoal de Nível Superior (CAPES)

Error Upper Bounds for a Computational Method for Nonlinear Boundary and Initial-Value Problems

This work develops a computational approach for boundary and initial-value problems by using operational matrices, in order to run an evolutive process in a Hilbert space. Besides, upper bounds for errors in the solutions and in their derivatives can be estimated providing accuracy measures.

Probabilistic crack growth analyses using a boundary element model: Applications in linear elastic fracture and fatigue problems

This paper addresses the numerical solution of random crack propagation problems using the coupling boundary element method (BEM) and reliability algorithms. Crack propagation phenomenon is efficiently modelled using BEM, due to its mesh reduction features. The BEM model is based on the dual BEM formulation, in which singular and hyper-singular integral equations are adopted to construct the system of algebraic equations. Two reliability algorithms are coupled with BEM model. The first is the well known response surface method, in which local, adaptive polynomial approximations of the mechanical response are constructed in search of the design point. Different experiment designs and adaptive schemes are considered. The alternative approach direct coupling, in which the limit state function remains implicit and its gradients are calculated directly from the numerical mechanical response, is also considered. The performance of both coupling methods is compared in application to some crack propagation problems. The investigation shows that direct coupling scheme converged for all problems studied, irrespective of the problem nonlinearity. The computational cost of direct coupling has shown to be a fraction of the cost of response surface solutions, regardless of experiment design or adaptive scheme considered. (C) 2012 Elsevier Ltd. All rights reserved.

Optimal mean-variance control for discrete-time linear systems with Markovian jumps and multiplicative noises

In this paper, we consider the stochastic optimal control problem of discrete-time linear systems subject to Markov jumps and multiplicative noises under two criteria. The first one is an unconstrained mean-variance trade-off performance criterion along the time, and the second one is a minimum variance criterion along the time with constraints on the expected output. We present explicit conditions for the existence of an optimal control strategy for the problems, generalizing previous results in the literature. We conclude the paper by presenting a numerical example of a multi-period portfolio selection problem with regime switching in which it is desired to minimize the sum of the variances of the portfolio along the time under the restriction of keeping the expected value of the portfolio greater than some minimum values specified by the investor. (C) 2011 Elsevier Ltd. All rights reserved.

A unified framework for linear function approximation of value functions in stochastic control

This paper contributes with a unified formulation that merges previ- ous analysis on the prediction of the performance ( value function ) of certain sequence of actions ( policy ) when an agent operates a Markov decision process with large state-space. When the states are represented by features and the value function is linearly approxi- mated, our analysis reveals a new relationship between two common cost functions used to obtain the optimal approximation. In addition, this analysis allows us to propose an efficient adaptive algorithm that provides an unbiased linear estimate. The performance of the pro- posed algorithm is illustrated by simulation, showing competitive results when compared with the state-of-the-art solutions.

Scattering theory and the discrete linear optimal control problem

This paper deals with the interpretation of the discrete-time optimal control problem as a scattering process in a discrete medium. We treat the discrete optimal linear regulator, constrained end-point and servo and tracking problems, providing a unified approach to these problems. This approach results in an easy derivation of the desired results as well as several new ones.

Control technique for non-linear and unbalance compensation of an integrated magnetics-based three-phase series-shunt-compensated uninterruptible power supply

Handling unbalanced and non-linear loads in a three-phase AC power supply has always been a difficult issue. This has been addressed in the literature by either using fast controllers in the fundamental rotating reference frame or using separate controllers in reference frames specific to the harmonics. In the former case, the controller needs to be fast and in the latter case, besides the need for many controllers, negative-sequence components need to be extracted from the measured signal. This study proposes a control scheme for harmonic and unbalance compensation of a three-phase uninterruptible power supply wherein the problems mentioned above are addressed. The control takes place in the fundamental positive-sequence reference frame using only a set of feedback and feed-forward compensators. The harmonic components are extracted by a process of frame transformations and used as feed-forward compensation terms in the positive-sequence fundamental reference frame. This study uses a method wherein the measured signal itself is used for fundamental negative-sequence compensation. As the feed-forward compensator handles the high-bandwidth components, the feedback compensator can be a simple low-bandwidth one. This control algorithm is explained and validated experimentally.

Control technique for non-linear and unbalance compensation of an integrated magnetics-based three-phase series-shunt-compensated uninterruptible power supply

Handling unbalanced and non-linear loads in a three-phase AC power supply has always been a difficult issue. This has been addressed in the literature by either using fast controllers in the fundamental rotating reference frame or using separate controllers in reference frames specific to the harmonics. In the former case, the controller needs to be fast and in the latter case, besides the need for many controllers, negative-sequence components need to be extracted from the measured signal. This study proposes a control scheme for harmonic and unbalance compensation of a three-phase uninterruptible power supply wherein the problems mentioned above are addressed. The control takes place in the fundamental positive-sequence reference frame using only a set of feedback and feed-forward compensators. The harmonic components are extracted by a process of frame transformations and used as feed-forward compensation terms in the positive-sequence fundamental reference frame. This study uses a method wherein the measured signal itself is used for fundamental negative-sequence compensation. As the feed-forward compensator handles the high-bandwidth components, the feedback compensator can be a simple low-bandwidth one. This control algorithm is explained and validated experimentally.

Control technique for non-linear and unbalance compensation of an integrated magnetics-based three-phase series-shunt-compensated uninterruptible power supply

Handling unbalanced and non-linear loads in a three-phase AC power supply has always been a difficult issue. This has been addressed in the literature by either using fast controllers in the fundamental rotating reference frame or using separate controllers in reference frames specific to the harmonics. In the former case, the controller needs to be fast and in the lattercase, besides the need for many controllers, negative-sequence components need to be extracted from the measured signal.This study proposes a control scheme for harmonic and unbalance compensation of a three-phase uninterruptible power supply wherein the problems mentioned above are addressed. The control takes place in the fundamental positive-sequence reference frame using only a set of feedback and feed-forward compensators. The harmonic components are extracted by process of frame transformations and used as feed-forward compensation terms in the positive-sequence fundamental reference frame. This study uses a method wherein the measured signal itself is used for fundamental negative-sequence compensation. As the feed-forward compensator handles the high-bandwidth components, the feedback compensator can be a simple low-bandwidth one. This control algorithm is explained and validated experimentally.

A MATLAB Code for Three Dimensional Linear Elastostatics using Constant Boundary Elements

Present work presents a code written in the very simple programming language MATLAB, for three dimensional linear elastostatics, using constant boundary elements. The code, in full or in part, is not a translation or a copy of any of the existing codes. Present paper explains how the code is written, and lists all the formulae used. Code is verified by using the code to solve a simple problem which has the well known approximate analytical solution. Of course, present work does not make any contribution to research on boundary elements, in terms of theory. But the work is justified by the fact that, to the best of author’s knowledge, as of now, one cannot find an open access MATLAB code for three dimensional linear elastostatics using constant boundary elements. Author hopes this paper to be of help to beginners who wish to understand how a simple but complete boundary element code works, so that they can build upon and modify the present open access code to solve complex engineering problems quickly and easily. The code is available online for open access (as supplementary file for the present paper), and may be downloaded from the website for the present journal.

A POSTERIORI ERROR CONTROL OF DISCONTINUOUS GALERKIN METHODS FOR ELLIPTIC OBSTACLE PROBLEMS

In this article, we derive an a posteriori error estimator for various discontinuous Galerkin (DG) methods that are proposed in (Wang, Han and Cheng, SIAM J. Numer. Anal., 48: 708-733, 2010) for an elliptic obstacle problem. Using a key property of DG methods, we perform the analysis in a general framework. The error estimator we have obtained for DG methods is comparable with the estimator for the conforming Galerkin (CG) finite element method. In the analysis, we construct a non-linear smoothing function mapping DG finite element space to CG finite element space and use it as a key tool. The error estimator consists of a discrete Lagrange multiplier associated with the obstacle constraint. It is shown for non-over-penalized DG methods that the discrete Lagrange multiplier is uniformly stable on non-uniform meshes. Finally, numerical results demonstrating the performance of the error estimator are presented.

Cooperative and market-based solutions to pollution abatement problems

This work concerns itself with the possibility of solutions, both cooperative and market based, to pollution abatement problems. In particular, we are interested in pollutant emissions in Southern California and possible solutions to the abatement problems enumerated in the 1990 Clean Air Act. A tradable pollution permit program has been implemented to reduce emissions, creating property rights associated with various pollutants.

Before we discuss the performance of market-based solutions to LA's pollution woes, we consider the existence of cooperative solutions. In Chapter 2, we examine pollutant emissions as a trans boundary public bad. We show that for a class of environments in which pollution moves in a bi-directional, acyclic manner, there exists a sustainable coalition structure and associated levels of emissions. We do so via a new core concept, one more appropriate to modeling cooperative emissions agreements (and potential defection from them) than the standard definitions.

However, this leaves the question of implementing pollution abatement programs unanswered. While the existence of a cost-effective permit market equilibrium has long been understood, the implementation of such programs has been difficult. The design of Los Angeles' REgional CLean Air Incentives Market (RECLAIM) alleviated some of the implementation problems, and in part exacerbated them. For example, it created two overlapping cycles of permits and two zones of permits for different geographic regions. While these design features create a market that allows some measure of regulatory control, they establish a very difficult trading environment with the potential for inefficiency arising from the transactions costs enumerated above and the illiquidity induced by the myriad assets and relatively few participants in this market.

It was with these concerns in mind that the ACE market (Automated Credit Exchange) was designed. The ACE market utilizes an iterated combined-value call market (CV Market). Before discussing the performance of the RECLAIM program in general and the ACE mechanism in particular, we test experimentally whether a portfolio trading mechanism can overcome market illiquidity. Chapter 3 experimentally demonstrates the ability of a portfolio trading mechanism to overcome portfolio rebalancing problems, thereby inducing sufficient liquidity for markets to fully equilibrate.

With experimental evidence in hand, we consider the CV Market's performance in the real world. We find that as the allocation of permits reduces to the level of historical emissions, prices are increasing. As of April of this year, prices are roughly equal to the cost of the Best Available Control Technology (BACT). This took longer than expected, due both to tendencies to mis-report emissions under the old regime, and abatement technology advances encouraged by the program. Vve also find that the ACE market provides liquidity where needed to encourage long-term planning on behalf of polluting facilities.