Discrete-Time Controlled Markov Processes with Average Cost Criterion: A Survey

This work is a survey of the average cost control problem for discrete-time Markov processes. The authors have attempted to put together a comprehensive account of the considerable research on this problem over the past three decades. The exposition ranges from finite to Borel state and action spaces and includes a variety of methodologies to find and characterize optimal policies. The authors have included a brief historical perspective of the research efforts in this area and have compiled a substantial yet not exhaustive bibliography. The authors have also identified several important questions that are still open to investigation.

On the Milito-Cruz adaptive control scheme for Markov chains

Milito and Cruz have introduced a novel adaptive control scheme for finite Markov chains when a finite parametrized family of possible transition matrices is available. The scheme involves the minimization of a composite functional of the observed history of the process incorporating both control and estimation aspects. We prove the a.s. optimality of a similar scheme when the state space is countable and the parameter space a compact subset ofR.

Transformation formula for exponential sums involving fourier coefficients of modular forms

In 1984 Jutila [5] obtained a transformation formula for certain exponential sums involving the Fourier coefficients of a holomorphic cusp form for the full modular group SL(2, Z). With the help of the transformation formula he obtained good estimates for the distance between consecutive zeros on the critical line of the Dirichlet series associated with the cusp form and for the order of the Dirichlet series on the critical line, [7]. In this paper we follow Jutila to obtain a transformation formula for exponential sums involving the Fourier coefficients of either holomorphic cusp forms or certain Maass forms for congruence subgroups of SL(2, Z) and prove similar estimates for the corresponding Dirichlet series.

Risk Minimizing Option Pricing for a Class of Exotic Options in a Markov-Modulated Market

We address risk minimizing option pricing in a regime switching market where the floating interest rate depends on a finite state Markov process. The growth rate and the volatility of the stock also depend on the Markov process. Using the minimal martingale measure, we show that the locally risk minimizing prices for certain exotic options satisfy a system of Black-Scholes partial differential equations with appropriate boundary conditions. We find the corresponding hedging strategies and the residual risk. We develop suitable numerical methods to compute option prices.

A formula for the solution of general Abel integral equation

A new formula for the solution of the general Abel Integral equation is derived, and an important special case is checked with the known result.

Supercritical age-dependent branching Markov processes and their scaling limits

This paper studies the long-time behavior of the empirical distribution of age and normalized position of an age-dependent supercritical branching Markov process. The motion of each individual during its life is a random function of its age. It is shown that the empirical distribution of the age and the normalized position of all individuals alive at time t converges as t -> infinity to a deterministic product measure.

An actor-critic algorithm with function approximation for discounted cost constrained Markov decision processes

We develop in this article the first actor-critic reinforcement learning algorithm with function approximation for a problem of control under multiple inequality constraints. We consider the infinite horizon discounted cost framework in which both the objective and the constraint functions are suitable expected policy-dependent discounted sums of certain sample path functions. We apply the Lagrange multiplier method to handle the inequality constraints. Our algorithm makes use of multi-timescale stochastic approximation and incorporates a temporal difference (TD) critic and an actor that makes a gradient search in the space of policy parameters using efficient simultaneous perturbation stochastic approximation (SPSA) gradient estimates. We prove the asymptotic almost sure convergence of our algorithm to a locally optimal policy. (C) 2010 Elsevier B.V. All rights reserved.

Extended kac-akhiezer formulae and the fredholm determinant of finite section hilbert-schmidt kernels

This paper deals with some results (known as Kac-Akhiezer formulae) on generalized Fredholm determinants for Hilbert-Schmidt operators on L2-spaces, available in the literature for convolution kernels on intervals. The Kac-Akhiezer formulae have been obtained for kernels which are not necessarily of convolution nature and for domains in R(n).

Matrix method for eigenvalue assignment: The single input case

The eigenvalue assignment/pole placement procedure has found application in a wide variety of control problems. The associated literature is rather extensive with a number of techniques discussed to that end. In this paper a method for assigning eigenvalues to a Linear Time Invariant (LTI) single input system is proposed. The algorithm determines a matrix, which has eigenvalues at the desired locations. It is obtained from the knowledge of the open-loop system and the desired eigenvalues. Solution of the matrix equation, involving unknown controller gains, open-loop system matrices and desired eigenvalues, results in the state feedback controller. The proposed algorithm requires the closed-loop eigenvalues to be different from those of the open-loop case. This apparent constraint is easily overcome by a negligible shift in the values. Two examples are considered to verify the proposed algorithm. The first one pertains to the in-plane libration of a Tethered Satellite System (TSS) while the second is concerned with control of the short period dynamics of a flexible airplane. Finally, the method is extended to determine the Controllability Grammian, corresponding to the specified closed-loop eigenvalues, without computing the controller gains.

Matrix method for eigenvalue assignment: The single input case

The eigenvalue assignment/pole placement procedure has found application in a wide variety of control problems. The associated literature is rather extensive with a number of techniques discussed to that end. In this paper a method for assigning eigenvalues to a Linear Time Invariant (LTI) single input system is proposed. The algorithm determines a matrix, which has eigenvalues at the desired locations. It is obtained from the knowledge of the open-loop system and the desired eigenvalues. Solution of the matrix equation, involving unknown controller gains, open-loop system matrices and desired eigenvalues, results in the state feedback controller. The proposed algorithm requires the closed-loop eigenvalues to be different from those of the open-loop case. This apparent constraint is easily overcome by a negligible shift in the values. Two examples are considered to verify the proposed algorithm. The first one pertains to the in-plane libration of a Tethered Satellite System (TSS) while the second is concerned with control of the short period dynamics of a flexible airplane. Finally, the method is extended to determine the Controllability Grammian, corresponding to the specified closed-loop eigenvalues, without computing the controller gains.

Phosphate derivatives of 123 cuprates of the formula LnSr(2)Cu(3)O(7) (Ln=Gd, Dy, Ho)

Partial substitution of Cu in the chain by the phosphate ion stabilizes LnSr(2)Cu(3)O(7) (Ln = Gd, Dy or Ho) in the 123 structure. The LnSr(2)Cu(2.8)(PO4)(0.2)O-y derivatives exhibit incommensurately modulated structures. The holmium oxy-phosphate derivative has been rendered superconducting by the partial substitution of Ho by Ca.

Multiscale stochastic approximation for parametric optimization of hidden Markov models

A two-time scale stochastic approximation algorithm is proposed for simulation-based parametric optimization of hidden Markov models, as an alternative to the traditional approaches to ''infinitesimal perturbation analysis.'' Its convergence is analyzed, and a queueing example is presented.