Discrete-Time Controlled Markov Processes with Average Cost Criterion: A Survey

This work is a survey of the average cost control problem for discrete-time Markov processes. The authors have attempted to put together a comprehensive account of the considerable research on this problem over the past three decades. The exposition ranges from finite to Borel state and action spaces and includes a variety of methodologies to find and characterize optimal policies. The authors have included a brief historical perspective of the research efforts in this area and have compiled a substantial yet not exhaustive bibliography. The authors have also identified several important questions that are still open to investigation.

A global convergence criterion for discrete-time multivariable adaptive systems

The paper deals with the basic problem of adjusting a matrix gain in a discrete-time linear multivariable system. The object is to obtain a global convergence criterion, i.e. conditions under which a specified error signal asymptotically approaches zero and other signals in the system remain bounded for arbitrary initial conditions and for any bounded input to the system. It is shown that for a class of up-dating algorithms for the adjustable gain matrix, global convergence is crucially dependent on a transfer matrix G(z) which has a simple block diagram interpretation. When w(z)G(z) is strictly discrete positive real for a scalar w(z) such that w-1(z) is strictly proper with poles and zeros within the unit circle, an augmented error scheme is suggested and is proved to result in global convergence. The solution avoids feeding back a quadratic term as recommended in other schemes for single-input single-output systems.

Rendering stiffer walls: a hybrid haptic system using continuous and discrete time feedback

Instability in conventional haptic rendering destroys the perception of rigid objects in virtual environments. Inherent limitations in the conventional haptic loop restrict the maximum stiffness that can be rendered. In this paper we present a method to render virtual walls that are much stiffer than those achieved by conventional techniques. By removing the conventional digital haptic loop and replacing it with a part-continuous and part-discrete time hybrid haptic loop, we were able to render stiffer walls. The control loop is implemented as a combinational logic circuit on an field-programmable gate array. We compared the performance of the conventional haptic loop and our hybrid haptic loop on the same haptic device, and present mathematical analysis to show the limit of stability of our device. Our hybrid method removes the computer-intensive haptic loop from the CPU-this can free a significant amount of resources that can be used for other purposes such as graphical rendering and physics modeling. It is our hope that, in the future, similar designs will lead to a haptics processing unit (HPU).

Optimal infinite-horizon feedback laws for a general class of constrained discrete-time systems: stability and moving-horizon approximations

Stability results are given for a class of feedback systems arising from the regulation of time-varying discrete-time systems using optimal infinite-horizon and moving-horizon feedback laws. The class is characterized by joint constraints on the state and the control, a general nonlinear cost function and nonlinear equations of motion possessing two special properties. It is shown that weak conditions on the cost function and the constraints are sufficient to guarantee uniform asymptotic stability of both the optimal infinite-horizon and movinghorizon feedback systems. The infinite-horizon cost associated with the moving-horizon feedback law approaches the optimal infinite-horizon cost as the moving horizon is extended.

T. E. Harris and branching processes

T. E. Harris was a pioneer par excellence in many fields of probability theory. In this paper, we give a brief survey of the many fundamental contributions of Harris to the theory of branching processes, starting with his doctoral work at Princeton in the late forties and culminating in his fundamental book ``The Theory of Branching Processes,'' published in 1963.

Some algorithms for discrete time queues with finite capacity

We consider a discrete time queue with finite capacity and i.i.d. and Markov modulated arrivals, Efficient algorithms are developed to calculate the moments and the distributions of the first time to overflow and the regeneration length, Results are extended to the multiserver queue. Some illustrative numerical examples are provided.

Performance analysis of a fluid queue with random service rate in discrete time

We consider a fluid queue in discrete time with random service rate. Such a queue has been used in several recent studies on wireless networks where the packets can be arbitrarily fragmented. We provide conditions on finiteness of moments of stationary delay, its Laplace-Stieltjes transform and various approximations under heavy traffic. Results are extended to the case where the wireless link can transmit in only a few slots during a frame.

Delay optimal scheduling of a discrete-time batch service queue for point-to-point channel code rate selection

We consider the problem of characterizing the minimum average delay, or equivalently the minimum average queue length, of message symbols randomly arriving to the transmitter queue of a point-to-point link which dynamically selects a (n, k) block code from a given collection. The system is modeled by a discrete time queue with an IID batch arrival process and batch service. We obtain a lower bound on the minimum average queue length, which is the optimal value for a linear program, using only the mean (λ) and variance (σ2) of the batch arrivals. For a finite collection of (n, k) codes the minimum achievable average queue length is shown to be Θ(1/ε) as ε ↓ 0 where ε is the difference between the maximum code rate and λ. We obtain a sufficient condition for code rate selection policies to achieve this optimal growth rate. A simple family of policies that use only one block code each as well as two other heuristic policies are shown to be weakly optimal in the sense of achieving the 1/ε growth rate. An appropriate selection from the family of policies that use only one block code each is also shown to achieve the optimal coefficient σ2/2 of the 1/ε growth rate. We compare the performance of the heuristic policies with the minimum achievable average queue length and the lower bound numerically. For a countable collection of (n, k) codes, the optimal average queue length is shown to be Ω(1/ε). We illustrate the selectivity among policies of the growth rate optimality criterion for both finite and countable collections of (n, k) block codes.

Discrete time second order sliding mode observer for uncertain linear multi-output system

This paper presents a second order sliding mode observer (SOSMO) design for discrete time uncertain linear multi-output system. The design procedure is effective for both matched and unmatched bounded uncertainties and/or disturbances. A second order sliding function and corresponding sliding manifold for discrete time system are defined similar to the lines of continuous time counterpart. A boundary layer concept is employed to avoid switching across the defined sliding manifold and the sliding trajectory is confined to a boundary layer once it converges to it. The condition for existence of convergent quasi-sliding mode (QSM) is derived. The observer estimation errors satisfying given stability conditions converge to an ultimate finite bound (within the specified boundary layer) with thickness O(T-2) where T is the sampling period. A relation between sliding mode gain and boundary layer is established for the existence of second order discrete sliding motion. The design strategy is very simple to apply and is demonstrated for three examples with different class of disturbances (matched and unmatched) to show the effectiveness of the design. Simulation results to show the robustness with respect to the measurement noise are given for SOSMO and the performance is compared with pseudo-linear Kalman filter (PLKF). (C) 2013 Published by Elsevier Ltd. on behalf of The Franklin Institute

Risk-Sensitive Ergodic Control of Continuous Time Markov Processes With Denumerable State Space

In this article, we study risk-sensitive control problem with controlled continuous time Markov chain state dynamics. Using multiplicative dynamic programming principle along with the atomic structure of the state dynamics, we prove the existence and a characterization of optimal risk-sensitive control under geometric ergodicity of the state dynamics along with a smallness condition on the running cost.

Scattering theory and the discrete linear optimal control problem

This paper deals with the interpretation of the discrete-time optimal control problem as a scattering process in a discrete medium. We treat the discrete optimal linear regulator, constrained end-point and servo and tracking problems, providing a unified approach to these problems. This approach results in an easy derivation of the desired results as well as several new ones.

Discrete parameter simulation optimization algorithms with applications to admission control with dependent service times

We propose certain discrete parameter variants of well known simulation optimization algorithms. Two of these algorithms are based on the smoothed functional (SF) technique while two others are based on the simultaneous perturbation stochastic approximation (SPSA) method. They differ from each other in the way perturbations are obtained and also the manner in which projections and parameter updates are performed. All our algorithms use two simulations and two-timescale stochastic approximation. As an application setting, we consider the important problem of admission control of packets in communication networks under dependent service times. We consider a discrete time slotted queueing model of the system and consider two different scenarios - one where the service times have a dependence on the system state and the other where they depend on the number of arrivals in a time slot. Under our settings, the simulated objective function appears ill-behaved with multiple local minima and a unique global minimum characterized by a sharp dip in the objective function in a small region of the parameter space. We compare the performance of our algorithms on these settings and observe that the two SF algorithms show the best results overall. In fact, in many cases studied, SF algorithms converge to the global minimum.

Stability of scheduled message communication over degraded broadcast channels

We develop a multi-class discrete-time processor-sharing queueing model for scheduled message communication over a discrete memoryless degraded broadcast channel. The framework we consider here models both the random message arrivals and the subsequent reliable communication by suitably combining techniques from queueing theory and information theory. Requests for message transmissions are assumed to arrive according to i.i.d. arrival processes. Then, (i) we derive an outer bound to the stability region of message arrival rate vectors achievable by the class of stationary scheduling policies, (ii) we show for any message arrival rate vector that satisfies the outer bound, that there exists a stationary "state-independent" policy that results in a stable system for the corresponding message arrival processes, and (iii) under an asymptotic regime, we show that the stability region of information arrival rate vectors is the information-theoretic capacity region of a degraded broadcast channel.

Robustness of early warning signals of regime shifts in time-delayed ecological models

Various ecological and other complex dynamical systems may exhibit abrupt regime shifts or critical transitions, wherein they reorganize from one stable state to another over relatively short time scales. Because of potential losses to ecosystem services, forecasting such unexpected shifts would be valuable. Using mathematical models of regime shifts, ecologists have proposed various early warning signals of imminent shifts. However, their generality and applicability to real ecosystems remain unclear because these mathematical models are considered too simplistic. Here, we investigate the robustness of recently proposed early warning signals of regime shifts in two well-studied ecological models, but with the inclusion of time-delayed processes. We find that the average variance may either increase or decrease prior to a regime shift and, thus, may not be a robust leading indicator in time-delayed ecological systems. In contrast, changing average skewness, increasing autocorrelation at short time lags, and reddening power spectra of time series of the ecological state variable all show trends consistent with those of models with no time delays. Our results provide insights into the robustness of early warning signals of regime shifts in a broader class of ecological systems.