Ergodic control of partially observed Markov chains

The ergodic or long-run average cost control problem for a partially observed finite-state Markov chain is studied via the associated fully observed separated control problem for the nonlinear filter. Dynamic programming equations for the latter are derived, leading to existence and characterization of optimal stationary policies.

Asymptotic behaviour of point processes with Markov-dependent intervals

This splitting techniques for MARKOV chains developed by NUMMELIN (1978a) and ATHREYA and NEY (1978b) are used to derive an imbedded renewal process in WOLD's point process with MARKOV-correlated intervals. This leads to a simple proof of renewal theorems for such processes. In particular, a key renewal theorem is proved, from which analogues to both BLACKWELL's and BREIMAN's forms of the renewal theorem can be deduced.

Risk Minimizing Option Pricing in a Semi-Markov Modulated Market

We address risk minimizing option pricing in a semi-Markov modulated market where the floating interest rate depends on a finite state semi-Markov process. The growth rate and the volatility of the stock also depend on the semi-Markov process. Using the Föllmer–Schweizer decomposition we find the locally risk minimizing price for European options and the corresponding hedging strategy. We develop suitable numerical methods for computing option prices.

Portfolio Optimization in a Semi-Markov Modulated Market

We address a portfolio optimization problem in a semi-Markov modulated market. We study both the terminal expected utility optimization on finite time horizon and the risk-sensitive portfolio optimization on finite and infinite time horizon. We obtain optimal portfolios in relevant cases. A numerical procedure is also developed to compute the optimal expected terminal utility for finite horizon problem.

Optimal maintenance and replacement via a semi-Markov decision model

Optimal bang-coast maintenance policies for a machine, subject to failure, are considered. The approach utilizes a semi-Markov model for the system. A simplified model for modifying the probability of machine failure with maintenance is employed. A numerical example is presented to illustrate the procedure and results.

Asymptotic analysis of option pricing in a Markov modulated market

We address asymptotic analysis of option pricing in a regime switching market where the risk free interest rate, growth rate and the volatility of the stocks depend on a finite state Markov chain. We study two variations of the chain namely, when the chain is moving very fast compared to the underlying asset price and when it is moving very slow. Using quadratic hedging and asymptotic expansion, we derive corrections on the locally risk minimizing option price.

Control aspects in proportionality Markov manpower systems

In this paper, the control aspects of a hierarchical organization under the influence of "proportionality" policies are analyzed. Proportionality policies are those that restrict the recruitment to every level of the hierarchy (except the bottom most level or base level) to be in strict proportion to the promotions into that level. Both long term and short term control analysis have been discussed. In long term control the specific roles of the parameters of the system with regard to control of the shape and size of the system have been analyzed and yield suitable control strategies. In short term control, the attainability of a target or goal structure within a specific time from a given initial structure has been analyzed and yields the required recruitment strategies. The theoretical analyses have been illustrated with computational examples and also with real world data.

Control aspects in proportionality Markov manpower systems

In this paper, the control aspects of a hierarchical organization under the influence of "proportionality" policies are analyzed. Proportionality policies are those that restrict the recruitment to every level of the hierarchy (except the bottom most level or base level) to be in strict proportion to the promotions into that level. Both long term and short term control analysis have been discussed. In long term control the specific roles of the parameters of the system with regard to control of the shape and size of the system have been analyzed and yield suitable control strategies. In short term control, the attainability of a target or goal structure within a specific time from a given initial structure has been analyzed and yields the required recruitment strategies. The theoretical analyses have been illustrated with computational examples and also with real world data. The control of such proportionality systems is then compared with that of the general systems (which do not follow such policies) with some significant conclusions. The control relations of such proportionality systems are found to be simpler and more practically feasible than those of general Markov systems, which do not have such restrictions. Such proportionality systems thus not only retain and match the flexibility of general Markov systems but also have the added advantage of simpler and more practically feasible controls. The proportionality policies hence act as an alternative and more practicably feasible means of control. (C) 2004 Elsevier Inc. All rights reserved.

A reinforcement learning based algorithm for finite horizon Markov decision processes

We develop a simulation based algorithm for finite horizon Markov decision processes with finite state and finite action space. Illustrative numerical experiments with the proposed algorithm are shown for problems in flow control of communication networks and capacity switching in semiconductor fabrication.

Markov Chain Model for Cracking Behavior of Reinforced Concrete Beams

In this paper, nonhomogeneous Markov chains are proposed for modeling the cracking behavior of reinforced concrete beams subjected to monotonically increasing loads. The model facilitates prediction of the maximum crackwidth at a given load given the crackwidth at a lower load level, and thus leads to a better understanding of the cracking phenomenon. To illustrate the methodology developed, the results of three reinforced concrete beams tested in the laboratory are analyzed and presented.

Critical age-dependent branching Markov processes and their scaling limits

This paper studies:(i)the long-time behaviour of the empirical distribution of age and normalized position of an age-dependent critical branching Markov process conditioned on non-extinction;and (ii) the super-process limit of a sequence of age-dependent critical branching Brownian motions.

Discrete-Time Controlled Markov Processes with Average Cost Criterion: A Survey

This work is a survey of the average cost control problem for discrete-time Markov processes. The authors have attempted to put together a comprehensive account of the considerable research on this problem over the past three decades. The exposition ranges from finite to Borel state and action spaces and includes a variety of methodologies to find and characterize optimal policies. The authors have included a brief historical perspective of the research efforts in this area and have compiled a substantial yet not exhaustive bibliography. The authors have also identified several important questions that are still open to investigation.