Control aspects in proportionality Markov manpower systems

In this paper, the control aspects of a hierarchical organization under the influence of "proportionality" policies are analyzed. Proportionality policies are those that restrict the recruitment to every level of the hierarchy (except the bottom most level or base level) to be in strict proportion to the promotions into that level. Both long term and short term control analysis have been discussed. In long term control the specific roles of the parameters of the system with regard to control of the shape and size of the system have been analyzed and yield suitable control strategies. In short term control, the attainability of a target or goal structure within a specific time from a given initial structure has been analyzed and yields the required recruitment strategies. The theoretical analyses have been illustrated with computational examples and also with real world data.

Control aspects in proportionality Markov manpower systems

In this paper, the control aspects of a hierarchical organization under the influence of "proportionality" policies are analyzed. Proportionality policies are those that restrict the recruitment to every level of the hierarchy (except the bottom most level or base level) to be in strict proportion to the promotions into that level. Both long term and short term control analysis have been discussed. In long term control the specific roles of the parameters of the system with regard to control of the shape and size of the system have been analyzed and yield suitable control strategies. In short term control, the attainability of a target or goal structure within a specific time from a given initial structure has been analyzed and yields the required recruitment strategies. The theoretical analyses have been illustrated with computational examples and also with real world data. The control of such proportionality systems is then compared with that of the general systems (which do not follow such policies) with some significant conclusions. The control relations of such proportionality systems are found to be simpler and more practically feasible than those of general Markov systems, which do not have such restrictions. Such proportionality systems thus not only retain and match the flexibility of general Markov systems but also have the added advantage of simpler and more practically feasible controls. The proportionality policies hence act as an alternative and more practicably feasible means of control. (C) 2004 Elsevier Inc. All rights reserved.

Ergodic control of partially observed Markov chains

The ergodic or long-run average cost control problem for a partially observed finite-state Markov chain is studied via the associated fully observed separated control problem for the nonlinear filter. Dynamic programming equations for the latter are derived, leading to existence and characterization of optimal stationary policies.

Optimal maintenance and replacement via a semi-Markov decision model

Optimal bang-coast maintenance policies for a machine, subject to failure, are considered. The approach utilizes a semi-Markov model for the system. A simplified model for modifying the probability of machine failure with maintenance is employed. A numerical example is presented to illustrate the procedure and results.

A mathematical programming approach to optimal Markovian switching of Poisson arrival streams to queueing systems

Motivated by certain situations in manufacturing systems and communication networks, we look into the problem of maximizing the profit in a queueing system with linear reward and cost structure and having a choice of selecting the streams of Poisson arrivals according to an independent Markov chain. We view the system as a MMPP/GI/1 queue and seek to maximize the profits by optimally choosing the stationary probabilities of the modulating Markov chain. We consider two formulations of the optimization problem. The first one (which we call the PUT problem) seeks to maximize the profit per unit time whereas the second one considers the maximization of the profit per accepted customer (the PAC problem). In each of these formulations, we explore three separate problems. In the first one, the constraints come from bounding the utilization of an infinite capacity server; in the second one the constraints arise from bounding the mean queue length of the same queue; and in the third one the finite capacity of the buffer reflect as a set of constraints. In the problems bounding the utilization factor of the queue, the solutions are given by essentially linear programs, while the problems with mean queue length constraints are linear programs if the service is exponentially distributed. The problems modeling the finite capacity queue are non-convex programs for which global maxima can be found. There is a rich relationship between the solutions of the PUT and PAC problems. In particular, the PUT solutions always make the server work at a utilization factor that is no less than that of the PAC solutions.

An actor-critic algorithm with function approximation for discounted cost constrained Markov decision processes

We develop in this article the first actor-critic reinforcement learning algorithm with function approximation for a problem of control under multiple inequality constraints. We consider the infinite horizon discounted cost framework in which both the objective and the constraint functions are suitable expected policy-dependent discounted sums of certain sample path functions. We apply the Lagrange multiplier method to handle the inequality constraints. Our algorithm makes use of multi-timescale stochastic approximation and incorporates a temporal difference (TD) critic and an actor that makes a gradient search in the space of policy parameters using efficient simultaneous perturbation stochastic approximation (SPSA) gradient estimates. We prove the asymptotic almost sure convergence of our algorithm to a locally optimal policy. (C) 2010 Elsevier B.V. All rights reserved.

Monte-Carlo estimation of time-dependent statistical characteristics of random dynamical systems

The problem of estimating the time-dependent statistical characteristics of a random dynamical system is studied under two different settings. In the first, the system dynamics is governed by a differential equation parameterized by a random parameter, while in the second, this is governed by a differential equation with an underlying parameter sequence characterized by a continuous time Markov chain. We propose, for the first time in the literature, stochastic approximation algorithms for estimating various time-dependent process characteristics of the system. In particular, we provide efficient estimators for quantities such as the mean, variance and distribution of the process at any given time as well as the joint distribution and the autocorrelation coefficient at different times. A novel aspect of our approach is that we assume that information on the parameter model (i.e., its distribution in the first case and transition probabilities of the Markov chain in the second) is not available in either case. This is unlike most other work in the literature that assumes availability of such information. Also, most of the prior work in the literature is geared towards analyzing the steady-state system behavior of the random dynamical system while our focus is on analyzing the time-dependent statistical characteristics which are in general difficult to obtain. We prove the almost sure convergence of our stochastic approximation scheme in each case to the true value of the quantity being estimated. We provide a general class of strongly consistent estimators for the aforementioned statistical quantities with regular sample average estimators being a specific instance of these. We also present an application of the proposed scheme on a widely used model in population biology. Numerical experiments in this framework show that the time-dependent process characteristics as obtained using our algorithm in each case exhibit excellent agreement with exact results. (C) 2010 Elsevier Inc. All rights reserved.

Stochastic modelling of flexible manufacturing systems

Mathematical modelling plays a vital role in the design, planning and operation of flexible manufacturing systems (FMSs). In this paper, attention is focused on stochastic modelling of FMSs using Markov chains, queueing networks, and stochastic Petri nets. We bring out the role of these modelling tools in FMS performance evaluation through several illustrative examples and provide a critical comparative evaluation. We also include a discussion on the modelling of deadlocks which constitute an important source of performance degradation in fully automated FMSs.

Structured systems methodology for evaluation of random interruptions in continuous process type manufacturing systems

A structured systems methodology was developed to analyse the problems of production interruptions occurring at random intervals in continuous process type manufacturing systems. At a macro level the methodology focuses on identifying suitable investment policies to reduce interruptions of a total manufacturing system that is a combination of several process plants. An interruption-tree-based simulation model was developed for macroanalysis. At a micro level the methodology focuses on finding the effects of alternative configurations of individual process plants on the overall system performance. A Markov simulation model was developed for microlevel analysis. The methodology was tested with an industry-specific application.

On the optimality of exhaustive service policies in multiclass queueing systems with modulated arrivals and switchovers

Consider a single-server multiclass queueing system with K classes where the individual queues are fed by K-correlated interrupted Poisson streams generated in the states of a K-state stationary modulating Markov chain. The service times for all the classes are drawn independently from the same distribution. There is a setup time (and/or a setup cost) incurred whenever the server switches from one queue to another. It is required to minimize the sum of discounted inventory and setup costs over an infinite horizon. We provide sufficient conditions under which exhaustive service policies are optimal. We then present some simulation results for a two-class queueing system to show that exhaustive, threshold policies outperform non-exhaustive policies.

Interference in Multiprocessor Systems with Localized Memory Access Probabilities

Past studies of memory interference in multiprocessor systems have generally assumed that the references of each processor are uniformly distributed among the memory modules. In this paper we develop a model with local referencing, which reflects more closely the behavior of real-life programs. This model is analyzed using Markov chain techniques and expressions are derived for the multiprocessor performance. New expressions are also obtained for the performance in the traditional uniform reference model and are compared with other expressions-available in the literature. Results of a simulation study are given to show the accuracy of the expressions for both models.

Approach to the problem of ferroresonance in e.h.v. systems

An analytical analysis of ferroresonance with possible cases of its occurrence in series-and shunt-compensated systems is presented. A term `percentage unstable zoneÿ is defined to compare the jump severity of different nonlinearities. A direct analytical method has been shown to yield complete information. An attempt has been made to find all four critical points: jump-from and jump-to points of ferroresonance jump phenomena. The systems considered for analysis are typical 500 kV transmission systems of various lengths.

Modeling stochastic hybrid systems

Stochastic hybrid systems arise in numerous applications of systems with multiple models; e.g., air traffc management, flexible manufacturing systems, fault tolerant control systems etc. In a typical hybrid system, the state space is hybrid in the sense that some components take values in a Euclidean space, while some other components are discrete. In this paper we propose two stochastic hybrid models, both of which permit diffusion and hybrid jump. Such models are essential for studying air traffic management in a stochastic framework.

Stochastic optimization for adaptive labor staffing in service systems

Service systems are labor intensive. Further, the workload tends to vary greatly with time. Adapting the staffing levels to the workloads in such systems is nontrivial due to a large number of parameters and operational variations, but crucial for business objectives such as minimal labor inventory. One of the central challenges is to optimize the staffing while maintaining system steady-state and compliance to aggregate SLA constraints. We formulate this problem as a parametrized constrained Markov process and propose a novel stochastic optimization algorithm for solving it. Our algorithm is a multi-timescale stochastic approximation scheme that incorporates a SPSA based algorithm for ‘primal descent' and couples it with a ‘dual ascent' scheme for the Lagrange multipliers. We validate this optimization scheme on five real-life service systems and compare it with a state-of-the-art optimization tool-kit OptQuest. Being two orders of magnitude faster than OptQuest, our scheme is particularly suitable for adaptive labor staffing. Also, we observe that it guarantees convergence and finds better solutions than OptQuest in many cases.