Non-Stationary Semi-Markov Decision Processes on a Finite Horizon

We introduce and study a class of non-stationary semi-Markov decision processes on a finite horizon. By constructing an equivalent Markov decision process, we establish the existence of a piecewise open loop relaxed control which is optimal for the finite horizon problem.

A novel Q-learning algorithm with function approximation for constrained Markov decision processes

We present a novel multi-timescale Q-learning algorithm for average cost control in a Markov decision process subject to multiple inequality constraints. We formulate a relaxed version of this problem through the Lagrange multiplier method. Our algorithm is different from Q-learning in that it updates two parameters - a Q-value parameter and a policy parameter. The Q-value parameter is updated on a slower time scale as compared to the policy parameter. Whereas Q-learning with function approximation can diverge in some cases, our algorithm is seen to be convergent as a result of the aforementioned timescale separation. We show the results of experiments on a problem of constrained routing in a multistage queueing network. Our algorithm is seen to exhibit good performance and the various inequality constraints are seen to be satisfied upon convergence of the algorithm.

A Markov Decision Theoretic Approach to Pilot Allocation and Receive Antenna Selection

This paper considers antenna selection (AS) at a receiver equipped with multiple antenna elements but only a single radio frequency chain for packet reception. As information about the channel state is acquired using training symbols (pilots), the receiver makes its AS decisions based on noisy channel estimates. Additional information that can be exploited for AS includes the time-correlation of the wireless channel and the results of the link-layer error checks upon receiving the data packets. In this scenario, the task of the receiver is to sequentially select (a) the pilot symbol allocation, i.e., how to distribute the available pilot symbols among the antenna elements, for channel estimation on each of the receive antennas; and (b) the antenna to be used for data packet reception. The goal is to maximize the expected throughput, based on the past history of allocation and selection decisions, and the corresponding noisy channel estimates and error check results. Since the channel state is only partially observed through the noisy pilots and the error checks, the joint problem of pilot allocation and AS is modeled as a partially observed Markov decision process (POMDP). The solution to the POMDP yields the policy that maximizes the long-term expected throughput. Using the Finite State Markov Chain (FSMC) model for the wireless channel, the performance of the POMDP solution is compared with that of other existing schemes, and it is illustrated through numerical evaluation that the POMDP solution significantly outperforms them.

Risk-sensitive control of continuous time Markov chains

We study risk-sensitive control of continuous time Markov chains taking values in discrete state space. We study both finite and infinite horizon problems. In the finite horizon problem we characterize the value function via Hamilton Jacobi Bellman equation and obtain an optimal Markov control. We do the same for infinite horizon discounted cost case. In the infinite horizon average cost case we establish the existence of an optimal stationary control under certain Lyapunov condition. We also develop a policy iteration algorithm for finding an optimal control.

LANDSCAPE DYNAMICS MODELING THROUGH INTEGRATED MARKOV, FUZZY-AHP AND CELLULAR AUTOMATA

Multi temporal land use information were derived using two decades remote sensing data and simulated for 2012 and 2020 with Cellular Automata (CA) considering scenarios, change probabilities (through Markov chain) and Multi Criteria Evaluation (MCE). Agents and constraints were considered for modeling the urbanization process. Agents were nornmlized through fiizzyfication and priority weights were assigned through Analytical Hierarchical Process (AHP) pairwise comparison for each factor (in MCE) to derive behavior-oriented rules of transition for each land use class. Simulation shows a good agreement with the classified data. Fuzzy and AHP helped in analyzing the effects of agents of growth clearly and CA-Markov proved as a powerful tool in modelling and helped in capturing and visualizing the spatiotemporal patterns of urbanization. This provided rapid land evaluation framework with the essential insights of the urban trajectory for effective sustainable city planning.

Markov chains, R-trivial monoids and representation theory

We develop a general theory of Markov chains realizable as random walks on R-trivial monoids. It provides explicit and simple formulas for the eigenvalues of the transition matrix, for multiplicities of the eigenvalues via Mobius inversion along a lattice, a condition for diagonalizability of the transition matrix and some techniques for bounding the mixing time. In addition, we discuss several examples, such as Toom-Tsetlin models, an exchange walk for finite Coxeter groups, as well as examples previously studied by the authors, such as nonabelian sandpile models and the promotion Markov chain on posets. Many of these examples can be viewed as random walks on quotients of free tree monoids, a new class of monoids whose combinatorics we develop.

Markov chain splitting methods in structural reliability integral estimation

Monte Carlo simulation methods involving splitting of Markov chains have been used in evaluation of multi-fold integrals in different application areas. We examine in this paper the performance of these methods in the context of evaluation of reliability integrals from the point of view of characterizing the sampling fluctuations. The methods discussed include the Au-Beck subset simulation, Holmes-Diaconis-Ross method, and generalized splitting algorithm. A few improvisations based on first order reliability method are suggested to select algorithmic parameters of the latter two methods. The bias and sampling variance of the alternative estimators are discussed. Also, an approximation to the sampling distribution of some of these estimators is obtained. Illustrative examples involving component and series system reliability analyses are presented with a view to bring out the relative merits of alternative methods. (C) 2015 Elsevier Ltd. All rights reserved.

Risk-Sensitive Ergodic Control of Continuous Time Markov Processes With Denumerable State Space

In this article, we study risk-sensitive control problem with controlled continuous time Markov chain state dynamics. Using multiplicative dynamic programming principle along with the atomic structure of the state dynamics, we prove the existence and a characterization of optimal risk-sensitive control under geometric ergodicity of the state dynamics along with a smallness condition on the running cost.

Bayesian model selection for time series using Markov chain Monte Carlo

We present a stochastic simulation technique for subset selection in time series models, based on the use of indicator variables with the Gibbs sampler within a hierarchical Bayesian framework. As an example, the method is applied to the selection of subset linear AR models, in which only significant lags are included. Joint sampling of the indicators and parameters is found to speed convergence. We discuss the possibility of model mixing where the model is not well determined by the data, and the extension of the approach to include non-linear model terms.

Caracterización y evaluación de siete clones de pitahaya (Hylocereus spp.) en el Centro Experimental Campos Azules (CECA), Masatepe (Masaya)

La intensificación del cultivo de la Pitahaya (Hylocereus spp.) y la creciente demanda de frutos por el mercado nacional así como la exportación, ha requerido de la búsqueda de nuevas variedades que respondan adecuadamente a las exigencias de los mercados en sabor, color, dulzor, apariencia, y a las expectativas de los productores como la resistencia a plagas y enfermedades. La presente investigación se realizó entre los meses de Junio y Noviembre del año 2003 en el Centro Experimental Campos Azules (CECA); ubicado en el municipio de Masatepe, departamento de Masaya. El objetivo del trabajo es principalmente validar una Guía de Descriptores de Pitahaya. Además de caracterizar morfológicamente cada uno de los siete clones de Pitahaya (Amarilla, Cebra, Lisa, Orejona, Rosa, Sin Espina y San Ignacio) encontradas en el banco de germoplasma del CECA. Se tomaron las muestras bajo un Diseño Completo al Azar (D.C.A.) ya que la plantación estaba establecida y se realizó la caracterización de flores y frutos en el laboratorio del REGEN, exceptuando el levantamiento de datos vegetativos (cladodios) el que se realizó en el campo. Los datos se procesaron mediante análisis de varianzas y desviación estándar (entre y dentro de los clones con la tabla Tukey al 5%) en cuanto a clones, fechas de muestreo e interacción clon * fecha, Análisis de Agrupamiento (AA), Análisis de Componentes Principales (ACP), análisis de correlación y un análisis del empleo de la Guía de Descriptores. El clon Amarilla presentó los más altos valores en el ANDEVA respecto a los cladodios para las variables ancho de cladodio, longitud de espinas y número de espinas, respecto al resto de los clones (7.34 cm, 12.74cm y 6.75 espinas respectivamente), el clon Sin Espina obtuvo el más alto valor en la variable distancia de areola (3.71 cm). El Análisis de Varianza en cuanto a frutos mostró diferencias estadísticas en las variables tamaño de semilla y número de brácteas, correspondientes a los clones Sin Espina y Cebra (49.11s/g y 32.88 brácteas respectivamente); El ANDEVA en cuanto a fechas mostró que en la fecha 4 (Noviembre) se obtuvieron los mejores resultados; reflejando significancia estadística en todas las variables. El análisis de Componentes Principales determinó que el 40.17% de la variación total del germoplasma la aportan los dos primeros CP y las variables que la integran son volumen de fruto, peso de fruto, peso de pulpa, diámetro de fruto, volumen de pulpa, número de pétalos, número de brácteas, color de estigma, ancho y forma de pétalos. El Análisis de Agrupamiento mostró que existen cuatro (4) grupos formados por los clones Amarilla, I; Cebra, Sin Espina y San Ignacio, II; Lisa y Orejona, III y Rosa, IV. El análisis de correlación demuestra que las variables de fruto están íntimamente asociadas.

Parameter Estimation for Hidden Markov Models with Intractable Likelihoods

Approximate Bayesian computation (ABC) is a popular technique for analysing data for complex models where the likelihood function is intractable. It involves using simulation from the model to approximate the likelihood, with this approximate likelihood then being used to construct an approximate posterior. In this paper, we consider methods that estimate the parameters by maximizing the approximate likelihood used in ABC. We give a theoretical analysis of the asymptotic properties of the resulting estimator. In particular, we derive results analogous to those of consistency and asymptotic normality for standard maximum likelihood estimation. We also discuss how sequential Monte Carlo methods provide a natural method for implementing our likelihood-based ABC procedures.

Bayesian learning of noisy Markov decision processes

This work addresses the problem of estimating the optimal value function in a Markov Decision Process from observed state-action pairs. We adopt a Bayesian approach to inference, which allows both the model to be estimated and predictions about actions to be made in a unified framework, providing a principled approach to mimicry of a controller on the basis of observed data. A new Markov chain Monte Carlo (MCMC) sampler is devised for simulation from theposterior distribution over the optimal value function. This step includes a parameter expansion step, which is shown to be essential for good convergence properties of the MCMC sampler. As an illustration, the method is applied to learning a human controller.