Non-Stationary Semi-Markov Decision Processes on a Finite Horizon

We introduce and study a class of non-stationary semi-Markov decision processes on a finite horizon. By constructing an equivalent Markov decision process, we establish the existence of a piecewise open loop relaxed control which is optimal for the finite horizon problem.

A novel Q-learning algorithm with function approximation for constrained Markov decision processes

We present a novel multi-timescale Q-learning algorithm for average cost control in a Markov decision process subject to multiple inequality constraints. We formulate a relaxed version of this problem through the Lagrange multiplier method. Our algorithm is different from Q-learning in that it updates two parameters - a Q-value parameter and a policy parameter. The Q-value parameter is updated on a slower time scale as compared to the policy parameter. Whereas Q-learning with function approximation can diverge in some cases, our algorithm is seen to be convergent as a result of the aforementioned timescale separation. We show the results of experiments on a problem of constrained routing in a multistage queueing network. Our algorithm is seen to exhibit good performance and the various inequality constraints are seen to be satisfied upon convergence of the algorithm.

A Markov Decision Theoretic Approach to Pilot Allocation and Receive Antenna Selection

This paper considers antenna selection (AS) at a receiver equipped with multiple antenna elements but only a single radio frequency chain for packet reception. As information about the channel state is acquired using training symbols (pilots), the receiver makes its AS decisions based on noisy channel estimates. Additional information that can be exploited for AS includes the time-correlation of the wireless channel and the results of the link-layer error checks upon receiving the data packets. In this scenario, the task of the receiver is to sequentially select (a) the pilot symbol allocation, i.e., how to distribute the available pilot symbols among the antenna elements, for channel estimation on each of the receive antennas; and (b) the antenna to be used for data packet reception. The goal is to maximize the expected throughput, based on the past history of allocation and selection decisions, and the corresponding noisy channel estimates and error check results. Since the channel state is only partially observed through the noisy pilots and the error checks, the joint problem of pilot allocation and AS is modeled as a partially observed Markov decision process (POMDP). The solution to the POMDP yields the policy that maximizes the long-term expected throughput. Using the Finite State Markov Chain (FSMC) model for the wireless channel, the performance of the POMDP solution is compared with that of other existing schemes, and it is illustrated through numerical evaluation that the POMDP solution significantly outperforms them.

Risk-sensitive control of continuous time Markov chains

We study risk-sensitive control of continuous time Markov chains taking values in discrete state space. We study both finite and infinite horizon problems. In the finite horizon problem we characterize the value function via Hamilton Jacobi Bellman equation and obtain an optimal Markov control. We do the same for infinite horizon discounted cost case. In the infinite horizon average cost case we establish the existence of an optimal stationary control under certain Lyapunov condition. We also develop a policy iteration algorithm for finding an optimal control.

LANDSCAPE DYNAMICS MODELING THROUGH INTEGRATED MARKOV, FUZZY-AHP AND CELLULAR AUTOMATA

Multi temporal land use information were derived using two decades remote sensing data and simulated for 2012 and 2020 with Cellular Automata (CA) considering scenarios, change probabilities (through Markov chain) and Multi Criteria Evaluation (MCE). Agents and constraints were considered for modeling the urbanization process. Agents were nornmlized through fiizzyfication and priority weights were assigned through Analytical Hierarchical Process (AHP) pairwise comparison for each factor (in MCE) to derive behavior-oriented rules of transition for each land use class. Simulation shows a good agreement with the classified data. Fuzzy and AHP helped in analyzing the effects of agents of growth clearly and CA-Markov proved as a powerful tool in modelling and helped in capturing and visualizing the spatiotemporal patterns of urbanization. This provided rapid land evaluation framework with the essential insights of the urban trajectory for effective sustainable city planning.

Markov chains, R-trivial monoids and representation theory

We develop a general theory of Markov chains realizable as random walks on R-trivial monoids. It provides explicit and simple formulas for the eigenvalues of the transition matrix, for multiplicities of the eigenvalues via Mobius inversion along a lattice, a condition for diagonalizability of the transition matrix and some techniques for bounding the mixing time. In addition, we discuss several examples, such as Toom-Tsetlin models, an exchange walk for finite Coxeter groups, as well as examples previously studied by the authors, such as nonabelian sandpile models and the promotion Markov chain on posets. Many of these examples can be viewed as random walks on quotients of free tree monoids, a new class of monoids whose combinatorics we develop.

Markov chain splitting methods in structural reliability integral estimation

Monte Carlo simulation methods involving splitting of Markov chains have been used in evaluation of multi-fold integrals in different application areas. We examine in this paper the performance of these methods in the context of evaluation of reliability integrals from the point of view of characterizing the sampling fluctuations. The methods discussed include the Au-Beck subset simulation, Holmes-Diaconis-Ross method, and generalized splitting algorithm. A few improvisations based on first order reliability method are suggested to select algorithmic parameters of the latter two methods. The bias and sampling variance of the alternative estimators are discussed. Also, an approximation to the sampling distribution of some of these estimators is obtained. Illustrative examples involving component and series system reliability analyses are presented with a view to bring out the relative merits of alternative methods. (C) 2015 Elsevier Ltd. All rights reserved.

Risk-Sensitive Ergodic Control of Continuous Time Markov Processes With Denumerable State Space

In this article, we study risk-sensitive control problem with controlled continuous time Markov chain state dynamics. Using multiplicative dynamic programming principle along with the atomic structure of the state dynamics, we prove the existence and a characterization of optimal risk-sensitive control under geometric ergodicity of the state dynamics along with a smallness condition on the running cost.

Inclusión de harina de hoja de Marango (Moringa oleifera) en la alimentación de conejos de engorde y su efecto en el comportamiento productivo

El objetivo del presente trabajo fue evaluar el efecto de la inclusión de harina de Marango (Moringa oleifera) en dietas para conejos de engorde y su efecto en el comportamiento productivo. Se utilizaron 36 conejos de camadas homogéneas con 37 d de edad, de las razas Neozelandés (18) y California (18) con pesos iniciales promedio de 554 g (133.8), los cuales fueron distribuidos en igual proporción de razas y sexos, en un Diseño Completamente al Azar con tres tratamientos: Tl: Concentrado Comercial, T2: Concentrado Isométrico, T3: Concentrado lsoprotéico. Las variables estudiadas fueron: consumo de alimento diario (CAD), ganancia media diaria (GMD), índice de conversión alimenticia (!CA), digestibilidad aparente de la materia seca (DMS), peso final (PF). Se realizó análisis de varianza y comparaciones de medias con la Prueba de Tukey utilizando el paquete estadístico MINITAB~:Ver. 12.0. Los resultados indican que para el CAD no se encontró diferencia significativa (P>0.05) siendo los valores promedio de 11 1.73g d"1 respectivamente. Sin embargo, para la GMD y la DMS se encontró diferencias (Pde !8.63g d" 1 para la GMD y para DMS la dieta isoprotéica obtuvo el mayor valor (70.79%), así mismo para ICA y PF se observó diferencia (P<0.05) 5.541 para la T3 vs 5.964 y 7.336 para TI y T2. El peso final fue en promedio de 2247g. No se encontraron diferencias (P>0.05) para el efecto de raza y sexo. El análisis financiero favorece a la dieta isoprotéica, presentando un valor de USA$ 0.98 por encima de la dieta comercial. En conclusión la harina de Marango puede ser utilizada como sustituto parcial o total de la proteína en dietas para conejos de engorde sin causar efectos negativos en los parámetros productivos.

Bayesian model selection for time series using Markov chain Monte Carlo

We present a stochastic simulation technique for subset selection in time series models, based on the use of indicator variables with the Gibbs sampler within a hierarchical Bayesian framework. As an example, the method is applied to the selection of subset linear AR models, in which only significant lags are included. Joint sampling of the indicators and parameters is found to speed convergence. We discuss the possibility of model mixing where the model is not well determined by the data, and the extension of the approach to include non-linear model terms.

Diagnóstico agro-socioeconómico del sistema agrario de la zona Posoltega - Quezalguaque en la Cordillera de los Maribios

El presente trabajo consistió en un diagnóstico agro socioeconómico del sistema agrario de una zona entre los municipios de Posoltega y Quezalguaque de los departamentos de Chinandega y León que abarca un área de 23 km2 aproximadamente. El estudio se desarrolla bajo la metodología del análisis de sistemas, abarcando los sistemas de cultivo, pecuario y forestales, condiciones económicas y limitaciones de los productores de la zona. La metodología del estudio de zona comprende; la selección de la zona, fuentes secundarias (material bibliográfico, cartográfico de la zona ), un sondeo de zona, división de la zona en sub-zonas homogéneas, historia agraria, realización de encuestas base con la finalidad de elaborar las pre-tipologías ( 45 entrevistas a productores que representan el cien por ciento de productores de la zona). Dando como resultado de las encuestas base tres tipos de pre-tipologías que son; Pequeños productores individuales, Sector cooperativo y Medianos productores individuales, en base a la clasificación que realizó Nitlapan ( 1993 ), de estos se seleccionaron siete casos para un estudio profundo de los sistemas de finca, quedando tres casos para Pequeños productores individuales, tres casos para el Sector cooperativo y un caso para Medianos productores, en base a su representatividad dentro de la zona de estudio. Se dio un seguimiento semanal durante todo el ciclo de primera y postrera del año 1994. dando como resultado al finalizar el ciclo de postrera la realización de los modelos, conocidos como tipologías definidas de sistemas de producción, quedando tres tipos fundamentales de productores que son: Productores de granos básicos que se caracterizan por ser beneficiados de la reforma agraria de la década de los ochenta, Con áreas reducidas para el desarrollo de las distintas actividades y sin posibilidad de aumento, no poseen capital circulante, las actividades no agropecuarias representan mucha importancia, presentan un sistema de cultivo maíz ( Zea mays L ) y frijol común ( Phaseolus vulgaris L.), utiliza sólo mano de obra familiar poco uso de productos químicos por falta de dinero, un sistema forestal bien desarrollado y sus principales limitantes son; la obtención de agua y mala comercialización de sus productos. Los productores con diversificación agropecuaria son individuales privados que se ubican cerca del Umbral de Reproducción Simple ( URS ), con una tendencia hacia el aumento de sus áreas, producto de una gran diversidad de cultivos como; maiz ( Zea mays L), frijol común (Phaseolus vulgaris L ), ajonjoli ( Sesamum indica L ), sorgo ( Sorghum bicolor L ), sorgo escobero ( Sorghum vulgare Pers ), yuca ( Manihot esculenta Cranz), utilizando en cierto grado tracción mécanica y productos químicos, un sistema ganadero reducido y en vías de aumento y con especies menores de mucha importancia para el mantenimiento de la finca, en donde la mayor mano de obra para las labores del campo es familiar y en menor grado mano de obra contratada. Y por último los Medianos productores ganaderos-soyeros que poseen extensas áreas para el desarrollo agrícola y pecuario, son productores individuales privados que sobrepasan en más del cincuenta por ciento, el Umbral de Reproducción Simple, poseen capital circulante para realizar todas sus actividades propias del campo, obtienen préstamos de las entidades bancarias para el desarrollo agrícola en cultivos como soya ( Glycime max (L.) Merrill ) y ajonjolí ( Sesamum indica L ), presentan un desarrollo ganadero de mucha importancia que les proporciona buenos ingresos económicos a la finca y en donde la mayoría de la mano de obra es contratada.

Parameter Estimation for Hidden Markov Models with Intractable Likelihoods

Approximate Bayesian computation (ABC) is a popular technique for analysing data for complex models where the likelihood function is intractable. It involves using simulation from the model to approximate the likelihood, with this approximate likelihood then being used to construct an approximate posterior. In this paper, we consider methods that estimate the parameters by maximizing the approximate likelihood used in ABC. We give a theoretical analysis of the asymptotic properties of the resulting estimator. In particular, we derive results analogous to those of consistency and asymptotic normality for standard maximum likelihood estimation. We also discuss how sequential Monte Carlo methods provide a natural method for implementing our likelihood-based ABC procedures.

Bayesian learning of noisy Markov decision processes

This work addresses the problem of estimating the optimal value function in a Markov Decision Process from observed state-action pairs. We adopt a Bayesian approach to inference, which allows both the model to be estimated and predictions about actions to be made in a unified framework, providing a principled approach to mimicry of a controller on the basis of observed data. A new Markov chain Monte Carlo (MCMC) sampler is devised for simulation from theposterior distribution over the optimal value function. This step includes a parameter expansion step, which is shown to be essential for good convergence properties of the MCMC sampler. As an illustration, the method is applied to learning a human controller.