Bayesian learning of noisy Markov decision processes

This work addresses the problem of estimating the optimal value function in a Markov Decision Process from observed state-action pairs. We adopt a Bayesian approach to inference, which allows both the model to be estimated and predictions about actions to be made in a unified framework, providing a principled approach to mimicry of a controller on the basis of observed data. A new Markov chain Monte Carlo (MCMC) sampler is devised for simulation from theposterior distribution over the optimal value function. This step includes a parameter expansion step, which is shown to be essential for good convergence properties of the MCMC sampler. As an illustration, the method is applied to learning a human controller.

Filtering via approximate Bayesian computation

Approximate Bayesian computation (ABC) has become a popular technique to facilitate Bayesian inference from complex models. In this article we present an ABC approximation designed to perform biased filtering for a Hidden Markov Model when the likelihood function is intractable. We use a sequential Monte Carlo (SMC) algorithm to both fit and sample from our ABC approximation of the target probability density. This approach is shown to, empirically, be more accurate w.r.t.~the original filter than competing methods. The theoretical bias of our method is investigated; it is shown that the bias goes to zero at the expense of increased computational effort. Our approach is illustrated on a constrained sequential lasso for portfolio allocation to 15 constituents of the FTSE 100 share index.

La Teoría del Diseño Inteligente : un análisis desde el tomismo

Resumen: Michael Behe y William Dembski son dos de los líderes de la Teoría del Diseño Inteligente, una propuesta surgida como respuesta a los modelos evolucionistas y anti-finalistas prevalentes en ciertos ambientes académicos e intelectuales, especialmente del mundo anglosajón. Las especulaciones de Behe descansan en el concepto de “sistema de complejidad irreductible”, entendido como un conjunto ordenado de partes cuya funcionalidad depende estrictamente de su indemnidad estructural, y que su origen resulta, por tanto, refractario a explicaciones gradualistas. Estos sistemas, según Behe, están presentes en los vivientes, lo que permitiría inferir que ellos no son el producto de mecanismos ciegos y azarosos, sino el resultado de un diseño. Dembski, por su parte, ha abordado el problema desde una perspectiva más cuantitativa, desarrollando un algoritmo probabilístico conocido como “filtro explicatorio”, que permitiría, según el autor, inferir científicamente la presencia de un diseño, tanto en entidades artificiales como naturales. Trascendiendo las descalificaciones del neodarwinismo, examinamos la propuesta de estos autores desde los fundamentos filosóficos de la escuela tomista. A nuestro parecer, hay en el trabajo de estos autores algunas intuiciones valiosas, las que sin embargo suelen pasar desapercibidas por la escasa formalidad en que vienen presentadas, y por la aproximación eminentemente mecanicista y artefactual con que ambos enfrentan la cuestión. Es precisamente a la explicitación de tales intuiciones a las que se dirige el artículo.

Forward Smoothing Using Sequential Monte Carlo

Sequential Monte Carlo (SMC) methods are a widely used set of computational tools for inference in non-linear non-Gaussian state-space models. We propose a new SMC algorithm to compute the expectation of additive functionals recursively. Essentially, it is an on-line or "forward only" implementation of a forward filtering backward smoothing SMC algorithm proposed by Doucet, Godsill and Andrieu (2000). Compared to the standard \emph{path space} SMC estimator whose asymptotic variance increases quadratically with time even under favorable mixing assumptions, the non asymptotic variance of the proposed SMC estimator only increases linearly with time. We show how this allows us to perform recursive parameter estimation using an SMC implementation of an on-line version of the Expectation-Maximization algorithm which does not suffer from the particle path degeneracy problem.

La abstracción formal y la validación del razonamiento inductivo

Resumen: Suele admitirse que uno de los problemas pendientes, al menos desde Hume, en la teoría de la ciencia es la justificación crítica de los procesos inductivos, que son los que siguen las ciencias experimentales, como son las ciencias naturales. Frente a las ineficaces explicaciones aportadas por el empirismo o neopositivismo, así como por los racionalismos o idealismos, que son radicalmente incapaces para resolver el problema, nosotros presentamos la respuesta del realismo aristotélico, basada en el correcto concepto de abstracción formal, ignorado o malentendido incluso por muchos aristotélicos tanto antiguos como modernos. La respuesta consiste, en suma, en ver que el término del proceso inductivo, en cuanto llega a conclusiones universales a partir de lo particular, debe estar mediado por un proceso previo de abstracción de la forma, bien entendida, en lo mismo concreto y particular. Ello permite ver que la conclusión universal no desborda las premisas, que es el problema clásico de la epagogé aristotélica.

Sequential decision making in general state space models

Many problems in control and signal processing can be formulated as sequential decision problems for general state space models. However, except for some simple models one cannot obtain analytical solutions and has to resort to approximation. In this thesis, we have investigated problems where Sequential Monte Carlo (SMC) methods can be combined with a gradient based search to provide solutions to online optimisation problems. We summarise the main contributions of the thesis as follows. Chapter 4 focuses on solving the sensor scheduling problem when cast as a controlled Hidden Markov Model. We consider the case in which the state, observation and action spaces are continuous. This general case is important as it is the natural framework for many applications. In sensor scheduling, our aim is to minimise the variance of the estimation error of the hidden state with respect to the action sequence. We present a novel SMC method that uses a stochastic gradient algorithm to find optimal actions. This is in contrast to existing works in the literature that only solve approximations to the original problem. In Chapter 5 we presented how an SMC can be used to solve a risk sensitive control problem. We adopt the use of the Feynman-Kac representation of a controlled Markov chain flow and exploit the properties of the logarithmic Lyapunov exponent, which lead to a policy gradient solution for the parameterised problem. The resulting SMC algorithm follows a similar structure with the Recursive Maximum Likelihood(RML) algorithm for online parameter estimation. In Chapters 6, 7 and 8, dynamic Graphical models were combined with with state space models for the purpose of online decentralised inference. We have concentrated more on the distributed parameter estimation problem using two Maximum Likelihood techniques, namely Recursive Maximum Likelihood (RML) and Expectation Maximization (EM). The resulting algorithms can be interpreted as an extension of the Belief Propagation (BP) algorithm to compute likelihood gradients. In order to design an SMC algorithm, in Chapter 8 uses a nonparametric approximations for Belief Propagation. The algorithms were successfully applied to solve the sensor localisation problem for sensor networks of small and medium size.

Sequential Monte Carlo computation of the score and observed information matrix in state-space models with application to parameter estimation

Sequential Monte Carlo (SMC) methods are popular computational tools for Bayesian inference in non-linear non-Gaussian state-space models. For this class of models, we propose SMC algorithms to compute the score vector and observed information matrix recursively in time. We propose two different SMC implementations, one with computational complexity $\mathcal{O}(N)$ and the other with complexity $\mathcal{O}(N^{2})$ where $N$ is the number of importance sampling draws. Although cheaper, the performance of the $\mathcal{O}(N)$ method degrades quickly in time as it inherently relies on the SMC approximation of a sequence of probability distributions whose dimension is increasing linearly with time. In particular, even under strong \textit{mixing} assumptions, the variance of the estimates computed with the $\mathcal{O}(N)$ method increases at least quadratically in time. The $\mathcal{O}(N^{2})$ is a non-standard SMC implementation that does not suffer from this rapid degrade. We then show how both methods can be used to perform batch and recursive parameter estimation.

Self-consistent kinetic modeling of low-pressure inductively coupled radio-frequency discharges

An efficient method for solving the spatially inhomogeneous Boltzmann equation in a two-term approximation for low-pressure inductively coupled plasmas has been developed. The electron distribution function (EDF), a function of total electron energy and two spatial coordinates, is found self-consistently with the static space-charge potential which is computed from a 2D fluid model, and the rf electric field profile which is calculated from the Maxwell equations. The EDF and the spatial distributions of the electron density, potential, temperature, ionization rate, and the inductive electric field are calculated and discussed. (C) 1996 American Institute of Physics.

Electric characteristics of pseudo-spark during breakdown phase

An analysis of the time-dependent resistive voltage and power deposition during the breakdown phase of pseudo-spark is presented. The voltage and current were measured by specially designed low-inductance capacitive voltage divider and current measuring resistor. The measured waveforms of voltage and current are digitized and processed by a computer program to remove the inductive component, so as to obtain resistive voltage and power deposition. The influence of pressure, cathode geometry and charging voltage of storage capacitors on the electrical properties in the breakdown phase are investigated. The results suggest that the breakdown phase of pseudo-spark consists of three stages. The first stage is mainly hollow cathode discharge. In the second stage, field-enhanced thermionic emission takes place, resulting in a fast voltage drop and sharp rise of discharge current. The third stage of discharge depends simply on the parameters of the discharge circuit.

On the Informational Role of Term Structure in the U.S. Monetary Policy Rule

This paper uses a structural approach based on the indirect inference principle to estimate a standard version of the new Keynesian monetary (NKM) model augmented with term structure using both revised and real-time data. The estimation results show that the term spread and policy inertia are both important determinants of the U.S. estimated monetary policy rule whereas the persistence of shocks plays a small but significant role when revised and real-time data of output and inflation are both considered. More importantly, the relative importance of term spread and persistent shocks in the policy rule and the shock transmission mechanism drastically change when it is taken into account that real-time data are not well behaved.

Gestão do conhecimento estratégico : uma proposta de modelo integrado

Pesquisa focada na definição de um modelo teórico-sistêmico de Gestão do Conhecimento Estratégico (GCE), estando inserida nos estudos da Gestão do Conhecimento (GC) e da Gestão da Informação (GI), considerando conceitos relacionados ao conhecimento (tácito e explícito), a estratégias (perspectivas e abordagens) e aos agentes envolvidos (decisores e estrategistas; novatos e experientes). A construção do modelo se vale de visões da Ciência da Informação, da Administração e da Psicologia Cognitiva. A metodologia empregada utiliza o método abdutivo de pesquisa (uso concomitante dos métodos indutivo e dedutivo), valendo-se da análise bibliográfica (para sustentação teórica do modelo), do estudo comparado (para a avaliação de diferentes modelos de GC e de abordagens e perspectivas estratégicas) e da pesquisa descritiva ou de campo (para validação do modelo junto a profissionais da área em estudo). Os resultados indicam que é possível definir-se um modelo de Gestão do Conhecimento estratégico e que muitos trabalhos podem ser desenvolvidos, derivados da proposta apresentada nesta tese.

The Importance of Stock Market Returns in Estimated Monetary Policy Rules: a Structural Approach

This paper estimates a standard version of the New Keynesian Monetary (NKM) model augmented with financial variables in order to analyze the relative importance of stock market returns and term spread in the estimated U.S. monetary policy rule. The estimation procedure implemented is a classical structural method based on the indirect inference principle. The empirical results show that the Fed seems to respond to the macroeconomic outlook and to the stock market return but does not seem to respond to the term spread. Moreover, policy inertia and persistent policy shocks are also significant features of the estimated policy rule.

Term Structure and the Estimated Monetary Policy Rule in the Eurozone

Published as an article in: Spanish Economic Review, 2008, vol. 10, issue 4, pages 251-277.