Stability and ergodicity of piecewise deterministic Markov processes

The main goal of this paper is to establish some equivalence results on stability, recurrence, and ergodicity between a piecewise deterministic Markov process ( PDMP) {X( t)} and an embedded discrete-time Markov chain {Theta(n)} generated by a Markov kernel G that can be explicitly characterized in terms of the three local characteristics of the PDMP, leading to tractable criterion results. First we establish some important results characterizing {Theta(n)} as a sampling of the PDMP {X( t)} and deriving a connection between the probability of the first return time to a set for the discrete-time Markov chains generated by G and the resolvent kernel R of the PDMP. From these results we obtain equivalence results regarding irreducibility, existence of sigma-finite invariant measures, and ( positive) recurrence and ( positive) Harris recurrence between {X( t)} and {Theta(n)}, generalizing the results of [ F. Dufour and O. L. V. Costa, SIAM J. Control Optim., 37 ( 1999), pp. 1483-1502] in several directions. Sufficient conditions in terms of a modified Foster-Lyapunov criterion are also presented to ensure positive Harris recurrence and ergodicity of the PDMP. We illustrate the use of these conditions by showing the ergodicity of a capacity expansion model.

AVERAGE CONTINUOUS CONTROL OF PIECEWISE DETERMINISTIC MARKOV PROCESSES

This paper deals with the long run average continuous control problem of piecewise deterministic Markov processes (PDMPs) taking values in a general Borel space and with compact action space depending on the state variable. The control variable acts on the jump rate and transition measure of the PDMP, and the running and boundary costs are assumed to be positive but not necessarily bounded. Our first main result is to obtain an optimality equation for the long run average cost in terms of a discrete-time optimality equation related to the embedded Markov chain given by the postjump location of the PDMP. Our second main result guarantees the existence of a feedback measurable selector for the discrete-time optimality equation by establishing a connection between this equation and an integro-differential equation. Our final main result is to obtain some sufficient conditions for the existence of a solution for a discrete-time optimality inequality and an ordinary optimal feedback control for the long run average cost using the so-called vanishing discount approach. Two examples are presented illustrating the possible applications of the results developed in the paper.

Bayesian online algorithms for learning in discrete Hidden Markov Models

We propose and analyze two different Bayesian online algorithms for learning in discrete Hidden Markov Models and compare their performance with the already known Baldi-Chauvin Algorithm. Using the Kullback-Leibler divergence as a measure of generalization we draw learning curves in simplified situations for these algorithms and compare their performances.

Experimental studies of di-jet survival and surface emission bias in Au plus Au collisions via angular correlations with respect to back-to-back leading hadrons

We report first results from an analysis based on a new multi-hadron correlation technique, exploring jet-medium interactions and di-jet surface emission bias at the BNL Relativistic Heavy Ion Collider (RHIC). Pairs of back-to-back high-transverse-momentum hadrons are used for triggers to study associated hadron distributions. In contrast with two-and three-particle correlations with a single trigger with similar kinematic selections, the associated hadron distribution of both trigger sides reveals no modification in either relative pseudorapidity Delta eta or relative azimuthal angle Delta phi from d + Au to central Au + Au collisions. We determine associated hadron yields and spectra as well as production rates for such correlated back-to-back triggers to gain additional insights on medium properties.

Azimuthal di-hadron correlations in d plus Au and Au plus Au collisions at root s(NN)=200 GeV measured at the STAR detector

Yields, correlation shapes, and mean transverse momenta p(T) of charged particles associated with intermediate-to high-p(T) trigger particles (2.5 < p(T) < 10 GeV/c) in d + Au and Au + Au collisions at root s(NN) = 200 GeV are presented. For associated particles at higher p(T) greater than or similar to 2.5 GeV/c, narrow correlation peaks are seen in d + Au and Au + Au, indicating that the main production mechanism is jet fragmentation. At lower associated particle pT < 2 GeV/c, a large enhancement of the near- (Delta phi similar to 0) and away-side (Delta phi similar to pi) associated yields is found, together with a strong broadening of the away-side azimuthal distributions in Au + Au collisions compared to d + Au measurements, suggesting that other particle production mechanisms play a role. This is further supported by the observed significant softening of the away-side associated particle yield distribution at Delta phi similar to pi in central Au + Au collisions.

The Policy Iteration Algorithm for Average Continuous Control of Piecewise Deterministic Markov Processes

The main goal of this paper is to apply the so-called policy iteration algorithm (PIA) for the long run average continuous control problem of piecewise deterministic Markov processes (PDMP`s) taking values in a general Borel space and with compact action space depending on the state variable. In order to do that we first derive some important properties for a pseudo-Poisson equation associated to the problem. In the sequence it is shown that the convergence of the PIA to a solution satisfying the optimality equation holds under some classical hypotheses and that this optimal solution yields to an optimal control strategy for the average control problem for the continuous-time PDMP in a feedback form.

THE VANISHING DISCOUNT APPROACH FOR THE AVERAGE CONTINUOUS CONTROL OF PIECEWISE DETERMINISTIC MARKOV PROCESSES

This work is concerned with the existence of an optimal control strategy for the long-run average continuous control problem of piecewise-deterministic Markov processes (PDMPs). In Costa and Dufour (2008), sufficient conditions were derived to ensure the existence of an optimal control by using the vanishing discount approach. These conditions were mainly expressed in terms of the relative difference of the alpha-discount value functions. The main goal of this paper is to derive tractable conditions directly related to the primitive data of the PDMP to ensure the existence of an optimal control. The present work can be seen as a continuation of the results derived in Costa and Dufour (2008). Our main assumptions are written in terms of some integro-differential inequalities related to the so-called expected growth condition, and geometric convergence of the post-jump location kernel associated to the PDMP. An example based on the capacity expansion problem is presented, illustrating the possible applications of the results developed in the paper.

On the Filtering Problem for Continuous-Time Markov Jump Linear Systems with no Observation of the Markov Chain

We consider in this paper the optimal stationary dynamic linear filtering problem for continuous-time linear systems subject to Markovian jumps in the parameters (LSMJP) and additive noise (Wiener process). It is assumed that only an output of the system is available and therefore the values of the jump parameter are not accessible. It is a well known fact that in this setting the optimal nonlinear filter is infinite dimensional, which makes the linear filtering a natural numerically, treatable choice. The goal is to design a dynamic linear filter such that the closed loop system is mean square stable and minimizes the stationary expected value of the mean square estimation error. It is shown that an explicit analytical solution to this optimal filtering problem is obtained from the stationary solution associated to a certain Riccati equation. It is also shown that the problem can be formulated using a linear matrix inequalities (LMI) approach, which can be extended to consider convex polytopic uncertainties on the parameters of the possible modes of operation of the system and on the transition rate matrix of the Markov process. As far as the authors are aware of this is the first time that this stationary filtering problem (exact and robust versions) for LSMJP with no knowledge of the Markov jump parameters is considered in the literature. Finally, we illustrate the results with an example.

Singular Perturbation for the Discounted Continuous Control of Piecewise Deterministic Markov Processes

This paper deals with the expected discounted continuous control of piecewise deterministic Markov processes (PDMP`s) using a singular perturbation approach for dealing with rapidly oscillating parameters. The state space of the PDMP is written as the product of a finite set and a subset of the Euclidean space a""e (n) . The discrete part of the state, called the regime, characterizes the mode of operation of the physical system under consideration, and is supposed to have a fast (associated to a small parameter epsilon > 0) and a slow behavior. By using a similar approach as developed in Yin and Zhang (Continuous-Time Markov Chains and Applications: A Singular Perturbation Approach, Applications of Mathematics, vol. 37, Springer, New York, 1998, Chaps. 1 and 3) the idea in this paper is to reduce the number of regimes by considering an averaged model in which the regimes within the same class are aggregated through the quasi-stationary distribution so that the different states in this class are replaced by a single one. The main goal is to show that the value function of the control problem for the system driven by the perturbed Markov chain converges to the value function of this limit control problem as epsilon goes to zero. This convergence is obtained by, roughly speaking, showing that the infimum and supremum limits of the value functions satisfy two optimality inequalities as epsilon goes to zero. This enables us to show the result by invoking a uniqueness argument, without needing any kind of Lipschitz continuity condition.

Generalized Coupled Algebraic Riccati Equations for Discrete-time Markov Jump with Multiplicative Noise Systems

In this paper we consider the existence of the maximal and mean square stabilizing solutions for a set of generalized coupled algebraic Riccati equations (GCARE for short) associated to the infinite-horizon stochastic optimal control problem of discrete-time Markov jump with multiplicative noise linear systems. The weighting matrices of the state and control for the quadratic part are allowed to be indefinite. We present a sufficient condition, based only on some positive semi-definite and kernel restrictions on some matrices, under which there exists the maximal solution and a necessary and sufficient condition under which there exists the mean square stabilizing solution fir the GCARE. We also present a solution for the discounted and long run average cost problems when the performance criterion is assumed be composed by a linear combination of an indefinite quadratic part and a linear part in the state and control variables. The paper is concluded with a numerical example for pension fund with regime switching.

Linear minimum mean square filter for discrete-time linear systems with Markov jumps and multiplicative noises

In this paper we obtain the linear minimum mean square estimator (LMMSE) for discrete-time linear systems subject to state and measurement multiplicative noises and Markov jumps on the parameters. It is assumed that the Markov chain is not available. By using geometric arguments we obtain a Kalman type filter conveniently implementable in a recurrence form. The stationary case is also studied and a proof for the convergence of the error covariance matrix of the LMMSE to a stationary value under the assumption of mean square stability of the system and ergodicity of the associated Markov chain is obtained. It is shown that there exists a unique positive semi-definite solution for the stationary Riccati-like filter equation and, moreover, this solution is the limit of the error covariance matrix of the LMMSE. The advantage of this scheme is that it is very easy to implement and all calculations can be performed offline. (c) 2011 Elsevier Ltd. All rights reserved.

A generalized multi-period mean-variance portfolio optimization with Markov switching parameters

In this paper, we deal with a generalized multi-period mean-variance portfolio selection problem with market parameters Subject to Markov random regime switchings. Problems of this kind have been recently considered in the literature for control over bankruptcy, for cases in which there are no jumps in market parameters (see [Zhu, S. S., Li, D., & Wang, S. Y. (2004). Risk control over bankruptcy in dynamic portfolio selection: A generalized mean variance formulation. IEEE Transactions on Automatic Control, 49, 447-457]). We present necessary and Sufficient conditions for obtaining an optimal control policy for this Markovian generalized multi-period meal-variance problem, based on a set of interconnected Riccati difference equations, and oil a set of other recursive equations. Some closed formulas are also derived for two special cases, extending some previous results in the literature. We apply the results to a numerical example with real data for Fisk control over bankruptcy Ill a dynamic portfolio selection problem with Markov jumps selection problem. (C) 2008 Elsevier Ltd. All rights reserved.

Neuropharmacological effects in mice of Lychnophora species (Vernonieae, Asteraceae) and anticonvulsant activity of 4,5-di-O-[E]-caffeoylquinic acid isolated from the stem of L-rupestris and L-staavioides

Aiming at contributing with the search for neuroactive substances from natural sources, we report for the first time antinociceptive and anticonvulsant effects of some Lychnophora species. We verify the protective effects of polar extracts (600 mg/kg, intraperitoneally), and methanolic fractions of L. staavioides and L. rupestris (100 mg/kg, intraperitoneally) in pentylenetetrazole-induced seizures on mice. Previously, a screening was accomplished, evaluating the antinociceptive central activity (hot plate test), with different extracts of L. rupestris, L. staavioides and L. diamantinana. It was possible to select the possible extracts of Lychnophora with central nervous system activity. Some of the active extracts were submitted to fractionation and purification process and the methanolic fractions of L. rupestris (stem) and L. staavioides (stem), with anticonvulsant properties (100 mg/kg, intraperitoneally), yielded 4,5-di-O-[E]-caffeoylquinic acid. This substance was injected intraperitoneally in mice and showed anticonvulsant effect against pentylenetetrazole-induced seizures at doses of 25 and 50 mg/kg. It has often been shown that seizures induced by pentylenetetrazole are involved in inhibition and/or attenuation of GABAergic neurotransmission. However, other systems of the central nervous system such as adenosinergic and glutamatergic could be involved in the caffeoylquinic acid effects. Further studies should be conducted to verify that the target receptor could be participating in this anticonvulsant property. Although other investigations have reported a series of biological activities from Lychnophora species, this is the first report of central analgesic and anticonvulsant activity in species of this genus.

Neighborhood and Efficiency in Manufacturing in Brazilian Regions: A Spatial Markov Chain Analysis

This paper analyzes the geography of regional competitiveness in manufacturing in Brazil. The authors estimate stochastic frontiers to calculate regional efficiency of representative firms in 137 regions in the period 2000-2006, in four sectors defined by technological intensity. The efficiency results are analyzed using Markov Spatial Transition Matrices to provide insights into the transition of regions between efficiency levels, considering their local spatial context. The results indicate that geography plays an important role in manufacturing competitiveness. In particular, regions with more competitive neighbors are more likely to improve their relative efficiency (pull effect) over time, and regions with less competitive neighbors are more likely to lose relative efficiency (drag effect). The authors find that the pull effect is stronger than the drag effect.

Synthesis and electrochemical properties of vanadyl phosphate di-hydrate/polyaniline derivatives hybrid films

Vanadyl phosphate and its hybrid compounds have proven to undergo electrochemical intercalation and de-intercalation of lithium ions, which enables its use as cathode material for Li ion rechargeable batteries. In this context, vanadyl phosphate di-hydrate/polyaniline derivatives hybrid films were synthesized via the exfoliation and reconstruction approach in order to evaluate their potential use as cathode in ion lithium batteries. X-ray diffraction patterns indicate that the lamellar structure of the inorganic matrix is maintained, consistent with the topotactic process. In the scanning electron micrographs, hybrid films exhibit rough surface consisting of warped and cracked crystallites, quite different from vanadyl phosphate di-hydrate square platelets crystallites. Electrochemical evaluation using cyclic voltammetry and charge-discharge galvanostatic techniques shows small differences between the charge and the discharge curves, indicating an irreversibility of the hybrid systems. (C) 2009 Elsevier B.V. All rights reserved.