Efficient solutions to factored MDPs with imprecise transition probabilities

When modeling real-world decision-theoretic planning problems in the Markov Decision Process (MDP) framework, it is often impossible to obtain a completely accurate estimate of transition probabilities. For example, natural uncertainty arises in the transition specification due to elicitation of MOP transition models from an expert or estimation from data, or non-stationary transition distributions arising from insufficient state knowledge. In the interest of obtaining the most robust policy under transition uncertainty, the Markov Decision Process with Imprecise Transition Probabilities (MDP-IPs) has been introduced to model such scenarios. Unfortunately, while various solution algorithms exist for MDP-IPs, they often require external calls to optimization routines and thus can be extremely time-consuming in practice. To address this deficiency, we introduce the factored MDP-IP and propose efficient dynamic programming methods to exploit its structure. Noting that the key computational bottleneck in the solution of factored MDP-IPs is the need to repeatedly solve nonlinear constrained optimization problems, we show how to target approximation techniques to drastically reduce the computational overhead of the nonlinear solver while producing bounded, approximately optimal solutions. Our results show up to two orders of magnitude speedup in comparison to traditional ""flat"" dynamic programming approaches and up to an order of magnitude speedup over the extension of factored MDP approximate value iteration techniques to MDP-IPs while producing the lowest error of any approximation algorithm evaluated. (C) 2011 Elsevier B.V. All rights reserved.

Stability and ergodicity of piecewise deterministic Markov processes

The main goal of this paper is to establish some equivalence results on stability, recurrence, and ergodicity between a piecewise deterministic Markov process ( PDMP) {X( t)} and an embedded discrete-time Markov chain {Theta(n)} generated by a Markov kernel G that can be explicitly characterized in terms of the three local characteristics of the PDMP, leading to tractable criterion results. First we establish some important results characterizing {Theta(n)} as a sampling of the PDMP {X( t)} and deriving a connection between the probability of the first return time to a set for the discrete-time Markov chains generated by G and the resolvent kernel R of the PDMP. From these results we obtain equivalence results regarding irreducibility, existence of sigma-finite invariant measures, and ( positive) recurrence and ( positive) Harris recurrence between {X( t)} and {Theta(n)}, generalizing the results of [ F. Dufour and O. L. V. Costa, SIAM J. Control Optim., 37 ( 1999), pp. 1483-1502] in several directions. Sufficient conditions in terms of a modified Foster-Lyapunov criterion are also presented to ensure positive Harris recurrence and ergodicity of the PDMP. We illustrate the use of these conditions by showing the ergodicity of a capacity expansion model.

AVERAGE CONTINUOUS CONTROL OF PIECEWISE DETERMINISTIC MARKOV PROCESSES

This paper deals with the long run average continuous control problem of piecewise deterministic Markov processes (PDMPs) taking values in a general Borel space and with compact action space depending on the state variable. The control variable acts on the jump rate and transition measure of the PDMP, and the running and boundary costs are assumed to be positive but not necessarily bounded. Our first main result is to obtain an optimality equation for the long run average cost in terms of a discrete-time optimality equation related to the embedded Markov chain given by the postjump location of the PDMP. Our second main result guarantees the existence of a feedback measurable selector for the discrete-time optimality equation by establishing a connection between this equation and an integro-differential equation. Our final main result is to obtain some sufficient conditions for the existence of a solution for a discrete-time optimality inequality and an ordinary optimal feedback control for the long run average cost using the so-called vanishing discount approach. Two examples are presented illustrating the possible applications of the results developed in the paper.

The Policy Iteration Algorithm for Average Continuous Control of Piecewise Deterministic Markov Processes

The main goal of this paper is to apply the so-called policy iteration algorithm (PIA) for the long run average continuous control problem of piecewise deterministic Markov processes (PDMP`s) taking values in a general Borel space and with compact action space depending on the state variable. In order to do that we first derive some important properties for a pseudo-Poisson equation associated to the problem. In the sequence it is shown that the convergence of the PIA to a solution satisfying the optimality equation holds under some classical hypotheses and that this optimal solution yields to an optimal control strategy for the average control problem for the continuous-time PDMP in a feedback form.

THE VANISHING DISCOUNT APPROACH FOR THE AVERAGE CONTINUOUS CONTROL OF PIECEWISE DETERMINISTIC MARKOV PROCESSES

This work is concerned with the existence of an optimal control strategy for the long-run average continuous control problem of piecewise-deterministic Markov processes (PDMPs). In Costa and Dufour (2008), sufficient conditions were derived to ensure the existence of an optimal control by using the vanishing discount approach. These conditions were mainly expressed in terms of the relative difference of the alpha-discount value functions. The main goal of this paper is to derive tractable conditions directly related to the primitive data of the PDMP to ensure the existence of an optimal control. The present work can be seen as a continuation of the results derived in Costa and Dufour (2008). Our main assumptions are written in terms of some integro-differential inequalities related to the so-called expected growth condition, and geometric convergence of the post-jump location kernel associated to the PDMP. An example based on the capacity expansion problem is presented, illustrating the possible applications of the results developed in the paper.

Singular Perturbation for the Discounted Continuous Control of Piecewise Deterministic Markov Processes

This paper deals with the expected discounted continuous control of piecewise deterministic Markov processes (PDMP`s) using a singular perturbation approach for dealing with rapidly oscillating parameters. The state space of the PDMP is written as the product of a finite set and a subset of the Euclidean space a""e (n) . The discrete part of the state, called the regime, characterizes the mode of operation of the physical system under consideration, and is supposed to have a fast (associated to a small parameter epsilon > 0) and a slow behavior. By using a similar approach as developed in Yin and Zhang (Continuous-Time Markov Chains and Applications: A Singular Perturbation Approach, Applications of Mathematics, vol. 37, Springer, New York, 1998, Chaps. 1 and 3) the idea in this paper is to reduce the number of regimes by considering an averaged model in which the regimes within the same class are aggregated through the quasi-stationary distribution so that the different states in this class are replaced by a single one. The main goal is to show that the value function of the control problem for the system driven by the perturbed Markov chain converges to the value function of this limit control problem as epsilon goes to zero. This convergence is obtained by, roughly speaking, showing that the infimum and supremum limits of the value functions satisfy two optimality inequalities as epsilon goes to zero. This enables us to show the result by invoking a uniqueness argument, without needing any kind of Lipschitz continuity condition.

A semi-Markov multistate model for estimation of the mean quality-adjusted survival for non-progressive processes

We discuss the estimation of the expected value of the quality-adjusted survival, based on multistate models. We generalize an earlier work, considering the sojourn times in health states are not identically distributed, for a given vector of covariates. Approaches based on semiparametric and parametric (exponential and Weibull distributions) methodologies are considered. A simulation study is conducted to evaluate the performance of the proposed estimator and the jackknife resampling method is used to estimate the variance of such estimator. An application to a real data set is also included.

Frequency and genotypic distribution of GB virus C (GBV-C) among Colombian population with Hepatitis B (HBV) or Hepatitis C (HCV) infection

Background: GB virus C (GBV-C) is an enveloped positive-sense ssRNA virus belonging to the Flaviviridae family. Studies on the genetic variability of the GBV-C reveals the existence of six genotypes: genotype 1 predominates in West Africa, genotype 2 in Europe and America, genotype 3 in Asia, genotype 4 in Southwest Asia, genotype 5 in South Africa and genotype 6 in Indonesia. The aim of this study was to determine the frequency and genotypic distribution of GBV-C in the Colombian population. Methods: Two groups were analyzed: i) 408 Colombian blood donors infected with HCV (n = 250) and HBV (n = 158) from Bogota and ii) 99 indigenous people with HBV infection from Leticia, Amazonas. A fragment of 344 bp from the 5' untranslated region (5' UTR) was amplified by nested RT PCR. Viral sequences were genotyped by phylogenetic analysis using reference sequences from each genotype obtained from GenBank (n = 160). Bayesian phylogenetic analyses were conducted using Markov chain Monte Carlo (MCMC) approach to obtain the MCC tree using BEAST v. 1.5.3. Results: Among blood donors, from 158 HBsAg positive samples, eight 5.06% (n = 8) were positive for GBV-C and from 250 anti-HCV positive samples, 3.2%(n = 8) were positive for GBV-C. Also, 7.7% (n = 7) GBV-C positive samples were found among indigenous people from Leticia. A phylogenetic analysis revealed the presence of the following GBV-C genotypes among blood donors: 2a (41.6%), 1 (33.3%), 3 (16.6%) and 2b (8.3%). All genotype 1 sequences were found in co-infection with HBV and 4/5 sequences genotype 2a were found in co-infection with HCV. All sequences from indigenous people from Leticia were classified as genotype 3. The presence of GBV-C infection was not correlated with the sex (p = 0.43), age (p = 0.38) or origin (p = 0.17). Conclusions: It was found a high frequency of GBV-C genotype 1 and 2 in blood donors. The presence of genotype 3 in indigenous population was previously reported from Santa Marta region in Colombia and in native people from Venezuela and Bolivia. This fact may be correlated to the ancient movements of Asian people to South America a long time ago.

Inferences of evolutionary and ecological events that influenced the population structure of Plebeia remota, a stingless bee from Brazil

The present study characterised the population genetic structure of Plebeia remota through mitochondrial DNA (mtDNA) analysis and evaluated evolutionary and ecological processes that may have contributed to the species current genetic scenario. Seventy feral nests were sampled representing four geographic regions (Cunha, Curitiba, Prudentopolis, and Blumenau). Fifteen composite mtDNA haplotypes were determined and a high genetic structure was detected among all populations. The current population structure may be a result of queen philopatry and vegetation shifts caused by palaeoclimatic changes and uplift of Brazilian coastal ranges. Finally, this study strongly suggests a revision of the taxonomic status of P. remota from the Prudentopolis region.

Random perturbations of stochastic processes with unbounded variable length memory

We consider binary infinite order stochastic chains perturbed by a random noise. This means that at each time step, the value assumed by the chain can be randomly and independently flipped with a small fixed probability. We show that the transition probabilities of the perturbed chain are uniformly close to the corresponding transition probabilities of the original chain. As a consequence, in the case of stochastic chains with unbounded but otherwise finite variable length memory, we show that it is possible to recover the context tree of the original chain, using a suitable version of the algorithm Context, provided that the noise is small enough.

Maternal investment in egg production: environmental and population-specific effects on offspring performance in the snapping shrimp Alpheus nuttingi (Schmitt, 1924) (Decapoda, Alpheidae)

Among marine invertebrates, the overall biomass invested in egg production varies widely within populations, which can result from the interaction of endogenous and exogenous factors. Species that have constant reproduction throughout the year can be good models to study the influence of environmental factors on reproductive processes. We conducted a seasonal comparison of egg production in the intertidal snapping shrimp Alpheus nuttingi, which shows a continuous reproductive pattern, to examine the hypothesis that differences in egg production are driven by environmental conditions and population features. This population showed an uncommon strategy, characterized by females that produce eggs of varying sizes within their clutches, with reduced egg volume when the number of eggs is higher (Spring-Summer). In these seasons, higher temperatures and greater food availability may allow the production of more eggs compared to the Autumn-Winter seasons. Compared to other alpheid shrimps, this population produces small eggs, but in larger numbers. Despite the higher fecundity, the reproductive output is relatively low, this production being supported by the large size of females from the southern Atlantic region. Our findings showed that the egg production of A. nuttingi was greatly influenced by environmental factors. Therefore, this shrimp, and probably other decapods that possess continuous reproduction, adopt different reproductive strategies during the year. (C) Koninklijke Brill NV, Leiden, 2010.

Association of haplotypes in the IL8 gene with susceptibility to chronic periodontitis in a Brazilian population

Background: Interleukin 8 (IL-8) is a chemokine related to the initiation and amplification of acute and chronic inflammatory processes. Polymorphisms in the IL8 gene have been associated with inflammatory diseases. We investigated whether the - 845(T/C) and - 738(T/A) single nucleotide polymorphisms (SNPs) in the IL8 gene, as well as the haplotypes they form together with the previously investigated -353(A/T), are associated with susceptibility to chronic periodontitis. Methods: DNA was extracted from buccal epithelial cells of 400 Brazilian individuals (control n =182, periodontitis n=218). SNPs were genotyped by the polymerase chain reaction-restriction fragment length polymorphism (PCR-RFLP) method. Disease associations were analyzed by the chi(2) test, Exact Fisher test and Clump program. Haplotypes were reconstructed using the expectation-maximization algorithm and differences in haplotype distribution between the groups were analyzed to estimate genetic susceptibility for chronic periodontitis development. Results: When analyzed individually, no SNPs showed different distributions between the control and chronic periodontitis groups. Although, nonsmokers carrying the TTA/CAT (OR = 2.35, 95% CI = 1.03-5.36) and TAT/CTA (OR= 6.05, 95% CI = 1.32-27.7) haplotypes were genetically susceptible to chronic periodontitis. The ITT/TAA haplotype was associated with protection against the development of periodontitis (for nonsmokers OR= 0.22, 95% CI = 0.10-0.46). Conclusion: Although none of the investigated SNPs in the IL8 gene was individually associated with periodontitis, some haplotypes showed significant association with susceptibility to, or protection against, chronic periodontitis in a Brazilian population. (C) 2010 Elsevier B.V. All rights reserved.

Population dynamics of genetically diverse Plasmodium falciparum lineages: community-based prospective study in rural Amazonia

Temporal changes in the prevalence of antigenic variants in Plasmodium falciparum populations have been interpreted as evidence of immune-mediated frequency-dependent selection, but evolutively neutral processes may generate similar patterns of serotype replacement. Over 4 years, we investigated the population dynamics of P. falciparum polymorphisms the community level by using 11 putatively neutral microsatellite markers. Plasmodium falciparum Populations were less diverse than sympatric P. vivax isolates, with less multiple-clone infections, lower number of alleles per locus and lower Virtual heterozygosity, but both species showed significant multilocus linkage disequilibrium. Evolutively neutral P. falciparum polymorphisms showed a high turnover rate, with few lineages persisting for several months in the population. Similar results had previously been obtained, in the same community, for sympatric P. vivax isolates. In contrast, the prevalence of the 2 dimorphic types of a major antigen, MSP-2, remained remarkably stable throughout the Study period. We Suggest that the relatively fast turnover of parasite lineages represents the typical population dynamics of neutral polymorphisms in small populations, with clear implications for the detection of frequency-dependent selection of polymorphisms.

Recording from Two Neurons: Second-Order Stimulus Reconstruction from Spike Trains and Population Coding

We study the reconstruction of visual stimuli from spike trains, representing the reconstructed stimulus by a Volterra series up to second order. We illustrate this procedure in a prominent example of spiking neurons, recording simultaneously from the two H1 neurons located in the lobula plate of the fly Chrysomya megacephala. The fly views two types of stimuli, corresponding to rotational and translational displacements. Second-order reconstructions require the manipulation of potentially very large matrices, which obstructs the use of this approach when there are many neurons. We avoid the computation and inversion of these matrices using a convenient set of basis functions to expand our variables in. This requires approximating the spike train four-point functions by combinations of two-point functions similar to relations, which would be true for gaussian stochastic processes. In our test case, this approximation does not reduce the quality of the reconstruction. The overall contribution to stimulus reconstruction of the second-order kernels, measured by the mean squared error, is only about 5% of the first-order contribution. Yet at specific stimulus-dependent instants, the addition of second-order kernels represents up to 100% improvement, but only for rotational stimuli. We present a perturbative scheme to facilitate the application of our method to weakly correlated neurons.

RUMOR PROCESSES ON N

We study four discrete-time stochastic systems on N, modeling processes of rumor spreading. The involved individuals can either have an active or a passive role, speaking up or asking for the rumor. The appetite for spreading or hearing the rumor is represented by a set of random variables whose distributions may depend on the individuals. Our goal is to understand-based on the distribution of the random variables-whether the probability of having an infinite set of individuals knowing the rumor is positive or not.