Inexact uniformization method for computing transient distributions of Markov chains

The uniformization method (also known as randomization) is a numerically stable algorithm for computing transient distributions of a continuous time Markov chain. When the solution is needed after a long run or when the convergence is slow, the uniformization method involves a large number of matrix-vector products. Despite this, the method remains very popular due to its ease of implementation and its reliability in many practical circumstances. Because calculating the matrix-vector product is the most time-consuming part of the method, overall efficiency in solving large-scale problems can be significantly enhanced if the matrix-vector product is made more economical. In this paper, we incorporate a new relaxation strategy into the uniformization method to compute the matrix-vector products only approximately. We analyze the error introduced by these inexact matrix-vector products and discuss strategies for refining the accuracy of the relaxation while reducing the execution cost. Numerical experiments drawn from computer systems and biological systems are given to show that significant computational savings are achieved in practical applications.

Adaptive estimation of hidden semi-Markov chains with parameterised transition probabilities and exponential decaying states

This paper develops maximum likelihood (ML) estimation schemes for finite-state semi-Markov chains in white Gaussian noise. We assume that the semi-Markov chain is characterised by transition probabilities of known parametric from with unknown parameters. We reformulate this hidden semi-Markov model (HSM) problem in the scalar case as a two-vector homogeneous hidden Markov model (HMM) problem in which the state consist of the signal augmented by the time to last transition. With this reformulation we apply the expectation Maximumisation (EM ) algorithm to obtain ML estimates of the transition probabilities parameters, Markov state levels and noise variance. To demonstrate our proposed schemes, motivated by neuro-biological applications, we use a damped sinusoidal parameterised function for the transition probabilities.

A Bayesian‐Markov process for reliability prediction

Accurate reliability prediction for large-scale, long lived engineering is a crucial foundation for effective asset risk management and optimal maintenance decision making. However, a lack of failure data for assets that fail infrequently, and changing operational conditions over long periods of time, make accurate reliability prediction for such assets very challenging. To address this issue, we present a Bayesian-Marko best approach to reliability prediction using prior knowledge and condition monitoring data. In this approach, the Bayesian theory is used to incorporate prior information about failure probabilities and current information about asset health to make statistical inferences, while Markov chains are used to update and predict the health of assets based on condition monitoring data. The prior information can be supplied by domain experts, extracted from previous comparable cases or derived from basic engineering principles. Our approach differs from existing hybrid Bayesian models which are normally used to update the parameter estimation of a given distribution such as the Weibull-Bayesian distribution or the transition probabilities of a Markov chain. Instead, our new approach can be used to update predictions of failure probabilities when failure data are sparse or nonexistent, as is often the case for large-scale long-lived engineering assets.

Large body size in an island-dwelling bird : a microevolutionary analysis

Island races of passerine birds display repeated evolution towards larger body size compared with their continental ancestors. The Capricorn silvereye (Zosterops lateralis chlorocephalus) has become up to six phenotypic standard deviations bigger in several morphological measures since colonization of an island approximately 4000 years ago. We estimated the genetic variance-covariance (G) matrix using full-sib and 'animal model' analyses, and selection gradients, for six morphological traits under field conditions in three consecutive cohorts of nestlings. Significant levels of genetic variance were found for all traits. Significant directional selection was detected for wing and tail lengths in one year and quadratic selection on culmen depth in another year. Although selection gradients on many traits were negative, the predicted evolutionary response to selection of these traits for all cohorts was uniformly positive. These results indicate that the G matrix and predicted evolutionary responses are consistent with those of a population evolving in the manner observed in the island passerine trend, that is, towards larger body size.

Infinite-horizon policy-gradient estimation

Gradient-based approaches to direct policy search in reinforcement learning have received much recent attention as a means to solve problems of partial observability and to avoid some of the problems associated with policy degradation in value-function methods. In this paper we introduce GPOMDP, a simulation-based algorithm for generating a biased estimate of the gradient of the average reward in Partially Observable Markov Decision Processes (POMDPs) controlled by parameterized stochastic policies. A similar algorithm was proposed by Kimura, Yamamura, and Kobayashi (1995). The algorithm's chief advantages are that it requires storage of only twice the number of policy parameters, uses one free parameter β ∈ [0,1) (which has a natural interpretation in terms of bias-variance trade-off), and requires no knowledge of the underlying state. We prove convergence of GPOMDP, and show how the correct choice of the parameter β is related to the mixing time of the controlled POMDP. We briefly describe extensions of GPOMDP to controlled Markov chains, continuous state, observation and control spaces, multiple-agents, higher-order derivatives, and a version for training stochastic policies with internal states. In a companion paper (Baxter, Bartlett, & Weaver, 2001) we show how the gradient estimates generated by GPOMDP can be used in both a traditional stochastic gradient algorithm and a conjugate-gradient procedure to find local optima of the average reward. ©2001 AI Access Foundation and Morgan Kaufmann Publishers. All rights reserved.

Convergence and divergence during the adaptation to similar environments by an Australian Groundsel

Adaptation to replicate environments is often achieved through similar phenotypic solutions. Whether selection also produces convergent genomic changes in these situations remains largely unknown. The variable groundsel, Senecio lautus, is an excellent system to investigate the genetic underpinnings of convergent evolution, because morphologically similar forms of these plants have adapted to the same environments along the coast of Australia. We compared range-wide patterns of genomic divergence in natural populations of this plant and searched for regions putatively affected by natural selection. Our results indicate that environmental adaptation followed complex genetic trajectories, affecting multiple loci, implying both the parallel recruitment of the same alleles and the divergence of completely different genomic regions across geography. An analysis of the biological functions of candidate genes suggests that adaptation to coastal environments may have occurred through the recruitment of different genes participating in similar processes. The relatively low genetic convergence that characterizes the parallel evolution of S. lautus forms suggests that evolution is more constrained at higher levels of biological organization.

The land hoarding and land inspector dilemma in China : an evolutionary game theoretic perspective

China has experienced considerable economic growth since 1978, which was accompanied by unprecedented growth in urbanization and, more recently, by associated rising urban housing and land banking issues. One such issue is that of land hoarding - where real estate developers purchase land to hold unused in the rising market for a future lucrative sale, often several years later. This practice is outlawed in China, where land use is controlled by increasingly strengthened Government policies and inspectors. Despite this, land hoarding continues apace, with the main culprits being the developers and inspectors working subversively. This resembles a game between two players - the inspector and the developer - which provides the setting for this paper in developing an evolutionary game theory model to provide insights into dealing with the dilemmas faced by the players. The logic and dilemma of land banking strategy and illegal land banking issues are analysed, along with the land inspector’s role from a game theory perspective by determining the replication dynamic mechanism and evolutionary stable strategies under the various conditions that the players face. The major factors influencing the actions of land inspectors, on the other hand, are the costs of inspection, no matter if it is strict or indolent, conflict costs, and income and penalties from corruption. From this, it is shown that, when the net loss for corruption (income from corruption minus the penalties for corruption and cost of strict inspections) is less than the cost of strict inspections, the final evolutionary stable strategy of the inspectors is to carry out indolent inspections. Then, whether penalising developers for hoarding is severe or not, the evolutionary strategy for the developer is to hoard. The implications for land use control mechanisms and associated developer-inspector actions and counteractions are then examined in the light of the model's properties.

Dynamic Bayesian network inferencing for non-homogeneous complex systems

Dynamic Bayesian Networks (DBNs) provide a versatile platform for predicting and analysing the behaviour of complex systems. As such, they are well suited to the prediction of complex ecosystem population trajectories under anthropogenic disturbances such as the dredging of marine seagrass ecosystems. However, DBNs assume a homogeneous Markov chain whereas a key characteristics of complex ecosystems is the presence of feedback loops, path dependencies and regime changes whereby the behaviour of the system can vary based on past states. This paper develops a method based on the small world structure of complex systems networks to modularise a non-homogeneous DBN and enable the computation of posterior marginal probabilities given evidence in forwards inference. It also provides an approach for an approximate solution for backwards inference as convergence is not guaranteed for a path dependent system. When applied to the seagrass dredging problem, the incorporation of path dependency can implement conditional absorption and allows release from the zero state in line with environmental and ecological observations. As dredging has a marked global impact on seagrass and other marine ecosystems of high environmental and economic value, using such a complex systems model to develop practical ways to meet the needs of conservation and industry through enhancing resistance and/or recovery is of paramount importance.

Neighborhood disadvantage and physical activity : baseline results from the HABITAT multilevel longitudinal study

PURPOSE: To examine the association between neighborhood disadvantage and physical activity (PA). ---------- METHODS: We use data from the HABITAT multilevel longitudinal study of PA among mid-aged (40-65 years) men and women (n=11, 037, 68.5% response rate) living in 200 neighborhoods in Brisbane, Australia. PA was measured using three questions from the Active Australia Survey (general walking, moderate, and vigorous activity), one indicator of total activity, and two questions about walking and cycling for transport. The PA measures were operationalized using multiple categories based on time and estimated energy expenditure that were interpretable with reference to the latest PA recommendations. The association between neighborhood disadvantage and PA was examined using multilevel multinomial logistic regression and Markov Chain Monte Carlo simulation. The contribution of neighborhood disadvantage to between-neighborhood variation in PA was assessed using the 80% interval odds ratio. ---------- RESULTS: After adjustment for sex, age, living arrangement, education, occupation, and household income, reported participation in all measures and levels of PA varied significantly across Brisbane’s neighborhoods, and neighborhood disadvantage accounted for some of this variation. Residents of advantaged neighborhoods reported significantly higher levels of total activity, general walking, moderate, and vigorous activity; however, they were less likely to walk for transport. There was no statistically significant association between neighborhood disadvantage and cycling for transport. In terms of total PA, residents of advantaged neighborhoods were more likely to exceed PA recommendations. ---------- CONCLUSIONS: Neighborhoods may exert a contextual effect on residents’ likelihood of participating in PA. The greater propensity of residents in advantaged neighborhoods to do high levels of total PA may contribute to lower rates of cardiovascular disease and obesity in these areas

Reading the play : adaptation by prediction of agent motion

An adaptive agent improves its performance by learning from experience. This paper describes an approach to adaptation based on modelling dynamic elements of the environment in order to make predictions of likely future state. This approach is akin to an elite sports player being able to “read the play”, allowing for decisions to be made based on predictions of likely future outcomes. Modelling of the agent‟s likely future state is performed using Markov Chains and a technique called “Motion and Occupancy Grids”. The experiments in this paper compare the performance of the planning system with and without the use of this predictive model. The results of the study demonstrate a surprising decrease in performance when using the predictions of agent occupancy. The results are derived from statistical analysis of the agent‟s performance in a high fidelity simulation of a world leading real robot soccer team.

Correcting mean-field approximations for birth-death-movement processes

On the microscale, migration, proliferation and death are crucial in the development, homeostasis and repair of an organism; on the macroscale, such effects are important in the sustainability of a population in its environment. Dependent on the relative rates of migration, proliferation and death, spatial heterogeneity may arise within an initially uniform field; this leads to the formation of spatial correlations and can have a negative impact upon population growth. Usually, such effects are neglected in modeling studies and simple phenomenological descriptions, such as the logistic model, are used to model population growth. In this work we outline some methods for analyzing exclusion processes which include agent proliferation, death and motility in two and three spatial dimensions with spatially homogeneous initial conditions. The mean-field description for these types of processes is of logistic form; we show that, under certain parameter conditions, such systems may display large deviations from the mean field, and suggest computationally tractable methods to correct the logistic-type description.

Corrected mean-field models for spatially-dependent advection-diffusion-reaction phenomena

In the exclusion-process literature, mean-field models are often derived by assuming that the occupancy status of lattice sites is independent. Although this assumption is questionable, it is the foundation of many mean-field models. In this work we develop methods to relax the independence assumption for a range of discrete exclusion process-based mechanisms motivated by applications from cell biology. Previous investigations that focussed on relaxing the independence assumption have been limited to studying initially-uniform populations and ignored any spatial variations. By ignoring spatial variations these previous studies were greatly simplified due to translational invariance of the lattice. These previous corrected mean-field models could not be applied to many important problems in cell biology such as invasion waves of cells that are characterised by moving fronts. Here we propose generalised methods that relax the independence assumption for spatially inhomogeneous problems, leading to corrected mean-field descriptions of a range of exclusion process-based models that incorporate (i) unbiased motility, (ii) biased motility, and (iii) unbiased motility with agent birth and death processes. The corrected mean-field models derived here are applicable to spatially variable processes including invasion wave type problems. We show that there can be large deviations between simulation data and traditional mean-field models based on invoking the independence assumption. Furthermore, we show that the corrected mean-field models give an improved match to the simulation data in all cases considered.

A phenomenological model for the structure-composition relationship of the high Tc cuprates based on simple chemical principles

A simple phenomenological model for the relationship between structure and composition of the high Tc cuprates is presented. The model is based on two simple crystal chemistry principles: unit cell doping and charge balance within unit cells. These principles are inspired by key experimental observations of how the materials accommodate large deviations from stoichiometry. Consistent explanations for significant HTSC properties can be explained without any additional assumptions while retaining valuable insight for geometric interpretation. Combining these two chemical principles with a review of Crystal Field Theory (CFT) or Ligand Field Theory (LFT), it becomes clear that the two oxidation states in the conduction planes (typically d8 and d9) belong to the most strongly divergent d-levels as a function of deformation from regular octahedral coordination. This observation offers a link to a range of coupling effects relating vibrations and spin waves through application of Hund’s rules. An indication of this model’s capacity to predict physical properties for HTSC is provided and will be elaborated in subsequent publications. Simple criteria for the relationship between structure and composition in HTSC systems may guide chemical syntheses within new material systems.

On adaptive HMM state estimation

In this paper new online adaptive hidden Markov model (HMM) state estimation schemes are developed, based on extended least squares (ELS) concepts and recursive prediction error (RPE) methods. The best of the new schemes exploit the idempotent nature of Markov chains and work with a least squares prediction error index, using a posterior estimates, more suited to Markov models then traditionally used in identification of linear systems.

Strategy in developing cell based therapy for large bone

Regeneration of osseous defects by tissue-engineering approach provides a novel means of treatment utilizing cell biology, materials science, and molecular biology. The concept of in vitro cultured osteoblasts having an ability to induce new bone formation has been demonstrated in the critical size defects using small animal models. The bone derived cells can be incorporated into bioengineered scaffolds and synthesize bone matrix, which on implantation can induce new bone formation. In search of optimal cell delivery materials, the extracellular matrix as cell carriers for the repair and regeneration of tissues is receiving increased attention. We have investigated extracellular matrix formed by osteoblasts in vitro as a scaffold for osteoblasts transplantation and found a mineralized matrix, formed by human osteoblasts in vitro, can initiate bone formation by activating endogenous mesenchymal cells. To repair the large bone defects, osteogenic or stem cells need to be prefabricated in a large three dimensional scaffold usually made of synthetic biomaterials, which have inadequate interaction with cells and lead to in vivo foreign body reactions. The interstitial extracellular matrix has been applied to modify biomaterials surface and identified vitronectin, which binds the heparin domain and RGD (Arg-Gly-Asp) sequence can modulate cell spreading, migration and matrix formation on biomaterials. We also synthesized a tri-block copolymer, methoxy-terminated poly(ethylene glycol)(MPEG)-polyL-lactide(PLLA)-polylysine(PLL) for human osteoblasts delivery. We identified osteogenic activity can be regulated by the molecular weight and composition of the triblock copolymers. Due to the sequential loss of lineage differentiation potential during the culture of bone marrow stromal cells that hinderers their potential clinical application, we have developed a clonal culture system and established several stem cell clones with fast growing and multi-differentiation properties. Using proteomics and subtractive immunization, several differential proteins have been identified and verified their potential application in stem cell characterization and tissue regeneration