Infinite-horizon policy-gradient estimation

Gradient-based approaches to direct policy search in reinforcement learning have received much recent attention as a means to solve problems of partial observability and to avoid some of the problems associated with policy degradation in value-function methods. In this paper we introduce GPOMDP, a simulation-based algorithm for generating a biased estimate of the gradient of the average reward in Partially Observable Markov Decision Processes (POMDPs) controlled by parameterized stochastic policies. A similar algorithm was proposed by Kimura, Yamamura, and Kobayashi (1995). The algorithm's chief advantages are that it requires storage of only twice the number of policy parameters, uses one free parameter β ∈ [0,1) (which has a natural interpretation in terms of bias-variance trade-off), and requires no knowledge of the underlying state. We prove convergence of GPOMDP, and show how the correct choice of the parameter β is related to the mixing time of the controlled POMDP. We briefly describe extensions of GPOMDP to controlled Markov chains, continuous state, observation and control spaces, multiple-agents, higher-order derivatives, and a version for training stochastic policies with internal states. In a companion paper (Baxter, Bartlett, & Weaver, 2001) we show how the gradient estimates generated by GPOMDP can be used in both a traditional stochastic gradient algorithm and a conjugate-gradient procedure to find local optima of the average reward. ©2001 AI Access Foundation and Morgan Kaufmann Publishers. All rights reserved.

Generalized binomial Tau-leap method for biochemical kinetics incorporating both delay and intrinsic noise

The delay stochastic simulation algorithm (DSSA) by Barrio et al. [Plos Comput. Biol.2, 117–E (2006)] was developed to simulate delayed processes in cell biology in the presence of intrinsic noise, that is, when there are small-to-moderate numbers of certain key molecules present in a chemical reaction system. These delayed processes can faithfully represent complex interactions and mechanisms that imply a number of spatiotemporal processes often not explicitly modeled such as transcription and translation, basic in the modeling of cell signaling pathways. However, for systems with widely varying reaction rate constants or large numbers of molecules, the simulation time steps of both the stochastic simulation algorithm (SSA) and the DSSA can become very small causing considerable computational overheads. In order to overcome the limit of small step sizes, various τ-leap strategies have been suggested for improving computational performance of the SSA. In this paper, we present a binomial τ- DSSA method that extends the τ-leap idea to the delay setting and avoids drawing insufficient numbers of reactions, a common shortcoming of existing binomial τ-leap methods that becomes evident when dealing with complex chemical interactions. The resulting inaccuracies are most evident in the delayed case, even when considering reaction products as potential reactants within the same time step in which they are produced. Moreover, we extend the framework to account for multicellular systems with different degrees of intercellular communication. We apply these ideas to two important genetic regulatory models, namely, the hes1 gene, implicated as a molecular clock, and a Her1/Her 7 model for coupled oscillating cells.

Oscillatory regulation of Hes1: Discrete stochastic delay modelling and simulation

Discrete stochastic simulations are a powerful tool for understanding the dynamics of chemical kinetics when there are small-to-moderate numbers of certain molecular species. In this paper we introduce delays into the stochastic simulation algorithm, thus mimicking delays associated with transcription and translation. We then show that this process may well explain more faithfully than continuous deterministic models the observed sustained oscillations in expression levels of hes1 mRNA and Hes1 protein.

An infinite-horizon robust filter for uncertain hidden Markov Models with conditional relative entropy constraints

We consider a robust filtering problem for uncertain discrete-time, homogeneous, first-order, finite-state hidden Markov models (HMMs). The class of uncertain HMMs considered is described by a conditional relative entropy constraint on measures perturbed from a nominal regular conditional probability distribution given the previous posterior state distribution and the latest measurement. Under this class of perturbations, a robust infinite horizon filtering problem is first formulated as a constrained optimization problem before being transformed via variational results into an unconstrained optimization problem; the latter can be elegantly solved using a risk-sensitive information-state based filtering.

Child characteristics associated with delay of gratification in children with Down syndrome

Aim: Children with Down syndrome have been identified as having difficulty delaying gratification when compared to mental age matched children who are developing typically. This study investigated the association between individual characteristics hypopthesized to be associated with ability to delay as well as the strategies children used in a waiting task. Method: Thirty-two children with Down syndrome and 50 typically developing children matched for mental age completed the tasks. Observations of their behaviour while waiting were video-recorded for later analysis. In addition, parents completed questionnaires with respect to their child’s personality and behaviour. Results: Children with Down syndrome were significantly less able to delay gratification than the comparison group. Different patterns of association were found for the two groups between the observational and questionnaire measures and delay time. Conclusions: Children with Down syndrome have greater difficulty delaying gratification than would be predicted on the basis of their mental age. The contributions to delay appear to differ from those for typically developing children and these differences need to be considered when planning interventions for developing this skill

Strategy use of children with Down syndrome in a delay of gratification situation

Background: The capacity to delay gratification has been shown to be a very important developmental task for children who are developing typically. There is evidence that children with Down syndrome have more difficulty with a delay of gratification task than typically developing children of the same mental age. This study focused on the strategies children with Down syndrome use while in a delay of gratification situation to ascertain if these contribute to the differences in delay times from those of typically developing children. Method: Thirty-two children with Down syndrome (15 females) and 50 typically developing children participated in the study. Children with Down syndrome had a mental age, as measured by the Stanford-Binet IV, between 36 and 66 months (M = 45.66). The typically developing children had a mean chronological age of 45.76 months. Children participated in a delay of gratification task where they were offered two or one small treats and asked which they preferred. They were then told that they could have the two treats if they waited for the researcher to return (an undisclosed time of 15 min). If they did not want to wait any longer they could call the researcher back but then they could have only one treat. Twenty-two of the children with Down syndrome and 43 of the typically developing children demonstrated understanding of the task and their data are included here. Sessions were videotaped for later analysis. Results: There were significant differences in the mean waiting times of the two groups. The mean of the waiting times for children with Down syndrome was 181.32 s (SD = 347.62) and was 440.21 s (SD = 377.59) for the typically developing children. Eighteen percent of the group with Down syndrome waited for the researcher to return in comparison to 35% of the typically developing group. Sixty-four percent of children with Down syndrome called the researcher back and the remainder (18%) violated. In the typically developing group 37% called the researcher back and 28% violated. The mean waiting time for the group of children with Down syndrome who called the researcher back was 24 s. Examination of strategy use in this group was therefore very limited. There appeared to be quite similar strategy use across the groups who waited the full 15 min. Conclusions: These results confirm the difficulty children with Down syndrome have in delaying gratification. Teaching strategies for waiting, using information drawn from the behaviours of children who are developing typically may be a useful undertaking. Examination of other contributors to delay ability (e.g., language skills) is also likely to be helpful in understanding the difficulties demonstrated in delaying gratification.

Establishing initial trust in autonomous delay tolerant networks without centralised PKI

A Delay Tolerant Network (DTN) is one where nodes can be highly mobile, with long message delay times forming dynamic and fragmented networks. Traditional centralised network security is difficult to implement in such a network, therefore distributed security solutions are more desirable in DTN implementations. Establishing effective trust in distributed systems with no centralised Public Key Infrastructure (PKI) such as the Pretty Good Privacy (PGP) scheme usually requires human intervention. Our aim is to build and compare different de- centralised trust systems for implementation in autonomous DTN systems. In this paper, we utilise a key distribution model based on the Web of Trust principle, and employ a simple leverage of common friends trust system to establish initial trust in autonomous DTN’s. We compare this system with two other methods of autonomously establishing initial trust by introducing a malicious node and measuring the distribution of malicious and fake keys. Our results show that the new trust system not only mitigates the distribution of fake malicious keys by 40% at the end of the simulation, but it also improved key distribution between nodes. This paper contributes a comparison of three de-centralised trust systems that can be employed in autonomous DTN systems.

A Bayesian approach to the determination of ignition delay

A novel in-cylinder pressure method for determining ignition delay has been proposed and demonstrated. This method proposes a new Bayesian statistical model to resolve the start of combustion, defined as being the point at which the band-pass in-cylinder pressure deviates from background noise and the combustion resonance begins. Further, it is demonstrated that this method is still accurate in situations where there is noise present. The start of combustion can be resolved for each cycle without the need for ad hoc methods such as cycle averaging. Therefore, this method allows for analysis of consecutive cycles and inter-cycle variability studies. Ignition delay obtained by this method and by the net rate of heat release have been shown to give good agreement. However, the use of combustion resonance to determine the start of combustion is preferable over the net rate of heat release method because it does not rely on knowledge of heat losses and will still function accurately in the presence of noise. Results for a six-cylinder turbo-charged common-rail diesel engine run with neat diesel fuel at full, three quarters and half load have been presented. Under these conditions the ignition delay was shown to increase as the load was decreased with a significant increase in ignition delay at half load, when compared with three quarter and full loads.

Ignition delay in an ethanol fumigated common rail diesel engine

A novel method for determining ignition delay is presented. This method utilises combustion resonance as a means of determining the onset of ignition. Results are shown from an ethanol fumigation study comprising of substitutions up to 50% at full, three-quarter and half load. It has been demonstrated that at full load there is a decrease in ignition delay with increasing ethanol substitutions, whereas at half load there is an increase in ignition delay with increasing ethanol substitutions. It is suggested that this conflicting result is a consequence of the auto ignition of ethanol.

Determining operations affected by delay in predictive train timetables

Constructing train schedules is vital in railways. This complex and time consuming task is however made more difficult by additional requirements to make train schedules robust to delays and other disruptions. For a timetable to be regarded as robust, it should be insensitive to delays of a specified level and its performance with respect to a given metric, should be within given tolerances. In other words the effect of delays should be identifiable and should be shown to be minimal. To this end, a sensitivity analysis is proposed that identifies affected operations. More specifically a sensitivity analysis for determining what operation delays cause each operation to be affected is proposed. The information provided by this analysis gives another measure of timetable robustness and also provides control information that can be used when delays occur in practice. Several algorithms are proposed to identify this information and they utilise a disjunctive graph model of train operations. Upon completion the sets of affected operations can also be used to define the impact of all delays without further disjunctive graph evaluations.

Ignition delay of bio-fuels in a common-rail compression ignition engine

The utility of a novel technique for determining the ignition delay in a compression ignition engine has been shown. This method utilises statistical modelling in the Bayesian paradigm to accurately resolve the start of combustion from a band-pass in-cylinder pressure signal. Applied to neat diesel and six biofuels, including four fractionations of palm oil of varying carbon chain length and degree of unsaturation, the relationships between ignition delay, cetane number and oxygen content have been explored. It is noted that the expected negative relationship between ignition delay and cetane number held, as did the positive relationship between ignition delay and oxygen content. The degree of unsaturation was also identified as a potential factor influencing the ignition delay.

The optimal duration and delay of first aid treatment for deep partial thickness burn injuries

Using our porcine model of deep dermal partial thickness burn injury, various durations (10min, 20min, 30min or 1h) and delays (immediate, 10min, 1h, 3h) of 15 degrees C running water first aid were applied to burns and compared to untreated controls. The subdermal temperatures were monitored during the treatment and wounds observed weekly for 6 weeks, for re-epithelialisation, wound surface area and cosmetic appearance. At 6 weeks after the burn, tissue biopsies were taken of the scar for histological analysis. Results showed that immediate application of cold running water for 20min duration is associated with an improvement in re-epithelialisation over the first 2 weeks post-burn and decreased scar tissue at 6 weeks. First aid application of cold water for as little as 10min duration or up to 1h delay still provides benefit.

Public sector project delay : the Malaysian perspective and the way forward

Construction delay has been a protracted problem for the Malaysian construction industry. Recent report showed that 80% of public sector projects are behind schedule. This underachieving time performance has led to many problems including public complaints, loss of reputation and revenue for the government and a slump in the industry’s GDP contribution. Research in the area of project delay has mushroomed worldwide with attempts to place mitigation plans, but delay remains a global phenomenon. There is now an urgent need for revolutionizing construction practices and past research, backed up with few successful cases suggests that Supply Chain Management (SCM) could prove beneficial to reduce or eliminate delays in construction. SCM which originated from the automotive manufacturing industry promotes a more collaborative approach to construction management and has recently gained attention of the construction industry. However every country, including Malaysia, would certainly have disparities of their own compared to others being it from the cultural point of view, nature of problems, locality or improvements needed. Therefore, this paper will present part of a Ph.D. research which aims at illustrating the Malaysian construction industry experts’ perception of the Malaysian public sector project delay, provide insight into these dilemmas, highlights the problems with current practices, its effects and the improvements needed. Subsequently, this paper would propose ratification to the problems using SCM. A semi-structured interview has been conducted to practitioners with at least 20 years’ experience in the industry. The findings showed that Malaysia may be unique compared to other countries and that by considering a number of additional factors, SCM could prove beneficial to increase efficiency of the Malaysian public sector projects.