Infinite-horizon policy-gradient estimation

Gradient-based approaches to direct policy search in reinforcement learning have received much recent attention as a means to solve problems of partial observability and to avoid some of the problems associated with policy degradation in value-function methods. In this paper we introduce GPOMDP, a simulation-based algorithm for generating a biased estimate of the gradient of the average reward in Partially Observable Markov Decision Processes (POMDPs) controlled by parameterized stochastic policies. A similar algorithm was proposed by Kimura, Yamamura, and Kobayashi (1995). The algorithm's chief advantages are that it requires storage of only twice the number of policy parameters, uses one free parameter β ∈ [0,1) (which has a natural interpretation in terms of bias-variance trade-off), and requires no knowledge of the underlying state. We prove convergence of GPOMDP, and show how the correct choice of the parameter β is related to the mixing time of the controlled POMDP. We briefly describe extensions of GPOMDP to controlled Markov chains, continuous state, observation and control spaces, multiple-agents, higher-order derivatives, and a version for training stochastic policies with internal states. In a companion paper (Baxter, Bartlett, & Weaver, 2001) we show how the gradient estimates generated by GPOMDP can be used in both a traditional stochastic gradient algorithm and a conjugate-gradient procedure to find local optima of the average reward. ©2001 AI Access Foundation and Morgan Kaufmann Publishers. All rights reserved.

Mortality, fertility, and persistent income inequality

Many developing countries are plagued by persistent inequality in income distribution. While a growing body of economic-demographic literature emphasizes differential fertility channel, this paper investigates differential child mortality--differences in child mortality across income groups--as a critical link through which income inequality persists. Using an overlapping generations model in which both child mortality and fertility are endogenously determined by parental choice, this paper demonstrates that differential child mortality and its interaction with differential fertility may generate an "income inequality trap." The trap is characterized by higher child mortality and lower degree of skill formation among the poorer households. The model can also explain the behavior of aggregate fertility and mortality rates for countries at various stages of development, consonant with patterns of demographic transition. The results indicate that provision of public health that raises the productivity of private health spending may be an effective way to reduce income inequality

Motorcycle protective clothing : protection from injury or just the weather?

Background Apart from helmets, little is known about the effectiveness of motorcycle protective clothing in reducing injuries in crashes. The study aimed to quantify the association between usage of motorcycle clothing and injury in crashes. Methods and findings Cross-sectional analytic study. Crashed motorcyclists (n = 212, 71% of identified eligible cases) were recruited through hospitals and motorcycle repair services. Data was obtained through structured face-to-face interviews. The main outcome was hospitalization and motorcycle crash-related injury. Poisson regression was used to estimate relative risk (RR) and 95% confidence intervals for injury adjusting for potential confounders. Results Motorcyclists were significantly less likely to be admitted to hospital if they crashed wearing motorcycle jackets (RR = 0.79, 95% CI: 0.69–0.91), pants (RR = 0.49, 95% CI: 0.25–0.94), or gloves (RR = 0.41, 95% CI: 0.26–0.66). When garments included fitted body armour there was a significantly reduced risk of injury to the upper body (RR = 0.77, 95% CI: 0.66–0.89), hands and wrists (RR = 0.55, 95% CI: 0.38–0.81), legs (RR = 0.60, 95% CI: 0.40–0.90), feet and ankles (RR = 0.54, 95% CI: 0.35–0.83). Non-motorcycle boots were also associated with a reduced risk of injury compared to shoes or joggers (RR = 0.46, 95% CI: 0.28–0.75). No association between use of body armour and risk of fracture injuries was detected. A substantial proportion of motorcycle designed gloves (25.7%), jackets (29.7%) and pants (28.1%) were assessed to have failed due to material damage in the crash. Conclusions Motorcycle protective clothing is associated with reduced risk and severity of crash related injury and hospitalization, particularly when fitted with body armour. The proportion of clothing items that failed under crash conditions indicates a need for improved quality control. While mandating usage of protective clothing is not recommended, consideration could be given to providing incentives for usage of protective clothing, such as tax exemptions for safety gear, health insurance premium reductions and rebates.

An economic evaluation of the Safe Motherhood programme in Guangxi, China

Maternal and infant mortality is a global health issue with a significant social and economic impact. Each year, over half a million women worldwide die due to complications related to pregnancy or childbirth, four million infants die in the first 28 days of life, and eight million infants die in the first year. Ninety-nine percent of maternal and infant deaths are in developing countries. Reducing maternal and infant mortality is among the key international development goals. In China, the national maternal mortality ratio and infant mortality rate were reduced greatly in the past two decades, yet a large discrepancy remains between urban and rural areas. To address this problem, a large-scale Safe Motherhood Programme was initiated in 2000. The programme was implemented in Guangxi in 2003. Interventions in the programme included both demand-side and supply side-interventions focusing on increasing health service use and improving birth outcomes. Little is known about the effects and economic outcomes of the Safe Motherhood Programme in Guangxi, although it has been implemented for seven years. The aim of this research is to estimate the effectiveness and cost-effectiveness of the interventions in the Safe Motherhood Programme in Guangxi, China. The objectives of this research include: 1. To evaluate whether the changes of health service use and birth outcomes are associated with the interventions in the Safe Motherhood Programme. 2. To estimate the cost-effectiveness of the interventions in the Safe Motherhood Programme and quantify the uncertainty surrounding the decision. 3. To assess the expected value of perfect information associated with both the whole decision and individual parameters, and interpret the findings to inform priority setting in further research and policy making in this area. A quasi-experimental study design was used in this research to assess the effectiveness of the programme in increasing health service use and improving birth outcomes. The study subjects were 51 intervention counties and 30 control counties. Data on the health service use, birth outcomes and socio-economic factors from 2001 to 2007 were collected from the programme database and statistical yearbooks. Based on the profile plots of the data, general linear mixed models were used to evaluate the effectiveness of the programme while controlling for the effects of baseline levels of the response variables, change of socio-economic factors over time and correlations among repeated measurements from the same county. Redundant multicollinear variables were deleted from the mixed model using the results of the multicollinearity diagnoses. For each response variable, the best covariance structure was selected from 15 alternatives according to the fit statistics including Akaike information criterion, Finite-population corrected Akaike information criterion, and Schwarz.s Bayesian information criterion. Residual diagnostics were used to validate the model assumptions. Statistical inferences were made to show the effect of the programme on health service use and birth outcomes. A decision analytic model was developed to evaluate the cost-effectiveness of the programme, quantify the decision uncertainty, and estimate the expected value of perfect information associated with the decision. The model was used to describe the transitions between health states for women and infants and reflect the change of both costs and health benefits associated with implementing the programme. Result gained from the mixed models and other relevant evidence identified were synthesised appropriately to inform the input parameters of the model. Incremental cost-effectiveness ratios of the programme were calculated for the two groups of intervention counties over time. Uncertainty surrounding the parameters was dealt with using probabilistic sensitivity analysis, and uncertainty relating to model assumptions was handled using scenario analysis. Finally the expected value of perfect information for both the whole model and individual parameters in the model were estimated to inform priority setting in further research in this area.The annual change rates of the antenatal care rate and the institutionalised delivery rate were improved significantly in the intervention counties after the programme was implemented. Significant improvements were also found in the annual change rates of the maternal mortality ratio, the infant mortality rate, the incidence rate of neonatal tetanus and the mortality rate of neonatal tetanus in the intervention counties after the implementation of the programme. The annual change rate of the neonatal mortality rate was also improved, although the improvement was only close to statistical significance. The influences of the socio-economic factors on the health service use indicators and birth outcomes were identified. The rural income per capita had a significant positive impact on the health service use indicators, and a significant negative impact on the birth outcomes. The number of beds in healthcare institutions per 1,000 population and the number of rural telephone subscribers per 1,000 were found to be positively significantly related to the institutionalised delivery rate. The length of highway per square kilometre negatively influenced the maternal mortality ratio. The percentage of employed persons in the primary industry had a significant negative impact on the institutionalised delivery rate, and a significant positive impact on the infant mortality rate and neonatal mortality rate. The incremental costs of implementing the programme over the existing practice were US $11.1 million from the societal perspective, and US $13.8 million from the perspective of the Ministry of Health. Overall, 28,711 life years were generated by the programme, producing an overall incremental cost-effectiveness ratio of US $386 from the societal perspective, and US $480 from the perspective of the Ministry of Health, both of which were below the threshold willingness-to-pay ratio of US $675. The expected net monetary benefit generated by the programme was US $8.3 million from the societal perspective, and US $5.5 million from the perspective of the Ministry of Health. The overall probability that the programme was cost-effective was 0.93 and 0.89 from the two perspectives, respectively. The incremental cost-effectiveness ratio of the programme was insensitive to the different estimates of the three parameters relating to the model assumptions. Further research could be conducted to reduce the uncertainty surrounding the decision, in which the upper limit of investment was US $0.6 million from the societal perspective, and US $1.3 million from the perspective of the Ministry of Health. It is also worthwhile to get a more precise estimate of the improvement of infant mortality rate. The population expected value of perfect information for individual parameters associated with this parameter was US $0.99 million from the societal perspective, and US $1.14 million from the perspective of the Ministry of Health. The findings from this study have shown that the interventions in the Safe Motherhood Programme were both effective and cost-effective in increasing health service use and improving birth outcomes in rural areas of Guangxi, China. Therefore, the programme represents a good public health investment and should be adopted and further expanded to an even broader area if possible. This research provides economic evidence to inform efficient decision making in improving maternal and infant health in developing countries.

The analysis of social capital in online social communities

Social networks have proven to be an attractive avenue of investigation for researchers since humans are social creatures. Numerous literature have explored the term “social networks” from different perspectives and in diverse research fields. With the popularity of the Internet, social networking has taken on a new dimension. Online social communities therefore have become an emerging social avenue for people to communicate in today’s information age. People use online social communities to share their interests, maintain friendships, and extend their so-called circle of “friends”. Likewise, social capital, also known as human capital, is an important theory in sociology. Researchers usually utilise social capital theory when they investigate the topic relating to social networks. However, there is little literature that can provide an explicit and strong assertion in that research area due to the complexity of social capital. This thesis therefore focuses on the issue related to providing a better understanding about the relationship between social capital and online social communities. To enhance the value within the scope of this analysis, an online survey was conducted to examine the effects of the dimensions of social capital: relational capital, structural capital, and cognitive capital, determining the intensity of using online social communities. The data were derived from a total of 350 self-selected respondents completing an online survey during the research period. The main results indicate that social capital exists in online social communities under normal circumstances. Finally, this thesis also presents three contributions for both theory and practice in Chapter 5. The main results contribute to the understanding of connectivity in the interrelationships between individual social capital exchange within online social networks. Secondly, social trust was found to have a weak effect in influencing the intensity of individuals using online social communities. Third, the perpetual role of information sharing has an indirect influence on individual users participating in online social communities. This study also benefits online marketing consultants as marketers can not only gain consumer information easier from online social communities but also this understanding assists in designing effective communication within online social communities. The cross-sectional study, the reliability of Internet survey data, and sampling issues are the major three limitations in this research. The thesis provides a new research model and recommends that the mediating effects, privacy paradox, and social trust on online social communities should be further explored in future research.

Integrating collaborative filtering and search-based techniques for personalized online product recommendation

The existing Collaborative Filtering (CF) technique that has been widely applied by e-commerce sites requires a large amount of ratings data to make meaningful recommendations. It is not directly applicable for recommending products that are not frequently purchased by users, such as cars and houses, as it is difficult to collect rating data for such products from the users. Many of the e-commerce sites for infrequently purchased products are still using basic search-based techniques whereby the products that match with the attributes given in the target user's query are retrieved and recommended to the user. However, search-based recommenders cannot provide personalized recommendations. For different users, the recommendations will be the same if they provide the same query regardless of any difference in their online navigation behaviour. This paper proposes to integrate collaborative filtering and search-based techniques to provide personalized recommendations for infrequently purchased products. Two different techniques are proposed, namely CFRRobin and CFAg Query. Instead of using the target user's query to search for products as normal search based systems do, the CFRRobin technique uses the products in which the target user's neighbours have shown interest as queries to retrieve relevant products, and then recommends to the target user a list of products by merging and ranking the returned products using the Round Robin method. The CFAg Query technique uses the products that the user's neighbours have shown interest in to derive an aggregated query, which is then used to retrieve products to recommend to the target user. Experiments conducted on a real e-commerce dataset show that both the proposed techniques CFRRobin and CFAg Query perform better than the standard Collaborative Filtering (CF) and the Basic Search (BS) approaches, which are widely applied by the current e-commerce applications. The CFRRobin and CFAg Query approaches also outperform the e- isting query expansion (QE) technique that was proposed for recommending infrequently purchased products.

Evolution of life drama program

Life Drama is a program of drama-based experiential learning activities involving groups of community leaders and members. The three-year project evolved from a theatre-in-education approach to an intercultural theatre approach incorporating Papua New Guinean performance traditions. It involved Australian, English and Papua New Guinean researchers at four key sites: Tari, Southern Highlands Province; Port Moresby, National Capital District; Madang, Madang Province; and Karkar Island, Madang Province. The project was innovative in a number of ways, including: a Participatory Action Research approach, involving community leaders at various levels as co-researchers; a participatory theatre approach as opposed to a performance approach; emphasis on sexual health promotion and HIV prevention through an experiential learning paradigm; addressing the norms and realities of the community rather than targeting only individual behaviour; an International Theatre Research Laboratory to explore the fusion of traditional cultural elements with contemporary health promotion aims; and an innovative method-assemblage approach to collecting and triangulating quantitative, qualitative, and performative data. The project attracted over $350,000 in funding and support from the Australian Research Council, National AIDS Secretariat in PNG, and private sector and non-government partners. Findings were presented at various conferences and symposia including the annual Medical Symposium in Wewak (2010), the triennial Research in Drama Education conference in Exeter (2011), and the International Research in Drama Education conference (Sydney 2009 and Limerick 2012). A number of peer-reviewed journal articles have been published. Elements of the program have been incorporated into the University of Goroka's compulsory HIV awareness program for undergraduate students. A national dissemination strategy for Life Drama in Papua New Guinea is now underway, with seed funding of AUD$74,000 from the National AIDS Council Secretariat, PNG.

Useful field of view predicts driving in the presence of distracters

Purpose. The Useful Field of View (UFOV(R)) test has been shown to be highly effective in predicting crash risk among older adults. An important question which we examined in this study is whether this association is due to the ability of the UFOV to predict difficulties in attention-demanding driving situations that involve either visual or auditory distracters. Methods. Participants included 92 community-living adults (mean age 73.6 +/- 5.4 years; range 65-88 years) who completed all three subtests of the UFOV involving assessment of visual processing speed (subtest 1), divided attention (subtest 2), and selective attention (subtest 3); driving safety risk was also classified using the UFOV scoring system. Driving performance was assessed separately on a closed-road circuit while driving under three conditions: no distracters, visual distracters, and auditory distracters. Driving outcome measures included road sign recognition, hazard detection, gap perception, time to complete the course, and performance on the distracter tasks. Results. Those rated as safe on the UFOV (safety rating categories 1 and 2), as well as those responding faster than the recommended cut-off on the selective attention subtest (350 msec), performed significantly better in terms of overall driving performance and also experienced less interference from distracters. Of the three UFOV subtests, the selective attention subtest best predicted overall driving performance in the presence of distracters. Conclusions. Older adults who were rated as higher risk on the UFOV, particularly on the selective attention subtest, demonstrated poorest driving performance in the presence of distracters. This finding suggests that the selective attention subtest of the UFOV may be differentially more effective in predicting driving difficulties in situations of divided attention which are commonly associated with crashes.

Hot temperatures and morbidity : a systematic review and meta–analysis

Extreme temperatures have been shown to have a detrimental effect on health. Hot temperatures can increase the risk of mortality, particularly in people suffering from cardiorespiratory diseases. Given the onset of climate change, it is critical that the impact of temperature on health is understood, so that effective public health strategies can correctly identify vulnerable groups within the population. However, while effects on mortality have been extensively studied, temperature–related morbidity has received less attention. This study applied a systematic review and meta–analysis to examine the current literature relating to hot temperatures and morbidity.