Infinite-horizon policy-gradient estimation

Gradient-based approaches to direct policy search in reinforcement learning have received much recent attention as a means to solve problems of partial observability and to avoid some of the problems associated with policy degradation in value-function methods. In this paper we introduce GPOMDP, a simulation-based algorithm for generating a biased estimate of the gradient of the average reward in Partially Observable Markov Decision Processes (POMDPs) controlled by parameterized stochastic policies. A similar algorithm was proposed by Kimura, Yamamura, and Kobayashi (1995). The algorithm's chief advantages are that it requires storage of only twice the number of policy parameters, uses one free parameter β ∈ [0,1) (which has a natural interpretation in terms of bias-variance trade-off), and requires no knowledge of the underlying state. We prove convergence of GPOMDP, and show how the correct choice of the parameter β is related to the mixing time of the controlled POMDP. We briefly describe extensions of GPOMDP to controlled Markov chains, continuous state, observation and control spaces, multiple-agents, higher-order derivatives, and a version for training stochastic policies with internal states. In a companion paper (Baxter, Bartlett, & Weaver, 2001) we show how the gradient estimates generated by GPOMDP can be used in both a traditional stochastic gradient algorithm and a conjugate-gradient procedure to find local optima of the average reward. ©2001 AI Access Foundation and Morgan Kaufmann Publishers. All rights reserved.

Boosting the margin: A new explanation for the effectiveness of voting methods

One of the surprising recurring phenomena observed in experiments with boosting is that the test error of the generated classifier usually does not increase as its size becomes very large, and often is observed to decrease even after the training error reaches zero. In this paper, we show that this phenomenon is related to the distribution of margins of the training examples with respect to the generated voting classification rule, where the margin of an example is simply the difference between the number of correct votes and the maximum number of votes received by any incorrect label. We show that techniques used in the analysis of Vapnik's support vector classifiers and of neural networks with small weights can be applied to voting methods to relate the margin distribution to the test error. We also show theoretically and experimentally that boosting is especially effective at increasing the margins of the training examples. Finally, we compare our explanation to those based on the bias-variance decomposition.

Heat strain, hydration status, and symptoms of heat illness in surface mine workers

The aim of the research program was to evaluate the heat strain, hydration status, and heat illness symptoms experienced by surface mine workers. An initial investigation involved 91 surface miners completing a heat stress questionnaire; assessing the work environment, hydration practices, and heat illness symptom experience. The key findings included 1) more than 80 % of workers experienced at least one symptom of heat illness over a 12 month period; and 2) the risk of moderate symptoms of heat illness increased with the severity of dehydration. These findings highlight a health and safety concern for surface miners, as experiencing symptoms of heat illness is an indication that the physiological systems of the body may be struggling to meet the demands of thermoregulation. To illuminate these findings a field investigation to monitor the heat strain and hydration status of surface miners was proposed. Two preliminary studies were conducted to ensure accurate and reliable data collection techniques. Firstly, a study was undertaken to determine a calibration procedure to ensure the accuracy of core body temperature measurement via an ingestible sensor. A water bath was heated to several temperatures between 23 . 51 ¢ªC, allowing for comparison of the temperature recorded by the sensors and a traceable thermometer. A positive systematic bias was observed and indicated a need for calibration. It was concluded that a linear regression should be developed for each sensor prior to ingestion, allowing for a correction to be applied to the raw data. Secondly, hydration status was to be assessed through urine specific gravity measurement. It was foreseeable that practical limitations on mine sites would delay the time between urine collection and analysis. A study was undertaken to assess the reliability of urine analysis over time. Measurement of urine specific gravity was found to be reliable up to 24 hours post urine collection and was suitable to be used in the field study. Twenty-nine surface miners (14 drillers [winter] and 15 blast crew [summer]) were monitored during a normal work shift. Core body temperature was recorded continuously. Average mean core body temperature was 37.5 and 37.4 ¢ªC for blast crew and drillers, with average maximum body temperatures of 38.0 and 37.9 ¢ªC respectively. The highest body temperature recorded was 38.4 ¢ªC. Urine samples were collected at each void for specific gravity measurement. The average mean urine specific gravity was 1.024 and 1.021 for blast crew and drillers respectively. The Heat Illness Symptoms Index was used to evaluate the experience of heat illness symptoms on shift. Over 70 % of drillers and over 80 % of blast crew reported at least one symptom. It was concluded that 1) heat strain remained within the recommended limits for acclimatised workers; and 2) the majority of workers were dehydrated before commencing their shift, and tend to remain dehydrated for the duration. Dehydration was identified as the primary issue for surface miners working in the heat. Therefore continued study focused on investigating a novel approach to monitoring hydration status. The final aim of this research program was to investigate the influence dehydration has on intraocular pressure (IOP); and subsequently, whether IOP could provide a novel indicator of hydration status. Seven males completed 90 minutes of walking in both a cool and hot climate with fluid restriction. Hydration variables and intraocular pressure were measured at baseline and at 30 minute intervals. Participants became dehydrated during the trial in the heat but maintained hydration status in the cool. Intraocular pressure progressively declined in the trial in the heat but remained relatively stable when hydration was maintained. A significant relationship was observed between intraocular pressure and both body mass loss and plasma osmolality. This evidence suggests that intraocular pressure is influenced by changes in hydration status. Further research is required to determine if intraocular pressure could be utilised as an indirect indicator of hydration status.

Practical stability with respect to model mismatch of approximate discrete-time output feedback control

This paper establishes a practical stability result for discrete-time output feedback control involving mismatch between the exact system to be stabilised and the approximating system used to design the controller. The practical stability is in the sense of an asymptotic bound on the amount of error bias introduced by the model approximation, and is established using local consistency properties of the systems. Importantly, the practical stability established here does not require the approximating system to be of the same model type as the exact system. Examples are presented to illustrate the nature of our practical stability result.

Malnutrition and clinical outcomes in elderly patients from a Singapore acute hospital

Older adults, especially those acutely ill, are vulnerable to developing malnutrition due to a range of risk factors. The high prevalence and extensive consequences of malnutrition in hospitalised older adults have been reported extensively. However, there are few well-designed longitudinal studies that report the independent relationship between malnutrition and clinical outcomes after adjustment for a wide range of covariates. Acutely ill older adults are exceptionally prone to nutritional decline during hospitalisation, but few reports have studied this change and impact on clinical outcomes. In the rapidly ageing Singapore population, all this evidence is lacking, and the characteristics associated with the risk of malnutrition are also not well-documented. Despite the evidence on malnutrition prevalence, it is often under-recognised and under-treated. It is therefore crucial that validated nutrition screening and assessment tools are used for early identification of malnutrition. Although many nutrition screening and assessment tools are available, there is no universally accepted method for defining malnutrition risk and nutritional status. Most existing tools have been validated amongst Caucasians using various approaches, but they are rarely reported in the Asian elderly and none has been validated in Singapore. Due to the multiethnicity, cultural, and language differences in Singapore older adults, the results from non-Asian validation studies may not be applicable. Therefore it is important to identify validated population and setting specific nutrition screening and assessment methods to accurately detect and diagnose malnutrition in Singapore. The aims of this study are therefore to: i) characterise hospitalised elderly in a Singapore acute hospital; ii) describe the extent and impact of admission malnutrition; iii) identify and evaluate suitable methods for nutritional screening and assessment; and iv) examine changes in nutritional status during admission and their impact on clinical outcomes. A total of 281 participants, with a mean (+SD) age of 81.3 (+7.6) years, were recruited from three geriatric wards in Tan Tock Seng Hospital over a period of eight months. They were predominantly Chinese (83%) and community-dwellers (97%). They were screened within 72 hours of admission by a single dietetic technician using four nutrition screening tools [Tan Tock Seng Hospital Nutrition Screening Tool (TTSH NST), Nutritional Risk Screening 2002 (NRS 2002), Mini Nutritional Assessment-Short Form (MNA-SF), and Short Nutritional Assessment Questionnaire (SNAQ©)] that were administered in no particular order. The total scores were not computed during the screening process so that the dietetic technician was blinded to the results of all the tools. Nutritional status was assessed by a single dietitian, who was blinded to the screening results, using four malnutrition assessment methods [Subjective Global Assessment (SGA), Mini Nutritional Assessment (MNA), body mass index (BMI), and corrected arm muscle area (CAMA)]. The SGA rating was completed prior to computation of the total MNA score to minimise bias. Participants were reassessed for weight, arm anthropometry (mid-arm circumference, triceps skinfold thickness), and SGA rating at discharge from the ward. The nutritional assessment tools and indices were validated against clinical outcomes (length of stay (LOS) >11days, discharge to higher level care, 3-month readmission, 6-month mortality, and 6-month Modified Barthel Index) using multivariate logistic regression. The covariates included age, gender, race, dementia (defined using DSM IV criteria), depression (defined using a single question “Do you often feel sad or depressed?”), severity of illness (defined using a modified version of the Severity of Illness Index), comorbidities (defined using Charlson Comorbidity Index, number of prescribed drugs and admission functional status (measured using Modified Barthel Index; MBI). The nutrition screening tools were validated against the SGA, which was found to be the most appropriate nutritional assessment tool from this study (refer section 5.6) Prevalence of malnutrition on admission was 35% (defined by SGA), and it was significantly associated with characteristics such as swallowing impairment (malnourished vs well-nourished: 20% vs 5%), poor appetite (77% vs 24%), dementia (44% vs 28%), depression (34% vs 22%), and poor functional status (MBI 48.3+29.8 vs 65.1+25.4). The SGA had the highest completion rate (100%) and was predictive of the highest number of clinical outcomes: LOS >11days (OR 2.11, 95% CI [1.17- 3.83]), 3-month readmission (OR 1.90, 95% CI [1.05-3.42]) and 6-month mortality (OR 3.04, 95% CI [1.28-7.18]), independent of a comprehensive range of covariates including functional status, disease severity and cognitive function. SGA is therefore the most appropriate nutritional assessment tool for defining malnutrition. The TTSH NST was identified as the most suitable nutritional screening tool with the best diagnostic performance against the SGA (AUC 0.865, sensitivity 84%, specificity 79%). Overall, 44% of participants experienced weight loss during hospitalisation, and 27% had weight loss >1% per week over median LOS 9 days (range 2-50). Wellnourished (45%) and malnourished (43%) participants were equally prone to experiencing decline in nutritional status (defined by weight loss >1% per week). Those with reduced nutritional status were more likely to be discharged to higher level care (adjusted OR 2.46, 95% CI [1.27-4.70]). This study is the first to characterise malnourished hospitalised older adults in Singapore. It is also one of the very few studies to (a) evaluate the association of admission malnutrition with clinical outcomes in a multivariate model; (b) determine the change in their nutritional status during admission; and (c) evaluate the validity of nutritional screening and assessment tools amongst hospitalised older adults in an Asian population. Results clearly highlight that admission malnutrition and deterioration in nutritional status are prevalent and are associated with adverse clinical outcomes in hospitalised older adults. With older adults being vulnerable to risks and consequences of malnutrition, it is important that they are systematically screened so timely and appropriate intervention can be provided. The findings highlighted in this thesis provide an evidence base for, and confirm the validity of the current nutrition screening and assessment tools used among hospitalised older adults in Singapore. As the older adults may have developed malnutrition prior to hospital admission, or experienced clinically significant weight loss of >1% per week of hospitalisation, screening of the elderly should be initiated in the community and continuous nutritional monitoring should extend beyond hospitalisation.

Platinum/graphene nanosheet/SiC contacts and their application for hydrogen gas sensing

Pt/graphene nanosheet/SiC based devices are fabricated and characterized and their performances toward hydrogen gas are investigated. The graphene nanosheets are synthesized via the reduction of spray-coated graphite oxide deposited onto SiC substrates. Raman and X-ray photoelectron spectroscopies indicate incomplete reduction of the graphite oxide, resulting in partially oxidized graphene nanosheet layers of less than 10 nm thickness. The effects of interfaces on the nonlinear behavior of the Pt/graphene and graphene/SiC junctions are investigated. Current-voltage measurements of the sensors toward 1% hydrogen in synthetic air gas mixture at various temperatures ranging up to 100. ° C are performed. From the dynamic response, a voltage shift of ∼100 mV is recorded for 1% hydrogen at a constant current bias of 1 mA at 100. °C. © 2010 American Chemical Society.

Reverse biased Pt/nanostructured MoO3/SiC Schottky diode based hydrogen gas sensors

Pt/nanostructured molybdenum oxide (MoO3) /SiC Schottky diode based gas sensors were fabricated for hydrogen (H2) gas sensing. Due to the enhanced performance, which is ascribed to the application of MoO3 nanostructures, these devices were used in reversed bias. MoO3 characterization by scanning electron microscopy showed morphology of randomly orientated nanoplatelets with thicknesses between 50 and 500 nm. An α-Β mixed phase crystallographic structure of MoO3 was characterized by x-ray diffraction. At 180 °C, 1.343 V voltage shift in the reverse I-V curve and a Pt/ MoO3 barrier height change of 20 meV were obtained after exposure to 1% H2 gas in synthetic air. © 2009 American Institute of Physics.

Teachers' and education students' perceptions of and reactions to children with and without the diagnostic label 'ADHD'

Thirty-four elementary school teachers and 32 education students from Canada rated their reactions towards vignettes describing children who met attention-deficit/hyperactivity disorder (ADHD) symptom criteria that included or did not include the label “ADHD.” “ADHD”-labeled vignettes elicited greater perceptions of the child's impairment as well as more negative emotions and less confidence in the participants, although it also increased participants' willingness to implement treatment interventions. Ratings were similar to vignettes of boys versus girls; however, important differences in ratings between teachers and education students emerged and are discussed. Finally, we investigated the degree to which teachers' professional backgrounds influenced bias based on the label “ADHD.” Training specific to ADHD consistently predicted label bias, whereas teachers' experience working with children with ADHD did not.

Unbiased estimation of the half-life to price index convergence among U.S. cities

Estimates of the half-life to convergence of prices across a panel of cities are subject to bias from three potential sources: inappropriate cross-sectional aggregation of heterogeneous coefficients, presence of lagged dependent variables in a model with individual fixed effects, and time aggregation of commodity prices. This paper finds no evidence of heterogeneity bias in annual CPI data for 17 U.S. cities from 1918 to 2006, but correcting for the “Nickell bias” and time aggregation bias produces a half-life of 7.5 years, shorter than estimates from previous studies.

A forced titration study of the antioxidant and immunomodulatory effects of Ambrotose AO supplement

Background Oxidative stress plays a role in acute and chronic inflammatory disease and antioxidant supplementation has demonstrated beneficial effects in the treatment of these conditions. This study was designed to determine the optimal dose of an antioxidant supplement in healthy volunteers to inform a Phase 3 clinical trial. Methods The study was designed as a combined Phase 1 and 2 open label, forced titration dose response study in healthy volunteers (n = 21) to determine both acute safety and efficacy. Participants received a dietary supplement in a forced titration over five weeks commencing with a no treatment baseline through 1, 2, 4 and 8 capsules. The primary outcome measurement was ex vivo changes in serum oxygen radical absorbance capacity (ORAC). The secondary outcome measures were undertaken as an exploratory investigation of immune function. Results A significant increase in antioxidant activity (serum ORAC) was observed between baseline (no capsules) and the highest dose of 8 capsules per day (p = 0.040) representing a change of 36.6%. A quadratic function for dose levels was fitted in order to estimate a dose response curve for estimating the optimal dose. The quadratic component of the curve was significant (p = 0.047), with predicted serum ORAC scores increasing from the zero dose to a maximum at a predicted dose of 4.7 capsules per day and decreasing for higher doses. Among the secondary outcome measures, a significant dose effect was observed on phagocytosis of granulocytes, and a significant increase was also observed on Cox 2 expression. Conclusion This study suggests that Ambrotose AO® capsules appear to be safe and most effective at a dosage of 4 capsules/day. It is important that this study is not over interpreted; it aimed to find an optimal dose to assess the dietary supplement using a more rigorous clinical trial design. The study achieved this aim and demonstrated that the dietary supplement has the potential to increase antioxidant activity. The most significant limitation of this study was that it was open label Phase 1/Phase 2 trial and is subject to potential bias that is reduced with the use of randomization and blinding. To confirm the benefits of this dietary supplement these effects now need to be demonstrated in a Phase 3 randomised controlled trial (RCT).

REGAL : a regularization based algorithm for reinforcement learning in weakly communicating MDPs

We provide an algorithm that achieves the optimal regret rate in an unknown weakly communicating Markov Decision Process (MDP). The algorithm proceeds in episodes where, in each episode, it picks a policy using regularization based on the span of the optimal bias vector. For an MDP with S states and A actions whose optimal bias vector has span bounded by H, we show a regret bound of ~ O(HS p AT ). We also relate the span to various diameter-like quantities associated with the MDP, demonstrating how our results improve on previous regret bounds.

Multicultural education : the understandings of preschool teachers in Singapore

For multicultural education to be effective, it must be introduced in the early years and reinforced throughout the school years. It is important for teachers of young children to be equipped with the appropriate knowledge, skills and positive attitudes to promote multiculturalism in their classrooms. This paper reports on a survey involving preschool teachers in Singapore who were participating in an in-service training course. The teachers completed a questionnaire to indicate their understandings of multicultural education and their perceptions of its importance in early childhood education. The findings indicated that the teachers endorsed the need for multicultural education in preschools and the importance of children learning tolerance and understanding of other cultures in the Singaporean context. However, while preschool teachers held positive attitudes towards multicultural education, there was limited understanding of the challenges in implementing an anti-bias teaching and curriculum approach. The implications of these findings for preschool teacher education programs in Singapore and in other countries focus on supporting teachers in developing more critical and deeper understandings of multiculturalism.

Visceral hedonic rhetoric

Emotional responses can incite and entice consumers to select a particular product from a row of similar items and thus have a considerable impact on purchase decisions. Consequently, more and more companies are challenging designers to address the emotional impact of their work and to design for emotion and consumerproduct relationships. Furthermore, the creation of emotional attachment to one’s possessions is one way of approaching a sustainable consumer-product relationship. The aim of this research is to gain a deeper understanding of the instantaneous emotional attachment that consumers form with products and its subsequent implications for product development. The foci of the study are visceral design, consumer hedonics and product rhetoric. Studied in a conglomerate they become an area of new investigation: visceral hedonic rhetoric. In this context, the term “visceral hedonic rhetoric” is defined as the properties of a product that persuasively elicit the pursuit of pleasure at an instinctual level of cognition. This study explores visceral hedonic rhetoric evident in the design of interactive products and resides within the context of emotional design research. It employs an empirical approach to understand how consumers respond hedonically on a visceral level to rhetoric in products. Specifically, it examines visceral hedonic responses given by thirty participants to the stimuli of six mobile telephones, six Mp3 players and six USB memory flash drives. The study findings demonstrate a hierarchy of visceral hedonic rhetoric evident in interactive products. This hierarchy of visceral hedonic attributes include: colour, size, shape, intrigue, material, perceived usability, portability, perceived function, novelty, analogy, brand, quality, texture and gender. However, it is the interrelationships between these visceral hedonic attributes that are the most significant findings of this research. Certain associations were revealed between product attribute combinations and consumer perception. The most predominant of these were: gender bias associated with colour selection; the creation of intrigue through a vibrant attention-grabbing colour; perceived ease of use and function; product confidence as a result of brand familiarity and perceived usability; analogous association through familiarity with similar objects and shapes; and the association of longevity with quality, novelty or recent technology. A significant outcome of the research is the distillation of visceral hedonic rhetoric design principles, and a tool to assist designers in harnessing the full potential of visceral hedonic rhetoric. This study contributes to the identification of the emerging research field of visceral hedonic rhetoric. Application of this study’s findings has the potential to provide a hedonic consumer-product relationship that is more meaningful, less disposable and more sustainable. This theory of visceral hedonic rhetoric is not only a significant contribution to design knowledge but is also generally transferable to other research domains, as later suggested in future research avenues.

A hydrogen gas sensor based on Pt/nanostructured WO3/SiC Schottky diode

Pt/nanostructured WO3/SiC Schottky diodes were fabricated and applied for hydrogen gas sensing applications. The nanostructured WO3 films were synthesized from tungsten coated SiC substrates via an acid-etching method using a 1.5 M HNO3 solution for 1 hr, 2 hrs and 3 hrs duration. Scanning electron microscopy of the developed films revealed platelet crystals with thicknesses in the order of 20-60 nm and lengths between 100-700 nm. X-ray diffraction analysis revealed that the rate of oxidation of tungsten increases as the duration of acid-etching increases. The devices were tested towards hydrogen gas balanced in air at different temperatures from 25°C to 200°C. At 200°C, voltage shifts of 0.45 V, 0.93 V and 2.37 V were recorded for devices acid-etched for 1 hr, 2 hrs and 3 hrs duration, respectively upon exposure to 1% hydrogen, under a constant forward bias current of 500 µA.

An ultraviolet selective photodetector based on a nanocrystalline TiO2 photoelectrochemical cell

Solar ultraviolet (UV) radiation causes a range of skin disorders as well as affecting vision and the immune system. It also inhibits development of plants and animals. UV radiation monitoring is used routinely in some locations in order to alert the population to harmful solar radiation levels. There is ongoing research to develop UV-selective-sensors [1–3]. A personal, inexpensive and simple UV-selective-sensor would be desirable to measure UV intensity exposure. A prototype of such a detector has been developed and evaluated in our laboratory. It comprises a sealed two-electrode photoelectrochemical cell (PEC) based on nanocrystalline TiO2. This abundant semiconducting oxide, which is innocuous and very sta-ble, is the subject of intense study at present due to its application in dye sensitized solar cells (DSSC) [4]. Since TiO2 has a wide band gap (EG = 3.0 eV for rutile and EG = 3.2 eV for anatase), it is inher-ently UV-selective, so that UV filters are not required. This further reduces the cost of the proposed photodetector in comparison with conventional silicon detectors. The PEC is a semiconductor–electrolyte device that generates a photovoltage when it is illuminated and a corresponding photocur-rent if the external circuit is closed. The device does not require external bias, and the short circuit current is generally a linear function of illumination intensity. This greatly simplifies the elec-trical circuit needed when using the PEC as a photodetector. DSSC technology, which is based on a PEC containing nanocrystalline TiO2 sensitized with a ruthenium dye, holds out the promise of solar cells that are significantly cheaper than traditional silicon solar cells. The UV-sensor proposed in this paper relies on the cre-ation of electron–hole pairs in the TiO2 by UV radiation, so that it would be even cheaper than a DSSC since no sensitizer dye is needed. Although TiO2 has been reported as a suitable material for UV sensing [3], to the best of our knowledge, the PEC configuration described in the present paper is a new approach. In the present study, a novel double-layer TiO2 structure has been investigated. Fabrication is based on a simple and inexpensive technique for nanostructured TiO2 deposition using microwave-activated chemical bath deposition (MW-CBD) that has been reported recently [5]. The highly transparent TiO2 (anatase) films obtained are densely packed, and they adhere very well to the transparent oxide (TCO) substrate [6]. These compact layers have been studied as contacting layers in double-layer TiO2 structures for DSSC since improvement of electron extraction at the TiO2–TCO interface is expected [7]. Here we compare devices incorporating a single mesoporous nanocrystalline TiO2 structure with devices based on a double structure in which a MW-CBD film is situated between the TCO and the mesoporous nanocrystalline TiO2 layer. Besides improving electron extraction, this film could also help to block recombination of electrons transferred to the TCO with oxidized species in the electrolyte, as has been reported in the case of DSSC for compact TiO2 films obtained by other deposition tech-niques [8,9]. The two types of UV-selective sensors were characterized in detail. The current voltage characteristics, spectral response, inten-sity dependence of short circuit current and response times were measured and analyzed in order to evaluate the potential of sealed mesoporous TiO2-based photoelectrochemical cells (PEC) as low cost personal UV-photodetectors.