Infinite-horizon policy-gradient estimation

Gradient-based approaches to direct policy search in reinforcement learning have received much recent attention as a means to solve problems of partial observability and to avoid some of the problems associated with policy degradation in value-function methods. In this paper we introduce GPOMDP, a simulation-based algorithm for generating a biased estimate of the gradient of the average reward in Partially Observable Markov Decision Processes (POMDPs) controlled by parameterized stochastic policies. A similar algorithm was proposed by Kimura, Yamamura, and Kobayashi (1995). The algorithm's chief advantages are that it requires storage of only twice the number of policy parameters, uses one free parameter β ∈ [0,1) (which has a natural interpretation in terms of bias-variance trade-off), and requires no knowledge of the underlying state. We prove convergence of GPOMDP, and show how the correct choice of the parameter β is related to the mixing time of the controlled POMDP. We briefly describe extensions of GPOMDP to controlled Markov chains, continuous state, observation and control spaces, multiple-agents, higher-order derivatives, and a version for training stochastic policies with internal states. In a companion paper (Baxter, Bartlett, & Weaver, 2001) we show how the gradient estimates generated by GPOMDP can be used in both a traditional stochastic gradient algorithm and a conjugate-gradient procedure to find local optima of the average reward. ©2001 AI Access Foundation and Morgan Kaufmann Publishers. All rights reserved.

Randomised controlled trial of a supervised exercise rehabilitation program for colorectal cancer survivors immediately after chemotherapy: study protocol

Background Colorectal cancer (CRC) diagnosis and the ensuing treatments can have a substantial impact on the physical and psychological health of survivors. As the number of CRC survivors increases, so too does the need to develop viable rehabilitation programs to help these survivors return to good health as quickly as possible. Exercise has the potential to address many of the adverse effects of CRC treatment; however, to date, the role of exercise in the rehabilitation of cancer patients immediately after the completion of treatment has received limited research attention. This paper presents the design of a randomised controlled trial which will evaluate the feasibility and efficacy of a 12-week supervised aerobic exercise program (ImPACT Program) on the physiological and psychological markers of rehabilitation, in addition to biomarkers of standard haematological outcomes and the IGF axis. Methods/Design Forty CRC patients will be recruited through oncology clinics and randomised to an exercise group or a usual care control group. Baseline assessment will take place within 4 weeks of the patient completing adjuvant chemotherapy treatment. The exercise program for patients in the intervention group will commence a week after the baseline assessment. The program consists of three supervised moderate-intensity aerobic exercise sessions per week for 12 weeks. All participants will have assessments at baseline (0 wks), mid-intervention (6 wks), post-intervention (12 wks) and at a 6-week follow-up (18 wks). Outcome measures include cardio-respiratory fitness, biomarkers associated with health and survival, and indices of fatigue and quality of life. Process measures are participants' acceptability of, adherence to, and compliance with the exercise program, in addition to the safety of the program. Discussion The results of this study will provide valuable insight into the role of supervised exercise in improving life after CRC. Additionally, process analyses will inform the feasibility of implementing the program in a population of CRC patients immediately after completing chemotherapy.

Investigating the cost-effectiveness of video telephone based support for newly diagnosed paediatric oncology patients and their families: design of a randomised controlled trial

Background Providing ongoing family centred support is an integral part of childhood cancer care. For families living in regional and remote areas, opportunities to receive specialist support are limited by the availability of health care professionals and accessibility, which is often reduced due to distance, time, cost and transport. The primary aim of this work is to investigate the cost-effectiveness of videotelephony to support regional and remote families returning home for the first time with a child newly diagnosed with cancer Methods/design We will recruit 162 paediatric oncology patients and their families to a single centre randomised controlled trial. Patients from regional and remote areas, classified by Accessibility/Remoteness Index of Australia (ARIA+) greater than 0.2, will be randomised to a videotelephone support intervention or a usual support control group. Metropolitan families (ARIA+ ≤ 0.2) will be recruited as an additional usual support control group. Families allocated to the videotelephone support intervention will have access to usual support plus education, communication, counselling and monitoring with specialist multidisciplinary team members via a videotelephone service for a 12-week period following first discharge home. Families in the usual support control group will receive standard care i.e., specialist multidisciplinary team members provide support either face-to-face during inpatient stays, outpatient clinic visits or home visits, or via telephone for families who live far away from the hospital. The primary outcome measure is parental health related quality of life as measured using the Medical Outcome Survey (MOS) Short Form SF-12 measured at baseline, 4 weeks, 8 weeks and 12 weeks. The secondary outcome measures are: parental informational and emotional support; parental perceived stress, parent reported patient quality of life and parent reported sibling quality of life, parental satisfaction with care, cost of providing improved support, health care utilisation and financial burden for families. Discussion This investigation will establish the feasibility, acceptability and cost-effectiveness of using videotelephony to improve the clinical and psychosocial support provided to regional and remote paediatric oncology patients and their families.

Towards the biofortification of banana fruit for enhanced micronutrient content

In Uganda, vitamin A deficiency (VAD) and iron deficiency anaemia (IDA) are major public health problems with between 15-32% of children under 5 years of age showing VAD and 73% being anaemic. This is largely due to the fact that the staple food crop of the country, banana, is low in pro-vitamin A and iron, therefore leading to dietary deficiencies. Although worldwide progress has been made to control VAD and IDA through supplementation, food fortification and diet diversification, their long term sustainability and impact in developing countries such as Uganda is limited. The approach taken by researchers at Queensland University of Technology (QUT), Australia, in collaboration with the National Agricultural Research Organization (NARO), Uganda, to address this problem, is to generate consumer acceptable banana varieties with significantly increased levels of pro-vitamin A and iron in the fruit using genetic engineering techniques. Such an approach requires the use of suitable, well characterised genes and promoters for targeted transgene expression. Recently, a new banana phytoene synthase gene (APsy2a) involved in the synthesis of pro-vitamin A (pVA) carotenoids was isolated from a high â-carotene banana (F’ei cv Asupina). In addition, sequences of banana ferritin, an iron storage protein, have been isolated from Cavendish banana. The aim of the research described in this thesis was to evaluate the function of these genes to assess their suitability for the biofortification of banana fruit. In addition, a range of banana-derived promoters were characterised to determine their suitability for controlling the expression of transgenes in banana fruit. Due to the time constraints involved with generating transgenic banana fruit, rice was used as the model crop to investigate the functionality of the banana-derived APsy2a and ferritin genes. Using Agrobacterium-mediated transformation, rice callus was transformed with APsy2a +/- the bacterial-derived carotene desaturase gene (CrtI) each under the control of the constitutive maize poly-ubiquitin promoter (ZmUbi) or seed-specific rice glutelin1 (Gt1) promoter. The maize phytoene synthase (ZmPsy1) gene was included as a control. On selective media, with the exception of ZmUbi-CrtI-transgenic callus, all antibiotic resistant callus displayed a yellow-orange colour from which the presence of â-carotene was demonstrated using Raman spectroscopy. Although the regeneration of plants from yellow-orange callus was difficult, 16 transgenic plants were obtained and characterised from callus transformed with ZmUbi-APys2a alone. At least 50% of the T1 seeds developed a yellow-orange coloured callus which was found to contain levels of â-carotene ranging from 4.6-fold to 72-fold higher than that in non-transgenic rice callus. Using the seed-specific Gt1 promoter, 38 transgenic rice plants were generated from APsy2a-CrtI-transformed callus while 32 plants were regenerated from ZmPsy1-CrtI-transformed callus. However, when analysed for presence of transgene by PCR, all transgenic plants contained the APsy2a, ZmPsy1 or CrtI transgene, with none of the plants found to be co-transformed. Using Raman spectroscopy, no â-carotene was detected in-situ in representative T1 seeds. To investigate the potential of the banana-derived ferritin gene (BanFer1) to enhance iron content, rice callus was transformed with constitutively expressed BanFer1 using the soybean ferritin gene (SoyFer) as a control. A total of 12 and 11 callus lines independently transformed with BanFer1 and SoyFer, respectively, were multiplied and transgene expression was verified by RT-PCR. Pearl’s Prussian blue staining for in-situ detection of ferric iron showed a stronger blue colour in rice callus transformed with BanFer1 compared to SoyFer. Using flame atomic absorption spectrometry, the highest mean amount of iron quantified in callus transformed with BanFer1 was 30-fold while that obtained using the SoyFer was 14-fold higher than the controls. In addition, ~78% of BanFer1-transgenic callus lines and ~27% of SoyFer-transgenic callus lines had significantly higher iron content than the non-transformed controls. Since the genes used for enhancing micronutrient content need to be expressed in banana fruit, the activity of a range of banana-derived, potentially fruit-active promoters in banana was investigated. Using uidA (GUS) as a reporter gene, the function of the Expansin1 (MaExp1), Expansin1 containing the rice actin intron (MaExp1a), Expansin4 (MaExp4), Extensin (MaExt), ACS (MaACS), ACO (MaACO), Metallothionein (MaMT2a) and phytoene synthase (APsy2a) promoters were transiently analysed in intact banana fruit using two transformation methods, particle bombardment and Agrobacterium-mediated infiltration (agro-infiltration). Although a considerable amount of variation in promoter activity was observed both within and between experiments, similar trends were obtained using both transformation methods. The MaExp1 and MaExp1a directed high levels of GUS expression in banana fruit which were comparable to those observed from the ZmUbi and Banana bunchy top virus-derived BT4 promoters that were included as positive controls. Lower levels of promoter activity were obtained in both methods using the MaACO and MaExt promoters while the MaExp4, MaACS, and APsy2a promoters directed the lowest GUS activity in banana fruit. An attempt was subsequently made to use agro-infiltration to assess the expression of pVA biosynthesis genes in banana fruit by infiltrating fruit with constructs in which the ZmUbi promoter controlled the expression of APsy2a +/- CrtI, and with the maize phytoene synthase gene (ZmPsy1) included as a control. Unfortunately, the large amount of variation and inconsistency observed within and between experiments precluded any meaningful conclusions to be drawn. The final component of this research was to assess the level of promoter activity and specificity in non-target tissue. These analyses were done on leaves obtained from glasshouse-grown banana plants stably transformed with MaExp1, MaACO, APsy2a, BT4 and ZmUbi promoters driving the expression of the GUS gene in addition to leaves from a selection of the same transgenic plants which were growing in a field trial in North Queensland. The results from both histochemical and fluorometric GUS assays showed that the MaExp1 and MaACO promoters directed very low GUS activities in leaves of stably transformed banana plants compared to the constitutive ZmUbi and BT4 promoters. In summary, the results from this research provide evidence that the banana phytoene synthase gene (APsy2a) and the banana ferritin gene (BanFer1) are functional, since the constitutive over-expression of each of these transgenes led to increased levels of pVA carotenoids (for APsy2a) and iron content (for BanFer1) in transgenic rice callus. Further work is now required to determine the functionality of these genes in stably-transformed banana fruit. This research also demonstrated that the MaExp1 and MaACO promoters are fruit-active but have low activity in non-target tissue (leaves), characteristics that make them potentially useful for the biofortification of banana fruit. Ultimately, however, analysis of fruit from field-grown transgenic plants will be required to fully evaluate the suitability of pVA biosynthesis genes and the fruit-active promoters for fruit biofortification.

Mindfulness-based stress management: Findings from randomized controlled trials

The use of Mindfulness has become increasingly popular in the field of psychotherapy and counselling, and is now finding application in unexpected places. Is it just another therapeutic fad or an evidence-based intervention with lasting value? ROBERT KING provides a timely review of the research literature to provide a summary of the findings of randomised control trials of the clinical efficacy or effectiveness of Mindfulness as a stress reduction intervention.

Development of an autonomous unmanned aerial system to collect time-stamped samples from the atmosphere and localize potential pathogen sources

This paper presents the hardware development and testing of a new concept for air sampling via the integration of a prototype spore trap onboard an unmanned aerial system (UAS).We propose the integration of a prototype spore trap onboard a UAS to allow multiple capture of spores of pathogens in single remote locations at high or low altitude, otherwise not possible with stationary sampling devices.We also demonstrate the capability of this system for the capture of multiple time-stamped samples during a single mission.Wind tunnel testing was followed by simulation, and flight testing was conducted to measure and quantify the spread during simulated airborne air sampling operations. During autonomous operations, the onboard autopilot commands the servo to rotate the sampling device to a new indexed location once the UAS vehicle reaches the predefined waypoint or set of waypoints (which represents the region of interest). Time-stamped UAS data are continuously logged during the flight to assist with analysis of the particles collected. Testing and validation of the autopilot and spore trap integration, functionality, and performance is described. These tools may enhance the ability to detect new incursions of spores