Infinite-horizon policy-gradient estimation

Gradient-based approaches to direct policy search in reinforcement learning have received much recent attention as a means to solve problems of partial observability and to avoid some of the problems associated with policy degradation in value-function methods. In this paper we introduce GPOMDP, a simulation-based algorithm for generating a biased estimate of the gradient of the average reward in Partially Observable Markov Decision Processes (POMDPs) controlled by parameterized stochastic policies. A similar algorithm was proposed by Kimura, Yamamura, and Kobayashi (1995). The algorithm's chief advantages are that it requires storage of only twice the number of policy parameters, uses one free parameter β ∈ [0,1) (which has a natural interpretation in terms of bias-variance trade-off), and requires no knowledge of the underlying state. We prove convergence of GPOMDP, and show how the correct choice of the parameter β is related to the mixing time of the controlled POMDP. We briefly describe extensions of GPOMDP to controlled Markov chains, continuous state, observation and control spaces, multiple-agents, higher-order derivatives, and a version for training stochastic policies with internal states. In a companion paper (Baxter, Bartlett, & Weaver, 2001) we show how the gradient estimates generated by GPOMDP can be used in both a traditional stochastic gradient algorithm and a conjugate-gradient procedure to find local optima of the average reward. ©2001 AI Access Foundation and Morgan Kaufmann Publishers. All rights reserved.

Decision support system for contractor pre-qualification : artificial neural network model

The selection criteria for contractor pre-qualification are characterized by the co-existence of both quantitative and qualitative data. The qualitative data is non-linear, uncertain and imprecise. An ideal decision support system for contractor pre-qualification should have the ability of handling both quantitative and qualitative data, and of mapping the complicated nonlinear relationship of the selection criteria, such that rational and consistent decisions can be made. In this research paper, an artificial neural network model was developed to assist public clients identifying suitable contractors for tendering. The pre-qualification criteria (variables) were identified for the model. One hundred and twelve real pre-qualification cases were collected from civil engineering projects in Hong Kong, and eighty-eight hypothetical pre-qualification cases were also generated according to the “If-then” rules used by professionals in the pre-qualification process. The results of the analysis totally comply with current practice (public developers in Hong Kong). Each pre-qualification case consisted of input ratings for candidate contractors’ attributes and their corresponding pre-qualification decisions. The training of the neural network model was accomplished by using the developed program, in which a conjugate gradient descent algorithm was incorporated for improving the learning performance of the network. Cross-validation was applied to estimate the generalization errors based on the “re-sampling” of training pairs. The case studies show that the artificial neural network model is suitable for mapping the complicated nonlinear relationship between contractors’ attributes and their corresponding pre-qualification (disqualification) decisions. The artificial neural network model can be concluded as an ideal alternative for performing the contractor pre-qualification task.

Alleviating ‘overfitting’ via genetically-regularised neural network

A hybrid genetic algorithm/scaled conjugate gradient regularisation method is designed to alleviate ANN `over-fitting'. In application to day-ahead load forecasting, the proposed algorithm performs better than early-stopping and Bayesian regularisation, showing promising initial results.

Lateral buckling strength of simply supported LiteSteel beams subject to moment gradient effects

The flexural capacity of of a new cold-formed hollow flange channel section known as LiteSteel beam (LSB) is limited by lateral distortional buckling for intermediate spans, which is characterised by simultaneous lateral deflection, twist and web distortion. Recent research has developed suitable design rules for the member capacity of LSBs. However, they are limited to a uniform moment distribution that rarely exists in practice. Many steel design codes have adopted equivalent uniform moment distribution factors to accommodate the effect of non-uniform moment distributions in design. But they were derived mostly based on the data for conventional hot-rolled, doubly symmetric I-beams subject to lateral torsional buckling. The effect of moment distribution for LSBs, and the suitability of the current steel design code rules to include this effect for LSBs are not yet known. This paper presents the details of a research study based on finite element analyses of the lateral buckling strength of simply supported LSBs subject to moment gradient effects. It also presents the details of a number of LSB lateral buckling experiments undertaken to validate the results of finite element analyses. Finally, it discusses the suitability of the current design methods, and provides design recommendations for simply supported LSBs subject to moment gradient effects.

Exponentiated gradient algorithms for conditional random fields and max-margin Markov Networks

Log-linear and maximum-margin models are two commonly-used methods in supervised machine learning, and are frequently used in structured prediction problems. Efficient learning of parameters in these models is therefore an important problem, and becomes a key factor when learning from very large data sets. This paper describes exponentiated gradient (EG) algorithms for training such models, where EG updates are applied to the convex dual of either the log-linear or max-margin objective function; the dual in both the log-linear and max-margin cases corresponds to minimizing a convex function with simplex constraints. We study both batch and online variants of the algorithm, and provide rates of convergence for both cases. In the max-margin case, O(1/ε) EG updates are required to reach a given accuracy ε in the dual; in contrast, for log-linear models only O(log(1/ε)) updates are required. For both the max-margin and log-linear cases, our bounds suggest that the online EG algorithm requires a factor of n less computation to reach a desired accuracy than the batch EG algorithm, where n is the number of training examples. Our experiments confirm that the online algorithms are much faster than the batch algorithms in practice. We describe how the EG updates factor in a convenient way for structured prediction problems, allowing the algorithms to be efficiently applied to problems such as sequence learning or natural language parsing. We perform extensive evaluation of the algorithms, comparing them to L-BFGS and stochastic gradient descent for log-linear models, and to SVM-Struct for max-margin models. The algorithms are applied to a multi-class problem as well as to a more complex large-scale parsing task. In all these settings, the EG algorithms presented here outperform the other methods.

Does a 10-valent pneumococcal-Haemophilus influenzae protein D conjugate vaccine prevent respiratory exacerbations in children with recurrent protracted bacterial bronchitis, chronic suppurative lung disease and bronchiectasis : protocol for a randomised controlled trial

Background Recurrent protracted bacterial bronchitis (PBB), chronic suppurative lung disease (CSLD) and bronchiectasis are characterised by a chronic wet cough and are important causes of childhood respiratory morbidity globally. Haemophilus influenzae and Streptococcus pneumoniae are the most commonly associated pathogens. As respiratory exacerbations impair quality of life and may be associated with disease progression, we will determine if the novel 10-valent pneumococcal-Haemophilus influenzae protein D conjugate vaccine (PHiD-CV) reduces exacerbations in these children. Methods A multi-centre, parallel group, double-blind, randomised controlled trial in tertiary paediatric centres from three Australian cities is planned. Two hundred six children aged 18 months to 14 years with recurrent PBB, CSLD or bronchiectasis will be randomised to receive either two doses of PHiD-CV or control meningococcal (ACYW(135)) conjugate vaccine 2 months apart and followed for 12 months after the second vaccine dose. Randomisation will be stratified by site, age (<6 years and >= 6 years) and aetiology (recurrent PBB or CSLD/bronchiectasis). Clinical histories, respiratory status (including spirometry in children aged >= 6 years), nasopharyngeal and saliva swabs, and serum will be collected at baseline and at 2, 3, 8 and 14 months post-enrolment. Local and systemic reactions will be recorded on daily diaries for 7 and 30 days, respectively, following each vaccine dose and serious adverse events monitored throughout the trial. Fortnightly, parental contact will help record respiratory exacerbations. The primary outcome is the incidence of respiratory exacerbations in the 12 months following the second vaccine dose. Secondary outcomes include: nasopharyngeal carriage of H. influenzae and S. pneumoniae vaccine and vaccine-related serotypes; systemic and mucosal immune responses to H. influenzae proteins and S. pneumoniae vaccine and vaccine-related serotypes; impact upon lung function in children aged >= 6 years; and vaccine safety. Discussion As H. influenzae is the most common bacterial pathogen associated with these chronic respiratory diseases in children, a novel pneumococcal conjugate vaccine that also impacts upon H. influenzae and helps prevent respiratory exacerbations would assist clinical management with potential short- and long-term health benefits. Our study will be the first to assess vaccine efficacy targeting H. influenzae in children with recurrent PBB, CSLD and bronchiectasis.

Effectiveness of 7-valent pneumococcal conjugate vaccine against radiologically diagnosed pneumonia in indigenous infants in Australia

Objective To evaluate the effectiveness of the 7-valent pneumococcal conjugate vaccine (PCV7) in preventing pneumonia, diagnosed radiologically according to World Health Organization (WHO) criteria, among indigenous infants in the Northern Territory of Australia. Methods We conducted a historical cohort study of consecutive indigenous birth cohorts between 1 April 1998 and 28 February 2005. Children were followed up to 18 months of age. The PCV7 programme commenced on 1 June 2001. All chest X-rays taken within 3 days of any hospitalization were assessed. The primary endpoint was a first episode of WHO-defined pneumonia requiring hospitalization. Cox proportional hazards models were used to compare disease incidence. Findings There were 526 pneumonia events among 10 600 children - an incidence of 3.3 per 1000 child-months; 183 episodes (34.8%) occurred before 5 months of age and 247 (47.0%) by 7 months. Of the children studied, 27% had received 3 doses of vaccine by 7 months of age. Hazard ratios for endpoint pneumonia were 1.01 for 1 versus 0 doses; 1.03 for 2 versus 0 doses; and 0.84 for 3 versus 0 doses. Conclusion There was limited evidence that PCV7 reduced the incidence of radiologically confirmed pneumonia among Northern Territory indigenous infants, although there was a non-significant trend towards an effect after receipt of the third dose. These findings might be explained by lack of timely vaccination and/or occurrence of disease at an early age. Additionally, the relative contribution of vaccine-type pneumococcus to severe pneumonia in a setting where multiple other pathogens are prevalent may differ with respect to other settings where vaccine efficacy has been clearly established.

Development of a transient gradient enhanced non local continuum damage mechanics model for masonry

Due to the advent of varied types of masonry systems a comprehensive failure mechanism of masonry essential for the understanding of its behaviour is impossible to be determined from experimental testing. As masonry is predominantly used in wall structures a biaxial stress state dominates its failure mechanism. Biaxial testing will therefore be necessary for each type of masonry, which is expensive and time consuming. A computational method would be advantageous; however masonry is complex to model which requires advanced computational modelling methods. This thesis has formulated a damage mechanics inspired modelling method and has shown that the method effectively determines the failure mechanisms and deformation characteristics of masonry under biaxial states of loading.