Selection bias and the cross-validation of regression models for prediction

Strategies are compared for the development of a linear regression model with stochastic (multivariate normal) regressor variables and the subsequent assessment of its predictive ability. Bias and mean squared error of four estimators of predictive performance are evaluated in simulated samples of 32 population correlation matrices. Models including all of the available predictors are compared with those obtained using selected subsets. The subset selection procedures investigated include two stopping rules, C$\sb{\rm p}$ and S$\sb{\rm p}$, each combined with an 'all possible subsets' or 'forward selection' of variables. The estimators of performance utilized include parametric (MSEP$\sb{\rm m}$) and non-parametric (PRESS) assessments in the entire sample, and two data splitting estimates restricted to a random or balanced (Snee's DUPLEX) 'validation' half sample. The simulations were performed as a designed experiment, with population correlation matrices representing a broad range of data structures.^ The techniques examined for subset selection do not generally result in improved predictions relative to the full model. Approaches using 'forward selection' result in slightly smaller prediction errors and less biased estimators of predictive accuracy than 'all possible subsets' approaches but no differences are detected between the performances of C$\sb{\rm p}$ and S$\sb{\rm p}$. In every case, prediction errors of models obtained by subset selection in either of the half splits exceed those obtained using all predictors and the entire sample.^ Only the random split estimator is conditionally (on $\\beta$) unbiased, however MSEP$\sb{\rm m}$ is unbiased on average and PRESS is nearly so in unselected (fixed form) models. When subset selection techniques are used, MSEP$\sb{\rm m}$ and PRESS always underestimate prediction errors, by as much as 27 percent (on average) in small samples. Despite their bias, the mean squared errors (MSE) of these estimators are at least 30 percent less than that of the unbiased random split estimator. The DUPLEX split estimator suffers from large MSE as well as bias, and seems of little value within the context of stochastic regressor variables.^ To maximize predictive accuracy while retaining a reliable estimate of that accuracy, it is recommended that the entire sample be used for model development, and a leave-one-out statistic (e.g. PRESS) be used for assessment. ^

Use of a modified delphi process for establishing occupational exposure limits in support of health surveillance for space travelers

On-orbit exposures can come from numerous factors related to the space environment as evidenced by almost 50 years of environmental samples collected for water analysis, air analysis, radiation analysis, and physiologic parameters. For astronauts and spaceflight participants the occupational exposures can be very different from those experienced by workers performing similar tasks in workplaces on Earth, because the duration of the exposure could be continuous for very long orbital, and eventually interplanetary, missions. The establishment of long-term exposure standards is vital to controlling the quality of the spacecraft environment over long periods. NASA often needs to update and revise its prior exposure standards (Spacecrafts Maximum Allowable Concentrations (SMACs)). Traditional standards-setting processes are often lengthy, so a more rapid method to review and establish standards would be a substantial advancement in this area. This project investigates use of the Delphi method for this purpose. ^ In order to achieve the objectives of this study a modified Delphi methodology was tested in three trials executed by doctoral students and a panel of experts in disciplines related to occupational safety and health. During each test/trial modifications were made to the methodology. Prior to submission of the Delphi Questionnaire to the panel of experts a pilot study/trial was conducted using five doctoral students with the goals of testing and adjusting the Delphi questionnaire to improve comprehension, work out any procedural issues and evaluate the effectiveness of the questionnaire in drawing the desired responses. The remainder of the study consisted of two trials of the Modified Delphi process using 6 chemicals that currently have the potential of causing occupational exposures to NASA astronauts or spaceflight participants. To assist in setting Occupational Exposure Limits (OEL), the expert panel was established consisting of experts from academia, government and industry. Evidence was collected and used to create close-ended questionnaires which were submitted to the Delphi panel of experts for the establishment of OEL values for three chemicals from the list of six originally selected (trial 1). Once the first Delphi trial was completed, adjustments were made to the Delphi questionnaires and the process above was repeated with the remaining 3 chemicals (trial 2). ^ Results indicate that experience in occupational safety and health and with OEL methodologies can have a positive effect in minimizing the time experts take in completing this process. Based on the results of the questionnaires and comparison of the results with the SMAC already established by NASA, we conclude that use of the Delphi methodology is appropriate for use in the decision-making process for the selection of OELs.^