Selection bias and the cross-validation of regression models for prediction

Strategies are compared for the development of a linear regression model with stochastic (multivariate normal) regressor variables and the subsequent assessment of its predictive ability. Bias and mean squared error of four estimators of predictive performance are evaluated in simulated samples of 32 population correlation matrices. Models including all of the available predictors are compared with those obtained using selected subsets. The subset selection procedures investigated include two stopping rules, C$\sb{\rm p}$ and S$\sb{\rm p}$, each combined with an 'all possible subsets' or 'forward selection' of variables. The estimators of performance utilized include parametric (MSEP$\sb{\rm m}$) and non-parametric (PRESS) assessments in the entire sample, and two data splitting estimates restricted to a random or balanced (Snee's DUPLEX) 'validation' half sample. The simulations were performed as a designed experiment, with population correlation matrices representing a broad range of data structures.^ The techniques examined for subset selection do not generally result in improved predictions relative to the full model. Approaches using 'forward selection' result in slightly smaller prediction errors and less biased estimators of predictive accuracy than 'all possible subsets' approaches but no differences are detected between the performances of C$\sb{\rm p}$ and S$\sb{\rm p}$. In every case, prediction errors of models obtained by subset selection in either of the half splits exceed those obtained using all predictors and the entire sample.^ Only the random split estimator is conditionally (on $\\beta$) unbiased, however MSEP$\sb{\rm m}$ is unbiased on average and PRESS is nearly so in unselected (fixed form) models. When subset selection techniques are used, MSEP$\sb{\rm m}$ and PRESS always underestimate prediction errors, by as much as 27 percent (on average) in small samples. Despite their bias, the mean squared errors (MSE) of these estimators are at least 30 percent less than that of the unbiased random split estimator. The DUPLEX split estimator suffers from large MSE as well as bias, and seems of little value within the context of stochastic regressor variables.^ To maximize predictive accuracy while retaining a reliable estimate of that accuracy, it is recommended that the entire sample be used for model development, and a leave-one-out statistic (e.g. PRESS) be used for assessment. ^

Assessment of the effect on statistical power of regression model misspecification by using techniques of mathematical statistics and simulation study

Objectives. This paper seeks to assess the effect on statistical power of regression model misspecification in a variety of situations. ^ Methods and results. The effect of misspecification in regression can be approximated by evaluating the correlation between the correct specification and the misspecification of the outcome variable (Harris 2010).In this paper, three misspecified models (linear, categorical and fractional polynomial) were considered. In the first section, the mathematical method of calculating the correlation between correct and misspecified models with simple mathematical forms was derived and demonstrated. In the second section, data from the National Health and Nutrition Examination Survey (NHANES 2007-2008) were used to examine such correlations. Our study shows that comparing to linear or categorical models, the fractional polynomial models, with the higher correlations, provided a better approximation of the true relationship, which was illustrated by LOESS regression. In the third section, we present the results of simulation studies that demonstrate overall misspecification in regression can produce marked decreases in power with small sample sizes. However, the categorical model had greatest power, ranging from 0.877 to 0.936 depending on sample size and outcome variable used. The power of fractional polynomial model was close to that of linear model, which ranged from 0.69 to 0.83, and appeared to be affected by the increased degrees of freedom of this model.^ Conclusion. Correlations between alternative model specifications can be used to provide a good approximation of the effect on statistical power of misspecification when the sample size is large. When model specifications have known simple mathematical forms, such correlations can be calculated mathematically. Actual public health data from NHANES 2007-2008 were used as examples to demonstrate the situations with unknown or complex correct model specification. Simulation of power for misspecified models confirmed the results based on correlation methods but also illustrated the effect of model degrees of freedom on power.^

Assessment of cone beam computed tomography techniques for imaging lung damage in mice in vivo

Lung damage is a common side effect of chemotherapeutic drugs such as bleomycin. This study used a bleomycin mouse model which simulates the lung damage observed in humans. Noninvasive, in vivo cone-beam computed tomography (CBCT) was used to visualize and quantify fibrotic and inflammatory damage over the entire lung volume of mice. Bleomycin was used to induce pulmonary damage in vivo and the results from two CBCT systems, a micro-CT and flat panel CT (fpCT), were compared to histologic measurements, the standard method of murine lung damage quantification. Twenty C57BL/6 mice were given either 3 U/kg of bleomycin or saline intratracheally. The mice were scanned at baseline, before the administration of bleomycin, and then 10, 14, and 21 days afterward. At each time point, a subset of mice was sacrificed for histologic analysis. The resulting CT images were used to assess lung volume. Percent lung damage (PLD) was calculated for each mouse on both the fpCT (PLDfpcT) and the micro-CT (PLDμCT). Histologic PLD (PLDH) was calculated for each histologic section at each time point (day 10, n = 4; day 14, n = 4; day 21, n = 5; control group, n = 5). A linear regression was applied to the PLDfpCT vs. PLDH, PLDμCT vs. PLDH and PLDfpCT vs. PLDμCT distributions. This study did not demonstrate strong correlations between PLDCT and PLDH. The coefficient of determination, R, was 0.68 for PLDμCT vs. PLDH and 0.75 for the PLD fpCT vs. PLDH. The experimental issues identified from this study were: (1) inconsistent inflation of the lungs from scan to scan, (2) variable distribution of damage (one histologic section not representative of overall lung damage), (3) control mice not scanned with each group of bleomycin mice, (4) two CT systems caused long anesthesia time for the mice, and (5) respiratory gating did not hold the volume of lung constant throughout the scan. Addressing these issues might allow for further improvement of the correlation between PLDCT and PLDH. ^