Validation of the Asthma Predictive Index and comparison with simpler clinical prediction rules

Background The loose and stringent Asthma Predictive Indices (API), developed in Tucson, are popular rules to predict asthma in preschool children. To be clinically useful, they require validation in different settings. Objective To assess the predictive performance of the API in an independent population and compare it with simpler rules based only on preschool wheeze. Methods We studied 1954 children of the population-based Leicester Respiratory Cohort, followed up from age 1 to 10 years. The API and frequency of wheeze were assessed at age 3 years, and we determined their association with asthma at ages 7 and 10 years by using logistic regression. We computed test characteristics and measures of predictive performance to validate the API and compare it with simpler rules. Results The ability of the API to predict asthma in Leicester was comparable to Tucson: for the loose API, odds ratios for asthma at age 7 years were 5.2 in Leicester (5.5 in Tucson), and positive predictive values were 26% (26%). For the stringent API, these values were 8.2 (9.8) and 40% (48%). For the simpler rule early wheeze, corresponding values were 5.4 and 21%; for early frequent wheeze, 6.7 and 36%. The discriminative ability of all prediction rules was moderate (c statistic ≤ 0.7) and overall predictive performance low (scaled Brier score < 20%). Conclusion Predictive performance of the API in Leicester, although comparable to the original study, was modest and similar to prediction based only on preschool wheeze. This highlights the need for better prediction rules.

Prediction of 1-year mortality in patients with acute coronary syndromes undergoing percutaneous coronary intervention: validation of the logistic clinical SYNTAX (Synergy Between Percutaneous Coronary Interventions With Taxus and Cardiac Surgery) score

OBJECTIVES This study sought to validate the Logistic Clinical SYNTAX (Synergy Between Percutaneous Coronary Intervention With Taxus and Cardiac Surgery) score in patients with non-ST-segment elevation acute coronary syndromes (ACS), in order to further legitimize its clinical application. BACKGROUND The Logistic Clinical SYNTAX score allows for an individualized prediction of 1-year mortality in patients undergoing contemporary percutaneous coronary intervention. It is composed of a "Core" Model (anatomical SYNTAX score, age, creatinine clearance, and left ventricular ejection fraction), and "Extended" Model (composed of an additional 6 clinical variables), and has previously been cross validated in 7 contemporary stent trials (>6,000 patients). METHODS One-year all-cause death was analyzed in 2,627 patients undergoing percutaneous coronary intervention from the ACUITY (Acute Catheterization and Urgent Intervention Triage Strategy) trial. Mortality predictions from the Core and Extended Models were studied with respect to discrimination, that is, separation of those with and without 1-year all-cause death (assessed by the concordance [C] statistic), and calibration, that is, agreement between observed and predicted outcomes (assessed with validation plots). Decision curve analyses, which weight the harms (false positives) against benefits (true positives) of using a risk score to make mortality predictions, were undertaken to assess clinical usefulness. RESULTS In the ACUITY trial, the median SYNTAX score was 9.0 (interquartile range 5.0 to 16.0); approximately 40% of patients had 3-vessel disease, 29% diabetes, and 85% underwent drug-eluting stent implantation. Validation plots confirmed agreement between observed and predicted mortality. The Core and Extended Models demonstrated substantial improvements in the discriminative ability for 1-year all-cause death compared with the anatomical SYNTAX score in isolation (C-statistics: SYNTAX score: 0.64, 95% confidence interval [CI]: 0.56 to 0.71; Core Model: 0.74, 95% CI: 0.66 to 0.79; Extended Model: 0.77, 95% CI: 0.70 to 0.83). Decision curve analyses confirmed the increasing ability to correctly identify patients who would die at 1 year with the Extended Model versus the Core Model versus the anatomical SYNTAX score, over a wide range of thresholds for mortality risk predictions. CONCLUSIONS Compared to the anatomical SYNTAX score alone, the Core and Extended Models of the Logistic Clinical SYNTAX score more accurately predicted individual 1-year mortality in patients presenting with non-ST-segment elevation acute coronary syndromes undergoing percutaneous coronary intervention. These findings support the clinical application of the Logistic Clinical SYNTAX score.

Development and validation of GT3X accelerometer cut-off points in 5- to 9-year-old children based on indirect calorimetry measurements

The ActiGraph accelerometer is commonly used to measure physical activity in children. Count cut-off points are needed when using accelerometer data to determine the time a person spent in moderate or vigorous physical activity. For the GT3X accelerometer no cut-off points for young children have been published yet. The aim of the current study was thus to develop and validate count cut-off points for young children. Thirty-two children aged 5 to 9 years performed four locomotor and four play activities. Activity classification into the light-, moderate- or vigorous-intensity category was based on energy expenditure measurements with indirect calorimetry. Vertical axis as well as vector magnitude cut-off points were determined through receiver operating characteristic curve analyses with the data of two thirds of the study group and validated with the data of the remaining third. The vertical axis cut-off points were 133 counts per 5 sec for moderate to vigorous physical activity (MVPA), 193 counts for vigorous activity (VPA) corresponding to a metabolic threshold of 5 MET and 233 for VPA corresponding to 6 MET. The vector magnitude cut-off points were 246 counts per 5 sec for MVPA, 316 counts for VPA - 5 MET and 381 counts for VPA - 6 MET. When validated, the current cut-off points generally showed high recognition rates for each category, high sensitivity and specificity values and moderate agreement in terms of the Kappa statistic. These results were similar for vertical axis and vector magnitude cut-off points. The current cut-off points adequately reflect MVPA and VPA in young children. Cut-off points based on vector magnitude counts did not appear to reflect the intensity categories better than cut-off points based on vertical axis counts alone.

Potentially avoidable 30-day hospital readmissions in medical patients: derivation and validation of a prediction model

IMPORTANCE Because effective interventions to reduce hospital readmissions are often expensive to implement, a score to predict potentially avoidable readmissions may help target the patients most likely to benefit. OBJECTIVE To derive and internally validate a prediction model for potentially avoidable 30-day hospital readmissions in medical patients using administrative and clinical data readily available prior to discharge. DESIGN Retrospective cohort study. SETTING Academic medical center in Boston, Massachusetts. PARTICIPANTS All patient discharges from any medical services between July 1, 2009, and June 30, 2010. MAIN OUTCOME MEASURES Potentially avoidable 30-day readmissions to 3 hospitals of the Partners HealthCare network were identified using a validated computerized algorithm based on administrative data (SQLape). A simple score was developed using multivariable logistic regression, with two-thirds of the sample randomly selected as the derivation cohort and one-third as the validation cohort. RESULTS Among 10 731 eligible discharges, 2398 discharges (22.3%) were followed by a 30-day readmission, of which 879 (8.5% of all discharges) were identified as potentially avoidable. The prediction score identified 7 independent factors, referred to as the HOSPITAL score: h emoglobin at discharge, discharge from an o ncology service, s odium level at discharge, p rocedure during the index admission, i ndex t ype of admission, number of a dmissions during the last 12 months, and l ength of stay. In the validation set, 26.7% of the patients were classified as high risk, with an estimated potentially avoidable readmission risk of 18.0% (observed, 18.2%). The HOSPITAL score had fair discriminatory power (C statistic, 0.71) and had good calibration. CONCLUSIONS AND RELEVANCE This simple prediction model identifies before discharge the risk of potentially avoidable 30-day readmission in medical patients. This score has potential to easily identify patients who may need more intensive transitional care interventions.

External validation of the Hospital-patient One-year Mortality Risk (HOMR) model for predicting death within 1 year after hospital admission

BACKGROUND Predicting long-term survival after admission to hospital is helpful for clinical, administrative and research purposes. The Hospital-patient One-year Mortality Risk (HOMR) model was derived and internally validated to predict the risk of death within 1 year after admission. We conducted an external validation of the model in a large multicentre study. METHODS We used administrative data for all nonpsychiatric admissions of adult patients to hospitals in the provinces of Ontario (2003-2010) and Alberta (2011-2012), and to the Brigham and Women's Hospital in Boston (2010-2012) to calculate each patient's HOMR score at admission. The HOMR score is based on a set of parameters that captures patient demographics, health burden and severity of acute illness. We determined patient status (alive or dead) 1 year after admission using population-based registries. RESULTS The 3 validation cohorts (n = 2,862,996 in Ontario, 210 595 in Alberta and 66,683 in Boston) were distinct from each other and from the derivation cohort. The overall risk of death within 1 year after admission was 8.7% (95% confidence interval [CI] 8.7% to 8.8%). The HOMR score was strongly and significantly associated with risk of death in all populations and was highly discriminative, with a C statistic ranging from 0.89 (95% CI 0.87 to 0.91) to 0.92 (95% CI 0.91 to 0.92). Observed and expected outcome risks were similar (median absolute difference in percent dying in 1 yr 0.3%, interquartile range 0.05%-2.5%). INTERPRETATION The HOMR score, calculated using routinely collected administrative data, accurately predicted the risk of death among adult patients within 1 year after admission to hospital for nonpsychiatric indications. Similar performance was seen when the score was used in geographically and temporally diverse populations. The HOMR model can be used for risk adjustment in analyses of health administrative data to predict long-term survival among hospital patients.

Selection bias and the cross-validation of regression models for prediction

Strategies are compared for the development of a linear regression model with stochastic (multivariate normal) regressor variables and the subsequent assessment of its predictive ability. Bias and mean squared error of four estimators of predictive performance are evaluated in simulated samples of 32 population correlation matrices. Models including all of the available predictors are compared with those obtained using selected subsets. The subset selection procedures investigated include two stopping rules, C$\sb{\rm p}$ and S$\sb{\rm p}$, each combined with an 'all possible subsets' or 'forward selection' of variables. The estimators of performance utilized include parametric (MSEP$\sb{\rm m}$) and non-parametric (PRESS) assessments in the entire sample, and two data splitting estimates restricted to a random or balanced (Snee's DUPLEX) 'validation' half sample. The simulations were performed as a designed experiment, with population correlation matrices representing a broad range of data structures.^ The techniques examined for subset selection do not generally result in improved predictions relative to the full model. Approaches using 'forward selection' result in slightly smaller prediction errors and less biased estimators of predictive accuracy than 'all possible subsets' approaches but no differences are detected between the performances of C$\sb{\rm p}$ and S$\sb{\rm p}$. In every case, prediction errors of models obtained by subset selection in either of the half splits exceed those obtained using all predictors and the entire sample.^ Only the random split estimator is conditionally (on $\\beta$) unbiased, however MSEP$\sb{\rm m}$ is unbiased on average and PRESS is nearly so in unselected (fixed form) models. When subset selection techniques are used, MSEP$\sb{\rm m}$ and PRESS always underestimate prediction errors, by as much as 27 percent (on average) in small samples. Despite their bias, the mean squared errors (MSE) of these estimators are at least 30 percent less than that of the unbiased random split estimator. The DUPLEX split estimator suffers from large MSE as well as bias, and seems of little value within the context of stochastic regressor variables.^ To maximize predictive accuracy while retaining a reliable estimate of that accuracy, it is recommended that the entire sample be used for model development, and a leave-one-out statistic (e.g. PRESS) be used for assessment. ^

Validation of a new easy score to predict outcome in early stage renal cell carcinoma

Background e scopi dello studio. Il carcinoma renale rappresenta circa il 3% delle neoplasie e la sua incidenza è in aumento nel mondo. Il principale approccio terapeutico alla malattia in stadio precoce è rappresentato dalla chirurgia (nefrectomia parziale o radicale), sebbene circa il 30-40% dei pazienti vada incontro a recidiva di malattia dopo tale trattamento. La probabilità di recidivare può essere stimata per mezzo di alcuni noti modelli prognostici sviluppati integrando sia parametri clinici che anatomo-patologici. Il limite principale all’impiego nella pratica clinica di questi modelli è legata alla loro complessità di calcolo che li rende di difficile fruizione. Inoltre la stratificazione prognostica dei pazienti in questo ambito ha un ruolo rilevante nella pianificazione ed interpretazione dei risultati degli studi di terapia adiuvante dopo il trattamento chirurgico del carcinoma renale in stadio iniziale. Da un' analisi non pre-pianificata condotta nell’ambito di uno studio prospettico e randomizzato multicentrico italiano di recente pubblicazione, è stato sviluppato un nuovo modello predittivo e prognostico (“score”) che utilizza quattro semplici parametri: l’età del paziente, il grading istologico, lo stadio patologico del tumore (pT) e della componente linfonodale (pN). Lo scopo del presente studio era quello di validare esternamente tale score. Pazienti e Metodi. La validazione è stata condotta su due coorti retrospettive italiane (141 e 246 pazienti) e su una prospettica americana (1943 pazienti). Lo score testato prevedeva il confronto tra due gruppi di pazienti, uno a prognosi favorevole (pazienti con almeno due parametri positivi tra i seguenti: età < 60 anni, pT1-T3a, pN0, grading 1-2) e uno a prognosi sfavorevole (pazienti con meno di due fattori positivi). La statistica descrittiva è stata utilizzata per mostrare la distribuzione dei diversi parametri. Le analisi di sopravvivenza [recurrence free survival (RFS) e overall survival (OS)] sono state eseguite il metodo di Kaplan-Meier e le comparazioni tra i vari gruppi di pazienti sono state condotte utilizzando il Mantel-Haenszel log-rank test e il modello di regressione di Cox. Il metodo di Greenwood è stato utilizzato per stimare la varianza e la costruzione degli intervalli di confidenza al 95% (95% CI), la “C-statistic” è stata utilizzata per descrivere l’ accuratezza dello score. Risultati. I risultati della validazione dello score condotta sulle due casistiche retrospettive italiane, seppur non mostrando una differenza statisticamente significativa tra i due gruppi di pazienti (gruppo favorevole versus sfavorevole), sono stati ritenuti incoraggianti e meritevoli di ulteriore validazione sulla casistica prospettica americana. Lo score ha dimostrato di performare bene sia nel determinare la prognosi in termini di RFS [hazard ratio (HR) 1.85, 95% CI 1.57-2.17, p < 0.001] che di OS [HR 2.58, 95% CI 1.98-3.35, p < 0.001]. Inoltre in questa casistica lo score ha realizzato risultati sovrapponibili a quelli dello University of California Los Angeles Integrated Staging System. Conclusioni. Questo nuovo e semplice score ha dimostrato la sua validità in altre casistiche, sia retrospettive che prospettiche, in termini di impatto prognostico su RFS e OS. Ulteriori validazioni su casistiche internazionali sono in corso per confermare i risultati qui presentati e per testare l’eventuale ruolo predittivo di questo nuovo score.

Validation and refinement of survival models for liver retransplantation

Orthotopic liver retransplantation (re-OLT) is highly controversial. The objectives of this study were to determine the validity of a recently developed United Network for Organ Sharing (UNOS) multivariate model using an independent cohort of patients undergoing re-OLT outside the United States, to determine whether incorporation of other variables that were incomplete in the UNOS registry would provide additional prognostic information, to develop new models combining data sets from both cohorts, and to evaluate the validity of the model for end-stage liver disease (MELD) in patients undergoing re-OLT. Two hundred eighty-one adult patients undergoing re-OLT (between 1986 and 1999) at 6 foreign transplant centers comprised the validation cohort. We found good agreement between actual survival and predicted survival in the validation cohort; 1-year patient survival rates in the low-, intermediate-, and high-risk groups (as assigned by the original UNOS model) were 72%, 68%, and 36%, respectively (P < .0001). In the patients for whom the international normalized ratio (INR) of prothrombin time was available, MELD correlated with outcome following re-OLT; the median MELD scores for patients surviving at least 90 days compared with those dying within 90 days were 20.75 versus 25.9, respectively (P = .004). Utilizing both patient cohorts (n = 979), a new model, based on recipient age, total serum bilirubin, creatinine, and interval to re-OLT, was constructed (whole model χ(2) = 105, P < .0001). Using the c-statistic with 30-day, 90-day, 1-year, and 3-year mortality as the end points, the area under the receiver operating characteristic (ROC) curves for 4 different models were compared. In conclusion, prospective validation and use of these models as adjuncts to clinical decision making in the management of patients being considered for re-OLT are warranted.