Precision-recall operating characteristic (P-ROC) curves in imprecise environments

Traditionally, machine learning algorithms have been evaluated in applications where assumptions can be reliably made about class priors and/or misclassification costs. In this paper, we consider the case of imprecise environments, where little may be known about these factors and they may well vary significantly when the system is applied. Specifically, the use of precision-recall analysis is investigated and compared to the more well known performance measures such as error-rate and the receiver operating characteristic (ROC). We argue that while ROC analysis is invariant to variations in class priors, this invariance in fact hides an important factor of the evaluation in imprecise environments. Therefore, we develop a generalised precision-recall analysis methodology in which variation due to prior class probabilities is incorporated into a multi-way analysis of variance (ANOVA). The increased sensitivity and reliability of this approach is demonstrated in a remote sensing application.

Prospective independent validation of APACHE III models in an Australian tertiary adult intensive care unit

Evaluation of the performance of the APACHE III (Acute Physiology and Chronic Health Evaluation) ICU (intensive care unit) and hospital mortality models at the Princess Alexandra Hospital, Brisbane is reported. Prospective collection of demographic, diagnostic, physiological, laboratory, admission and discharge data of 5681 consecutive eligible admissions (1 January 1995 to 1 January 2000) was conducted at the Princess Alexandra Hospital, a metropolitan Australian tertiary referral medical/surgical adult ICU. ROC (receiver operating characteristic) curve areas for the APACHE III ICU mortality and hospital mortality models demonstrated excellent discrimination. Observed ICU mortality (9.1%) was significantly overestimated by the APACHE III model adjusted for hospital characteristics (10.1%), but did not significantly differ from the prediction of the generic APACHE III model (8.6%). In contrast, observed hospital mortality (14.8%) agreed well with the prediction of the APACHE III model adjusted for hospital characteristics (14.6%), but was significantly underestimated by the unadjusted APACHE III model (13.2%). Calibration curves and goodness-of-fit analysis using Hosmer-Lemeshow statistics, demonstrated that calibration was good with the unadjusted APACHE III ICU mortality model, and the APACHE III hospital mortality model adjusted for hospital characteristics. Post hoc analysis revealed a declining annual SMR (standardized mortality rate) during the study period. This trend was present in each of the non-surgical, emergency and elective surgical diagnostic groups, and the change was temporally related to increased specialist staffing levels. This study demonstrates that the APACHE III model performs well on independent assessment in an Australian hospital. Changes observed in annual SMR using such a validated model support an hypothesis of improved survival outcomes 1995-1999.

Myocardial blood volume and perfusion reserve responses to combined dipyridamole and exercise stress: A quantitative approach to contrast stress echocardiography

Background: Qualitative interpretation of myocardial contrast echocardiography (MCE) improves the accuracy of wall-motion analysis for assessment of coronary artery disease (CAD). We examined the feasibility and accuracy of quantitative MCE for diagnosis of CAD. Methods: Dipyridamole/exercise stress MCE (destruction-replenishment protocol with real-time imaging) was performed in 90 patients undergoing quantitative coronary angiography, 48 of whom had significant (> 50%) stenoses. MCE was repeated with exercise alone in 18 patients. Myocardial blood flow (A*beta) was obtained from blood volume (A) and time to refill (beta). Results: Quantification of flow reserve was feasible in 88%. The mean A*beta reserve in the anterior wall was significantly impaired for patients with left anterior descending coronary artery disease (n = 28) compared with those with no disease (1.6 +/- 1.2 vs; 4.0 +/- 2.5, P <=.001). This reflected impaired beta reserve, with no difference in the A reserve. Applying a receiver operating characteristic curve derived cutoff of 2.0 for A*beta reserve, quantitative MCE was 76% sensitive and 71% specific for the diagnosis of significant left anterior descending coronary artery stenosis. Posterior circulation results were similar, with 78% sensitivity and 59% specificity for detection of posterior CAD. Overall, quantitative MCE was similarly sensitive to qualitative approach for diagnosis of CAD (88% vs 93%), but with lower specificity (52% vs 65%, P =.07). In 18 patients restudied with pure exercise stress, the mean myocardial blood flow reserve was less than after combined stress (2.1 +/- 1.6 vs 3.7 +/- 1.9, P =.01). Conclusion: Quantitative MCE is feasible for the diagnosis of CAD with dipyridamole/exercise stress. Dipyridamole prolongs postexercise hyperemia, augmenting the degree of hyperemia at the time of imaging.

MULTIPRED: A computational system for prediction of promiscuous HLA binding peptides

MULTIPRED is a web-based computational system for the prediction of peptide binding to multiple molecules ( proteins) belonging to human leukocyte antigens (HLA) class I A2, A3 and class II DR supertypes. It uses hidden Markov models and artificial neural network methods as predictive engines. A novel data representation method enables MULTIPRED to predict peptides that promiscuously bind multiple HLA alleles within one HLA supertype. Extensive testing was performed for validation of the prediction models. Testing results show that MULTIPRED is both sensitive and specific and it has good predictive ability ( area under the receiver operating characteristic curve A(ROC) > 0.80). MULTIPRED can be used for the mapping of promiscuous T-cell epitopes as well as the regions of high concentration of these targets termed T-cell epitope hotspots. MULTIPRED is available at http:// antigen.i2r.a-star.edu.sg/ multipred/.

The clinical utility of the Hopkins Verbal Learning Test as a screening test for mild dementia

Objective. The purpose of this study was to determine whether the Hopkins Verbal Learning Test (HVLT) could be used as a valid and reliable screening test for mild dementia in older people, and to compare its performance to that of the Mini-Mental State Examination (MMSE). Method. Using a cross-sectional design, we studied three groups of older subjects recruited from a district geriatric psychiatry service: (1) 26 patients with DSM-IV dementia and MMSE scores of 18 or better; (2) 15 patients with psychiatric diagnoses other than dementia; and (3) 15 normal controls. The relationship of each potential cutting point on the HVLT and the MMSE was examined against the independently ascertained DSM-IV diagnoses of dementia using a Receiver Operating Characteristic (ROC) analysis. Results. The subjects consisted of 21 (37.5%) males and 35 (62.5%) females with a mean age of 74.7 (SD 6.1) years and a mean of 8.5 (SD 1.8) years of formal education. ROC analysis indicated that the optimal cutting point for detecting mild dementia in this group of subjects using the HVLT was 18/19 (sensitivity = 0.96, specificity = 0.80) and using the MMSE was 25/26 (sensitivity = 0.88, specificity = 0.93). Conclusions. The HVLT can be recommended as a valid and reliable screening test for mild dementia and as an adjunct in the clinical assessment of older people. The HVLT had better sensitivity than the MMSE in detecting patients with mild dementia, whereas the MMSE had better specificity. Copyright (C) 2000 John Wiley & Sons, Ltd.

Free mycophenolic acid should be monitored in renal transplant recipients with hypoalbuminemia

The current approach for therapeutic drug monitoring in renal transplant recipients receiving mycophenolate mofetil (MMF) is measurement of total mycophenolic acid (MPA) concentration. Because MPA is highly bound, during hypoalbuminemia the total concentration no longer reflects the free (pharmacologically active) concentration. The authors investigated what degree of hypoalbuminemia causes a significant change in protein binding and thus percentage free MPA. Forty-two renal transplant recipients were recruited for the study. Free and total concentrations of MPA (predose, and 1, 3, and 6 hours post-MMF dose samples) and plasma albumin concentrations were determined on day 5 posttransplantation. Six-hour area under the concentration-time curve (AUC(0-6)) values were calculated for free and total MPA, and percentage free MPA was determined for each patient. The authors found a significant relationship between low albumin concentrations and increased percentage free MPA (Spearman correlation = -0.54, P < 0.0001). Receiver operating characteristic (ROC) curve analysis was performed on the albumin versus percentage free MPA data. The cutoff value of albumin determined from the ROC analysis that differentiated normal from elevated percentage free MPA (defined as greater than or equal to3%) in this patient population was 31 g/L. At this cutoff value albumin was found to be a good predictor of altered free MPA percentage, with a sensitivity and specificity of 0.75 and 0.80, respectively, and an area under the ROC curve of 0.79. To rationalize MMF dosing regimens in hypoalbuminemic patients (plasma albumin less than or equal to 31 g/L), clinicians should consider monitoring the free MPA concentration.

A Test of Performance of Breast MRI Interpretation in a Multicentre Screening Study

Objectives: The aim of this study was to assess the consistency and performance of radiologists interpreting breast magnetic resonance imaging (MRI) examinations. Materials and Methods: Two test sets of eight cases comprising cancers, benign disease, technical problems and parenchymal enhancement were prepared from two manufacturers' equipment (X and Y) and reported by 15 radiologists using the recording form and scoring system of the UK MRI breast screening study [(MAgnetic Resonance Imaging in Breast Screening (MARIBS)]. Variations in assessments of morphology, kinetic scores and diagnosis were measured by assessing intraobserver and interobserver variability and agreement. The sensitivity and specificity of reporting performances was determined using receiver operating characteristic (ROC) curve analysis. Results: Intraobserver variation was seen in 13 (27.7%) of 47 of the radiologists' conclusions (four technical and seven pathological differences). Substantial interobserver variation was observed in the scores recorded for morphology, pattern of enhancement, quantification of enhancement and washout pattern. The overall sensitivity of breast MRI was high [88.6%, 95% confidence interval (CI) 77.4-94.7%], combined with a specificity of 69.2% (95% CI 60.5-76.7%). The sensitivities were similar for the two test sets (P=.3), but the specificity was significantly higher for the Manufacturer X dataset (P

Application and evaluation of classification trees for screening unwanted plants

Risk assessment systems for introduced species are being developed and applied globally, but methods for rigorously evaluating them are still in their infancy. We explore classification and regression tree models as an alternative to the current Australian Weed Risk Assessment system, and demonstrate how the performance of screening tests for unwanted alien species may be quantitatively compared using receiver operating characteristic (ROC) curve analysis. The optimal classification tree model for predicting weediness included just four out of a possible 44 attributes of introduced plants examined, namely: (i) intentional human dispersal of propagules; (ii) evidence of naturalization beyond native range; (iii) evidence of being a weed elsewhere; and (iv) a high level of domestication. Intentional human dispersal of propagules in combination with evidence of naturalization beyond a plants native range led to the strongest prediction of weediness. A high level of domestication in combination with no evidence of naturalization mitigated the likelihood of an introduced plant becoming a weed resulting from intentional human dispersal of propagules. Unlikely intentional human dispersal of propagules combined with no evidence of being a weed elsewhere led to the lowest predicted probability of weediness. The failure to include intrinsic plant attributes in the model suggests that either these attributes are not useful general predictors of weediness, or data and analysis were inadequate to elucidate the underlying relationship(s). This concurs with the historical pessimism that we will ever be able to accurately predict invasive plants. Given the apparent importance of propagule pressure (the number of individuals of an species released), future attempts at evaluating screening model performance for identifying unwanted plants need to account for propagule pressure when collating and/or analysing datasets. The classification tree had a cross-validated sensitivity of 93.6% and specificity of 36.7%. Based on the area under the ROC curve, the performance of the classification tree in correctly classifying plants as weeds or non-weeds was slightly inferior (Area under ROC curve = 0.83 +/- 0.021 (+/- SE)) to that of the current risk assessment system in use (Area under ROC curve = 0.89 +/- 0.018 (+/- SE)), although requires many fewer questions to be answered.

Predictive validity of three ActiGraph energy expenditure equations for children

Purpose: This Study evaluated the predictive validity of three previously published ActiGraph energy expenditure (EE) prediction equations developed for children and adolescents. Methods: A total of 45 healthy children and adolescents (mean age: 13.7 +/- 2.6 yr) completed four 5-min activity trials (normal walking. brisk walking, easy running, and fast running) in ail indoor exercise facility. During each trial, participants were all ActiGraph accelerometer oil the right hip. EE was monitored breath by breath using the Cosmed K4b(2) portable indirect calorimetry system. Differences and associations between measured and predicted EE were assessed using dependent t-tests and Pearson correlations, respectively. Classification accuracy was assessed using percent agreement, sensitivity, specificity, and area under the receiver operating characteristic (ROC) curve, Results: None of the equations accurately predicted mean energy expenditure during each of the four activity trials. Each equation, however, accurately predicted mean EE in at least one activity trial. The Puyau equation accurately predicted EE during slow walking. The Trost equation accurately predicted EE during slow running. The Freedson equation accurately predicted EE during fast running. None of the three equations accurately predicted EE during brisk walking. The equations exhibited fair to excellent classification accuracy with respect to activity intensity. with the Trost equation exhibiting the highest classification accuracy and the Puyau equation exhibiting the lowest. Conclusions: These data suggest that the three accelerometer prediction equations do not accurately predict EE on a minute-by-minute basis in children and adolescents during overground walking and running. The equations maybe, however, for estimating participation in moderate and vigorous activity.

Prediction of HLA-DQ3.2 beta ligands: evidence of multiple registers in class II binding peptides

Motivation: While processing of MHC class II antigens for presentation to helper T-cells is essential for normal immune response, it is also implicated in the pathogenesis of autoimmune disorders and hypersensitivity reactions. Sequence-based computational techniques for predicting HLA-DQ binding peptides have encountered limited success, with few prediction techniques developed using three-dimensional models. Methods: We describe a structure-based prediction model for modeling peptide-DQ3.2 beta complexes. We have developed a rapid and accurate protocol for docking candidate peptides into the DQ3.2 beta receptor and a scoring function to discriminate binders from the background. The scoring function was rigorously trained, tested and validated using experimentally verified DQ3.2 beta binding and non-binding peptides obtained from biochemical and functional studies. Results: Our model predicts DQ3.2 beta binding peptides with high accuracy [area under the receiver operating characteristic (ROC) curve A(ROC) > 0.90], compared with experimental data. We investigated the binding patterns of DQ3.2 beta peptides and illustrate that several registers exist within a candidate binding peptide. Further analysis reveals that peptides with multiple registers occur predominantly for high-affinity binders.

Comparison of calcaneal ultrasound and DXA in young women

Purpose: The purpose of the study was to assess quantitative ultrasound (QUS) parameters in collegiate female gymnasts, a population whose training incorporates high-impact loading, which is particularly osteogenic, and to determine the discriminative capacity of this relatively new radiation-free technique compared with bone densitometry in a young healthy population. Methods: We studied 19 collegiate gymnasts and 23 healthy controls undergoing regular weight-bearing activity, matched for age (gymnasts 19.2 +/- 1.2, controls 19.9 +/- 1.6 yr) and body weight (gymnasts 56.7 +/- 3.7, controls 57.7 +/- 7.8 kg). QUS parameters of the calcaneus (broadband ultrasound attenuation (BUA), bone velocity (BV), and speed of sound (SOS)) were measured by a Walker Sonix UBA 575+. Bone mineral density (BMD; g.cm(-2)) of the lumbar spine, hip (Femoral neck, trochanter. Ward's triangle) and whole body was assessed by dual energy x-ray absorptiometry (DXA, Hologic QDR 1000/W). Data analysis included unpaired two-tailed Student's t-tests, analysis of variance, Pearson product-moment, and Spearman rank-order correlations. Results: Regional and whole body BMD of gymnasts was greater than controls (P < 0.001), with the difference being 7-28%. Average QUS parameters of the right and left calcaneus were also higher (P < 0.001) in the gymnasts. BUA, BV, and SOS were significantly (P < 0.001) correlated to each bone site with r = 0.54-0.79. Analysis of receiver operating characteristic (ROC) curves indicated no significant difference in sensitivity and specificity for QUS and DXA measures. Conclusions: These results indicate that QUS parameters of the calcaneus are higher in young women gymnasts compared to individuals who undergo regular weight-bearing activity and that QUS parameters are able to discriminate between these two groups in a similar manner as does regional and whole body BMD.

Detecting insomnia: comparison of four self-report measures of sleep in a young adult population

The sensitivity and specificity of four self-report measures of disordered sleep - the Sleep Impairment Index (SII), the Sleep Disorders Questionnaire (SDQ), the Dysfunctional Beliefs and Attitudes About Sleep Scale (DBAS) and the Sleep-Wake Activity Inventory (SWAI) - were compared in subjects with insomnia and normal sleep. Nineteen young adult subjects met DSM-IV criteria for primary insomnia and another 19 were normal control subjects. Discriminatory characteristics of each measure were assessed using receiver operator characteristic curve analyses. Discriminatory power was maximised for each measure to produce cut-scores applicable for identification of individuals with insomnia. The DBAS, SII and SDQ psychiatric DIMS subscale were found to correlate, and discriminated well between the two groups. The SWAI nocturnal sleep subscale was not found to be an accurate discriminator. The results suggest differences in the measures in their ability to detect insomnia, and offer guidelines as to the optimal use of test scores to identify young adults suspected of insomnia.

CDT, GGT, and AST as markers of alcohol use: The WHO/ISBRA Collaborative Project

Background: Estimates of the performance of carbohydrate deficient transferrin (CDT) and gamma glutamyltransferase (GGT) as markers of alcohol consumption have varied widely. Studies have differed in design and subject characteristics. The WHO/ISBRA Collaborative Study allows assessment and comparison of CDT, GGT, and aspartate aminotransferase (AST) as markers of drinking in a large, well-characterized, multicenter sample. Methods: A total of 1863 subjects were recruited from five countries (Australia, Brazil, Canada, Finland, and Japan). Recruitment was stratified by alcohol use, age, and sex. Demographic characteristics, alcohol consumption, and presence of ICD-10 dependence were recorded using an interview schedule based on the AUDADIS, CDT was assayed using CDTect(TM) and GGT and AST by standard methods. Statistical techniques included receiver operating characteristic (ROC) analysis. Multiple regression was used to measure the impact of factors other than alcohol on test performance. Results: CDT and GGT had comparable performance on ROC analysis, with AST performing slightly less well. CDT was a slightly but significantly better marker of high-risk consumption in men. All were more effective for detection of high-risk rather than intermediate-risk drinking. CDT and GGT levels were influenced by body mass index, sex, age, and smoking status. Conclusions: CDT was little better than GGT in detecting high- or intermediate-risk alcohol consumption in this large, multicenter, predominantly community-based sample. As the two tests are relatively independent of each other, their combination is likely to provide better performance than either test alone, Test interpretation should take account sex, age. and body mass index.

Validation and refinement of survival models for liver retransplantation

Orthotopic liver retransplantation (re-OLT) is highly controversial. The objectives of this study were to determine the validity of a recently developed United Network for Organ Sharing (UNOS) multivariate model using an independent cohort of patients undergoing re-OLT outside the United States, to determine whether incorporation of other variables that were incomplete in the UNOS registry would provide additional prognostic information, to develop new models combining data sets from both cohorts, and to evaluate the validity of the model for end-stage liver disease (MELD) in patients undergoing re-OLT. Two hundred eighty-one adult patients undergoing re-OLT (between 1986 and 1999) at 6 foreign transplant centers comprised the validation cohort. We found good agreement between actual survival and predicted survival in the validation cohort; 1-year patient survival rates in the low-, intermediate-, and high-risk groups (as assigned by the original UNOS model) were 72%, 68%, and 36%, respectively (P < .0001). In the patients for whom the international normalized ratio (INR) of prothrombin time was available, MELD correlated with outcome following re-OLT; the median MELD scores for patients surviving at least 90 days compared with those dying within 90 days were 20.75 versus 25.9, respectively (P = .004). Utilizing both patient cohorts (n = 979), a new model, based on recipient age, total serum bilirubin, creatinine, and interval to re-OLT, was constructed (whole model χ(2) = 105, P < .0001). Using the c-statistic with 30-day, 90-day, 1-year, and 3-year mortality as the end points, the area under the receiver operating characteristic (ROC) curves for 4 different models were compared. In conclusion, prospective validation and use of these models as adjuncts to clinical decision making in the management of patients being considered for re-OLT are warranted.