Precision-recall operating characteristic (P-ROC) curves in imprecise environments

Traditionally, machine learning algorithms have been evaluated in applications where assumptions can be reliably made about class priors and/or misclassification costs. In this paper, we consider the case of imprecise environments, where little may be known about these factors and they may well vary significantly when the system is applied. Specifically, the use of precision-recall analysis is investigated and compared to the more well known performance measures such as error-rate and the receiver operating characteristic (ROC). We argue that while ROC analysis is invariant to variations in class priors, this invariance in fact hides an important factor of the evaluation in imprecise environments. Therefore, we develop a generalised precision-recall analysis methodology in which variation due to prior class probabilities is incorporated into a multi-way analysis of variance (ANOVA). The increased sensitivity and reliability of this approach is demonstrated in a remote sensing application.

Free mycophenolic acid should be monitored in renal transplant recipients with hypoalbuminemia

The current approach for therapeutic drug monitoring in renal transplant recipients receiving mycophenolate mofetil (MMF) is measurement of total mycophenolic acid (MPA) concentration. Because MPA is highly bound, during hypoalbuminemia the total concentration no longer reflects the free (pharmacologically active) concentration. The authors investigated what degree of hypoalbuminemia causes a significant change in protein binding and thus percentage free MPA. Forty-two renal transplant recipients were recruited for the study. Free and total concentrations of MPA (predose, and 1, 3, and 6 hours post-MMF dose samples) and plasma albumin concentrations were determined on day 5 posttransplantation. Six-hour area under the concentration-time curve (AUC(0-6)) values were calculated for free and total MPA, and percentage free MPA was determined for each patient. The authors found a significant relationship between low albumin concentrations and increased percentage free MPA (Spearman correlation = -0.54, P < 0.0001). Receiver operating characteristic (ROC) curve analysis was performed on the albumin versus percentage free MPA data. The cutoff value of albumin determined from the ROC analysis that differentiated normal from elevated percentage free MPA (defined as greater than or equal to3%) in this patient population was 31 g/L. At this cutoff value albumin was found to be a good predictor of altered free MPA percentage, with a sensitivity and specificity of 0.75 and 0.80, respectively, and an area under the ROC curve of 0.79. To rationalize MMF dosing regimens in hypoalbuminemic patients (plasma albumin less than or equal to 31 g/L), clinicians should consider monitoring the free MPA concentration.

A Test of Performance of Breast MRI Interpretation in a Multicentre Screening Study

Objectives: The aim of this study was to assess the consistency and performance of radiologists interpreting breast magnetic resonance imaging (MRI) examinations. Materials and Methods: Two test sets of eight cases comprising cancers, benign disease, technical problems and parenchymal enhancement were prepared from two manufacturers' equipment (X and Y) and reported by 15 radiologists using the recording form and scoring system of the UK MRI breast screening study [(MAgnetic Resonance Imaging in Breast Screening (MARIBS)]. Variations in assessments of morphology, kinetic scores and diagnosis were measured by assessing intraobserver and interobserver variability and agreement. The sensitivity and specificity of reporting performances was determined using receiver operating characteristic (ROC) curve analysis. Results: Intraobserver variation was seen in 13 (27.7%) of 47 of the radiologists' conclusions (four technical and seven pathological differences). Substantial interobserver variation was observed in the scores recorded for morphology, pattern of enhancement, quantification of enhancement and washout pattern. The overall sensitivity of breast MRI was high [88.6%, 95% confidence interval (CI) 77.4-94.7%], combined with a specificity of 69.2% (95% CI 60.5-76.7%). The sensitivities were similar for the two test sets (P=.3), but the specificity was significantly higher for the Manufacturer X dataset (P

Application and evaluation of classification trees for screening unwanted plants

Risk assessment systems for introduced species are being developed and applied globally, but methods for rigorously evaluating them are still in their infancy. We explore classification and regression tree models as an alternative to the current Australian Weed Risk Assessment system, and demonstrate how the performance of screening tests for unwanted alien species may be quantitatively compared using receiver operating characteristic (ROC) curve analysis. The optimal classification tree model for predicting weediness included just four out of a possible 44 attributes of introduced plants examined, namely: (i) intentional human dispersal of propagules; (ii) evidence of naturalization beyond native range; (iii) evidence of being a weed elsewhere; and (iv) a high level of domestication. Intentional human dispersal of propagules in combination with evidence of naturalization beyond a plants native range led to the strongest prediction of weediness. A high level of domestication in combination with no evidence of naturalization mitigated the likelihood of an introduced plant becoming a weed resulting from intentional human dispersal of propagules. Unlikely intentional human dispersal of propagules combined with no evidence of being a weed elsewhere led to the lowest predicted probability of weediness. The failure to include intrinsic plant attributes in the model suggests that either these attributes are not useful general predictors of weediness, or data and analysis were inadequate to elucidate the underlying relationship(s). This concurs with the historical pessimism that we will ever be able to accurately predict invasive plants. Given the apparent importance of propagule pressure (the number of individuals of an species released), future attempts at evaluating screening model performance for identifying unwanted plants need to account for propagule pressure when collating and/or analysing datasets. The classification tree had a cross-validated sensitivity of 93.6% and specificity of 36.7%. Based on the area under the ROC curve, the performance of the classification tree in correctly classifying plants as weeds or non-weeds was slightly inferior (Area under ROC curve = 0.83 +/- 0.021 (+/- SE)) to that of the current risk assessment system in use (Area under ROC curve = 0.89 +/- 0.018 (+/- SE)), although requires many fewer questions to be answered.

Predictive validity of three ActiGraph energy expenditure equations for children

Purpose: This Study evaluated the predictive validity of three previously published ActiGraph energy expenditure (EE) prediction equations developed for children and adolescents. Methods: A total of 45 healthy children and adolescents (mean age: 13.7 +/- 2.6 yr) completed four 5-min activity trials (normal walking. brisk walking, easy running, and fast running) in ail indoor exercise facility. During each trial, participants were all ActiGraph accelerometer oil the right hip. EE was monitored breath by breath using the Cosmed K4b(2) portable indirect calorimetry system. Differences and associations between measured and predicted EE were assessed using dependent t-tests and Pearson correlations, respectively. Classification accuracy was assessed using percent agreement, sensitivity, specificity, and area under the receiver operating characteristic (ROC) curve, Results: None of the equations accurately predicted mean energy expenditure during each of the four activity trials. Each equation, however, accurately predicted mean EE in at least one activity trial. The Puyau equation accurately predicted EE during slow walking. The Trost equation accurately predicted EE during slow running. The Freedson equation accurately predicted EE during fast running. None of the three equations accurately predicted EE during brisk walking. The equations exhibited fair to excellent classification accuracy with respect to activity intensity. with the Trost equation exhibiting the highest classification accuracy and the Puyau equation exhibiting the lowest. Conclusions: These data suggest that the three accelerometer prediction equations do not accurately predict EE on a minute-by-minute basis in children and adolescents during overground walking and running. The equations maybe, however, for estimating participation in moderate and vigorous activity.

Prediction of HLA-DQ3.2 beta ligands: evidence of multiple registers in class II binding peptides

Motivation: While processing of MHC class II antigens for presentation to helper T-cells is essential for normal immune response, it is also implicated in the pathogenesis of autoimmune disorders and hypersensitivity reactions. Sequence-based computational techniques for predicting HLA-DQ binding peptides have encountered limited success, with few prediction techniques developed using three-dimensional models. Methods: We describe a structure-based prediction model for modeling peptide-DQ3.2 beta complexes. We have developed a rapid and accurate protocol for docking candidate peptides into the DQ3.2 beta receptor and a scoring function to discriminate binders from the background. The scoring function was rigorously trained, tested and validated using experimentally verified DQ3.2 beta binding and non-binding peptides obtained from biochemical and functional studies. Results: Our model predicts DQ3.2 beta binding peptides with high accuracy [area under the receiver operating characteristic (ROC) curve A(ROC) > 0.90], compared with experimental data. We investigated the binding patterns of DQ3.2 beta peptides and illustrate that several registers exist within a candidate binding peptide. Further analysis reveals that peptides with multiple registers occur predominantly for high-affinity binders.

The clinical utility of the Hopkins Verbal Learning Test as a screening test for mild dementia

Objective. The purpose of this study was to determine whether the Hopkins Verbal Learning Test (HVLT) could be used as a valid and reliable screening test for mild dementia in older people, and to compare its performance to that of the Mini-Mental State Examination (MMSE). Method. Using a cross-sectional design, we studied three groups of older subjects recruited from a district geriatric psychiatry service: (1) 26 patients with DSM-IV dementia and MMSE scores of 18 or better; (2) 15 patients with psychiatric diagnoses other than dementia; and (3) 15 normal controls. The relationship of each potential cutting point on the HVLT and the MMSE was examined against the independently ascertained DSM-IV diagnoses of dementia using a Receiver Operating Characteristic (ROC) analysis. Results. The subjects consisted of 21 (37.5%) males and 35 (62.5%) females with a mean age of 74.7 (SD 6.1) years and a mean of 8.5 (SD 1.8) years of formal education. ROC analysis indicated that the optimal cutting point for detecting mild dementia in this group of subjects using the HVLT was 18/19 (sensitivity = 0.96, specificity = 0.80) and using the MMSE was 25/26 (sensitivity = 0.88, specificity = 0.93). Conclusions. The HVLT can be recommended as a valid and reliable screening test for mild dementia and as an adjunct in the clinical assessment of older people. The HVLT had better sensitivity than the MMSE in detecting patients with mild dementia, whereas the MMSE had better specificity. Copyright (C) 2000 John Wiley & Sons, Ltd.

Comparison of calcaneal ultrasound and DXA in young women

Purpose: The purpose of the study was to assess quantitative ultrasound (QUS) parameters in collegiate female gymnasts, a population whose training incorporates high-impact loading, which is particularly osteogenic, and to determine the discriminative capacity of this relatively new radiation-free technique compared with bone densitometry in a young healthy population. Methods: We studied 19 collegiate gymnasts and 23 healthy controls undergoing regular weight-bearing activity, matched for age (gymnasts 19.2 +/- 1.2, controls 19.9 +/- 1.6 yr) and body weight (gymnasts 56.7 +/- 3.7, controls 57.7 +/- 7.8 kg). QUS parameters of the calcaneus (broadband ultrasound attenuation (BUA), bone velocity (BV), and speed of sound (SOS)) were measured by a Walker Sonix UBA 575+. Bone mineral density (BMD; g.cm(-2)) of the lumbar spine, hip (Femoral neck, trochanter. Ward's triangle) and whole body was assessed by dual energy x-ray absorptiometry (DXA, Hologic QDR 1000/W). Data analysis included unpaired two-tailed Student's t-tests, analysis of variance, Pearson product-moment, and Spearman rank-order correlations. Results: Regional and whole body BMD of gymnasts was greater than controls (P < 0.001), with the difference being 7-28%. Average QUS parameters of the right and left calcaneus were also higher (P < 0.001) in the gymnasts. BUA, BV, and SOS were significantly (P < 0.001) correlated to each bone site with r = 0.54-0.79. Analysis of receiver operating characteristic (ROC) curves indicated no significant difference in sensitivity and specificity for QUS and DXA measures. Conclusions: These results indicate that QUS parameters of the calcaneus are higher in young women gymnasts compared to individuals who undergo regular weight-bearing activity and that QUS parameters are able to discriminate between these two groups in a similar manner as does regional and whole body BMD.

CDT, GGT, and AST as markers of alcohol use: The WHO/ISBRA Collaborative Project

Background: Estimates of the performance of carbohydrate deficient transferrin (CDT) and gamma glutamyltransferase (GGT) as markers of alcohol consumption have varied widely. Studies have differed in design and subject characteristics. The WHO/ISBRA Collaborative Study allows assessment and comparison of CDT, GGT, and aspartate aminotransferase (AST) as markers of drinking in a large, well-characterized, multicenter sample. Methods: A total of 1863 subjects were recruited from five countries (Australia, Brazil, Canada, Finland, and Japan). Recruitment was stratified by alcohol use, age, and sex. Demographic characteristics, alcohol consumption, and presence of ICD-10 dependence were recorded using an interview schedule based on the AUDADIS, CDT was assayed using CDTect(TM) and GGT and AST by standard methods. Statistical techniques included receiver operating characteristic (ROC) analysis. Multiple regression was used to measure the impact of factors other than alcohol on test performance. Results: CDT and GGT had comparable performance on ROC analysis, with AST performing slightly less well. CDT was a slightly but significantly better marker of high-risk consumption in men. All were more effective for detection of high-risk rather than intermediate-risk drinking. CDT and GGT levels were influenced by body mass index, sex, age, and smoking status. Conclusions: CDT was little better than GGT in detecting high- or intermediate-risk alcohol consumption in this large, multicenter, predominantly community-based sample. As the two tests are relatively independent of each other, their combination is likely to provide better performance than either test alone, Test interpretation should take account sex, age. and body mass index.

Quality of life assessment in the community-dwelling elderly: Validation of the Assessment of Quality of Life (AQoL) Instrument and comparison with the SF-36

Measurement of Health-Related Quality of Life (HRQoL) of the elderly requires instruments with demonstrated sensitivity, reliability, and validity, particularly with the increasing proportion of older people entering the health care system. This article reports the psychometric properties of the 12-item Assessment of Quality of Life (AQoL) instrument in chronically ill community-dwelling elderly people with an 18-month follow-up. Comparator instruments included the SF-36 and the OARS. Construct validity of the AQoL was strong when examined via factor analysis and convergent and divergent validity against other scales. Receiver Operator Characteristic (ROC) curve analyses and relative efficiency estimates indicated the AQoL is sensitive, responsive, and had the strongest predicative validity for nursing home entry. It was also sensitive to economic prediction over the follow-up. Given these robust psychometric properties and the brevity of the scale, AQoL appears to be a suitable instrument for epidemiologic studies where HRQoL and utility data are required from elderly populations. (C) 2003 Elsevier Science Inc. All rights reserved.

Validation and refinement of survival models for liver retransplantation

Orthotopic liver retransplantation (re-OLT) is highly controversial. The objectives of this study were to determine the validity of a recently developed United Network for Organ Sharing (UNOS) multivariate model using an independent cohort of patients undergoing re-OLT outside the United States, to determine whether incorporation of other variables that were incomplete in the UNOS registry would provide additional prognostic information, to develop new models combining data sets from both cohorts, and to evaluate the validity of the model for end-stage liver disease (MELD) in patients undergoing re-OLT. Two hundred eighty-one adult patients undergoing re-OLT (between 1986 and 1999) at 6 foreign transplant centers comprised the validation cohort. We found good agreement between actual survival and predicted survival in the validation cohort; 1-year patient survival rates in the low-, intermediate-, and high-risk groups (as assigned by the original UNOS model) were 72%, 68%, and 36%, respectively (P < .0001). In the patients for whom the international normalized ratio (INR) of prothrombin time was available, MELD correlated with outcome following re-OLT; the median MELD scores for patients surviving at least 90 days compared with those dying within 90 days were 20.75 versus 25.9, respectively (P = .004). Utilizing both patient cohorts (n = 979), a new model, based on recipient age, total serum bilirubin, creatinine, and interval to re-OLT, was constructed (whole model χ(2) = 105, P < .0001). Using the c-statistic with 30-day, 90-day, 1-year, and 3-year mortality as the end points, the area under the receiver operating characteristic (ROC) curves for 4 different models were compared. In conclusion, prospective validation and use of these models as adjuncts to clinical decision making in the management of patients being considered for re-OLT are warranted.

The use of a change in gentamicin clearance as an early predictor of gentamicin-induced nephrotoxicity

A retrospective review was undertaken in 744 patients who were dose-individualized with gentamicin once daily to evaluate a change in gentamicin clearance as a potential predictor of nephrotoxicity. The definition of nephrotoxicity was chosen to be a change in creatinine clearance greater than 20%. Similarly, a change in gentamicin clearance of greater than 20% was also considered a possible index of nephrotoxicity. Four criteria were developed to assess the usefulness of gentamicin clearance as a predictor of nephrotoxicity. Following the application of the inclusion/exclusion criteria, 132 patients were available for the analysis. The sensitivity, specificity, positive predictive value, and negative predictive value were assessed for each of the criteria. Receiver operating characteristic (ROC) curves were produced to determine if an optimum value in the change of gentamicin clearance could be found to maximize sensitivity and specificity. The overall incidence of nephrotoxicity based on a decrease in creatinine clearance by 20% or more was 3.8%. Women were overrepresented in the nephrotoxic group [71.4% versus 40.1% (P = 0.0025)]. Patients with nephrotoxicity had statistically longer treatment periods, increased cumulative dose, and more dosing predictions (P < 0.05 in each case). The sensitivity of the criteria ranged from 43 to 46%, and specificity ranged from 93 to 99%. The positive and negative predictive values ranged from 63 to 94% and 86 to 89%, respectively. In those patients in whom nephrotoxicity was predicted from a change in gentamicin clearance, this change occurred on average 3 days before the change in creatinine clearance (P < 0.05). A change in gentamicin clearance to predict nephrotoxicity may be a useful addition to current monitoring methods, although it is not the complete answer.

Prospective independent validation of APACHE III models in an Australian tertiary adult intensive care unit

Evaluation of the performance of the APACHE III (Acute Physiology and Chronic Health Evaluation) ICU (intensive care unit) and hospital mortality models at the Princess Alexandra Hospital, Brisbane is reported. Prospective collection of demographic, diagnostic, physiological, laboratory, admission and discharge data of 5681 consecutive eligible admissions (1 January 1995 to 1 January 2000) was conducted at the Princess Alexandra Hospital, a metropolitan Australian tertiary referral medical/surgical adult ICU. ROC (receiver operating characteristic) curve areas for the APACHE III ICU mortality and hospital mortality models demonstrated excellent discrimination. Observed ICU mortality (9.1%) was significantly overestimated by the APACHE III model adjusted for hospital characteristics (10.1%), but did not significantly differ from the prediction of the generic APACHE III model (8.6%). In contrast, observed hospital mortality (14.8%) agreed well with the prediction of the APACHE III model adjusted for hospital characteristics (14.6%), but was significantly underestimated by the unadjusted APACHE III model (13.2%). Calibration curves and goodness-of-fit analysis using Hosmer-Lemeshow statistics, demonstrated that calibration was good with the unadjusted APACHE III ICU mortality model, and the APACHE III hospital mortality model adjusted for hospital characteristics. Post hoc analysis revealed a declining annual SMR (standardized mortality rate) during the study period. This trend was present in each of the non-surgical, emergency and elective surgical diagnostic groups, and the change was temporally related to increased specialist staffing levels. This study demonstrates that the APACHE III model performs well on independent assessment in an Australian hospital. Changes observed in annual SMR using such a validated model support an hypothesis of improved survival outcomes 1995-1999.

MULTIPRED: A computational system for prediction of promiscuous HLA binding peptides

MULTIPRED is a web-based computational system for the prediction of peptide binding to multiple molecules ( proteins) belonging to human leukocyte antigens (HLA) class I A2, A3 and class II DR supertypes. It uses hidden Markov models and artificial neural network methods as predictive engines. A novel data representation method enables MULTIPRED to predict peptides that promiscuously bind multiple HLA alleles within one HLA supertype. Extensive testing was performed for validation of the prediction models. Testing results show that MULTIPRED is both sensitive and specific and it has good predictive ability ( area under the receiver operating characteristic curve A(ROC) > 0.80). MULTIPRED can be used for the mapping of promiscuous T-cell epitopes as well as the regions of high concentration of these targets termed T-cell epitope hotspots. MULTIPRED is available at http:// antigen.i2r.a-star.edu.sg/ multipred/.

Determining the fetal scalp lactate level that indicates the need for intervention in labour

Background: Fetal scalp lactate testing has been shown to be as useful as pH with added benefits. One remaining question is What level of lactate should trigger intervention in the first stage of labour?' Aims: This study aimed to establish the lactate level in the first stage of labour that indicates the need for intervention to ensure satisfactory outcomes for both babies and mothers. Methods: A prospective study at Mater Mothers' Hospital, Brisbane, Australia, a tertiary referral centre. One hundred and forty women in labour, with non-reassuring fetal heart rate traces, were tested using fetal blood scalp sampling of 5 mu L of capillary blood tested on an Accusport (Boeringer, Mannheim, East Sussex, UK) lactate meter. Decision to intervene in labour was based on clinical assessment plus a predetermined cut off. Main outcome measures were APGAR scores, cord arterial pH, meconium stained liquor and Intensive Care Nursery admission. Results: Two-graph receiver operating characteristic (TG-ROC) analysis showed optimal specificity, and sensitivity for predicting adverse neonatal outcomes was a scalp lactate level above 4.2 mmol/L. Conclusions: Fetal blood sampling remains the standard for further investigating-non-reassuring cardiotocograph (CTG) traces. Even so, it is a poor predictor of fetal outcomes. Scalp lactate has been shown to be at least as good a predictor as scalp pH, with the advantages of being easier, cheaper and with a lower rate of technical failure. Our study, found that a cut off fetal scalp lactate level of 4.2 mmol/L, in combination with an assessment of the entire clinical picture, is a useful tool in identifying those women who need intervention.