Precision-recall operating characteristic (P-ROC) curves in imprecise environments

Traditionally, machine learning algorithms have been evaluated in applications where assumptions can be reliably made about class priors and/or misclassification costs. In this paper, we consider the case of imprecise environments, where little may be known about these factors and they may well vary significantly when the system is applied. Specifically, the use of precision-recall analysis is investigated and compared to the more well known performance measures such as error-rate and the receiver operating characteristic (ROC). We argue that while ROC analysis is invariant to variations in class priors, this invariance in fact hides an important factor of the evaluation in imprecise environments. Therefore, we develop a generalised precision-recall analysis methodology in which variation due to prior class probabilities is incorporated into a multi-way analysis of variance (ANOVA). The increased sensitivity and reliability of this approach is demonstrated in a remote sensing application.

Comparison of calcaneal ultrasound and DXA in young women

Purpose: The purpose of the study was to assess quantitative ultrasound (QUS) parameters in collegiate female gymnasts, a population whose training incorporates high-impact loading, which is particularly osteogenic, and to determine the discriminative capacity of this relatively new radiation-free technique compared with bone densitometry in a young healthy population. Methods: We studied 19 collegiate gymnasts and 23 healthy controls undergoing regular weight-bearing activity, matched for age (gymnasts 19.2 +/- 1.2, controls 19.9 +/- 1.6 yr) and body weight (gymnasts 56.7 +/- 3.7, controls 57.7 +/- 7.8 kg). QUS parameters of the calcaneus (broadband ultrasound attenuation (BUA), bone velocity (BV), and speed of sound (SOS)) were measured by a Walker Sonix UBA 575+. Bone mineral density (BMD; g.cm(-2)) of the lumbar spine, hip (Femoral neck, trochanter. Ward's triangle) and whole body was assessed by dual energy x-ray absorptiometry (DXA, Hologic QDR 1000/W). Data analysis included unpaired two-tailed Student's t-tests, analysis of variance, Pearson product-moment, and Spearman rank-order correlations. Results: Regional and whole body BMD of gymnasts was greater than controls (P < 0.001), with the difference being 7-28%. Average QUS parameters of the right and left calcaneus were also higher (P < 0.001) in the gymnasts. BUA, BV, and SOS were significantly (P < 0.001) correlated to each bone site with r = 0.54-0.79. Analysis of receiver operating characteristic (ROC) curves indicated no significant difference in sensitivity and specificity for QUS and DXA measures. Conclusions: These results indicate that QUS parameters of the calcaneus are higher in young women gymnasts compared to individuals who undergo regular weight-bearing activity and that QUS parameters are able to discriminate between these two groups in a similar manner as does regional and whole body BMD.

Detecting insomnia: comparison of four self-report measures of sleep in a young adult population

The sensitivity and specificity of four self-report measures of disordered sleep - the Sleep Impairment Index (SII), the Sleep Disorders Questionnaire (SDQ), the Dysfunctional Beliefs and Attitudes About Sleep Scale (DBAS) and the Sleep-Wake Activity Inventory (SWAI) - were compared in subjects with insomnia and normal sleep. Nineteen young adult subjects met DSM-IV criteria for primary insomnia and another 19 were normal control subjects. Discriminatory characteristics of each measure were assessed using receiver operator characteristic curve analyses. Discriminatory power was maximised for each measure to produce cut-scores applicable for identification of individuals with insomnia. The DBAS, SII and SDQ psychiatric DIMS subscale were found to correlate, and discriminated well between the two groups. The SWAI nocturnal sleep subscale was not found to be an accurate discriminator. The results suggest differences in the measures in their ability to detect insomnia, and offer guidelines as to the optimal use of test scores to identify young adults suspected of insomnia.

Validation and refinement of survival models for liver retransplantation

Orthotopic liver retransplantation (re-OLT) is highly controversial. The objectives of this study were to determine the validity of a recently developed United Network for Organ Sharing (UNOS) multivariate model using an independent cohort of patients undergoing re-OLT outside the United States, to determine whether incorporation of other variables that were incomplete in the UNOS registry would provide additional prognostic information, to develop new models combining data sets from both cohorts, and to evaluate the validity of the model for end-stage liver disease (MELD) in patients undergoing re-OLT. Two hundred eighty-one adult patients undergoing re-OLT (between 1986 and 1999) at 6 foreign transplant centers comprised the validation cohort. We found good agreement between actual survival and predicted survival in the validation cohort; 1-year patient survival rates in the low-, intermediate-, and high-risk groups (as assigned by the original UNOS model) were 72%, 68%, and 36%, respectively (P < .0001). In the patients for whom the international normalized ratio (INR) of prothrombin time was available, MELD correlated with outcome following re-OLT; the median MELD scores for patients surviving at least 90 days compared with those dying within 90 days were 20.75 versus 25.9, respectively (P = .004). Utilizing both patient cohorts (n = 979), a new model, based on recipient age, total serum bilirubin, creatinine, and interval to re-OLT, was constructed (whole model χ(2) = 105, P < .0001). Using the c-statistic with 30-day, 90-day, 1-year, and 3-year mortality as the end points, the area under the receiver operating characteristic (ROC) curves for 4 different models were compared. In conclusion, prospective validation and use of these models as adjuncts to clinical decision making in the management of patients being considered for re-OLT are warranted.

The use of a change in gentamicin clearance as an early predictor of gentamicin-induced nephrotoxicity

A retrospective review was undertaken in 744 patients who were dose-individualized with gentamicin once daily to evaluate a change in gentamicin clearance as a potential predictor of nephrotoxicity. The definition of nephrotoxicity was chosen to be a change in creatinine clearance greater than 20%. Similarly, a change in gentamicin clearance of greater than 20% was also considered a possible index of nephrotoxicity. Four criteria were developed to assess the usefulness of gentamicin clearance as a predictor of nephrotoxicity. Following the application of the inclusion/exclusion criteria, 132 patients were available for the analysis. The sensitivity, specificity, positive predictive value, and negative predictive value were assessed for each of the criteria. Receiver operating characteristic (ROC) curves were produced to determine if an optimum value in the change of gentamicin clearance could be found to maximize sensitivity and specificity. The overall incidence of nephrotoxicity based on a decrease in creatinine clearance by 20% or more was 3.8%. Women were overrepresented in the nephrotoxic group [71.4% versus 40.1% (P = 0.0025)]. Patients with nephrotoxicity had statistically longer treatment periods, increased cumulative dose, and more dosing predictions (P < 0.05 in each case). The sensitivity of the criteria ranged from 43 to 46%, and specificity ranged from 93 to 99%. The positive and negative predictive values ranged from 63 to 94% and 86 to 89%, respectively. In those patients in whom nephrotoxicity was predicted from a change in gentamicin clearance, this change occurred on average 3 days before the change in creatinine clearance (P < 0.05). A change in gentamicin clearance to predict nephrotoxicity may be a useful addition to current monitoring methods, although it is not the complete answer.

The relative impact of scar and viable tissue in predicting global left ventricular remodeling in medically treated patients

We sought to determine the relative impact of myocardial scar and viability on post-infarct left ventricular (LV) remodeling in medically-treated patients with LV dysfunction. Forty patients with chronic ischemic heart disease (age 64±9, EF 40±11%) underwent rest-redistribution Tl201 SPECT (scar = 50% transmural extent), A global index of scarring for each patient (CMR scar score) was calculated as the sum of transmural extent scores in all segts. LV end diastolic volumes (LVEDV) and LV end systolic volumes (LVESV) were measured by real-time threedimensional echo at baseline and median of 12 months follow-up. There was a significant positive correlation between change in LVEDV with number of scar segts by all three imaging techniques (LVEDV: SPECT scar, r = 0.62, p < 0.001; DbE scar, r = 0.57, p < 0.001; CMR scar, r = 0.52, p < 0.001) but change in LV volumes did not the correlate with number of viable segments. ROC curve analysis showed that remodeling (LVEDV> 15%) was predicted bySPECTscars(AUC= 0.79),DbEscars(AUC= 0.76),CMR scars (AUC= 0.70), and CMR scar score (AUC 0.72). There were no significant differences between any of the ROC curves (Z score

Reduction in scar thickness post myocardial infarction is secondary to increase in regional left ventricular volumes

We sought to determine the relative impact of myocardial scar and viability on post-infarct left ventricular (LV) remodeling in medically-treated patients with LV dysfunction. Forty patients with chronic ischemic heart disease (age 64±9, EF 40±11%) underwent rest-redistribution Tl201 SPECT (scar = 50% transmural extent), A global index of scarring for each patient (CMR scar score) was calculated as the sum of transmural extent scores in all segts. LV end diastolic volumes (LVEDV) and LV end systolic volumes (LVESV) were measured by real-time threedimensional echo at baseline and median of 12 months follow-up. There was a significant positive correlation between change in LVEDV with number of scar segts by all three imaging techniques (LVEDV: SPECT scar, r = 0.62, p < 0.001; DbE scar, r = 0.57, p < 0.001; CMR scar, r = 0.52, p < 0.001) but change in LV volumes did not the correlate with number of viable segments. ROC curve analysis showed that remodeling (LVEDV> 15%) was predicted bySPECTscars(AUC= 0.79),DbEscars(AUC= 0.76),CMR scars (AUC= 0.70), and CMR scar score (AUC 0.72). There were no significant differences between any of the ROC curves (Z score

Relating body mass index to figural stimuli: population-based normative data for Caucasians

OBJECTIVE: To establish body mass index (BMI) norms for standard figural stimuli using a large Caucasian population-based sample. In addition, we sought to determine the effectiveness of the figural stimuli to identify individuals as obese or thin. DESIGN: All Caucasian twins born in Virginia between 1915 and 1971 were identified by public birth record. In addition, 3347 individual twins responded to a letter published in the newsletter of the American Association of Retired Persons (AARP). All adult twins (aged 18 and over) from both of these sources and their family members were mailed a 16 page 'Health and Lifestyle' questionnaire. SUBJECTS: BMI and silhouette data were available on 16 728 females and 11 366 males ranging in age from 18- 100. MEASUREMENTS: Self-report information on height-weight, current body size, desired body size and a discrepancy score using standard figural stimuli. RESULTS: Gender- and age-specific norms are presented linking BMI to each of the figural stimuli. Additional norms for desired body size and discrepancy scores are also presented. Receiver operating curves (ROC) indicate that the figural stimuli are effective in classifying individuals as obese or thin. CONCLUSIONS: With the establishment of these norms, the silhouettes used in standard body image assessment can now be linked to BMI. Differences were observed between women and men in terms of desired body size and discrepancy scores, with women preferring smaller sizes. The figural stimuli are a robust technique for classifying individuals as obese or thin.

The number of rational points of a class of Artin-Schreier curves

We determine the number of F-q-rational points of a class of Artin-Schreier curves by using recent results concerning evaluations of some exponential sums. In particular, we determine infinitely many new examples of maximal and minimal plane curves in the context of the Hasse-Weil bound. (C) 2002 Elsevier Science (USA).

Prospective independent validation of APACHE III models in an Australian tertiary adult intensive care unit

Evaluation of the performance of the APACHE III (Acute Physiology and Chronic Health Evaluation) ICU (intensive care unit) and hospital mortality models at the Princess Alexandra Hospital, Brisbane is reported. Prospective collection of demographic, diagnostic, physiological, laboratory, admission and discharge data of 5681 consecutive eligible admissions (1 January 1995 to 1 January 2000) was conducted at the Princess Alexandra Hospital, a metropolitan Australian tertiary referral medical/surgical adult ICU. ROC (receiver operating characteristic) curve areas for the APACHE III ICU mortality and hospital mortality models demonstrated excellent discrimination. Observed ICU mortality (9.1%) was significantly overestimated by the APACHE III model adjusted for hospital characteristics (10.1%), but did not significantly differ from the prediction of the generic APACHE III model (8.6%). In contrast, observed hospital mortality (14.8%) agreed well with the prediction of the APACHE III model adjusted for hospital characteristics (14.6%), but was significantly underestimated by the unadjusted APACHE III model (13.2%). Calibration curves and goodness-of-fit analysis using Hosmer-Lemeshow statistics, demonstrated that calibration was good with the unadjusted APACHE III ICU mortality model, and the APACHE III hospital mortality model adjusted for hospital characteristics. Post hoc analysis revealed a declining annual SMR (standardized mortality rate) during the study period. This trend was present in each of the non-surgical, emergency and elective surgical diagnostic groups, and the change was temporally related to increased specialist staffing levels. This study demonstrates that the APACHE III model performs well on independent assessment in an Australian hospital. Changes observed in annual SMR using such a validated model support an hypothesis of improved survival outcomes 1995-1999.

A Bronchoscopic scoring system for airway secretions - Airway cellularity and microbiological validation

There is currently no validated scoring system for quantification of airway secretions in children. A user friendly, valid scoring system of airway secretions during flexible bronchoscopy (FB) would be useful for comparative purposes in clinical medicine and research. The objective of this study was to validate our bronchoscopic secretion (BS) scoring system by examining the relationship between the amount of secretions seen at bronchoscopy with airway cellularity and microbiology. In 106 children undergoing FIB, the relationship of BS grades with bronchocalveolar lavage (BAL) cellularity and infective state (bacterial and viral infections) were examined using receptor operator curves (ROC). BAL was obtained according to European Respiratory Society guidelines; first lavage for microbiology and second lavage for cellularity Area under the ROC was significant for total cell count (TCC) and neutrophil % but not for lymphocyte %. BS grade significantly related to infection positive state (chi(2)(trend) = 5.85, P = 0,016). The area under the ROC for infection positive state versus BS grade was 0.645, 95% Cl 0.527-0.763. The BS scoring system is a valid method for quantifying airway secretions in children undergoing bronchoscopy The system related well to airway cellularity and neutrophilia, as well as to an airway infective state. However, the system is only complementary to cell counts and cultures and cannot replace these laboratory quantification techniques.

Species accumulation curves and their applications in parasite ecology

Species accumulation curves (SACs) chart the increase in recovery of new species as a function of some measure of sampling effort. Studies of parasite diversity can benefit from the application of SACs, both as empirical tools to guide sampling efforts and predict richness, and because their properties are informative about community patterns and the structure of parasite diversity. SACs can be used to infer interactivity in parasite infra-communities, to partition species richness into contributions from different spatial scales and different levels of the host hierarchy (individuals, populations and communities) or to identify modes of community assembly (niche versus dispersal). A historical tendency to treat individual hosts as statistically equivalent replicates (quadrats) seemingly satisfies the sample-based subgroup of SACs but care is required in this because of the inequality of hosts as sampling units. Knowledge of the true distribution of parasite richness over multiple host-derived and spatial scales is far from complete but SACs can improve the understanding of diversity patterns in parasite assemblages.