Decision curve analysis revisited: overall net benefit, relationships to ROC curve analysis, and application to case-control studies.

ABSTRACT: BACKGROUND: Decision curve analysis has been introduced as a method to evaluate prediction models in terms of their clinical consequences if used for a binary classification of subjects into a group who should and into a group who should not be treated. The key concept for this type of evaluation is the "net benefit", a concept borrowed from utility theory. METHODS: We recall the foundations of decision curve analysis and discuss some new aspects. First, we stress the formal distinction between the net benefit for the treated and for the untreated and define the concept of the "overall net benefit". Next, we revisit the important distinction between the concept of accuracy, as typically assessed using the Youden index and a receiver operating characteristic (ROC) analysis, and the concept of utility of a prediction model, as assessed using decision curve analysis. Finally, we provide an explicit implementation of decision curve analysis to be applied in the context of case-control studies. RESULTS: We show that the overall net benefit, which combines the net benefit for the treated and the untreated, is a natural alternative to the benefit achieved by a model, being invariant with respect to the coding of the outcome, and conveying a more comprehensive picture of the situation. Further, within the framework of decision curve analysis, we illustrate the important difference between the accuracy and the utility of a model, demonstrating how poor an accurate model may be in terms of its net benefit. Eventually, we expose that the application of decision curve analysis to case-control studies, where an accurate estimate of the true prevalence of a disease cannot be obtained from the data, is achieved with a few modifications to the original calculation procedure. CONCLUSIONS: We present several interrelated extensions to decision curve analysis that will both facilitate its interpretation and broaden its potential area of application.

Receiver operating characteristic (ROC) curve for classification of18F-NaF uptake on PET/CT

Abstract Objective: To assess the cutoff values established by ROC curves to classify18F-NaF uptake as normal or malignant. Materials and Methods: PET/CT images were acquired 1 hour after administration of 185 MBq of18F-NaF. Volumes of interest (VOIs) were drawn on three regions of the skeleton as follows: proximal right humerus diaphysis (HD), proximal right femoral diaphysis (FD) and first vertebral body (VB1), in a total of 254 patients, totalling 762 VOIs. The uptake in the VOIs was classified as normal or malignant on the basis of the radiopharmaceutical distribution pattern and of the CT images. A total of 675 volumes were classified as normal and 52 were classified as malignant. Thirty-five VOIs classified as indeterminate or nonmalignant lesions were excluded from analysis. The standardized uptake value (SUV) measured on the VOIs were plotted on an ROC curve for each one of the three regions. The area under the ROC (AUC) as well as the best cutoff SUVs to classify the VOIs were calculated. The best cutoff values were established as the ones with higher result of the sum of sensitivity and specificity. Results: The AUCs were 0.933, 0.889 and 0.975 for UD, FD and VB1, respectively. The best SUV cutoffs were 9.0 (sensitivity: 73%; specificity: 99%), 8.4 (sensitivity: 79%; specificity: 94%) and 21.0 (sensitivity: 93%; specificity: 95%) for UD, FD and VB1, respectively. Conclusion: The best cutoff value varies according to bone region of analysis and it is not possible to establish one value for the whole body.

Active Learning to Maximize Area Under the ROC Curve

In active learning, a machine learning algorithmis given an unlabeled set of examples U, and is allowed to request labels for a relatively small subset of U to use for training. The goal is then to judiciously choose which examples in U to have labeled in order to optimize some performance criterion, e.g. classification accuracy. We study how active learning affects AUC. We examine two existing algorithms from the literature and present our own active learning algorithms designed to maximize the AUC of the hypothesis. One of our algorithms was consistently the top performer, and Closest Sampling from the literature often came in second behind it. When good posterior probability estimates were available, our heuristics were by far the best.

Nonparametric estimation of time-dependent ROC curves conditional on a continuous covariate

The receiver-operating characteristic (ROC) curve is the most widely used measure for evaluating the performance of a diagnostic biomarker when predicting a binary disease outcome. The ROC curve displays the true positive rate (or sensitivity) and the false positive rate (or 1-specificity) for different cut-off values used to classify an individual as healthy or diseased. In time-to-event studies, however, the disease status (e.g. death or alive) of an individual is not a fixed characteristic, and it varies along the study. In such cases, when evaluating the performance of the biomarker, several issues should be taken into account: first, the time-dependent nature of the disease status; and second, the presence of incomplete data (e.g. censored data typically present in survival studies). Accordingly, to assess the discrimination power of continuous biomarkers for time-dependent disease outcomes, time-dependent extensions of true positive rate, false positive rate, and ROC curve have been recently proposed. In this work, we present new nonparametric estimators of the cumulative/dynamic time-dependent ROC curve that allow accounting for the possible modifying effect of current or past covariate measures on the discriminatory power of the biomarker. The proposed estimators can accommodate right-censored data, as well as covariate-dependent censoring. The behavior of the estimators proposed in this study will be explored through simulations and illustrated using data from a cohort of patients who suffered from acute coronary syndrome.

Aplicação da metodologia ROC na avaliação de desempenho de índices de gravidade clínica em unidades de neonatologia de Portugal

Tese de Doutoramento em Engenharia Industrial e de Sistemas

Model Checking for ROC Regression Analysis

The Receiver Operating Characteristic (ROC) curve is a prominent tool for characterizing the accuracy of continuous diagnostic test. To account for factors that might invluence the test accuracy, various ROC regression methods have been proposed. However, as in any regression analysis, when the assumed models do not fit the data well, these methods may render invalid and misleading results. To date practical model checking techniques suitable for validating existing ROC regression models are not yet available. In this paper, we develop cumulative residual based procedures to graphically and numerically assess the goodness-of-fit for some commonly used ROC regression models, and show how specific components of these models can be examined within this framework. We derive asymptotic null distributions for the residual process and discuss resampling procedures to approximate these distributions in practice. We illustrate our methods with a dataset from the Cystic Fibrosis registry.

ROC curves of obesity indicators have a predictive value for children hypertension aged 7-17 years

Objective: The aim of the study is to examine the distribution of integrated covariate and its association with blood pressure (BP) among children in Anhui province, China, and assess the predictive value of integrated covariate to children hypertension. Methods: A total of 2,828 subjects (1,588 male and 1,240 female) aged 7-17 years participated in this study. Height, weight, waistline, hipline and BP of all subjects were measured, obesity and overweight were defined by an international standard, specifying the measurement, the reference population, and the age and sex specific cut off points. High BP status was defined as systolic blood pressure (SBP) and/or diastolic blood pressure (DBP) > 95th percentile for age and gender. Results: Our results revealed that the prevalence of children hypertension was 11.03%, the SBP and DBP of obesity group were significantly higher than that of normal group. Anthropometric obesity indices such as body mass index (BMI) were positively correlated with SBP and DBP. Integrated covariate had a better performance than the single covariate in the receiver-operating characteristic (ROC) curve, the cut-off value; the sensitivity and the specificity of the integrated covariate were 0.112, 0.577, 0.683, respectively. Conclusion: Integrated covariate is a simple and effective anthropometric index to identify childhood hypertension.

Biochemical, Anthropometric And Body Composition Indicators As Predictors Of Hepatic Steatosis In Obese Adolescents.

To describe the prevalence of hepatic steatosis and to assess the performance of biochemical, anthropometric and body composition indicators for hepatic steatosis in obese teenagers. Cross-sectional study including 79 adolecents aged from ten to 18 years old. Hepatic steatosis was diagnosed by abdominal ultrasound in case of moderate or intense hepatorenal contrast and/or a difference in the histogram ≥7 on the right kidney cortex. The insulin resistance was determined by the Homeostasis Model Assessment-Insulin Resistance (HOMA-IR) index for values >3.16. Anthropometric and body composition indicators consisted of body mass index, body fat percentage, abdominal circumference and subcutaneous fat. Fasting glycemia and insulin, lipid profile and hepatic enzymes, such as aspartate aminotransferase, alanine aminotransferase, gamma-glutamyltransferase and alkaline phosphatase, were also evaluated. In order to assess the performance of these indicators in the diagnosis of hepatic steatosis in teenagers, a ROC curve analysis was applied. Hepatic steatosis was found in 20% of the patients and insulin resistance, in 29%. Gamma-glutamyltransferase and HOMA-IR were good indicators for predicting hepatic steatosis, with a cutoff of 1.06 times above the reference value for gamma-glutamyltransferase and 3.28 times for the HOMA-IR. The anthropometric indicators, the body fat percentage, the lipid profile, the glycemia and the aspartate aminotransferase did not present significant associations. Patients with high gamma-glutamyltransferase level and/or HOMA-IR should be submitted to abdominal ultrasound examination due to the increased chance of having hepatic steatosis.

Waist-to-height Ratio Percentiles And Cutoffs For Obesity: A Cross-sectional Study In Brazilian Adolescents.

This study aimed to describe the distribution of waist-to-height ratio (WHtR) percentiles and cutoffs for obesity in Brazilian adolescents. A cross-sectional study including adolescents aged 10 to 15 years was conducted in the city of São Paulo, Brazil; anthropometric measurements (weight, height, and waist-circumference) were taken, and WHtRs were calculated and then divided into percentiles derived by using Least Median of Squares (LMS) regression. The receiver operating characteristic (ROC) curve was used in determining cutoffs for obesity (BMI ≥ 97th percentile) and Mann-Whitney and Kruskal-Wallis tests were used for comparing variables. The study included 8,019 adolescents from 43 schools, of whom 54.5% were female, and 74.8% attended public schools. Boys had higher mean WHtR than girls (0.45 ± 0.06 vs 0.44 ± 0.05; p=0.002) and higher WHtR at the 95th percentile (0.56 vs 0.54; p<0.05). The WHtR cutoffs according to the WHO criteria ranged from 0.467 to 0.506 and 0.463 to 0.496 among girls and boys respectively, with high sensitivity (82.8-95%) and specificity (84-95.5%). The WHtR was significantly associated with body adiposity measured by BMI. Its age-specific percentiles and cutoffs may be used as additional surrogate markers of central obesity and its co-morbidities.

Aplicação do escore MELD em pacientes submetidos a transplante de fígado: análise retrospectiva da sobrevida e dos fatores preditivos a curto e longo prazo

BACKGROUND: The model for end-stage liver disease (MELD) was developed to predict short-term mortality in patients with cirrhosis. There are few reports studying the correlation between MELD and long-term posttransplantation survival. AIM: To assess the value of pretransplant MELD in the prediction of posttransplant survival. METHODS: The adult patients (age >18 years) who underwent liver transplantation were examined in a retrospective longitudinal cohort of patients, through the prospective data base. We excluded acute liver failure, retransplantation and reduced or split-livers. The liver donors were evaluated according to: age, sex, weight, creatinine, bilirubin, sodium, aspartate aminotransferase, personal antecedents, brain death cause, steatosis, expanded criteria donor number and index donor risk. The recipients' data were: sex, age, weight, chronic hepatic disease, Child-Turcotte-Pugh points, pretransplant and initial MELD score, pretransplant creatinine clearance, sodium, cold and warm ischemia times, hospital length of stay, blood requirements, and alanine aminotransferase (ALT >1,000 UI/L = liver dysfunction). The Kaplan-Meier method with the log-rank test was used for the univariable analyses of posttransplant patient survival. For the multivariable analyses the Cox proportional hazard regression method with the stepwise procedure was used with stratifying sodium and MELD as variables. ROC curve was used to define area under the curve for MELD and Child-Turcotte-Pugh. RESULTS: A total of 232 patients with 10 years follow up were available. The MELD cutoff was 20 and Child-Turcotte-Pugh cutoff was 11.5. For MELD score > 20, the risk factors for death were: red cell requirements, liver dysfunction and donor's sodium. For the patients with hyponatremia the risk factors were: negative delta-MELD score, red cell requirements, liver dysfunction and donor's sodium. The regression univariated analyses came up with the following risk factors for death: score MELD > 25, blood requirements, recipient creatinine clearance pretransplant and age donor >50. After stepwise analyses, only red cell requirement was predictive. Patients with MELD score < 25 had a 68.86%, 50,44% and 41,50% chance for 1, 5 and 10-year survival and > 25 were 39.13%, 29.81% and 22.36% respectively. Patients without hyponatremia were 65.16%, 50.28% and 41,98% and with hyponatremia 44.44%, 34.28% and 28.57% respectively. Patients with IDR > 1.7 showed 53.7%, 27.71% and 13.85% and index donor risk <1.7 was 63.62%, 51.4% and 44.08%, respectively. Age donor > 50 years showed 38.4%, 26.21% and 13.1% and age donor <50 years showed 65.58%, 26.21% and 13.1%. Association with delta-MELD score did not show any significant difference. Expanded criteria donors were associated with primary non-function and severe liver dysfunction. Predictive factors for death were blood requirements, hyponatremia, liver dysfunction and donor's sodium. CONCLUSION: In conclusion MELD over 25, recipient's hyponatremia, blood requirements, donor's sodium were associated with poor survival.

Comparison between the 1% and 2% ibopamine provocative test in primary open-angle glaucoma patients: sensitivity, specificity and tolerability

PURPOSE: To compare intraocular pressure (IOP) rise in normal individuals and primary open-angle glaucoma patients and the safety and efficacy of ibopamine eye drops in different concentrations as a provocative test for glaucoma. METHODS: Glaucoma patients underwent (same eye) the ibopamine provocative test with two concentrations, 1% and 2%, in a random sequence at least 3 weeks apart, but not more than 3 months. The normal individuals were randomly submitted to one of the concentrations of ibopamine (1% and 2%). The test was considered positive if there was an IOP rise greater than 3 or 4 mmHg at 30 or 45 minutes to test which subset of the test has the best sensitivity (Se)/specificity (Sp). RESULTS: There was no statistically significant difference in any of the IOP measurements, comparing 1% with 2% ibopamine. The IOP was significantly higher at 30 and 45 minutes with both concentrations (p<0.001). The best sensitivity/specificity ratio was achieved with the cutoff point set as greater than 3 mmHg at 45 minutes with 2% ibopamine (area under the ROC curve: 0.864, Se: 84.6%; Sp:73.3%). All patients described a slight burning after ibopamine's instillation. CONCLUSION: 2% ibopamine is recommended as a provocative test for glaucoma. Because both concentrations have similar ability to rise IOP, 1% ibopamine may be used to treat ocular hypotony.

Ibopamina 2% vs. sobrecarga hídrica como teste provocativo para glaucoma

PURPOSE: To compare the 2% ibopamine provocative test with the water drinking test as a provocative test for glaucoma. METHODS: Primary open-angle glaucoma patients and normal individuals were selected from CEROF-Universidade Federal de Goiânia UFG, and underwent the 2% ibopamine provocative test and the water drinking test in a randomized fashion, at least 1 week apart. Intraocular pressure (IOP) before and after both tests, Bland-Altman graph, sensitivity and specificity (as mesured by ROC curves) were obtained for both methods. RESULTS: Forty-seven eyes from 25 patients were included (27 eyes from 15 glaucoma patients and 20 eyes from 10 normal individuals), with a mean age of 54.2 ± 12.7 years. The mean MD of glaucoma patients was -2.8 ± 2.11 dB. There was no statistically difference in the baseline IOP (p=0.8) comparing glaucoma patients, but positive after the provocative tests (p=0.03), and in the IOP variation (4.4 ± 1.3 mmHg for ibopamine and 3.2 ± 2.2 mmHg for water drinking test, p=0.01). There was no difference in all studied parameters for normal individuals. The Bland-Altman graph showed high dispersion comparing both methods. The areas under the ROC curve were 0.987 for the ibopamine provocative test, and 0.807 for the water-drinking test. CONCLUSION: In this selected subgroup of glaucoma patients with early visual field defect, the ibopamine provocative test has shown better sensitivity/specificity than the water drinking test.

Validity and reliability of methods for the detection of secondary caries around amalgam restorations in primary teeth

Secondary caries has been reported as the main reason for restoration replacement. The aim of this in vitro study was to evaluate the performance of different methods - visual inspection, laser fluorescence (DIAGNOdent), radiography and tactile examination - for secondary caries detection in primary molars restored with amalgam. Fifty-four primary molars were photographed and 73 suspect sites adjacent to amalgam restorations were selected. Two examiners evaluated independently these sites using all methods. Agreement between examiners was assessed by the Kappa test. To validate the methods, a caries-detector dye was used after restoration removal. The best cut-off points for the sample were found by a Receiver Operator Characteristic (ROC) analysis, and the area under the ROC curve (Az), and the sensitivity, specificity and accuracy of the methods were calculated for enamel (D2) and dentine (D3) thresholds. These parameters were found for each method and then compared by the McNemar test. The tactile examination and visual inspection presented the highest inter-examiner agreement for the D2 and D3 thresholds, respectively. The visual inspection also showed better performance than the other methods for both thresholds (Az = 0.861 and Az = 0.841, respectively). In conclusion, the visual inspection presented the best performance for detecting enamel and dentin secondary caries in primary teeth restored with amalgam.

Classification of breast masses using a committee machine of artificial neural networks

This work proposes a new approach using a committee machine of artificial neural networks to classify masses found in mammograms as benign or malignant. Three shape factors, three edge-sharpness measures, and 14 texture measures are used for the classification of 20 regions of interest (ROIs) related to malignant tumors and 37 ROIs related to benign masses. A group of multilayer perceptrons (MLPs) is employed as a committee machine of neural network classifiers. The classification results are reached by combining the responses of the individual classifiers. Experiments involving changes in the learning algorithm of the committee machine are conducted. The classification accuracy is evaluated using the area A. under the receiver operating characteristics (ROC) curve. The A, result for the committee machine is compared with the A, results obtained using MLPs and single-layer perceptrons (SLPs), as well as a linear discriminant analysis (LDA) classifier Tests are carried out using the student's t-distribution. The committee machine classifier outperforms the MLP SLP, and LDA classifiers in the following cases: with the shape measure of spiculation index, the A, values of the four methods are, in order 0.93, 0.84, 0.75, and 0.76; and with the edge-sharpness measure of acutance, the values are 0.79, 0.70, 0.69, and 0.74. Although the features with which improvement is obtained with the committee machines are not the same as those that provided the maximal value of A(z) (A(z) = 0.99 with some shape features, with or without the committee machine), they correspond to features that are not critically dependent on the accuracy of the boundaries of the masses, which is an important result. (c) 2008 SPIE and IS&T.