Receiver operating characteristic (ROC) curve for classification of18F-NaF uptake on PET/CT

Abstract Objective: To assess the cutoff values established by ROC curves to classify18F-NaF uptake as normal or malignant. Materials and Methods: PET/CT images were acquired 1 hour after administration of 185 MBq of18F-NaF. Volumes of interest (VOIs) were drawn on three regions of the skeleton as follows: proximal right humerus diaphysis (HD), proximal right femoral diaphysis (FD) and first vertebral body (VB1), in a total of 254 patients, totalling 762 VOIs. The uptake in the VOIs was classified as normal or malignant on the basis of the radiopharmaceutical distribution pattern and of the CT images. A total of 675 volumes were classified as normal and 52 were classified as malignant. Thirty-five VOIs classified as indeterminate or nonmalignant lesions were excluded from analysis. The standardized uptake value (SUV) measured on the VOIs were plotted on an ROC curve for each one of the three regions. The area under the ROC (AUC) as well as the best cutoff SUVs to classify the VOIs were calculated. The best cutoff values were established as the ones with higher result of the sum of sensitivity and specificity. Results: The AUCs were 0.933, 0.889 and 0.975 for UD, FD and VB1, respectively. The best SUV cutoffs were 9.0 (sensitivity: 73%; specificity: 99%), 8.4 (sensitivity: 79%; specificity: 94%) and 21.0 (sensitivity: 93%; specificity: 95%) for UD, FD and VB1, respectively. Conclusion: The best cutoff value varies according to bone region of analysis and it is not possible to establish one value for the whole body.

ROC curves of obesity indicators have a predictive value for children hypertension aged 7-17 years

Objective: The aim of the study is to examine the distribution of integrated covariate and its association with blood pressure (BP) among children in Anhui province, China, and assess the predictive value of integrated covariate to children hypertension. Methods: A total of 2,828 subjects (1,588 male and 1,240 female) aged 7-17 years participated in this study. Height, weight, waistline, hipline and BP of all subjects were measured, obesity and overweight were defined by an international standard, specifying the measurement, the reference population, and the age and sex specific cut off points. High BP status was defined as systolic blood pressure (SBP) and/or diastolic blood pressure (DBP) > 95th percentile for age and gender. Results: Our results revealed that the prevalence of children hypertension was 11.03%, the SBP and DBP of obesity group were significantly higher than that of normal group. Anthropometric obesity indices such as body mass index (BMI) were positively correlated with SBP and DBP. Integrated covariate had a better performance than the single covariate in the receiver-operating characteristic (ROC) curve, the cut-off value; the sensitivity and the specificity of the integrated covariate were 0.112, 0.577, 0.683, respectively. Conclusion: Integrated covariate is a simple and effective anthropometric index to identify childhood hypertension.

Evaluation of non-contact sampling and detection of explosives using receiver operating characteristic curves

The growing need for fast sampling of explosives in high throughput areas has increased the demand for improved technology for the trace detection of illicit compounds. Detection of the volatiles associated with the presence of the illicit compounds offer a different approach for sensitive trace detection of these compounds without increasing the false positive alarm rate. This study evaluated the performance of non-contact sampling and detection systems using statistical analysis through the construction of Receiver Operating Characteristic (ROC) curves in real-world scenarios for the detection of volatiles in the headspace of smokeless powder, used as the model system for generalizing explosives detection. A novel sorbent coated disk coined planar solid phase microextraction (PSPME) was previously used for rapid, non-contact sampling of the headspace containers. The limits of detection for the PSPME coupled to IMS detection was determined to be 0.5-24 ng for vapor sampling of volatile chemical compounds associated with illicit compounds and demonstrated an extraction efficiency of three times greater than other commercially available substrates, retaining >50% of the analyte after 30 minutes sampling of an analyte spike in comparison to a non-detect for the unmodified filters. Both static and dynamic PSPME sampling was used coupled with two ion mobility spectrometer (IMS) detection systems in which 10-500 mg quantities of smokeless powders were detected within 5-10 minutes of static sampling and 1 minute of dynamic sampling time in 1-45 L closed systems, resulting in faster sampling and analysis times in comparison to conventional solid phase microextraction-gas chromatography-mass spectrometry (SPME-GC-MS) analysis. Similar real-world scenarios were sampled in low and high clutter environments with zero false positive rates. Excellent PSPME-IMS detection of the volatile analytes were visualized from the ROC curves, resulting with areas under the curves (AUC) of 0.85-1.0 and 0.81-1.0 for portable and bench-top IMS systems, respectively. Construction of ROC curves were also developed for SPME-GC-MS resulting with AUC of 0.95-1.0, comparable with PSPME-IMS detection. The PSPME-IMS technique provides less false positive results for non-contact vapor sampling, cutting the cost and providing an effective sampling and detection needed in high-throughput scenarios, resulting in similar performance in comparison to well-established techniques with the added advantage of fast detection in the field.

Avaliação do desempenho de indicadores com base na metodologia ROC (Receiver Operating Characteristic)

Dissertação de mestrado em Bioinformática

A nonparametric method of comparing the partial areas under two correlated receiver operating characteristic curves

A non-parametric method was developed and tested to compare the partial areas under two correlated Receiver Operating Characteristic curves. Based on the theory of generalized U-statistics the mathematical formulas have been derived for computing ROC area, and the variance and covariance between the portions of two ROC curves. A practical SAS application also has been developed to facilitate the calculations. The accuracy of the non-parametric method was evaluated by comparing it to other methods. By applying our method to the data from a published ROC analysis of CT image, our results are very close to theirs. A hypothetical example was used to demonstrate the effects of two crossed ROC curves. The two ROC areas are the same. However each portion of the area between two ROC curves were found to be significantly different by the partial ROC curve analysis. For computation of ROC curves with large scales, such as a logistic regression model, we applied our method to the breast cancer study with Medicare claims data. It yielded the same ROC area computation as the SAS Logistic procedure. Our method also provides an alternative to the global summary of ROC area comparison by directly comparing the true-positive rates for two regression models and by determining the range of false-positive values where the models differ. ^

Waist-to-height Ratio Percentiles And Cutoffs For Obesity: A Cross-sectional Study In Brazilian Adolescents.

This study aimed to describe the distribution of waist-to-height ratio (WHtR) percentiles and cutoffs for obesity in Brazilian adolescents. A cross-sectional study including adolescents aged 10 to 15 years was conducted in the city of São Paulo, Brazil; anthropometric measurements (weight, height, and waist-circumference) were taken, and WHtRs were calculated and then divided into percentiles derived by using Least Median of Squares (LMS) regression. The receiver operating characteristic (ROC) curve was used in determining cutoffs for obesity (BMI ≥ 97th percentile) and Mann-Whitney and Kruskal-Wallis tests were used for comparing variables. The study included 8,019 adolescents from 43 schools, of whom 54.5% were female, and 74.8% attended public schools. Boys had higher mean WHtR than girls (0.45 ± 0.06 vs 0.44 ± 0.05; p=0.002) and higher WHtR at the 95th percentile (0.56 vs 0.54; p<0.05). The WHtR cutoffs according to the WHO criteria ranged from 0.467 to 0.506 and 0.463 to 0.496 among girls and boys respectively, with high sensitivity (82.8-95%) and specificity (84-95.5%). The WHtR was significantly associated with body adiposity measured by BMI. Its age-specific percentiles and cutoffs may be used as additional surrogate markers of central obesity and its co-morbidities.

The influence of experience and training in a group of novice observers: a jackknife alternative free-response receiver operating characteristic analysis

Purpose - The study evaluates the pre- and post-training lesion localisation ability of a group of novice observers. Parallels are drawn with the performance of inexperienced radiographers taking part in preliminary clinical evaluation (PCE) and ‘red-dot’ systems, operating within radiography practice. Materials and methods - Thirty-four novice observers searched 92 images for simulated lesions. Pre-training and post-training evaluations were completed following the free-response the receiver operating characteristic (FROC) method. Training consisted of observer performance methodology, the characteristics of the simulated lesions and information on lesion frequency. Jackknife alternative FROC (JAFROC) and highest rating inferred ROC analyses were performed to evaluate performance difference on lesion-based and case-based decisions. The significance level of the test was set at 0.05 to control the probability of Type I error. Results - JAFROC analysis (F(3,33) = 26.34, p < 0.0001) and highest-rating inferred ROC analysis (F(3,33) = 10.65, p = 0.0026) revealed a statistically significant difference in lesion detection performance. The JAFROC figure-of-merit was 0.563 (95% CI 0.512,0.614) pre-training and 0.677 (95% CI 0.639,0.715) post-training. Highest rating inferred ROC figure-of-merit was 0.728 (95% CI 0.701,0.755) pre-training and 0.772 (95% CI 0.750,0.793) post-training. Conclusions - This study has demonstrated that novice observer performance can improve significantly. This study design may have relevance in the assessment of inexperienced radiographers taking part in PCE or commenting scheme for trauma.

Decision curve analysis revisited: overall net benefit, relationships to ROC curve analysis, and application to case-control studies.

ABSTRACT: BACKGROUND: Decision curve analysis has been introduced as a method to evaluate prediction models in terms of their clinical consequences if used for a binary classification of subjects into a group who should and into a group who should not be treated. The key concept for this type of evaluation is the "net benefit", a concept borrowed from utility theory. METHODS: We recall the foundations of decision curve analysis and discuss some new aspects. First, we stress the formal distinction between the net benefit for the treated and for the untreated and define the concept of the "overall net benefit". Next, we revisit the important distinction between the concept of accuracy, as typically assessed using the Youden index and a receiver operating characteristic (ROC) analysis, and the concept of utility of a prediction model, as assessed using decision curve analysis. Finally, we provide an explicit implementation of decision curve analysis to be applied in the context of case-control studies. RESULTS: We show that the overall net benefit, which combines the net benefit for the treated and the untreated, is a natural alternative to the benefit achieved by a model, being invariant with respect to the coding of the outcome, and conveying a more comprehensive picture of the situation. Further, within the framework of decision curve analysis, we illustrate the important difference between the accuracy and the utility of a model, demonstrating how poor an accurate model may be in terms of its net benefit. Eventually, we expose that the application of decision curve analysis to case-control studies, where an accurate estimate of the true prevalence of a disease cannot be obtained from the data, is achieved with a few modifications to the original calculation procedure. CONCLUSIONS: We present several interrelated extensions to decision curve analysis that will both facilitate its interpretation and broaden its potential area of application.

Precision-recall operating characteristic (P-ROC) curves in imprecise environments

Traditionally, machine learning algorithms have been evaluated in applications where assumptions can be reliably made about class priors and/or misclassification costs. In this paper, we consider the case of imprecise environments, where little may be known about these factors and they may well vary significantly when the system is applied. Specifically, the use of precision-recall analysis is investigated and compared to the more well known performance measures such as error-rate and the receiver operating characteristic (ROC). We argue that while ROC analysis is invariant to variations in class priors, this invariance in fact hides an important factor of the evaluation in imprecise environments. Therefore, we develop a generalised precision-recall analysis methodology in which variation due to prior class probabilities is incorporated into a multi-way analysis of variance (ANOVA). The increased sensitivity and reliability of this approach is demonstrated in a remote sensing application.

Nonparametric estimation of time-dependent ROC curves conditional on a continuous covariate

The receiver-operating characteristic (ROC) curve is the most widely used measure for evaluating the performance of a diagnostic biomarker when predicting a binary disease outcome. The ROC curve displays the true positive rate (or sensitivity) and the false positive rate (or 1-specificity) for different cut-off values used to classify an individual as healthy or diseased. In time-to-event studies, however, the disease status (e.g. death or alive) of an individual is not a fixed characteristic, and it varies along the study. In such cases, when evaluating the performance of the biomarker, several issues should be taken into account: first, the time-dependent nature of the disease status; and second, the presence of incomplete data (e.g. censored data typically present in survival studies). Accordingly, to assess the discrimination power of continuous biomarkers for time-dependent disease outcomes, time-dependent extensions of true positive rate, false positive rate, and ROC curve have been recently proposed. In this work, we present new nonparametric estimators of the cumulative/dynamic time-dependent ROC curve that allow accounting for the possible modifying effect of current or past covariate measures on the discriminatory power of the biomarker. The proposed estimators can accommodate right-censored data, as well as covariate-dependent censoring. The behavior of the estimators proposed in this study will be explored through simulations and illustrated using data from a cohort of patients who suffered from acute coronary syndrome.

Incidence and Predictors of Cardiovascular Complications and Death after Vascular Surgery

AbstractBackground:Patients undergoing arterial vascular surgery are considered at increased risk for post-operative complications.Objective:To assess the incidence and predictors of complications and death, as well as the performance of two models of risk stratification, in vascular surgery.Methods:This study determined the incidence of cardiovascular complications and deaths within 30 days from surgery in adults. Univariate comparison and logistic regression assessed the risk factors associated with the outcomes, and the receiver operating characteristic (ROC) curve assessed the discriminatory capacity of the revised cardiac risk index (RCRI) and vascular study group of New England cardiac risk index (VSG-CRI).Results:141 patients (mean age, 66 years; 65% men) underwent the following surgeries: carotid (15); lower limbs (65); abdominal aorta (56); and others (5). Cardiovascular complications and death occurred within 30 days in 28 (19.9%) and 20 (14.2%) patients, respectively. The risk predictors were: age, obesity, stroke, poor functional capacity, altered scintigraphy, surgery of the aorta, and troponin change. The scores RCRI and VSG-CRI had area under the curve of 0.635 and 0.639 for early cardiovascular complications, and 0.562 and 0.610 for death in 30 days.Conclusion:In this small and selected group of patients undergoing arterial vascular surgery, the incidence of adverse events was elevated. The risk assessment indices RCRI and VSG-CRI did not perform well for complications within 30 days.

Accuracy of computer-aided ultrasound as compared with magnetic resonance imaging in the evaluation of nonalcoholic fatty liver disease in obese and eutrophic adolescents

AbstractObjective:To compare the accuracy of computer-aided ultrasound (US) and magnetic resonance imaging (MRI) by means of hepatorenal gradient analysis in the evaluation of nonalcoholic fatty liver disease (NAFLD) in adolescents.Materials and Methods:This prospective, cross-sectional study evaluated 50 adolescents (aged 11–17 years), including 24 obese and 26 eutrophic individuals. All adolescents underwent computer-aided US, MRI, laboratory tests, and anthropometric evaluation. Sensitivity, specificity, positive and negative predictive values and accuracy were evaluated for both imaging methods, with subsequent generation of the receiver operating characteristic (ROC) curve and calculation of the area under the ROC curve to determine the most appropriate cutoff point for the hepatorenal gradient in order to predict the degree of steatosis, utilizing MRI results as the gold-standard.Results:The obese group included 29.2% girls and 70.8% boys, and the eutrophic group, 69.2% girls and 30.8% boys. The prevalence of NAFLD corresponded to 19.2% for the eutrophic group and 83% for the obese group. The ROC curve generated for the hepatorenal gradient with a cutoff point of 13 presented 100% sensitivity and 100% specificity. As the same cutoff point was considered for the eutrophic group, false-positive results were observed in 9.5% of cases (90.5% specificity) and false-negative results in 0% (100% sensitivity).Conclusion:Computer-aided US with hepatorenal gradient calculation is a simple and noninvasive technique for semiquantitative evaluation of hepatic echogenicity and could be useful in the follow-up of adolescents with NAFLD, population screening for this disease as well as for clinical studies.

Model Checking for ROC Regression Analysis

The Receiver Operating Characteristic (ROC) curve is a prominent tool for characterizing the accuracy of continuous diagnostic test. To account for factors that might invluence the test accuracy, various ROC regression methods have been proposed. However, as in any regression analysis, when the assumed models do not fit the data well, these methods may render invalid and misleading results. To date practical model checking techniques suitable for validating existing ROC regression models are not yet available. In this paper, we develop cumulative residual based procedures to graphically and numerically assess the goodness-of-fit for some commonly used ROC regression models, and show how specific components of these models can be examined within this framework. We derive asymptotic null distributions for the residual process and discuss resampling procedures to approximate these distributions in practice. We illustrate our methods with a dataset from the Cystic Fibrosis registry.

High serum follistatin levels in women with ovarian endometriosis

Follistatin is an activin-binding protein produced by several tissues, including endometrium and endometriotic implants. We aimed to quantify follistatin in patients with ovarian endometriosis and investigate its value as a diagnostic marker. Women undergoing laparoscopic excision of ovarian endometrioma (n = 52) or other benign ovarian cysts (n = 52) were studied, plus women with non-ovarian endometriosis (n = 11) and healthy controls (n = 27). Serum was collected from all subjects, and peritoneal and cystic fluid from a subset with endometrioma. Follistatin was measured by enzyme-linked immunosorbent assay. The diagnostic accuracy of follistatin to detect endometrioma was evaluated by receiver operating characteristic (ROC) curve and compared with cancer antigen (CA)-125. Serum follistatin was increased in women with ovarian endometrioma (2080 +/- 94 pg/ml) compared with controls (545 +/- 49 pg/ml, P < 0.001), other benign ovarian cysts (795 +/- 60 pg/ml, P < 0.001) or non-ovarian endometriosis (1271 +/- 115 pg/ml, P < 0.001). Cystic fluid showed a higher concentration of follistatin (9850 +/- 4461 pg/ml) than peritoneal fluid (1885 +/- 261 pg/ml, P < 0.001) and serum (P < 0.001). Follistatin levels detected 48/52 cases of endometrioma (92% sensitivity) at 1433 pg/ml cut-off, corresponding to 92% specificity. CA-125 detected only 44% of endometriomas with 90% specificity. ROC curve comparison showed follistatin was more accurate than CA-125 to discriminate women with endometrioma either from controls or women with other benign ovarian cysts (P < 0.0001). Serum follistatin is increased in women with endometriosis and allows clear distinction between endometrioma and other benign ovarian cysts. Follistatin has the sensitivity and specificity to become a useful clinical marker of ovarian endometrioma.

Validation of a Cutoff Value on Echo Doppler Analysis to Replace Right Heart Catheterization During Pulmonary Hypertension Evaluation in Heart Transplant Candidates

Background. Heart transplantation (OHT) has traditionally been contraindicated in the presence of severe pulmonary hypertension (PH), as detected by right heart catheterization. Noninvasive methods are still not reliably accurate to make this evaluation. Objectives. Determine the efficacy of echo Doppler analysis for the diagnosis of severe PH. Methods. One hundred thirty patients (mean age = 42 +/- 15 years, 82 men) showed severe left ventricular dysfunction (mean ejection fraction = 29 +/- 12%; functional class III-IV). We excluded patients with atrial fibrillation, heart failure secondary to congenital disease, and valvulopathy. The pulmonary parameters defined as severe PH were: systolic pulmonary artery pressure (sPAP) >= 60 mm Hg; a mean transpulmonary gradient >= 15; or pulmonary vascular resistance >= 5 Wood units. Patients underwent a right heart catheterization using a Swan-Ganz catheter to measure hemodynamic parameters and to noninvasively estimate right-sided pressures from spectral Doppler recordings of tricuspid regurgitation velocity (right ventricular systolic pressure [RVsP]). A Pearson correlation of sPAP was obtained with RVsP by; the sensitivity of RVsP for the diagnosis of PH was determined by a receiver operating characteristic (ROC) curve. Results. A good correlation between sPAP and RVsP was obtained by Pearson correlation analysis (r = 0.64; 95% confidence interval [CI] 0.50-0.75; P < .001). The ROC curve analysis showed a sensitivity of 100%, a specificity of 37.2%, (95% CI 0.69-0.83, P < .0001) of a RVsP < 45 mm Hg (cutoff) on the exclusion of severe PH. Conclusions. The cutoff of RVsP < 45 mm Hg, on noninvasive echo Doppler evaluation of PH is an efficient method to replace invasive heart catheterization in OHT candidates.