Receiver operating characteristic (ROC) curve for classification of18F-NaF uptake on PET/CT

Abstract Objective: To assess the cutoff values established by ROC curves to classify18F-NaF uptake as normal or malignant. Materials and Methods: PET/CT images were acquired 1 hour after administration of 185 MBq of18F-NaF. Volumes of interest (VOIs) were drawn on three regions of the skeleton as follows: proximal right humerus diaphysis (HD), proximal right femoral diaphysis (FD) and first vertebral body (VB1), in a total of 254 patients, totalling 762 VOIs. The uptake in the VOIs was classified as normal or malignant on the basis of the radiopharmaceutical distribution pattern and of the CT images. A total of 675 volumes were classified as normal and 52 were classified as malignant. Thirty-five VOIs classified as indeterminate or nonmalignant lesions were excluded from analysis. The standardized uptake value (SUV) measured on the VOIs were plotted on an ROC curve for each one of the three regions. The area under the ROC (AUC) as well as the best cutoff SUVs to classify the VOIs were calculated. The best cutoff values were established as the ones with higher result of the sum of sensitivity and specificity. Results: The AUCs were 0.933, 0.889 and 0.975 for UD, FD and VB1, respectively. The best SUV cutoffs were 9.0 (sensitivity: 73%; specificity: 99%), 8.4 (sensitivity: 79%; specificity: 94%) and 21.0 (sensitivity: 93%; specificity: 95%) for UD, FD and VB1, respectively. Conclusion: The best cutoff value varies according to bone region of analysis and it is not possible to establish one value for the whole body.

Evaluation of non-contact sampling and detection of explosives using receiver operating characteristic curves

The growing need for fast sampling of explosives in high throughput areas has increased the demand for improved technology for the trace detection of illicit compounds. Detection of the volatiles associated with the presence of the illicit compounds offer a different approach for sensitive trace detection of these compounds without increasing the false positive alarm rate. This study evaluated the performance of non-contact sampling and detection systems using statistical analysis through the construction of Receiver Operating Characteristic (ROC) curves in real-world scenarios for the detection of volatiles in the headspace of smokeless powder, used as the model system for generalizing explosives detection. A novel sorbent coated disk coined planar solid phase microextraction (PSPME) was previously used for rapid, non-contact sampling of the headspace containers. The limits of detection for the PSPME coupled to IMS detection was determined to be 0.5-24 ng for vapor sampling of volatile chemical compounds associated with illicit compounds and demonstrated an extraction efficiency of three times greater than other commercially available substrates, retaining >50% of the analyte after 30 minutes sampling of an analyte spike in comparison to a non-detect for the unmodified filters. Both static and dynamic PSPME sampling was used coupled with two ion mobility spectrometer (IMS) detection systems in which 10-500 mg quantities of smokeless powders were detected within 5-10 minutes of static sampling and 1 minute of dynamic sampling time in 1-45 L closed systems, resulting in faster sampling and analysis times in comparison to conventional solid phase microextraction-gas chromatography-mass spectrometry (SPME-GC-MS) analysis. Similar real-world scenarios were sampled in low and high clutter environments with zero false positive rates. Excellent PSPME-IMS detection of the volatile analytes were visualized from the ROC curves, resulting with areas under the curves (AUC) of 0.85-1.0 and 0.81-1.0 for portable and bench-top IMS systems, respectively. Construction of ROC curves were also developed for SPME-GC-MS resulting with AUC of 0.95-1.0, comparable with PSPME-IMS detection. The PSPME-IMS technique provides less false positive results for non-contact vapor sampling, cutting the cost and providing an effective sampling and detection needed in high-throughput scenarios, resulting in similar performance in comparison to well-established techniques with the added advantage of fast detection in the field.

A nonparametric method of comparing the partial areas under two correlated receiver operating characteristic curves

A non-parametric method was developed and tested to compare the partial areas under two correlated Receiver Operating Characteristic curves. Based on the theory of generalized U-statistics the mathematical formulas have been derived for computing ROC area, and the variance and covariance between the portions of two ROC curves. A practical SAS application also has been developed to facilitate the calculations. The accuracy of the non-parametric method was evaluated by comparing it to other methods. By applying our method to the data from a published ROC analysis of CT image, our results are very close to theirs. A hypothetical example was used to demonstrate the effects of two crossed ROC curves. The two ROC areas are the same. However each portion of the area between two ROC curves were found to be significantly different by the partial ROC curve analysis. For computation of ROC curves with large scales, such as a logistic regression model, we applied our method to the breast cancer study with Medicare claims data. It yielded the same ROC area computation as the SAS Logistic procedure. Our method also provides an alternative to the global summary of ROC area comparison by directly comparing the true-positive rates for two regression models and by determining the range of false-positive values where the models differ. ^

Precision-recall operating characteristic (P-ROC) curves in imprecise environments

Traditionally, machine learning algorithms have been evaluated in applications where assumptions can be reliably made about class priors and/or misclassification costs. In this paper, we consider the case of imprecise environments, where little may be known about these factors and they may well vary significantly when the system is applied. Specifically, the use of precision-recall analysis is investigated and compared to the more well known performance measures such as error-rate and the receiver operating characteristic (ROC). We argue that while ROC analysis is invariant to variations in class priors, this invariance in fact hides an important factor of the evaluation in imprecise environments. Therefore, we develop a generalised precision-recall analysis methodology in which variation due to prior class probabilities is incorporated into a multi-way analysis of variance (ANOVA). The increased sensitivity and reliability of this approach is demonstrated in a remote sensing application.

Avaliação do desempenho de indicadores com base na metodologia ROC (Receiver Operating Characteristic)

Dissertação de mestrado em Bioinformática

The influence of experience and training in a group of novice observers: a jackknife alternative free-response receiver operating characteristic analysis

Purpose - The study evaluates the pre- and post-training lesion localisation ability of a group of novice observers. Parallels are drawn with the performance of inexperienced radiographers taking part in preliminary clinical evaluation (PCE) and ‘red-dot’ systems, operating within radiography practice. Materials and methods - Thirty-four novice observers searched 92 images for simulated lesions. Pre-training and post-training evaluations were completed following the free-response the receiver operating characteristic (FROC) method. Training consisted of observer performance methodology, the characteristics of the simulated lesions and information on lesion frequency. Jackknife alternative FROC (JAFROC) and highest rating inferred ROC analyses were performed to evaluate performance difference on lesion-based and case-based decisions. The significance level of the test was set at 0.05 to control the probability of Type I error. Results - JAFROC analysis (F(3,33) = 26.34, p < 0.0001) and highest-rating inferred ROC analysis (F(3,33) = 10.65, p = 0.0026) revealed a statistically significant difference in lesion detection performance. The JAFROC figure-of-merit was 0.563 (95% CI 0.512,0.614) pre-training and 0.677 (95% CI 0.639,0.715) post-training. Highest rating inferred ROC figure-of-merit was 0.728 (95% CI 0.701,0.755) pre-training and 0.772 (95% CI 0.750,0.793) post-training. Conclusions - This study has demonstrated that novice observer performance can improve significantly. This study design may have relevance in the assessment of inexperienced radiographers taking part in PCE or commenting scheme for trauma.

Nonparametric estimation of time-dependent ROC curves conditional on a continuous covariate

The receiver-operating characteristic (ROC) curve is the most widely used measure for evaluating the performance of a diagnostic biomarker when predicting a binary disease outcome. The ROC curve displays the true positive rate (or sensitivity) and the false positive rate (or 1-specificity) for different cut-off values used to classify an individual as healthy or diseased. In time-to-event studies, however, the disease status (e.g. death or alive) of an individual is not a fixed characteristic, and it varies along the study. In such cases, when evaluating the performance of the biomarker, several issues should be taken into account: first, the time-dependent nature of the disease status; and second, the presence of incomplete data (e.g. censored data typically present in survival studies). Accordingly, to assess the discrimination power of continuous biomarkers for time-dependent disease outcomes, time-dependent extensions of true positive rate, false positive rate, and ROC curve have been recently proposed. In this work, we present new nonparametric estimators of the cumulative/dynamic time-dependent ROC curve that allow accounting for the possible modifying effect of current or past covariate measures on the discriminatory power of the biomarker. The proposed estimators can accommodate right-censored data, as well as covariate-dependent censoring. The behavior of the estimators proposed in this study will be explored through simulations and illustrated using data from a cohort of patients who suffered from acute coronary syndrome.

ROC curves of obesity indicators have a predictive value for children hypertension aged 7-17 years

Objective: The aim of the study is to examine the distribution of integrated covariate and its association with blood pressure (BP) among children in Anhui province, China, and assess the predictive value of integrated covariate to children hypertension. Methods: A total of 2,828 subjects (1,588 male and 1,240 female) aged 7-17 years participated in this study. Height, weight, waistline, hipline and BP of all subjects were measured, obesity and overweight were defined by an international standard, specifying the measurement, the reference population, and the age and sex specific cut off points. High BP status was defined as systolic blood pressure (SBP) and/or diastolic blood pressure (DBP) > 95th percentile for age and gender. Results: Our results revealed that the prevalence of children hypertension was 11.03%, the SBP and DBP of obesity group were significantly higher than that of normal group. Anthropometric obesity indices such as body mass index (BMI) were positively correlated with SBP and DBP. Integrated covariate had a better performance than the single covariate in the receiver-operating characteristic (ROC) curve, the cut-off value; the sensitivity and the specificity of the integrated covariate were 0.112, 0.577, 0.683, respectively. Conclusion: Integrated covariate is a simple and effective anthropometric index to identify childhood hypertension.

Sensitivity and specificity of machine learning classifiers for glaucoma diagnosis using Spectral Domain OCT and standard automated perimetry

PURPOSE: To evaluate the sensitivity and specificity of machine learning classifiers (MLCs) for glaucoma diagnosis using Spectral Domain OCT (SD-OCT) and standard automated perimetry (SAP). METHODS: Observational cross-sectional study. Sixty two glaucoma patients and 48 healthy individuals were included. All patients underwent a complete ophthalmologic examination, achromatic standard automated perimetry (SAP) and retinal nerve fiber layer (RNFL) imaging with SD-OCT (Cirrus HD-OCT; Carl Zeiss Meditec Inc., Dublin, California). Receiver operating characteristic (ROC) curves were obtained for all SD-OCT parameters and global indices of SAP. Subsequently, the following MLCs were tested using parameters from the SD-OCT and SAP: Bagging (BAG), Naive-Bayes (NB), Multilayer Perceptron (MLP), Radial Basis Function (RBF), Random Forest (RAN), Ensemble Selection (ENS), Classification Tree (CTREE), Ada Boost M1(ADA),Support Vector Machine Linear (SVML) and Support Vector Machine Gaussian (SVMG). Areas under the receiver operating characteristic curves (aROC) obtained for isolated SAP and OCT parameters were compared with MLCs using OCT+SAP data. RESULTS: Combining OCT and SAP data, MLCs' aROCs varied from 0.777(CTREE) to 0.946 (RAN).The best OCT+SAP aROC obtained with RAN (0.946) was significantly larger the best single OCT parameter (p<0.05), but was not significantly different from the aROC obtained with the best single SAP parameter (p=0.19). CONCLUSION: Machine learning classifiers trained on OCT and SAP data can successfully discriminate between healthy and glaucomatous eyes. The combination of OCT and SAP measurements improved the diagnostic accuracy compared with OCT data alone.

Validity and reliability of the Structured Clinical Interview for Mood Spectrum: Brazilian version (SCIMOODS-VB)

OBJECTIVE: The aim of this study was to translate the Structured Clinical Interview for Mood Spectrum into Brazilian Portuguese, measuring its reliability, validity, and defining scores for bipolar disorders. METHOD: Questionnaire was translated (into Brazilian Portuguese) and back-translated into English. Sample consisted of 47 subjects with bipolar disorder, 47 with major depressive disorder, 18 with schizophrenia and 22 controls. Inter-rater reliability was tested in 20 subjects with bipolar disorder and MDD. Internal consistency was measured using the Kuder Richardson formula. Forward stepwise discriminant analysis was performed. Scores were compared between groups; manic (M), depressive (D) and total (T) threshold scores were calculated through receiver operating characteristic (ROC) curves. RESULTS: Kuder Richardson coefficients were between 0.86 and 0.94. Intraclass correlation coefficient was 0.96 (CI 95 % 0.93-0.97). Subjects with bipolar disorder had higher M and T, and similar D scores, when compared to major depressive disorder (ANOVA, p < 0.001). The sub-domains that best discriminated unipolar and bipolar subjects were manic energy and manic mood. M had the best area under the curve (0.909), and values of M equal to or greater than 30 yielded 91.5% sensitivity and 74.5% specificity. CONCLUSION: Structured Clinical Interview for Mood Spectrum has good reliability and validity. Cut-off of 30 best differentiates subjects with bipolar disorder vs. unipolar depression. A cutoff score of 30 or higher in the mania sub-domain is appropriate to help make a distinction between subjects with bipolar disorder and those with unipolar depression.

Predictive Validity of Waterlow Scale for Pressure Ulcer Development Risk in Hospitalized Patients

AIM: We sought to evaluate the predictive validity of the Waterlow Scale in hospitalized patients. SUBJECTS AND SETTING: The study was conducted at a general private hospital with 220 beds and a mean time of hospitalization of 7.4 days and a mean occupation rate of approximately 80%. Adult patients with a Braden Scale score of 18 or less and a Waterlow Scale score of 16 or more were studied. The sample consisted of 98 patients with a mean age of 71.1 +/- 15.5 years. METHODS: Skin assessment and scoring by using the Waterlow and Braden scales were completed on alternate days. Patients were examined at least 3 times to be considered for analysis. The data were submitted to sensitivity and specificity analysis by using receiver operating characteristic (ROC) curves and positive (+LR) and negative (-LR) likelihood ratios. RESULTS: The cutoff scores were 17, 20, and 20 in the first, second, and third assessment, respectively. Sensitivity was 71.4%, 85.7%, and 85.7% and specificity was 67.0%, 40.7%, and 32.9%, respectively. Analysis of the area under the ROC curve revealed good accuracy (0.64, 95% confidence interval [CI]: 0.35-0.93) only for the cutoff score 17 in the first assessment. The results also showed probabilities of 14%, 10%, and 9% for the development of pressure ulcer when the test results were positive (+LR) and of 3% (-LR) when the test results were negative for the cutoff scores in the first, second, and third assessment, respectively. CONCLUSION: The Waterlow Scale achieved good predictive validity in predicting pressure ulcer in hospitalized patients when a cutoff score of 17 was used in the first assessment.

Comparison of calcaneal ultrasound and DXA in young women

Purpose: The purpose of the study was to assess quantitative ultrasound (QUS) parameters in collegiate female gymnasts, a population whose training incorporates high-impact loading, which is particularly osteogenic, and to determine the discriminative capacity of this relatively new radiation-free technique compared with bone densitometry in a young healthy population. Methods: We studied 19 collegiate gymnasts and 23 healthy controls undergoing regular weight-bearing activity, matched for age (gymnasts 19.2 +/- 1.2, controls 19.9 +/- 1.6 yr) and body weight (gymnasts 56.7 +/- 3.7, controls 57.7 +/- 7.8 kg). QUS parameters of the calcaneus (broadband ultrasound attenuation (BUA), bone velocity (BV), and speed of sound (SOS)) were measured by a Walker Sonix UBA 575+. Bone mineral density (BMD; g.cm(-2)) of the lumbar spine, hip (Femoral neck, trochanter. Ward's triangle) and whole body was assessed by dual energy x-ray absorptiometry (DXA, Hologic QDR 1000/W). Data analysis included unpaired two-tailed Student's t-tests, analysis of variance, Pearson product-moment, and Spearman rank-order correlations. Results: Regional and whole body BMD of gymnasts was greater than controls (P < 0.001), with the difference being 7-28%. Average QUS parameters of the right and left calcaneus were also higher (P < 0.001) in the gymnasts. BUA, BV, and SOS were significantly (P < 0.001) correlated to each bone site with r = 0.54-0.79. Analysis of receiver operating characteristic (ROC) curves indicated no significant difference in sensitivity and specificity for QUS and DXA measures. Conclusions: These results indicate that QUS parameters of the calcaneus are higher in young women gymnasts compared to individuals who undergo regular weight-bearing activity and that QUS parameters are able to discriminate between these two groups in a similar manner as does regional and whole body BMD.

Comparison of Fourier-Domain and Time-Domain Optical Coherence Tomography in the Detection of Band Atrophy of the Optic Nerve

PURPOSE: To compare the ability of Fourier-domain (FD) optical coherence tomography (3D OCT-1000; Top, con, Tokyo, Japan) and time domain (TD) OCT (Stratus; Carl Zeiss Meditec Inc, Dublin, California, USA) to detect axonal loss in eyes with band atrophy (BA) of the optic nerve. DESIGN: Cross-sectional study. METHODS: Thirty-six eyes from 36 patients with BA and temporal visual field (VF) defect from chiasmal compression and 36 normal eyes were studied. Subjects were submitted to standard automated perimetry and macular and retinal nerve fiber layer (RNFL) measurements were taken using 3D OCT-1000 and Stratus OCT. Receiver operating characteristic (ROC) curves were calculated for each parameter. Spearman correlation coefficients were obtained to evaluate the relationship between RNFL and macular thickness parameters and severity of VF loss. Measurements from the two devices were compared. RESULTS: Regardless of OCT device, all RNFL and macular thickness parameters were significantly lower in eyes with BA compared with normal eyes, but no statistically significant difference was found with regard to the area under the ROC curve. Structure-function relationships were also similar for the two devices. In both groups, RNFL and macular thickness measurements were generally and in some cases significantly smaller with 3D OCT-1000 than with Stratus OCT. CONCLUSIONS: The introduction of FD technology did not lead to better discrimination ability for detecting BA of the optic nerve compared with TD technology when using the software currently provided by the manufacturer. 3D OCT-1000 FD OCT RNFL and macular measurements were generally smaller than TD Stratus OCT measurements. Investigators should be aware of this fact when comparing measurements obtained with these two devices. (Am J Oplathalmol 2009;147: 56-63. (c) 2009 by Elsevier Inc. All rights reserved.)

Comparison of the GDx VCC scanning laser polarimeter and the stratus optical coherence tomograph in the detection of band atrophy of the optic nerve

Aim To compare the ability of scanning laser polarimeter (SLP) with variable corneal compensation (GDx VCC) and optical coherence tomograph (Stratus OCT) to discriminate between eyes with band atrophy (BA) of the optic nerve and healthy eyes. Methods The study included 37 eyes with BA and temporal visual field (VF) defects from chiasmal compression, and 29 normal eyes. Subjects underwent standard automated perimetry (SAP) and retinal nerve fibre layer (RNFL) scans using GDx VCC and Stratus OCT. The severity of the VF defects was evaluated by the temporal mean defect (TMD), calculated as the average of 22 values of the temporal total deviation plot on SAP. Receiver operating characteristic (ROC) curves were calculated. Pearson`s correlation coefficients were used to evaluate the relationship between RNFL thickness parameters and the TMD. Results No significant difference was found between the ROC curves areas (AUCs) for the GDx VCC and Stratus OCT with regard to average RNFL thickness (0.98 and 0.99, respectively) and the superior (0.94; 0.95), inferior (0.96; 0.97), and nasal (0.92; 0.96) quadrants. However, the AUC in the temporal quadrant (0.77) was significantly smaller (P < 0.001) with GDx VCC than with Stratus OCT (0.98). Lower TMD values were associated with smaller RNFL thickness in most parameters from both equipments. Conclusion Adding VCC resulted in improved performance in SLP when evaluating eyes with BA, and both technologies are sensitive in detecting average, superior, inferior, and nasal quadrant RNFL loss. However, GDx VCC still poorly discriminates RNFL loss in the temporal quadrant when compared with Stratus OCT.

Clinical usefulness of B-type natriuretic peptide in the diagnosis of pleural effusions due to heart failure

Background and objective: Light`s criteria are frequently used to evaluate the exudative or transudative nature of pleural effusions. However, misclassification resulting from the use of Light`s criteria has been reported, especially in the setting of diuretic use in patients with heart failure (HF). The objective of this study was to evaluate the utility of B-type natriuretic peptide (BNP) measurements as a diagnostic tool for determining the cardiac aetiology of pleural effusions. Methods: Patients with pleural effusions attributable to HF (n = 34), hepatic hydrothorax (n = 10), pleural effusions due to cancer (n = 21) and pleural effusions due to tuberculosis (n = 12) were studied. Diagnostic thoracentesis was performed for all 77 patients. Receiver operating characteristic (ROC) curves were constructed to determine the diagnostic accuracy of plasma BNP and pleural fluid BNP for the prediction of HF. Results: The areas under the ROC curves were 0.987 (95% CI 0.93-0.998) for plasma BNP and 0.949 (95% CI 0.874-0.986) for pleural fluid BNP, for distinguishing between patients with pleural effusions caused by HF (n = 34) and those with pleural effusions attributable to other causes (n = 43). The cut-off concentrations with the highest diagnostic accuracy for the diagnosis of HF as the cause of pleural effusion were 132 pg/mL for plasma BNP (sensitivity 97.1%, specificity 97.4%) and 127 pg/mL for pleural fluid BNP (sensitivity 97.1%, specificity 87.8%). Conclusions: In patients with pleural effusions of suspected cardiac origin, measurements of BNP in plasma and pleural fluid may be useful for the diagnosis of HF as the underlying cause.