A nonparametric method of comparing the partial areas under two correlated receiver operating characteristic curves

A non-parametric method was developed and tested to compare the partial areas under two correlated Receiver Operating Characteristic curves. Based on the theory of generalized U-statistics the mathematical formulas have been derived for computing ROC area, and the variance and covariance between the portions of two ROC curves. A practical SAS application also has been developed to facilitate the calculations. The accuracy of the non-parametric method was evaluated by comparing it to other methods. By applying our method to the data from a published ROC analysis of CT image, our results are very close to theirs. A hypothetical example was used to demonstrate the effects of two crossed ROC curves. The two ROC areas are the same. However each portion of the area between two ROC curves were found to be significantly different by the partial ROC curve analysis. For computation of ROC curves with large scales, such as a logistic regression model, we applied our method to the breast cancer study with Medicare claims data. It yielded the same ROC area computation as the SAS Logistic procedure. Our method also provides an alternative to the global summary of ROC area comparison by directly comparing the true-positive rates for two regression models and by determining the range of false-positive values where the models differ. ^

Evaluation of non-contact sampling and detection of explosives using receiver operating characteristic curves

The growing need for fast sampling of explosives in high throughput areas has increased the demand for improved technology for the trace detection of illicit compounds. Detection of the volatiles associated with the presence of the illicit compounds offer a different approach for sensitive trace detection of these compounds without increasing the false positive alarm rate. This study evaluated the performance of non-contact sampling and detection systems using statistical analysis through the construction of Receiver Operating Characteristic (ROC) curves in real-world scenarios for the detection of volatiles in the headspace of smokeless powder, used as the model system for generalizing explosives detection. A novel sorbent coated disk coined planar solid phase microextraction (PSPME) was previously used for rapid, non-contact sampling of the headspace containers. The limits of detection for the PSPME coupled to IMS detection was determined to be 0.5-24 ng for vapor sampling of volatile chemical compounds associated with illicit compounds and demonstrated an extraction efficiency of three times greater than other commercially available substrates, retaining >50% of the analyte after 30 minutes sampling of an analyte spike in comparison to a non-detect for the unmodified filters. Both static and dynamic PSPME sampling was used coupled with two ion mobility spectrometer (IMS) detection systems in which 10-500 mg quantities of smokeless powders were detected within 5-10 minutes of static sampling and 1 minute of dynamic sampling time in 1-45 L closed systems, resulting in faster sampling and analysis times in comparison to conventional solid phase microextraction-gas chromatography-mass spectrometry (SPME-GC-MS) analysis. Similar real-world scenarios were sampled in low and high clutter environments with zero false positive rates. Excellent PSPME-IMS detection of the volatile analytes were visualized from the ROC curves, resulting with areas under the curves (AUC) of 0.85-1.0 and 0.81-1.0 for portable and bench-top IMS systems, respectively. Construction of ROC curves were also developed for SPME-GC-MS resulting with AUC of 0.95-1.0, comparable with PSPME-IMS detection. The PSPME-IMS technique provides less false positive results for non-contact vapor sampling, cutting the cost and providing an effective sampling and detection needed in high-throughput scenarios, resulting in similar performance in comparison to well-established techniques with the added advantage of fast detection in the field.

Precision-recall operating characteristic (P-ROC) curves in imprecise environments

Traditionally, machine learning algorithms have been evaluated in applications where assumptions can be reliably made about class priors and/or misclassification costs. In this paper, we consider the case of imprecise environments, where little may be known about these factors and they may well vary significantly when the system is applied. Specifically, the use of precision-recall analysis is investigated and compared to the more well known performance measures such as error-rate and the receiver operating characteristic (ROC). We argue that while ROC analysis is invariant to variations in class priors, this invariance in fact hides an important factor of the evaluation in imprecise environments. Therefore, we develop a generalised precision-recall analysis methodology in which variation due to prior class probabilities is incorporated into a multi-way analysis of variance (ANOVA). The increased sensitivity and reliability of this approach is demonstrated in a remote sensing application.

Receiver operating characteristic (ROC) curve for classification of18F-NaF uptake on PET/CT

Abstract Objective: To assess the cutoff values established by ROC curves to classify18F-NaF uptake as normal or malignant. Materials and Methods: PET/CT images were acquired 1 hour after administration of 185 MBq of18F-NaF. Volumes of interest (VOIs) were drawn on three regions of the skeleton as follows: proximal right humerus diaphysis (HD), proximal right femoral diaphysis (FD) and first vertebral body (VB1), in a total of 254 patients, totalling 762 VOIs. The uptake in the VOIs was classified as normal or malignant on the basis of the radiopharmaceutical distribution pattern and of the CT images. A total of 675 volumes were classified as normal and 52 were classified as malignant. Thirty-five VOIs classified as indeterminate or nonmalignant lesions were excluded from analysis. The standardized uptake value (SUV) measured on the VOIs were plotted on an ROC curve for each one of the three regions. The area under the ROC (AUC) as well as the best cutoff SUVs to classify the VOIs were calculated. The best cutoff values were established as the ones with higher result of the sum of sensitivity and specificity. Results: The AUCs were 0.933, 0.889 and 0.975 for UD, FD and VB1, respectively. The best SUV cutoffs were 9.0 (sensitivity: 73%; specificity: 99%), 8.4 (sensitivity: 79%; specificity: 94%) and 21.0 (sensitivity: 93%; specificity: 95%) for UD, FD and VB1, respectively. Conclusion: The best cutoff value varies according to bone region of analysis and it is not possible to establish one value for the whole body.

Effect of combined bending strain and thermal cycling on the voltage-current characteristic curves of Bi-2223 tapes

High critical temperature superconductors are evolving from a scientific research subject into large-scale application devices. In order to meet this development demand they must withstand high current capacity under mechanical loads arising from thermal contraction during cooling from room temperature down to operating temperature (usually 77 K) and due to the electromagnetic forces generated by the current and the induced magnetic field. Among the HTS materials, the Bi2Sr2Ca2Cu3Ox, compound imbedded in an Ag/AgMg sheath has shown the best results in terms of critical current at 77 K and tolerance against mechanical strain. Aiming to evaluate the influence of thermal stress induced by a number of thermal shock cycles we have evaluated the V-I characteristic curves of samples mounted onto semicircular holders with different curvature radius (9.75 to 44.5 mm). The most deformed sample (epsilon = 1.08%) showed the largest reduction of critical current (40%) compared to the undeformed sample and the highest sensitivity to thermal stress (I-c/I-c0 = 0.5). The V-I characteristic curves were also fitted by a potential curve displaying n-exponents varying from 20 down to 10 between the initial and last thermal shock cycle.

The influence of experience and training in a group of novice observers: a jackknife alternative free-response receiver operating characteristic analysis

Purpose - The study evaluates the pre- and post-training lesion localisation ability of a group of novice observers. Parallels are drawn with the performance of inexperienced radiographers taking part in preliminary clinical evaluation (PCE) and ‘red-dot’ systems, operating within radiography practice. Materials and methods - Thirty-four novice observers searched 92 images for simulated lesions. Pre-training and post-training evaluations were completed following the free-response the receiver operating characteristic (FROC) method. Training consisted of observer performance methodology, the characteristics of the simulated lesions and information on lesion frequency. Jackknife alternative FROC (JAFROC) and highest rating inferred ROC analyses were performed to evaluate performance difference on lesion-based and case-based decisions. The significance level of the test was set at 0.05 to control the probability of Type I error. Results - JAFROC analysis (F(3,33) = 26.34, p < 0.0001) and highest-rating inferred ROC analysis (F(3,33) = 10.65, p = 0.0026) revealed a statistically significant difference in lesion detection performance. The JAFROC figure-of-merit was 0.563 (95% CI 0.512,0.614) pre-training and 0.677 (95% CI 0.639,0.715) post-training. Highest rating inferred ROC figure-of-merit was 0.728 (95% CI 0.701,0.755) pre-training and 0.772 (95% CI 0.750,0.793) post-training. Conclusions - This study has demonstrated that novice observer performance can improve significantly. This study design may have relevance in the assessment of inexperienced radiographers taking part in PCE or commenting scheme for trauma.

Avaliação do desempenho de indicadores com base na metodologia ROC (Receiver Operating Characteristic)

Dissertação de mestrado em Bioinformática

Sensitivity and specificity of machine learning classifiers for glaucoma diagnosis using Spectral Domain OCT and standard automated perimetry

PURPOSE: To evaluate the sensitivity and specificity of machine learning classifiers (MLCs) for glaucoma diagnosis using Spectral Domain OCT (SD-OCT) and standard automated perimetry (SAP). METHODS: Observational cross-sectional study. Sixty two glaucoma patients and 48 healthy individuals were included. All patients underwent a complete ophthalmologic examination, achromatic standard automated perimetry (SAP) and retinal nerve fiber layer (RNFL) imaging with SD-OCT (Cirrus HD-OCT; Carl Zeiss Meditec Inc., Dublin, California). Receiver operating characteristic (ROC) curves were obtained for all SD-OCT parameters and global indices of SAP. Subsequently, the following MLCs were tested using parameters from the SD-OCT and SAP: Bagging (BAG), Naive-Bayes (NB), Multilayer Perceptron (MLP), Radial Basis Function (RBF), Random Forest (RAN), Ensemble Selection (ENS), Classification Tree (CTREE), Ada Boost M1(ADA),Support Vector Machine Linear (SVML) and Support Vector Machine Gaussian (SVMG). Areas under the receiver operating characteristic curves (aROC) obtained for isolated SAP and OCT parameters were compared with MLCs using OCT+SAP data. RESULTS: Combining OCT and SAP data, MLCs' aROCs varied from 0.777(CTREE) to 0.946 (RAN).The best OCT+SAP aROC obtained with RAN (0.946) was significantly larger the best single OCT parameter (p<0.05), but was not significantly different from the aROC obtained with the best single SAP parameter (p=0.19). CONCLUSION: Machine learning classifiers trained on OCT and SAP data can successfully discriminate between healthy and glaucomatous eyes. The combination of OCT and SAP measurements improved the diagnostic accuracy compared with OCT data alone.

Coronary Artery Stenoses: Accuracy of 64-Detector Row CT Angiography in Segments with Mild, Moderate, or Severe Calcification-A Subanalysis of the CORE-64 Trial

Purpose: To evaluate the influence of cross-sectional arc calcification on the diagnostic accuracy of computed tomography (CT) angiography compared with conventional coronary angiography for the detection of obstructive coronary artery disease (CAD). Materials and Methods: Institutional Review Board approval and written informed consent were obtained from all centers and participants for this HIPAA-compliant study. Overall, 4511 segments from 371 symptomatic patients (279 men, 92 women; median age, 61 years [interquartile range, 53-67 years]) with clinical suspicion of CAD from the CORE-64 multi-center study were included in the analysis. Two independent blinded observers evaluated the percentage of diameter stenosis and the circumferential extent of calcium (arc calcium). The accuracy of quantitative multidetector CT angiography to depict substantial (>50%) stenoses was assessed by using quantitative coronary angiography (QCA). Cross-sectional arc calcium was rated on a segment level as follows: noncalcified or mild (<90 degrees), moderate (90 degrees-180 degrees), or severe (>180 degrees) calcification. Univariable and multivariable logistic regression, receiver operation characteristic curve, and clustering methods were used for statistical analyses. Results: A total of 1099 segments had mild calcification, 503 had moderate calcification, 338 had severe calcification, and 2571 segments were noncalcified. Calcified segments were highly associated (P < .001) with disagreement between CTA and QCA in multivariable analysis after controlling for sex, age, heart rate, and image quality. The prevalence of CAD was 5.4% in noncalcified segments, 15.0% in mildly calcified segments, 27.0% in moderately calcified segments, and 43.0% in severely calcified segments. A significant difference was found in area under the receiver operating characteristic curves (noncalcified: 0.86, mildly calcified: 0.85, moderately calcified: 0.82, severely calcified: 0.81; P < .05). Conclusion: In a symptomatic patient population, segment-based coronary artery calcification significantly decreased agreement between multidetector CT angiography and QCA to detect a coronary stenosis of at least 50%.

Anthropometric measures of increased central and overall adiposity in association with echocardiographic left ventricular hypertrophy

Left ventricular hypertrophy is an important predictor of cardiovascular risk and sudden death. This study explored the ability of four obesity indexes (body mass index, waist circumference, waist-hip ratio and waist-stature ratio) to identify left ventricular hypertrophy. A sample of the general population (n=682; 43.5% men) was surveyed to assess cardiovascular risk factors. Biochemical, anthropometric and blood pressure values were obtained in a clinic visit according to standard methods. Left ventricular mass was obtained from transthoracic echocardiogram. Left ventricular hypertrophy was defined using population-specific cutoff values for left ventricular mass indexed to height(2.7). The waist-stature ratio showed the strongest positive association with left ventricular mass. This correlation was stronger in women, even after controlling for age and systolic blood pressure. By multivariate analysis, the main predictors of left ventricular hypertrophy were waist-stature ratio (23%), systolic blood pressure (9%) and age (2%) in men, and waist-stature ratio (40%), age (6%) and systolic blood pressure (2%) in women. Receiver-operating characteristic curves showed the optimal cutoff values of the different anthropometric indexes associated with left ventricular hypertrophy. The waist-stature ratio was a significantly better predictor than the other indexes (except for the waist-hip ratio), independent of gender. It is noteworthy that a waist-stature ratio cutoff of 0.56 showed the highest combined sensitivity and specificity to detect left ventricular hypertrophy. Abdominal obesity identified by waist-stature ratio instead of overall obesity identified by body mass index is the simplest and best obesity index for assessing the risk of left ventricular hypertrophy, is a better predictor in women and has an optimal cutoff ratio of 0.56. Hypertension Research (2010) 33, 83-87; doi: 10.1038/hr.2009.188; published online 13 November 2009

FibroTest is an independent predictor of virologic response in chronic hepatitis C patients retreated with pegylated interferon alfa-2b and ribavirin in the EPIC(3) program

Background & Aims: EPIC-3 is a prospective, international study that has demonstrated the efficacy of PEG-IFN alfa-2b plus weight-based ribavirin in patients with chronic hepatitis C and significant fibrosis who previously failed any interferon-alfa/ribavirin therapy. The aim of the present study was to assess FibroTest (FT), a validated non-invasive marker of fibrosis in treatment-naive patients, as a possible alternative to biopsy as the baseline predictor of subsequent early virologic (EVR) and sustained virologic response (SVR) in previously treated patients. Methods: Of 2312 patients enrolled, 1459 had an available baseline FT, biopsy, and complete data. Uni- (UV) and multi-variable (MV) analyses were performed using FT and biopsy. Results: Baseline characteristics were similar as in the overall population; METAVIR stage: 28% F2, 29% F3, and 43% F4, previous relapsers 29%, previous PEG-IFN regimen 41%, high baseline viral load (BVL) 64%. 506 patients (35%) had undetectable HCV-RNA at TW12 (TW12neg), with 58% achieving SVR. The accuracy of FT was similar to that in naive patients: AUROC curve for the diagnosis of F4 vs F2 = 0.80 (p<0.00001). Five baseline factors were associated (p<0.001) with SVR in UV and MV analyses (odds ratio: UV/MV): fibrosis stage estimated using FT (4.5/5.9) or biopsy (1.5/1.6), genotype 2/3 (4.5/5.1), BVL (1.5/1.3), prior relapse (1.6/1.6), previous treatment with non-PEG-IFN (2.6/2.0). These same factors were associated (p <= 0.001) with EVR. Among patients TW12neg, two independent factors remained highly predictive of SVR by MV analysis (p <= 0.001): genotype 2/3 (odds ratio = 2.9), fibrosis estimated with FT (4.3) or by biopsy (1.5). Conclusions: FibroTest at baseline is a possible non-invasive alternative to biopsy for the prediction of EVR at 12 weeks and SVR, in patients with previous failures and advanced fibrosis, retreated with PEG-IFN alfa-2b and ribavirin. (C) 2010 European Association for the Study of the Liver. Published by Elsevier B.V. All rights reserved.

Comparison of nutritional risk screening tools for predicting clinical outcomes in hospitalized patients

Objective: International nutritional screening tools are recommended for screening hospitalized patients for nutritional risk, but no tool has been specifically evaluated in the Brazilian population. The aim of this study was to identify the most appropriate nutritional screening tool for predicting unfavorable clinical outcomes in patients admitted to a Brazilian public university hospital. Methods: The Nutritional Risk Screening 2002 (NRS 2002), Mini-Nutritional Assessment-Short Form (MNA-SF), and Malnutrition Universal Screening Tool (MUST) were administered to 705 patients within 48 h of hospital admission. Tool performance in predicting complications, very long length of hospital stay (LOS), and death was analyzed using receiver operating characteristic curves. Results: NRS 2002, MUST, and MNA-SF identified nutritional risk in 27.9%, 39.6%, and 73.2% of the patients, respectively. NRS 2002 (complications: 0.6531; very long LOS: 0.6508; death: 0.7948) and MNA-SF(complications: 0.6495; very long LOS: 0.6197; death: 0.7583) had largest areas under the ROC curve compared to MUST (complications: 0.6036; very long LOS: 0.6109; death: 0.6363). For elderly patients, NRS 2002 was not significantly different than MNA-SF (P>0.05) for predicting outcomes. Conclusion: Considering current criteria for nutritional risk, NRS 2002 and MNA-SF have similar performance to predict outcomes but NRS 2002 seems to provide a best yield. (C) 2010 Elsevier Inc. All rights reserved.

Comparison of Quantitative Imaging Devices and Subjective Optic Nerve Head Assessment by General Ophthalmologists to Differentiate Normal From Glaucomatous Eyes

Purpose: To compare the ability of Subjective assessment of optic nerve head (ONH) and retinal nerve fiber layer (RNFL) by general ophthalmologists and by a glaucoma expert with objective measurements by optical coherence tomography (Stratus OCT, Carl Zeiss Meditec Inc), confocal scanning laser ophthalmoscope (HRT III; Heidelberg Engineering, Heidelberg. Germany), and scanning laser polarimetry (GDx enhanced corneal compensation; Carl Zeiss Meditec Inc, Dublin, CA) in discriminating glaucomatous and normal eyes. Methods: Sixty-one glaucomatous and 57 normal eyes or 118 subjects Were included in the study. Three independent general ophthalmologists and I glaucoma expert evaluated ONH stereo-photographs. Receiver operating characteristic curves were constructed for each imaging technique and sensitivity at fixed specificity was estimated. Comparisons or areas under these curves (aROCs) and agreement (k) were determined between stereophoto grading and best parameter from each technique. Results: Best parameter from each technique showed larger aROC (Stratus OCT RNFL 0.92; Stratus OCT ONH vertical integrated area = 0.86; Stratus OCT macular thickness = 0.82; GDx enhanced corneal compensation = 0.91, HRT3 global cup-to-disc ratio = 0.83; HRT3 glaucoma probability score numeric area score 0.83) compared with stereophotograph grading by general ophthalmologists (0.80) in separating glaucomatous and normal eyes. Glaucoma expert stereophoto grading provided equal or larger aROC (0.92) than best parameter of each computerized imaging device. Stereophoto evaluated by a glaucoma expert showed better agreement with best parameter of each quantitative imaging technique in classifying eyes either as glaucomatous or normal compared with stereophoto grading by general ophthalmologists, The combination Of Subjective assessment of the optic disc by general ophthalmologists with RNFL objective parameters improved identification of glaucoma patients in a larger proportion than the combination of these objective parameters with Subjective assessment of the optic disc by a glaucoma expert (29.5% vs. 19.7%, respectively). Conclusions: Diagnostic ability of all imaging techniques showed better performance than subjective assessment of the ONH by general ophthalmologists, but not by It glaucoma expert, Objective RNFL measurements may provide improvement in glaucoma detection when combined with subjective assessment of the optic disc by general ophthalmologists or by a glaucoma expert.

Effect of Disease Severity and Optic Disc Size on Diagnostic Accuracy of RTVue Spectral Domain Optical Coherence Tomograph in Glaucoma

PURPOSE. To evaluate the effect of disease severity and optic disc size on the diagnostic accuracies of optic nerve head (ONH), retinal nerve fiber layer (RNFL), and macular parameters with RTVue (Optovue, Fremont, CA) spectral domain optical coherence tomography (SDOCT) in glaucoma. METHODS. 110 eyes of 62 normal subjects and 193 eyes of 136 glaucoma patients from the Diagnostic Innovations in Glaucoma Study underwent ONH, RNFL, and macular imaging with RTVue. Severity of glaucoma was based on visual field index (VFI) values from standard automated perimetry. Optic disc size was based on disc area measurement using the Heidelberg Retina Tomograph II (Heidelberg Engineering, Dossenheim, Germany). Influence of disease severity and disc size on the diagnostic accuracy of RTVue was evaluated by receiver operating characteristic (ROC) and logistic regression models. RESULTS. Areas under ROC curve (AUC) of all scanning areas increased (P < 0.05) as disease severity increased. For a VFI value of 99%, indicating early damage, AUCs for rim area, average RNLI thickness, and ganglion cell complex-root mean square were 0.693, 0.799, and 0.779, respectively. For a VFI of 70%, indicating severe damage, corresponding AUCs were 0.828, 0.985, and 0.992, respectively. Optic disc size did not influence the AUCs of any of the SDOCT scanning protocols of RTVue (P > 0.05). Sensitivity of the rim area increased and specificity decreased in large optic discs. CONCLUSIONS. Diagnostic accuracies of RTVue scanning protocols for glaucoma were significantly influenced by disease severity. Sensitivity of the rim area increased in large optic discs at the expense of specificity. (Invest Ophthalmol Vis Sci. 2011;92:1290-1296) DOI:10.1167/iovs.10-5516

Effect of Disease Severity on the Performance of Cirrus Spectral-Domain OCT for Glaucoma Diagnosis

PURPOSE. To evaluate the effect of disease severity on the diagnostic accuracy of the Cirrus Optical Coherence Tomograph (Cirrus HD-OCT; Carl Zeiss Meditec, Inc., Dublin, CA) for glaucoma detection. METHODS. One hundred thirty-five glaucomatous eyes of 99 patients and 79 normal eyes of 47 control subjects were recruited from the longitudinal Diagnostic Innovations in Glaucoma Study (DIGS). The severity of the disease was graded based on the visual field index (VFI) from standard automated perimetry. Imaging of the retinal nerve fiber layer (RNFL) was obtained using the optic disc cube protocol available on the Cirrus HD-OCT. Pooled receiver operating characteristic (ROC) curves were initially obtained for each parameter of the Cirrus HD-OCT. The effect of disease severity on diagnostic performance was evaluated by fitting an ROC regression model, with VFI used as a covariate, and calculating the area under the ROC curve (AUCs) for different levels of disease severity. RESULTS. The largest pooled AUCs were for average thickness (0.892), inferior quadrant thickness (0.881), and superior quadrant thickness (0.874). Disease severity had a significant influence on the detection of glaucoma. For the average RNFL thickness parameter, AUCs were 0.962, 0.932, 0.886, and 0.822 for VFIs of 70%, 80%, 90%, and 100%, respectively. CONCLUSIONS. Disease severity had a significant effect on the diagnostic performance of the Cirrus HD-OCT and thus should be considered when interpreting results from this device and when considering the potential applications of this instrument for diagnosing glaucoma in the various clinical settings. (Invest Ophthalmol Vis Sci. 2010;51:4104-4109) DOI:10.1167/iovs.094716