Receiver operating characteristic (ROC) curve for classification of18F-NaF uptake on PET/CT

Abstract Objective: To assess the cutoff values established by ROC curves to classify18F-NaF uptake as normal or malignant. Materials and Methods: PET/CT images were acquired 1 hour after administration of 185 MBq of18F-NaF. Volumes of interest (VOIs) were drawn on three regions of the skeleton as follows: proximal right humerus diaphysis (HD), proximal right femoral diaphysis (FD) and first vertebral body (VB1), in a total of 254 patients, totalling 762 VOIs. The uptake in the VOIs was classified as normal or malignant on the basis of the radiopharmaceutical distribution pattern and of the CT images. A total of 675 volumes were classified as normal and 52 were classified as malignant. Thirty-five VOIs classified as indeterminate or nonmalignant lesions were excluded from analysis. The standardized uptake value (SUV) measured on the VOIs were plotted on an ROC curve for each one of the three regions. The area under the ROC (AUC) as well as the best cutoff SUVs to classify the VOIs were calculated. The best cutoff values were established as the ones with higher result of the sum of sensitivity and specificity. Results: The AUCs were 0.933, 0.889 and 0.975 for UD, FD and VB1, respectively. The best SUV cutoffs were 9.0 (sensitivity: 73%; specificity: 99%), 8.4 (sensitivity: 79%; specificity: 94%) and 21.0 (sensitivity: 93%; specificity: 95%) for UD, FD and VB1, respectively. Conclusion: The best cutoff value varies according to bone region of analysis and it is not possible to establish one value for the whole body.

A nonparametric method of comparing the partial areas under two correlated receiver operating characteristic curves

A non-parametric method was developed and tested to compare the partial areas under two correlated Receiver Operating Characteristic curves. Based on the theory of generalized U-statistics the mathematical formulas have been derived for computing ROC area, and the variance and covariance between the portions of two ROC curves. A practical SAS application also has been developed to facilitate the calculations. The accuracy of the non-parametric method was evaluated by comparing it to other methods. By applying our method to the data from a published ROC analysis of CT image, our results are very close to theirs. A hypothetical example was used to demonstrate the effects of two crossed ROC curves. The two ROC areas are the same. However each portion of the area between two ROC curves were found to be significantly different by the partial ROC curve analysis. For computation of ROC curves with large scales, such as a logistic regression model, we applied our method to the breast cancer study with Medicare claims data. It yielded the same ROC area computation as the SAS Logistic procedure. Our method also provides an alternative to the global summary of ROC area comparison by directly comparing the true-positive rates for two regression models and by determining the range of false-positive values where the models differ. ^

The influence of experience and training in a group of novice observers: a jackknife alternative free-response receiver operating characteristic analysis

Purpose - The study evaluates the pre- and post-training lesion localisation ability of a group of novice observers. Parallels are drawn with the performance of inexperienced radiographers taking part in preliminary clinical evaluation (PCE) and ‘red-dot’ systems, operating within radiography practice. Materials and methods - Thirty-four novice observers searched 92 images for simulated lesions. Pre-training and post-training evaluations were completed following the free-response the receiver operating characteristic (FROC) method. Training consisted of observer performance methodology, the characteristics of the simulated lesions and information on lesion frequency. Jackknife alternative FROC (JAFROC) and highest rating inferred ROC analyses were performed to evaluate performance difference on lesion-based and case-based decisions. The significance level of the test was set at 0.05 to control the probability of Type I error. Results - JAFROC analysis (F(3,33) = 26.34, p < 0.0001) and highest-rating inferred ROC analysis (F(3,33) = 10.65, p = 0.0026) revealed a statistically significant difference in lesion detection performance. The JAFROC figure-of-merit was 0.563 (95% CI 0.512,0.614) pre-training and 0.677 (95% CI 0.639,0.715) post-training. Highest rating inferred ROC figure-of-merit was 0.728 (95% CI 0.701,0.755) pre-training and 0.772 (95% CI 0.750,0.793) post-training. Conclusions - This study has demonstrated that novice observer performance can improve significantly. This study design may have relevance in the assessment of inexperienced radiographers taking part in PCE or commenting scheme for trauma.

Avaliação do desempenho de indicadores com base na metodologia ROC (Receiver Operating Characteristic)

Dissertação de mestrado em Bioinformática

Evaluation of non-contact sampling and detection of explosives using receiver operating characteristic curves

The growing need for fast sampling of explosives in high throughput areas has increased the demand for improved technology for the trace detection of illicit compounds. Detection of the volatiles associated with the presence of the illicit compounds offer a different approach for sensitive trace detection of these compounds without increasing the false positive alarm rate. This study evaluated the performance of non-contact sampling and detection systems using statistical analysis through the construction of Receiver Operating Characteristic (ROC) curves in real-world scenarios for the detection of volatiles in the headspace of smokeless powder, used as the model system for generalizing explosives detection. A novel sorbent coated disk coined planar solid phase microextraction (PSPME) was previously used for rapid, non-contact sampling of the headspace containers. The limits of detection for the PSPME coupled to IMS detection was determined to be 0.5-24 ng for vapor sampling of volatile chemical compounds associated with illicit compounds and demonstrated an extraction efficiency of three times greater than other commercially available substrates, retaining >50% of the analyte after 30 minutes sampling of an analyte spike in comparison to a non-detect for the unmodified filters. Both static and dynamic PSPME sampling was used coupled with two ion mobility spectrometer (IMS) detection systems in which 10-500 mg quantities of smokeless powders were detected within 5-10 minutes of static sampling and 1 minute of dynamic sampling time in 1-45 L closed systems, resulting in faster sampling and analysis times in comparison to conventional solid phase microextraction-gas chromatography-mass spectrometry (SPME-GC-MS) analysis. Similar real-world scenarios were sampled in low and high clutter environments with zero false positive rates. Excellent PSPME-IMS detection of the volatile analytes were visualized from the ROC curves, resulting with areas under the curves (AUC) of 0.85-1.0 and 0.81-1.0 for portable and bench-top IMS systems, respectively. Construction of ROC curves were also developed for SPME-GC-MS resulting with AUC of 0.95-1.0, comparable with PSPME-IMS detection. The PSPME-IMS technique provides less false positive results for non-contact vapor sampling, cutting the cost and providing an effective sampling and detection needed in high-throughput scenarios, resulting in similar performance in comparison to well-established techniques with the added advantage of fast detection in the field.

Precision-recall operating characteristic (P-ROC) curves in imprecise environments

Traditionally, machine learning algorithms have been evaluated in applications where assumptions can be reliably made about class priors and/or misclassification costs. In this paper, we consider the case of imprecise environments, where little may be known about these factors and they may well vary significantly when the system is applied. Specifically, the use of precision-recall analysis is investigated and compared to the more well known performance measures such as error-rate and the receiver operating characteristic (ROC). We argue that while ROC analysis is invariant to variations in class priors, this invariance in fact hides an important factor of the evaluation in imprecise environments. Therefore, we develop a generalised precision-recall analysis methodology in which variation due to prior class probabilities is incorporated into a multi-way analysis of variance (ANOVA). The increased sensitivity and reliability of this approach is demonstrated in a remote sensing application.

Advancing Bag-of-visual-words Representations For Lesion Classification In Retinal Images.

Diabetic Retinopathy (DR) is a complication of diabetes that can lead to blindness if not readily discovered. Automated screening algorithms have the potential to improve identification of patients who need further medical attention. However, the identification of lesions must be accurate to be useful for clinical application. The bag-of-visual-words (BoVW) algorithm employs a maximum-margin classifier in a flexible framework that is able to detect the most common DR-related lesions such as microaneurysms, cotton-wool spots and hard exudates. BoVW allows to bypass the need for pre- and post-processing of the retinographic images, as well as the need of specific ad hoc techniques for identification of each type of lesion. An extensive evaluation of the BoVW model, using three large retinograph datasets (DR1, DR2 and Messidor) with different resolution and collected by different healthcare personnel, was performed. The results demonstrate that the BoVW classification approach can identify different lesions within an image without having to utilize different algorithms for each lesion reducing processing time and providing a more flexible diagnostic system. Our BoVW scheme is based on sparse low-level feature detection with a Speeded-Up Robust Features (SURF) local descriptor, and mid-level features based on semi-soft coding with max pooling. The best BoVW representation for retinal image classification was an area under the receiver operating characteristic curve (AUC-ROC) of 97.8% (exudates) and 93.5% (red lesions), applying a cross-dataset validation protocol. To assess the accuracy for detecting cases that require referral within one year, the sparse extraction technique associated with semi-soft coding and max pooling obtained an AUC of 94.2 ± 2.0%, outperforming current methods. Those results indicate that, for retinal image classification tasks in clinical practice, BoVW is equal and, in some instances, surpasses results obtained using dense detection (widely believed to be the best choice in many vision problems) for the low-level descriptors.

Neck Circumference As A New Anthropometric Indicator For Prediction Of Insulin Resistance And Components Of Metabolic Syndrome In Adolescents: Brazilian Metabolic Syndrome Study.

To evaluate the correlation between neck circumference and insulin resistance and components of metabolic syndrome in adolescents with different adiposity levels and pubertal stages, as well as to determine the usefulness of neck circumference to predict insulin resistance in adolescents. Cross-sectional study with 388 adolescents of both genders from ten to 19 years old. The adolescents underwent anthropometric and body composition assessment, including neck and waist circumferences, and biochemical evaluation. The pubertal stage was obtained by self-assessment, and the blood pressure, by auscultation. Insulin resistance was evaluated by the Homeostasis Model Assessment-Insulin Resistance. The correlation between two variables was evaluated by partial correlation coefficient adjusted for the percentage of body fat and pubertal stage. The performance of neck circumference to identify insulin resistance was tested by Receiver Operating Characteristic Curve. After the adjustment for percentage body fat and pubertal stage, neck circumference correlated with waist circumference, blood pressure, triglycerides and markers of insulin resistance in both genders. The results showed that the neck circumference is a useful tool for the detection of insulin resistance and changes in the indicators of metabolic syndrome in adolescents. The easiness of application and low cost of this measure may allow its use in Public Health services.

Motor Nerve-Conduction Studies in Obstetric Brachial Plexopathy for a Selection of Patients with a Poor Outcome

Background: The criteria and timing for nerve surgery in infants with obstetric brachial plexopathy remain controversial. Our aim was to develop a new method for early prognostic assessment to assist this decision process. Methods: Fifty-four patients with unilateral obstetric brachial plexopathy who were ten to sixty days old underwent bilateral motor-nerve-conduction studies of the axillary, musculocutaneous, proximal radial, distal radial, median, and ulnar nerves. The ratio between the amplitude of the compound muscle action potential of the affected limb and that of the healthy side was called the axonal viability index. The patients were followed and classified in three groups according to the clinical outcome. We analyzed the receiver operating characteristic curve of each index to define the best cutoff point to detect patients with a poor recovery. Results: The best cutoff points on the axonal viability index for each nerve (and its sensitivity and specificity) were <10% (88% and 89%, respectively) for the axillary nerve, 0% (88% and 73%) for the musculocutaneous nerve, <20% (82% and 97%) for the proximal radial nerve, <50% (82% and 97%) for the distal radial nerve, and <50% (59% and 97%) for the ulnar nerve. The indices from the proximal radial, distal radial, and ulnar nerves had better specificities compared with the most frequently used clinical criterion: absence of biceps function at three months of age. Conclusions: The axonal viability index yields an earlier and more specific prognostic estimation of obstetric brachial plexopathy than does the clinical criterion of biceps function, and we believe it may be useful in determining surgical indications in these patients.

COOP/WONCA Charts as a Screen for Mental Disorders in Primary Care

PURPOSE most people with mental disorders receive treatment in primary care. The charts developed by the Dartmouth Primary Care Cooperative Research Network (COOP) and the World Organization of National Colleges, Academies, and Academic Associations of General Practitioners/Family Physicians (WONCA) have not yet been evaluated as a screen for these disorders, using a structured psychiatric interview by an expert or considering diagnoses other than depression. We evaluated the validity and feasibility of the COOP/WONCA Charts as a mental disorders screen by comparing them both with other questionnaires previously validated and with the assessment of a mental health specialist using a structured diagnostic interview. METHODS We trained community health workers and nurse assistants working in a collaborative mental health care model to administer the COOP/WONCA Charts, the 20-item Self-Reporting Questionnaire (SRQ-20), and the World Health Organization Five Well-Being Index (WHO-5) to 120 primary care patients. A psychiatrist blinded to the patients' results on these questionnaires administered the SCID, or Structured Clinical Interview for the DSM-IV (Diagnostic and Statistical Manual of Mental Disorders, Fourth Edition). RESULTS The area under the receiver operating characteristic curve was at least 0.80 for single items, a 3-item combination, and the total score of the COOP/WONCA Charts, as well as for the SRQ-20 and the WHO-5, for screening both for all mental disorders and for depressive disorders. The accuracy, sensitivity, specificity, and positive and negative predictive values of these measures ranged between 0.77 and 0.92. Community health workers and nurse assistants rated the understandability, ease of use, and clinical relevance of all 3 questionnaires as satisfactory. CONCLUSIONS One-time assessment of patients with the COOP/WONCA Charts is a valid and feasible option for screening for mental disorders by primary care teams.

Predictive Validity of the Braden Scale for Pressure Ulcer Risk in Elderly Residents of Long-Term Care Facilities

The aim of this study was to evaluate the predictive validity of the Braden Scale for Predicting Pressure Sore Risk in elderly residents of long-term care facilities (LTCFs) in Brazil. The determination of the cutoff score for the Brazilian population is important for the comparison between Brazilian and international studies and establishment of guidelines for prevention of pressure ulcers in our health care facilities. This is the first study of its kind in Brazil. This was a secondary analysis of a prospective cohort study conducted with 233 LTCF residents aged 60 and over who underwent complete skin examination and Braden Scale rating every 2 days for 3 months. Two groups of patients were considered: the total group (N = 233) and risk group (n = 94, total scores <= 18). Data from the first and last assessments were analyzed for sensitivity, specificity, and likelihood ratios. The best results were obtained for the total group, with cutoff scores of 18 and 17, sensitivity of 75.9% and 74.1%, specificity of 70.3% and 75.4%, and area under the receiver operating characteristic curve (AUC-ROC) of 0.79 and 0.81 at the first and last assessments, respectively. For the risk group, the cutoff scores of 16 (first assessment) and 13 (last assessment) were associated with a smaller AUC-ROC and, therefore, lower predictive accuracy. The Braden Scale showed good predictive validity in elderly LTCF residents. (Geriatr Nurs 2010;31:95-104)

Performance of body fat and body mass index cutoffs in elevated blood pressure screening among male children and adolescents

For percentage of body fat (%BF), there are no internationally accepted cutoffs. The primary function of body fat cutoffs should be to identify not only excessive body fatness, but also the increased risk of unhealthy outcomes, such as hypertension. The purpose of this study was to analyze the accuracy of different %BF and body mass index (BMI) cutoffs as screening measures for EBP in pediatric populations. It was a cross-sectional study with a sample of 358 male subjects from 8 to 18 years old. BP was measured by the oscilometric method, and body composition was measured by dual-energy X-ray absorptiometry (DXA). The accuracy of three reference tables used for body fat cutoffs was assessed. The three body fat reference tables were highly specific, but insensitive, for elevated BP screening. For elevated BP screening, all body fat cutoffs presented similar sensitivity (range=48.3-53.7%) and specificity (range=79.2-84.1%). The body fat cutoffs performed no better than BMI in screening of children and adolescents at risk of elevated BP (EBP). BMI seems a more attractive tool for this function, as it performed similarly and can be applied in large surveys and with lower costs. Hypertension Research (2011) 34, 963-967; doi:10.1038/hr.2011.61; published online 26 May 2011

Spectral characteristics of atrial electrograms in sinus rhythm correlates with sites of ganglionated plexuses in patients with paroxysmal atrial fibrillation

Aims To verify whether spectral components of atrial electrograms (AE) during sinus rhythm (SR) correlate with cardiac ganglionated plexus (GP) sites. Methods and results Thirteen patients undergoing atrial fibrillation (AF) ablation were prospectively enrolled. Prior to radio frequency application, endocardial AE were recorded with a sequential point-by-point approach. Electrical stimuli were delivered at 20 Hz, amplitude 100 V, and pulse width of 4 ms. A vagal response was defined as a high-frequency stimulation (HFS) evoked atrioventricular block or a prolongation of RR interval. Spectral analysis was performed on single AE during SR, sampling rate of 1000 Hz, Hanning window. Overall, 1488 SR electrograms were analysed from 186 different left atrium sites, 129 of them corresponding to negative vagal response sites, and 57 to positive response sites. The electrogram duration and the number of deflections were similar in positive and negative response sites. Spectral power density of sites with vagal response was lower between 26 and 83 Hz and higher between 107 and 200 Hz compared with negative response sites. The area between 120 and 170 Hz normalized to the total spectrum area was tested as a diagnostic parameter. Receiver operating characteristic curve analysis demonstrated that an area120-170/area(total) value >0.14 identified vagal sites with 70.9% sensitivity and 72.1% specificity. Conclusion Spectral analysis of AE during SR in sites that correspond to the anatomical location of the GP is feasible and may be a simpler method of mapping the cardiac autonomic nervous system, compared with the HFS technique.

Adaptation of an apple sooty blotch and flyspeck warning system for the Upper Midwest United States

A warning system for sooty blotch and flyspeck (SBFS) of apple, developed in the southeastern United States, uses cumulative hours of leaf wetness duration (LWD) to predict the timing of the first appearance of signs. In the Upper Midwest United States, however, this warning system has resulted in sporadic disease control failures. The purpose of the present study was to determine whether the warning system`s algorithm could be modified to provide more reliable assessment of SBFS risk. Hourly LWD, rainfall, relative humidity (RH), and temperature data were collected from orchards in Iowa, North Carolina, and Wisconsin in 2005 and 2006. Timing of the first appearance of SBFS signs was determined by weekly scouting. Preliminary analysis using scatterplots and boxplots suggested that Cumulative hours of RH >= 97% could be a useful predictor of SBFS appearance. Receiver operating characteristic curve analysis was used to compare the predictive performance of cumulative LWD and cumulative hours of RH >= 97%. Cumulative hours of RH >= 97% was a more conservative and accurate predictor than cumulative LWD for 15 site years in the Upper Midwest, but not for four site years in North Carolina. Performance of the SBFS warning system in the Upper Midwest and climatically similar regions may be improved if cumulative hours of RH >= 97% were substituted for cumulative LWD to predict the first appearance of SBFS.

Discontinuation of continuous renal replacement therapy: A post hoc analysis of a prospective multicenter observational study

Objectives: To describe current practice for the discontinuation of continuous renal replacement therapy in a multinational setting and to identify variables associated with successful discontinuation. The approach to discontinue continuous renal replacement therapy may affect patient outcomes. However, there is lack of information on how and under what conditions continuous renal replacement therapy is discontinued. Design: Post hoc analysis of a prospective observational study. Setting. Fifty-four intensive care units in 23 countries. Patients: Five hundred twenty-nine patients (52.6%) who survived initial therapy among 1006 patients treated with continuous renal replacement therapy. Interventions: None. Measurements and Main Results., Three hundred thirteen patients were removed successfully from continuous renal replacement therapy and did not require any renal replacement therapy for at least 7 days and were classified as the ""success"" group and the rest (216 patients) were classified as the ""repeat-RRT"" (renal replacement therapy) group. Patients in the ""success"" group had lower hospital mortality (28.5% vs. 42.7%, p < .0001) compared with patients in the ""repeat-RRT"" group. They also had lower creatinine and urea concentrations and a higher urine output at the time of stopping continuous renal replacement therapy. Multivariate logistic regression analysis for successful discontinuation of continuous renal replacement therapy identified urine output (during the 24 hrs before stopping continuous renal replacement therapy: odds ratio, 1.078 per 100 mL/day increase) and creatinine (odds ratio, 0.996 per mu mol/L increase) as significant predictors of successful cessation. The area under the receiver operating characteristic curve to predict successful discontinuation of continuous renal replacement therapy was 0.808 for urine output and 0.635 for creatinine. The predictive ability of urine output was negatively affected by the use of diuretics (area under the receiver operating characteristic curve, 0.671 with diuretics and 0.845 without diuretics). Conclusions. We report on the current practice of discontinuing continuous renal replacement therapy in a multinational setting. Urine output at the time of initial cessation (if continuous renal replacement therapy was the most important predictor of successful discontinuation, especially if occurring without the administration of diuretics. (Crit Care Med 2009; 37:2576-2582)