Nonparametric estimation of time-dependent ROC curves conditional on a continuous covariate

The receiver-operating characteristic (ROC) curve is the most widely used measure for evaluating the performance of a diagnostic biomarker when predicting a binary disease outcome. The ROC curve displays the true positive rate (or sensitivity) and the false positive rate (or 1-specificity) for different cut-off values used to classify an individual as healthy or diseased. In time-to-event studies, however, the disease status (e.g. death or alive) of an individual is not a fixed characteristic, and it varies along the study. In such cases, when evaluating the performance of the biomarker, several issues should be taken into account: first, the time-dependent nature of the disease status; and second, the presence of incomplete data (e.g. censored data typically present in survival studies). Accordingly, to assess the discrimination power of continuous biomarkers for time-dependent disease outcomes, time-dependent extensions of true positive rate, false positive rate, and ROC curve have been recently proposed. In this work, we present new nonparametric estimators of the cumulative/dynamic time-dependent ROC curve that allow accounting for the possible modifying effect of current or past covariate measures on the discriminatory power of the biomarker. The proposed estimators can accommodate right-censored data, as well as covariate-dependent censoring. The behavior of the estimators proposed in this study will be explored through simulations and illustrated using data from a cohort of patients who suffered from acute coronary syndrome.

Precision-recall operating characteristic (P-ROC) curves in imprecise environments

Traditionally, machine learning algorithms have been evaluated in applications where assumptions can be reliably made about class priors and/or misclassification costs. In this paper, we consider the case of imprecise environments, where little may be known about these factors and they may well vary significantly when the system is applied. Specifically, the use of precision-recall analysis is investigated and compared to the more well known performance measures such as error-rate and the receiver operating characteristic (ROC). We argue that while ROC analysis is invariant to variations in class priors, this invariance in fact hides an important factor of the evaluation in imprecise environments. Therefore, we develop a generalised precision-recall analysis methodology in which variation due to prior class probabilities is incorporated into a multi-way analysis of variance (ANOVA). The increased sensitivity and reliability of this approach is demonstrated in a remote sensing application.

ROC Curves within the Framework of Neural Network Assembly Memory Model: Some Analytic Results

On the basis of convolutional (Hamming) version of recent Neural Network Assembly Memory Model (NNAMM) for intact two-layer autoassociative Hopfield network optimal receiver operating characteristics (ROCs) have been derived analytically. A method of taking into account explicitly a priori probabilities of alternative hypotheses on the structure of information initiating memory trace retrieval and modified ROCs (mROCs, a posteriori probabilities of correct recall vs. false alarm probability) are introduced. The comparison of empirical and calculated ROCs (or mROCs) demonstrates that they coincide quantitatively and in this way intensities of cues used in appropriate experiments may be estimated. It has been found that basic ROC properties which are one of experimental findings underpinning dual-process models of recognition memory can be explained within our one-factor NNAMM.

ROC curves of obesity indicators have a predictive value for children hypertension aged 7-17 years

Objective: The aim of the study is to examine the distribution of integrated covariate and its association with blood pressure (BP) among children in Anhui province, China, and assess the predictive value of integrated covariate to children hypertension. Methods: A total of 2,828 subjects (1,588 male and 1,240 female) aged 7-17 years participated in this study. Height, weight, waistline, hipline and BP of all subjects were measured, obesity and overweight were defined by an international standard, specifying the measurement, the reference population, and the age and sex specific cut off points. High BP status was defined as systolic blood pressure (SBP) and/or diastolic blood pressure (DBP) > 95th percentile for age and gender. Results: Our results revealed that the prevalence of children hypertension was 11.03%, the SBP and DBP of obesity group were significantly higher than that of normal group. Anthropometric obesity indices such as body mass index (BMI) were positively correlated with SBP and DBP. Integrated covariate had a better performance than the single covariate in the receiver-operating characteristic (ROC) curve, the cut-off value; the sensitivity and the specificity of the integrated covariate were 0.112, 0.577, 0.683, respectively. Conclusion: Integrated covariate is a simple and effective anthropometric index to identify childhood hypertension.

Aplicação da metodologia ROC na avaliação de desempenho de índices de gravidade clínica em unidades de neonatologia de Portugal

Tese de Doutoramento em Engenharia Industrial e de Sistemas

Receiver operating characteristic (ROC) curve for classification of18F-NaF uptake on PET/CT

Abstract Objective: To assess the cutoff values established by ROC curves to classify18F-NaF uptake as normal or malignant. Materials and Methods: PET/CT images were acquired 1 hour after administration of 185 MBq of18F-NaF. Volumes of interest (VOIs) were drawn on three regions of the skeleton as follows: proximal right humerus diaphysis (HD), proximal right femoral diaphysis (FD) and first vertebral body (VB1), in a total of 254 patients, totalling 762 VOIs. The uptake in the VOIs was classified as normal or malignant on the basis of the radiopharmaceutical distribution pattern and of the CT images. A total of 675 volumes were classified as normal and 52 were classified as malignant. Thirty-five VOIs classified as indeterminate or nonmalignant lesions were excluded from analysis. The standardized uptake value (SUV) measured on the VOIs were plotted on an ROC curve for each one of the three regions. The area under the ROC (AUC) as well as the best cutoff SUVs to classify the VOIs were calculated. The best cutoff values were established as the ones with higher result of the sum of sensitivity and specificity. Results: The AUCs were 0.933, 0.889 and 0.975 for UD, FD and VB1, respectively. The best SUV cutoffs were 9.0 (sensitivity: 73%; specificity: 99%), 8.4 (sensitivity: 79%; specificity: 94%) and 21.0 (sensitivity: 93%; specificity: 95%) for UD, FD and VB1, respectively. Conclusion: The best cutoff value varies according to bone region of analysis and it is not possible to establish one value for the whole body.

A nonparametric method of comparing the partial areas under two correlated receiver operating characteristic curves

A non-parametric method was developed and tested to compare the partial areas under two correlated Receiver Operating Characteristic curves. Based on the theory of generalized U-statistics the mathematical formulas have been derived for computing ROC area, and the variance and covariance between the portions of two ROC curves. A practical SAS application also has been developed to facilitate the calculations. The accuracy of the non-parametric method was evaluated by comparing it to other methods. By applying our method to the data from a published ROC analysis of CT image, our results are very close to theirs. A hypothetical example was used to demonstrate the effects of two crossed ROC curves. The two ROC areas are the same. However each portion of the area between two ROC curves were found to be significantly different by the partial ROC curve analysis. For computation of ROC curves with large scales, such as a logistic regression model, we applied our method to the breast cancer study with Medicare claims data. It yielded the same ROC area computation as the SAS Logistic procedure. Our method also provides an alternative to the global summary of ROC area comparison by directly comparing the true-positive rates for two regression models and by determining the range of false-positive values where the models differ. ^

Evaluation of non-contact sampling and detection of explosives using receiver operating characteristic curves

The growing need for fast sampling of explosives in high throughput areas has increased the demand for improved technology for the trace detection of illicit compounds. Detection of the volatiles associated with the presence of the illicit compounds offer a different approach for sensitive trace detection of these compounds without increasing the false positive alarm rate. This study evaluated the performance of non-contact sampling and detection systems using statistical analysis through the construction of Receiver Operating Characteristic (ROC) curves in real-world scenarios for the detection of volatiles in the headspace of smokeless powder, used as the model system for generalizing explosives detection. A novel sorbent coated disk coined planar solid phase microextraction (PSPME) was previously used for rapid, non-contact sampling of the headspace containers. The limits of detection for the PSPME coupled to IMS detection was determined to be 0.5-24 ng for vapor sampling of volatile chemical compounds associated with illicit compounds and demonstrated an extraction efficiency of three times greater than other commercially available substrates, retaining >50% of the analyte after 30 minutes sampling of an analyte spike in comparison to a non-detect for the unmodified filters. Both static and dynamic PSPME sampling was used coupled with two ion mobility spectrometer (IMS) detection systems in which 10-500 mg quantities of smokeless powders were detected within 5-10 minutes of static sampling and 1 minute of dynamic sampling time in 1-45 L closed systems, resulting in faster sampling and analysis times in comparison to conventional solid phase microextraction-gas chromatography-mass spectrometry (SPME-GC-MS) analysis. Similar real-world scenarios were sampled in low and high clutter environments with zero false positive rates. Excellent PSPME-IMS detection of the volatile analytes were visualized from the ROC curves, resulting with areas under the curves (AUC) of 0.85-1.0 and 0.81-1.0 for portable and bench-top IMS systems, respectively. Construction of ROC curves were also developed for SPME-GC-MS resulting with AUC of 0.95-1.0, comparable with PSPME-IMS detection. The PSPME-IMS technique provides less false positive results for non-contact vapor sampling, cutting the cost and providing an effective sampling and detection needed in high-throughput scenarios, resulting in similar performance in comparison to well-established techniques with the added advantage of fast detection in the field.

Ibopamina 2% vs. sobrecarga hídrica como teste provocativo para glaucoma

PURPOSE: To compare the 2% ibopamine provocative test with the water drinking test as a provocative test for glaucoma. METHODS: Primary open-angle glaucoma patients and normal individuals were selected from CEROF-Universidade Federal de Goiânia UFG, and underwent the 2% ibopamine provocative test and the water drinking test in a randomized fashion, at least 1 week apart. Intraocular pressure (IOP) before and after both tests, Bland-Altman graph, sensitivity and specificity (as mesured by ROC curves) were obtained for both methods. RESULTS: Forty-seven eyes from 25 patients were included (27 eyes from 15 glaucoma patients and 20 eyes from 10 normal individuals), with a mean age of 54.2 ± 12.7 years. The mean MD of glaucoma patients was -2.8 ± 2.11 dB. There was no statistically difference in the baseline IOP (p=0.8) comparing glaucoma patients, but positive after the provocative tests (p=0.03), and in the IOP variation (4.4 ± 1.3 mmHg for ibopamine and 3.2 ± 2.2 mmHg for water drinking test, p=0.01). There was no difference in all studied parameters for normal individuals. The Bland-Altman graph showed high dispersion comparing both methods. The areas under the ROC curve were 0.987 for the ibopamine provocative test, and 0.807 for the water-drinking test. CONCLUSION: In this selected subgroup of glaucoma patients with early visual field defect, the ibopamine provocative test has shown better sensitivity/specificity than the water drinking test.

Sensitivity and specificity of machine learning classifiers for glaucoma diagnosis using Spectral Domain OCT and standard automated perimetry

PURPOSE: To evaluate the sensitivity and specificity of machine learning classifiers (MLCs) for glaucoma diagnosis using Spectral Domain OCT (SD-OCT) and standard automated perimetry (SAP). METHODS: Observational cross-sectional study. Sixty two glaucoma patients and 48 healthy individuals were included. All patients underwent a complete ophthalmologic examination, achromatic standard automated perimetry (SAP) and retinal nerve fiber layer (RNFL) imaging with SD-OCT (Cirrus HD-OCT; Carl Zeiss Meditec Inc., Dublin, California). Receiver operating characteristic (ROC) curves were obtained for all SD-OCT parameters and global indices of SAP. Subsequently, the following MLCs were tested using parameters from the SD-OCT and SAP: Bagging (BAG), Naive-Bayes (NB), Multilayer Perceptron (MLP), Radial Basis Function (RBF), Random Forest (RAN), Ensemble Selection (ENS), Classification Tree (CTREE), Ada Boost M1(ADA),Support Vector Machine Linear (SVML) and Support Vector Machine Gaussian (SVMG). Areas under the receiver operating characteristic curves (aROC) obtained for isolated SAP and OCT parameters were compared with MLCs using OCT+SAP data. RESULTS: Combining OCT and SAP data, MLCs' aROCs varied from 0.777(CTREE) to 0.946 (RAN).The best OCT+SAP aROC obtained with RAN (0.946) was significantly larger the best single OCT parameter (p<0.05), but was not significantly different from the aROC obtained with the best single SAP parameter (p=0.19). CONCLUSION: Machine learning classifiers trained on OCT and SAP data can successfully discriminate between healthy and glaucomatous eyes. The combination of OCT and SAP measurements improved the diagnostic accuracy compared with OCT data alone.

Validity and reliability of the Structured Clinical Interview for Mood Spectrum: Brazilian version (SCIMOODS-VB)

OBJECTIVE: The aim of this study was to translate the Structured Clinical Interview for Mood Spectrum into Brazilian Portuguese, measuring its reliability, validity, and defining scores for bipolar disorders. METHOD: Questionnaire was translated (into Brazilian Portuguese) and back-translated into English. Sample consisted of 47 subjects with bipolar disorder, 47 with major depressive disorder, 18 with schizophrenia and 22 controls. Inter-rater reliability was tested in 20 subjects with bipolar disorder and MDD. Internal consistency was measured using the Kuder Richardson formula. Forward stepwise discriminant analysis was performed. Scores were compared between groups; manic (M), depressive (D) and total (T) threshold scores were calculated through receiver operating characteristic (ROC) curves. RESULTS: Kuder Richardson coefficients were between 0.86 and 0.94. Intraclass correlation coefficient was 0.96 (CI 95 % 0.93-0.97). Subjects with bipolar disorder had higher M and T, and similar D scores, when compared to major depressive disorder (ANOVA, p < 0.001). The sub-domains that best discriminated unipolar and bipolar subjects were manic energy and manic mood. M had the best area under the curve (0.909), and values of M equal to or greater than 30 yielded 91.5% sensitivity and 74.5% specificity. CONCLUSION: Structured Clinical Interview for Mood Spectrum has good reliability and validity. Cut-off of 30 best differentiates subjects with bipolar disorder vs. unipolar depression. A cutoff score of 30 or higher in the mania sub-domain is appropriate to help make a distinction between subjects with bipolar disorder and those with unipolar depression.

Comparação entre pontos de corte para excesso de peso na detecção de pressão arterial elevada em adolescentes

OBJETIVOS: Avaliar a acurácia de três pontos de corte na determinação da pressão arterial elevada em adolescentes, dada a forte relação entre o excesso de peso e valores elevados de pressão arterial. MÉTODOS: Participaram do estudo 1.021 adolescentes de ambos os sexos, selecionados de maneira aleatória nas escolas públicas e particulares de Londrina (PR). O peso corporal foi aferido por meio de balança digital, e a estatura, por um estadiômetro portátil com extensão máxima de 2 metros. A pressão arterial foi avaliada através de um aparelho automático. A capacidade do índice de massa corporal de detectar a pressão arterial elevada foi averiguada por meio da curva ROC e seus parâmetros (sensibilidade, especificidade e área sob a curva). RESULTADOS: Os pontos de corte da proposta nacional apresentaram maior acurácia (masculino: 0,636±0,038; feminino: 0,585±0,043) quando comparados aos pontos de corte das propostas internacional (masculino: 0,594±0,040; feminino: 0,570±0,044) e norte-americana (masculino: 0,612±0,039; feminino: 0,578±0,044). CONCLUSÃO: A proposta nacional foi a que apresentou melhor acurácia na indicação de valores elevados de pressão arterial.

Predictive Validity of Waterlow Scale for Pressure Ulcer Development Risk in Hospitalized Patients

AIM: We sought to evaluate the predictive validity of the Waterlow Scale in hospitalized patients. SUBJECTS AND SETTING: The study was conducted at a general private hospital with 220 beds and a mean time of hospitalization of 7.4 days and a mean occupation rate of approximately 80%. Adult patients with a Braden Scale score of 18 or less and a Waterlow Scale score of 16 or more were studied. The sample consisted of 98 patients with a mean age of 71.1 +/- 15.5 years. METHODS: Skin assessment and scoring by using the Waterlow and Braden scales were completed on alternate days. Patients were examined at least 3 times to be considered for analysis. The data were submitted to sensitivity and specificity analysis by using receiver operating characteristic (ROC) curves and positive (+LR) and negative (-LR) likelihood ratios. RESULTS: The cutoff scores were 17, 20, and 20 in the first, second, and third assessment, respectively. Sensitivity was 71.4%, 85.7%, and 85.7% and specificity was 67.0%, 40.7%, and 32.9%, respectively. Analysis of the area under the ROC curve revealed good accuracy (0.64, 95% confidence interval [CI]: 0.35-0.93) only for the cutoff score 17 in the first assessment. The results also showed probabilities of 14%, 10%, and 9% for the development of pressure ulcer when the test results were positive (+LR) and of 3% (-LR) when the test results were negative for the cutoff scores in the first, second, and third assessment, respectively. CONCLUSION: The Waterlow Scale achieved good predictive validity in predicting pressure ulcer in hospitalized patients when a cutoff score of 17 was used in the first assessment.

Comparison of calcaneal ultrasound and DXA in young women

Purpose: The purpose of the study was to assess quantitative ultrasound (QUS) parameters in collegiate female gymnasts, a population whose training incorporates high-impact loading, which is particularly osteogenic, and to determine the discriminative capacity of this relatively new radiation-free technique compared with bone densitometry in a young healthy population. Methods: We studied 19 collegiate gymnasts and 23 healthy controls undergoing regular weight-bearing activity, matched for age (gymnasts 19.2 +/- 1.2, controls 19.9 +/- 1.6 yr) and body weight (gymnasts 56.7 +/- 3.7, controls 57.7 +/- 7.8 kg). QUS parameters of the calcaneus (broadband ultrasound attenuation (BUA), bone velocity (BV), and speed of sound (SOS)) were measured by a Walker Sonix UBA 575+. Bone mineral density (BMD; g.cm(-2)) of the lumbar spine, hip (Femoral neck, trochanter. Ward's triangle) and whole body was assessed by dual energy x-ray absorptiometry (DXA, Hologic QDR 1000/W). Data analysis included unpaired two-tailed Student's t-tests, analysis of variance, Pearson product-moment, and Spearman rank-order correlations. Results: Regional and whole body BMD of gymnasts was greater than controls (P < 0.001), with the difference being 7-28%. Average QUS parameters of the right and left calcaneus were also higher (P < 0.001) in the gymnasts. BUA, BV, and SOS were significantly (P < 0.001) correlated to each bone site with r = 0.54-0.79. Analysis of receiver operating characteristic (ROC) curves indicated no significant difference in sensitivity and specificity for QUS and DXA measures. Conclusions: These results indicate that QUS parameters of the calcaneus are higher in young women gymnasts compared to individuals who undergo regular weight-bearing activity and that QUS parameters are able to discriminate between these two groups in a similar manner as does regional and whole body BMD.

Detecting insomnia: comparison of four self-report measures of sleep in a young adult population

The sensitivity and specificity of four self-report measures of disordered sleep - the Sleep Impairment Index (SII), the Sleep Disorders Questionnaire (SDQ), the Dysfunctional Beliefs and Attitudes About Sleep Scale (DBAS) and the Sleep-Wake Activity Inventory (SWAI) - were compared in subjects with insomnia and normal sleep. Nineteen young adult subjects met DSM-IV criteria for primary insomnia and another 19 were normal control subjects. Discriminatory characteristics of each measure were assessed using receiver operator characteristic curve analyses. Discriminatory power was maximised for each measure to produce cut-scores applicable for identification of individuals with insomnia. The DBAS, SII and SDQ psychiatric DIMS subscale were found to correlate, and discriminated well between the two groups. The SWAI nocturnal sleep subscale was not found to be an accurate discriminator. The results suggest differences in the measures in their ability to detect insomnia, and offer guidelines as to the optimal use of test scores to identify young adults suspected of insomnia.