A nonparametric method of comparing the partial areas under two correlated receiver operating characteristic curves

A non-parametric method was developed and tested to compare the partial areas under two correlated Receiver Operating Characteristic curves. Based on the theory of generalized U-statistics the mathematical formulas have been derived for computing ROC area, and the variance and covariance between the portions of two ROC curves. A practical SAS application also has been developed to facilitate the calculations. The accuracy of the non-parametric method was evaluated by comparing it to other methods. By applying our method to the data from a published ROC analysis of CT image, our results are very close to theirs. A hypothetical example was used to demonstrate the effects of two crossed ROC curves. The two ROC areas are the same. However each portion of the area between two ROC curves were found to be significantly different by the partial ROC curve analysis. For computation of ROC curves with large scales, such as a logistic regression model, we applied our method to the breast cancer study with Medicare claims data. It yielded the same ROC area computation as the SAS Logistic procedure. Our method also provides an alternative to the global summary of ROC area comparison by directly comparing the true-positive rates for two regression models and by determining the range of false-positive values where the models differ. ^

Evaluation of non-contact sampling and detection of explosives using receiver operating characteristic curves

The growing need for fast sampling of explosives in high throughput areas has increased the demand for improved technology for the trace detection of illicit compounds. Detection of the volatiles associated with the presence of the illicit compounds offer a different approach for sensitive trace detection of these compounds without increasing the false positive alarm rate. This study evaluated the performance of non-contact sampling and detection systems using statistical analysis through the construction of Receiver Operating Characteristic (ROC) curves in real-world scenarios for the detection of volatiles in the headspace of smokeless powder, used as the model system for generalizing explosives detection. A novel sorbent coated disk coined planar solid phase microextraction (PSPME) was previously used for rapid, non-contact sampling of the headspace containers. The limits of detection for the PSPME coupled to IMS detection was determined to be 0.5-24 ng for vapor sampling of volatile chemical compounds associated with illicit compounds and demonstrated an extraction efficiency of three times greater than other commercially available substrates, retaining >50% of the analyte after 30 minutes sampling of an analyte spike in comparison to a non-detect for the unmodified filters. Both static and dynamic PSPME sampling was used coupled with two ion mobility spectrometer (IMS) detection systems in which 10-500 mg quantities of smokeless powders were detected within 5-10 minutes of static sampling and 1 minute of dynamic sampling time in 1-45 L closed systems, resulting in faster sampling and analysis times in comparison to conventional solid phase microextraction-gas chromatography-mass spectrometry (SPME-GC-MS) analysis. Similar real-world scenarios were sampled in low and high clutter environments with zero false positive rates. Excellent PSPME-IMS detection of the volatile analytes were visualized from the ROC curves, resulting with areas under the curves (AUC) of 0.85-1.0 and 0.81-1.0 for portable and bench-top IMS systems, respectively. Construction of ROC curves were also developed for SPME-GC-MS resulting with AUC of 0.95-1.0, comparable with PSPME-IMS detection. The PSPME-IMS technique provides less false positive results for non-contact vapor sampling, cutting the cost and providing an effective sampling and detection needed in high-throughput scenarios, resulting in similar performance in comparison to well-established techniques with the added advantage of fast detection in the field.

Receiver operating characteristic (ROC) curve for classification of18F-NaF uptake on PET/CT

Abstract Objective: To assess the cutoff values established by ROC curves to classify18F-NaF uptake as normal or malignant. Materials and Methods: PET/CT images were acquired 1 hour after administration of 185 MBq of18F-NaF. Volumes of interest (VOIs) were drawn on three regions of the skeleton as follows: proximal right humerus diaphysis (HD), proximal right femoral diaphysis (FD) and first vertebral body (VB1), in a total of 254 patients, totalling 762 VOIs. The uptake in the VOIs was classified as normal or malignant on the basis of the radiopharmaceutical distribution pattern and of the CT images. A total of 675 volumes were classified as normal and 52 were classified as malignant. Thirty-five VOIs classified as indeterminate or nonmalignant lesions were excluded from analysis. The standardized uptake value (SUV) measured on the VOIs were plotted on an ROC curve for each one of the three regions. The area under the ROC (AUC) as well as the best cutoff SUVs to classify the VOIs were calculated. The best cutoff values were established as the ones with higher result of the sum of sensitivity and specificity. Results: The AUCs were 0.933, 0.889 and 0.975 for UD, FD and VB1, respectively. The best SUV cutoffs were 9.0 (sensitivity: 73%; specificity: 99%), 8.4 (sensitivity: 79%; specificity: 94%) and 21.0 (sensitivity: 93%; specificity: 95%) for UD, FD and VB1, respectively. Conclusion: The best cutoff value varies according to bone region of analysis and it is not possible to establish one value for the whole body.

The influence of experience and training in a group of novice observers: a jackknife alternative free-response receiver operating characteristic analysis

Purpose - The study evaluates the pre- and post-training lesion localisation ability of a group of novice observers. Parallels are drawn with the performance of inexperienced radiographers taking part in preliminary clinical evaluation (PCE) and ‘red-dot’ systems, operating within radiography practice. Materials and methods - Thirty-four novice observers searched 92 images for simulated lesions. Pre-training and post-training evaluations were completed following the free-response the receiver operating characteristic (FROC) method. Training consisted of observer performance methodology, the characteristics of the simulated lesions and information on lesion frequency. Jackknife alternative FROC (JAFROC) and highest rating inferred ROC analyses were performed to evaluate performance difference on lesion-based and case-based decisions. The significance level of the test was set at 0.05 to control the probability of Type I error. Results - JAFROC analysis (F(3,33) = 26.34, p < 0.0001) and highest-rating inferred ROC analysis (F(3,33) = 10.65, p = 0.0026) revealed a statistically significant difference in lesion detection performance. The JAFROC figure-of-merit was 0.563 (95% CI 0.512,0.614) pre-training and 0.677 (95% CI 0.639,0.715) post-training. Highest rating inferred ROC figure-of-merit was 0.728 (95% CI 0.701,0.755) pre-training and 0.772 (95% CI 0.750,0.793) post-training. Conclusions - This study has demonstrated that novice observer performance can improve significantly. This study design may have relevance in the assessment of inexperienced radiographers taking part in PCE or commenting scheme for trauma.

Avaliação do desempenho de indicadores com base na metodologia ROC (Receiver Operating Characteristic)

Dissertação de mestrado em Bioinformática

Relating body mass index to figural stimuli: population-based normative data for Caucasians

OBJECTIVE: To establish body mass index (BMI) norms for standard figural stimuli using a large Caucasian population-based sample. In addition, we sought to determine the effectiveness of the figural stimuli to identify individuals as obese or thin. DESIGN: All Caucasian twins born in Virginia between 1915 and 1971 were identified by public birth record. In addition, 3347 individual twins responded to a letter published in the newsletter of the American Association of Retired Persons (AARP). All adult twins (aged 18 and over) from both of these sources and their family members were mailed a 16 page 'Health and Lifestyle' questionnaire. SUBJECTS: BMI and silhouette data were available on 16 728 females and 11 366 males ranging in age from 18- 100. MEASUREMENTS: Self-report information on height-weight, current body size, desired body size and a discrepancy score using standard figural stimuli. RESULTS: Gender- and age-specific norms are presented linking BMI to each of the figural stimuli. Additional norms for desired body size and discrepancy scores are also presented. Receiver operating curves (ROC) indicate that the figural stimuli are effective in classifying individuals as obese or thin. CONCLUSIONS: With the establishment of these norms, the silhouettes used in standard body image assessment can now be linked to BMI. Differences were observed between women and men in terms of desired body size and discrepancy scores, with women preferring smaller sizes. The figural stimuli are a robust technique for classifying individuals as obese or thin.

Algoritmo para el diagnóstico precoz de la deficiencia de vitamina b12 en ancianos

BACKGROUND: The elderly population is particularly at risk for developing vitamin B12-deficiency. Serum cobalamin does not necessarily reflect a normal B12 status. The determination of methylmalonic acid is not available in all laboratories. Issues of sensitivity for holotranscobalamin and the low specificity of total homocysteine limit their utility. The aim of the present study is to establish a diagnostic algorithm by using a combination of these markers in place of a single measurement. METHODS: We compared the diagnostic efficiency of these markers for detection of vitamin B12 deficiency in a population (n = 218) of institutionalized elderly (median age 80 years). Biochemical, haematological and morphological data were used to categorize people with or without vitamin B12 deficiency. RESULTS: In receiver operating curves characteristics for detection on vitamin B12 deficiency using single measurements, serum folate has the greatest area under the curve (0.87) and homocysteine the lowest (0.67). The best specificity was observed for erythrocyte folate and methylmalonic acid (100% for both) but their sensitivity was very low (17% and 53%, respectively). The highest sensitivity was observed for homocysteine (81%) and serum folate (74%). When we combined these markers, starting with serum and erythrocyte folate, followed by holotranscobalamin and ending by methylmalonic acid measurements, the overall sensitivity and specificity of the algorithm were 100% and 90%, respectively. CONCLUSION: The proposed algorithm, which combines erythrocyte folate, serum folate, holotranscobalamin and methylmalonic acid, but eliminate B12 and tHcy measurements, is a useful alternative for vitamin B12 deficiency screening in an elderly institutionalized cohort.

Improvement of cardiovascular risk prediction: time to review current knowledge, debates, and fundamentals on how to assess test characteristics.

Cardiovascular risk assessment might be improved with the addition of emerging, new tests derived from atherosclerosis imaging, laboratory tests or functional tests. This article reviews relative risk, odds ratios, receiver-operating curves, posttest risk calculations based on likelihood ratios, the net reclassification improvement and integrated discrimination. This serves to determine whether a new test has an added clinical value on top of conventional risk testing and how this can be verified statistically. Two clinically meaningful examples serve to illustrate novel approaches. This work serves as a review and basic work for the development of new guidelines on cardiovascular risk prediction, taking into account emerging tests, to be proposed by members of the 'Taskforce on Vascular Risk Prediction' under the auspices of the Working Group 'Swiss Atherosclerosis' of the Swiss Society of Cardiology in the future.

Endorectal Magnetic Resonance Imaging and Spectroscopy Are Useful for Selecting Candidates for Biopsy among Patients with Persistently Elevated Prostate Specific Antigen

To evaluate the efficacy of endorectal Magnetic Resonance Imaging (MRI) and Magnetic Resonance Spetroscopic Imaging (MRSI) combined with total prostate-specific antigen (tPSA) and free prostate-specific antigen (fPSA) in selecting candidates for biopsy. Subjects and Methods: 246 patients with elevated tPSA (median: 7.81 ng/ml) underwent endorectal MRI and MRSI before Transrectal Ultrasound (TRUS) biopsy (10 peripheral + 2 central cores); patients with positive biopsies were treated with radical intention; those with negative biopsies were followed up and underwent MRSI before each additional biopsy if tPSA rose persistently. Mean follow-up: 27.6 months. We compared MRI, MRSI, tPSA, and fPSA with histopathology by sextant and determined the association between the Gleason score and MRI and MRSI. We determined the most accurate combination to detect prostate cancer (PCa) using receiver operating curves; we estimated the odds ratios (OR) and calculated sensitivity, specificity, and positive and negative predictive values. Results: No difference in tPSA was found between patients with and without PCa (p = 0.551). In the peripheral zone, the risk of PCa increased with MRSI grade; patients with high-grade MRSI had the greatest risk of PCa over time (OR = 328.6); the model including MRI, MRSI, tPSA, and fPSA was more accurate (Area under Curve: AUC = 95.7%) than MRI alone (AUC = 85.1%) or fPSA alone (AUC = 78.1%), but not than MRSI alone (94.5%). In the transitional zone, the model was less accurate (AUC = 84.4%). The association (p = 0.005) between MRSI and Gleason score was significant in both zones. Conclusions: MRSI is useful in patients with elevated tPSA. High-grade MRSI lesions call for repeated biopsies. Men with negative MRSI may forgo further biopsies because a significantly high Gleason lesion is very unlikely

Cardiovascular Risk Prediction with Ultrasound

This paper addresses primary care physicians, cardiologists, internists, angiologists and doctors desirous of improving vascular risk prediction in primary care. Many cardiovascular risk factors act aggressively on the arterial wall and result in atherosclerosis and atherothrombosis. Cardiovascular prognosis derived from ultrasound imaging is, however, excellent in subjects without formation of intimal thickening or atheromas. Since ultrasound visualises the arterial wall directly, the information derived from the arterial wall may add independent incremental information to the knowledge of risk derived from global risk assessment. This paper provides an overview on plaque imaging for vascular risk prediction in two parts: Part 1: Carotid IMT is frequently used as a surrogate marker for outcome in intervention studies addressing rather large cohorts of subjects. Carotid IMT as a risk prediction tool for the prevention of acute myocardial infarction and stroke has been extensively studied in many patients since 1987, and has yielded incremental hazard ratios for these cardiovascular events independently of established cardiovascular risk factors. However, carotid IMT measurements are not used uniformly and therefore still lack widely accepted standardisation. Hence, at an individual, practicebased level, carotid IMT is not recommended as a risk assessment tool. The total plaque area of the carotid arteries (TPA) is a measure of the global plaque burden within both carotid arteries. It was recently shown in a large Norwegian cohort involving over 6000 subjects that TPA is a very good predictor for future myocardial infarction in women with an area under the curve (AUC) using a receiver operating curves (ROC) value of 0.73 (in men: 0.63). Further, the AUC for risk prediction is high both for vascular death in a vascular prevention clinic group (AUC 0.77) and fatal or nonfatal myocardial infarction in a true primary care group (AUC 0.79). Since TPA has acceptable reproducibility, allows calculation of posttest risk and is easily obtained at low cost, this risk assessment tool may come in for more widespread use in the future and also serve as a tool for atherosclerosis tracking and guidance for intensity of preventive therapy. However, more studies with TPA are needed. Part 2: Carotid and femoral plaque formation as detected by ultrasound offers a global view of the extent of atherosclerosis. Several prospective cohort studies have shown that cardiovascular risk prediction is greater for plaques than for carotid IMT. The number of arterial beds affected by significant atheromas may simply be added numerically to derive additional information on the risk of vascular events. A new atherosclerosis burden score (ABS) simply calculates the sum of carotid and femoral plaques encountered during ultrasound scanning. ABS correlates well and independently with the presence of coronary atherosclerosis and stenosis as measured by invasive coronary angiogram. However, the prognostic power of ABS as an independent marker of risk still needs to be elucidated in prospective studies. In summary, the large number of ways to measure atherosclerosis and related changes in human arteries by ultrasound indicates that this technology is not yet sufficiently perfected and needs more standardisation and workup on clearly defined outcome studies before it can be recommended as a practice-based additional risk modifier.

Comparison of Griffiths-II and Bayley-II tests for the developmental assessment of high-risk infants.

INTRODUCTION: Two important risk factors for abnormal neurodevelopment are preterm birth and neonatal hypoxic ischemic encephalopathy. The new revisions of Griffiths Mental Development Scale (Griffiths-II, [1996]) and the Bayley Scales of Infant Development (BSID-II, [1993]) are two of the most frequently used developmental diagnostics tests. The Griffiths-II is divided into five subscales and a global development quotient (QD), and the BSID-II is divided into two scales, the Mental scale (MDI) and the Psychomotor scale (PDI). The main objective of this research was to establish the extent to which developmental diagnoses obtained using the new revisions of these two tests are comparable for a given child. MATERIAL AND METHODS: Retrospective study of 18-months-old high-risk children examined with both tests in the follow-up Unit of the Clinic of Neonatology of our tertiary care university Hospital between 2011 and 2012. To determine the concurrent validity of the two tests paired t-tests and Pearson product-moment correlation coefficients were computed. Using the BSID-II as a gold standard, the performance of the Griffiths-II was analyzed with receiver operating curves. RESULTS: 61 patients (80.3% preterm, 14.7% neonatal asphyxia) were examined. For the BSID-II the MDI mean was 96.21 (range 67-133) and the PDI mean was 87.72 (range 49-114). For the Griffiths-II, the QD mean was 96.95 (range 60-124), the locomotors subscale mean was 92.57 (range 49-119). The score of the Griffiths locomotors subscale was significantly higher than the PDI (p<0.001). Between the Griffiths-II QD and the BSID-II MDI no significant difference was found, and the area under the curve was 0.93, showing good validity. All correlations were high and significant with a Pearson product-moment correlation coefficient >0.8. CONCLUSIONS: The meaning of the results for a given child was the same for the two tests. Two scores were interchangeable, the Griffiths-II QD and the BSID-II MDI.

Nutrimetabolomics fingerprinting to identify biomarkers of bread exposure in a free-living population from the PREDIMED study cohort

Bread is one of the most widely consumed foods. Its impact on human health is currently of special interest for researchers. We aimed to identify biomarkers of bread consumption by applying a nutrimetabolomic approach to a free-living population. An untargeted HPLC q-TOF-MS and multivariate analysis was applied to human urine from 155 subjects stratified by habitual bread consumption in three groups: non-consumers of bread (n = 56), white-bread consumers (n = 48) and whole-grain bread consumers (n = 51). The most differential metabolites (variable importance for projection ≥1.5) included compounds originating from cereal plant phytochemicals such as benzoxazinoids and alkylresorcinol metabolites, and compounds produced by gut microbiota (such as enterolactones, hydroxybenzoic and dihydroferulic acid metabolites). Pyrraline, riboflavin, 3-indolecarboxylic acid glucuronide, 2,8-dihydroxyquinoline glucuronide and N-α-acetylcitrulline were also tentatively identified. In order to combine multiple metabolites in a model to predict bread consumption, a stepwise logistic regression analysis was used. Receiver operating curves were constructed to evaluate the global performance of individual metabolites and their combination. The area under the curve values [AUC (95 % CI)] of combined models ranged from 77.8 % (69.1 86.4 %) to 93.7 % (89.4 98.1 %), whereas the AUC for the metabolites included in the models had weak values when they were evaluated individually: from 58.1 % (46.6 69.7 %) to 78.4 % (69.8 87.1 %). Our study showed that a daily bread intake significantly impacted on the urinary metabolome, despite being examined under uncontrolled free-living conditions. We further concluded that a combination of several biomarkers of exposure is better than a single biomarker for the predictive ability of discriminative analysis.

Nutrimetabolomics fingerprinting to identify biomarkers of bread exposure in a free-living population from the PREDIMED study cohort

Bread is one of the most widely consumed foods. Its impact on human health is currently of special interest for researchers. We aimed to identify biomarkers of bread consumption by applying a nutrimetabolomic approach to a free-living population. An untargeted HPLC q-TOF-MS and multivariate analysis was applied to human urine from 155 subjects stratified by habitual bread consumption in three groups: non-consumers of bread (n = 56), white-bread consumers (n = 48) and whole-grain bread consumers (n = 51). The most differential metabolites (variable importance for projection ≥1.5) included compounds originating from cereal plant phytochemicals such as benzoxazinoids and alkylresorcinol metabolites, and compounds produced by gut microbiota (such as enterolactones, hydroxybenzoic and dihydroferulic acid metabolites). Pyrraline, riboflavin, 3-indolecarboxylic acid glucuronide, 2,8-dihydroxyquinoline glucuronide and N-α-acetylcitrulline were also tentatively identified. In order to combine multiple metabolites in a model to predict bread consumption, a stepwise logistic regression analysis was used. Receiver operating curves were constructed to evaluate the global performance of individual metabolites and their combination. The area under the curve values [AUC (95 % CI)] of combined models ranged from 77.8 % (69.1 86.4 %) to 93.7 % (89.4 98.1 %), whereas the AUC for the metabolites included in the models had weak values when they were evaluated individually: from 58.1 % (46.6 69.7 %) to 78.4 % (69.8 87.1 %). Our study showed that a daily bread intake significantly impacted on the urinary metabolome, despite being examined under uncontrolled free-living conditions. We further concluded that a combination of several biomarkers of exposure is better than a single biomarker for the predictive ability of discriminative analysis.

Développement et validation d’un index de prédiction des risques d’institutionnalisation et de décès pour le contrôle des variables confondantes non-mesurées dans la population âgée

Le biais de confusion est un défi majeur des études observationnelles, surtout s'ils sont induits par des caractéristiques difficiles, voire impossibles, à mesurer dans les banques de données administratives de soins de santé. Un des biais de confusion souvent présents dans les études pharmacoépidémiologiques est la prescription sélective (en anglais « prescription channeling »), qui se manifeste lorsque le choix du traitement dépend de l'état de santé du patient et/ou de son expérience antérieure avec diverses options thérapeutiques. Parmi les méthodes de contrôle de ce biais, on retrouve le score de comorbidité, qui caractérise l'état de santé d'un patient à partir de médicaments délivrés ou de diagnostics médicaux rapportés dans les données de facturations des médecins. La performance des scores de comorbidité fait cependant l'objet de controverses car elle semble varier de façon importante selon la population d'intérêt. Les objectifs de cette thèse étaient de développer, valider, et comparer les performances de deux scores de comorbidité (un qui prédit le décès et l’autre qui prédit l’institutionnalisation), développés à partir des banques de services pharmaceutiques de la Régie de l'assurance-maladie du Québec (RAMQ) pour leur utilisation dans la population âgée. Cette thèse vise également à déterminer si l'inclusion de caractéristiques non rapportées ou peu valides dans les banques de données administratives (caractéristiques socio-démographiques, troubles mentaux ou du sommeil), améliore la performance des scores de comorbidité dans la population âgée. Une étude cas-témoins intra-cohorte fut réalisée. La cohorte source consistait en un échantillon aléatoire de 87 389 personnes âgées vivant à domicile, répartie en une cohorte de développement (n=61 172; 70%) et une cohorte de validation (n=26 217; 30%). Les données ont été obtenues à partir des banques de données de la RAMQ. Pour être inclus dans l’étude, les sujets devaient être âgés de 66 ans et plus, et être membres du régime public d'assurance-médicaments du Québec entre le 1er janvier 2000 et le 31 décembre 2009. Les scores ont été développés à partir de la méthode du Framingham Heart Study, et leur performance évaluée par la c-statistique et l’aire sous les courbes « Receiver Operating Curves ». Pour le dernier objectif qui est de documenter l’impact de l’ajout de variables non-mesurées ou peu valides dans les banques de données au score de comorbidité développé, une étude de cohorte prospective (2005-2008) a été réalisée. La population à l'étude, de même que les données, sont issues de l'Étude sur la Santé des Aînés (n=1 494). Les variables d'intérêt incluaient statut marital, soutien social, présence de troubles de santé mentale ainsi que troubles du sommeil. Tel que décrit dans l'article 1, le Geriatric Comorbidity Score (GCS) basé sur le décès, a été développé et a présenté une bonne performance (c-statistique=0.75; IC95% 0.73-0.78). Cette performance s'est avérée supérieure à celle du Chronic Disease Score (CDS) lorsqu'appliqué dans la population à l'étude (c-statistique du CDS : 0.47; IC 95%: 0.45-0.49). Une revue de littérature exhaustive a montré que les facteurs associés au décès étaient très différents de ceux associés à l’institutionnalisation, justifiant ainsi le développement d'un score spécifique pour prédire le risque d'institutionnalisation. La performance de ce dernier s'est avérée non statistiquement différente de celle du score de décès (c-statistique institutionnalisation : 0.79 IC95% 0.77-0.81). L'inclusion de variables non rapportées dans les banques de données administratives n'a amélioré que de 11% la performance du score de décès; le statut marital et le soutien social ayant le plus contribué à l'amélioration observée. En conclusion, de cette thèse, sont issues trois contributions majeures. D'une part, il a été démontré que la performance des scores de comorbidité basés sur le décès dépend de la population cible, d'où l'intérêt du Geriatric Comorbidity Score, qui fut développé pour la population âgée vivant à domicile. D'autre part, les médicaments associés au risque d'institutionnalisation diffèrent de ceux associés au risque de décès dans la population âgé, justifiant ainsi le développement de deux scores distincts. Cependant, les performances des deux scores sont semblables. Enfin, les résultats indiquent que, dans la population âgée, l'absence de certaines caractéristiques ne compromet pas de façon importante la performance des scores de comorbidité déterminés à partir de banques de données d'ordonnances. Par conséquent, les scores de comorbidité demeurent un outil de recherche important pour les études observationnelles.