SAS macros for point and interval estimation of area under the receiver operating characteristic curve for non-proportional and proportional hazards Weibull models

Aims and objectives  For prediction of risk of cardiovascular end points using survival models the proportional hazards assumption is often not met. Thus, non-proportional hazards models are more appropriate for developing risk prediction equations in such situations. However, computer program for evaluating the prediction performance of such models has been rarely addressed. We therefore developed SAS macro programs for evaluating the discriminative ability of a non-proportional hazards Weibull model developed by Anderson (1991) and that of a proportional hazards Weibull model using the area under receiver operating characteristic (ROC) curve.

Method  Two SAS macro programs for non-proportional hazards Weibull model using Proc NLIN and Proc NLP respectively and model validation using area under ROC curve (with its confidence limits) were written with SAS IML language. A similar SAS macro for proportional hazards Weibull model was also written.

Results  The computer program was applied to data on coronary heart disease incidence for a Framingham population cohort. The five risk factors considered were current smoking, age, blood pressure, cholesterol and obesity. The predictive ability of the non-proportional hazard Weibull model was slightly higher than that of its proportional hazard counterpart. An advantage of SAS Proc NLP in terms of the example provided here is that it provides significance level for the parameter estimates whereas Proc NLIN does not.

Conclusion  The program is very useful for evaluating the predictive performance of non-proportional and proportional hazards Weibull models.

A nonparametric method of comparing the partial areas under two correlated receiver operating characteristic curves

A non-parametric method was developed and tested to compare the partial areas under two correlated Receiver Operating Characteristic curves. Based on the theory of generalized U-statistics the mathematical formulas have been derived for computing ROC area, and the variance and covariance between the portions of two ROC curves. A practical SAS application also has been developed to facilitate the calculations. The accuracy of the non-parametric method was evaluated by comparing it to other methods. By applying our method to the data from a published ROC analysis of CT image, our results are very close to theirs. A hypothetical example was used to demonstrate the effects of two crossed ROC curves. The two ROC areas are the same. However each portion of the area between two ROC curves were found to be significantly different by the partial ROC curve analysis. For computation of ROC curves with large scales, such as a logistic regression model, we applied our method to the breast cancer study with Medicare claims data. It yielded the same ROC area computation as the SAS Logistic procedure. Our method also provides an alternative to the global summary of ROC area comparison by directly comparing the true-positive rates for two regression models and by determining the range of false-positive values where the models differ. ^

Evaluation of non-contact sampling and detection of explosives using receiver operating characteristic curves

The growing need for fast sampling of explosives in high throughput areas has increased the demand for improved technology for the trace detection of illicit compounds. Detection of the volatiles associated with the presence of the illicit compounds offer a different approach for sensitive trace detection of these compounds without increasing the false positive alarm rate. This study evaluated the performance of non-contact sampling and detection systems using statistical analysis through the construction of Receiver Operating Characteristic (ROC) curves in real-world scenarios for the detection of volatiles in the headspace of smokeless powder, used as the model system for generalizing explosives detection. A novel sorbent coated disk coined planar solid phase microextraction (PSPME) was previously used for rapid, non-contact sampling of the headspace containers. The limits of detection for the PSPME coupled to IMS detection was determined to be 0.5-24 ng for vapor sampling of volatile chemical compounds associated with illicit compounds and demonstrated an extraction efficiency of three times greater than other commercially available substrates, retaining >50% of the analyte after 30 minutes sampling of an analyte spike in comparison to a non-detect for the unmodified filters. Both static and dynamic PSPME sampling was used coupled with two ion mobility spectrometer (IMS) detection systems in which 10-500 mg quantities of smokeless powders were detected within 5-10 minutes of static sampling and 1 minute of dynamic sampling time in 1-45 L closed systems, resulting in faster sampling and analysis times in comparison to conventional solid phase microextraction-gas chromatography-mass spectrometry (SPME-GC-MS) analysis. Similar real-world scenarios were sampled in low and high clutter environments with zero false positive rates. Excellent PSPME-IMS detection of the volatile analytes were visualized from the ROC curves, resulting with areas under the curves (AUC) of 0.85-1.0 and 0.81-1.0 for portable and bench-top IMS systems, respectively. Construction of ROC curves were also developed for SPME-GC-MS resulting with AUC of 0.95-1.0, comparable with PSPME-IMS detection. The PSPME-IMS technique provides less false positive results for non-contact vapor sampling, cutting the cost and providing an effective sampling and detection needed in high-throughput scenarios, resulting in similar performance in comparison to well-established techniques with the added advantage of fast detection in the field.

Precision-recall operating characteristic (P-ROC) curves in imprecise environments

Traditionally, machine learning algorithms have been evaluated in applications where assumptions can be reliably made about class priors and/or misclassification costs. In this paper, we consider the case of imprecise environments, where little may be known about these factors and they may well vary significantly when the system is applied. Specifically, the use of precision-recall analysis is investigated and compared to the more well known performance measures such as error-rate and the receiver operating characteristic (ROC). We argue that while ROC analysis is invariant to variations in class priors, this invariance in fact hides an important factor of the evaluation in imprecise environments. Therefore, we develop a generalised precision-recall analysis methodology in which variation due to prior class probabilities is incorporated into a multi-way analysis of variance (ANOVA). The increased sensitivity and reliability of this approach is demonstrated in a remote sensing application.

Women who miscarry : the effectiveness and clinical utility of the Kessler 10 questionnaire in screening for ongoing psychological distress

Background: Early pregnancy loss has been linked to enduring psychological morbidity. Aims: This study aimed to investigate the utility of the Kessler 10 (K10) questionnaire as a brief screening instrument to identify women at risk for the development of psychiatric diagnoses three months post-miscarriage. Method: Participants were 117 consecutive women presenting at a public hospital emergency department and receiving a diagnosis of miscarriage. Main outcome measures: K10 screen for psychological distress and the Structured Clinical Interview for DSM Disorders to determine psychiatric diagnoses. Results: A majority of women (81.2%) experienced elevated levels of distress initially, 24.8% in the very high range. They were not at increased risk of psychiatric diagnoses at three months compared with the general population; however, they were significantly more likely to report subsyndromal symptoms at this time compared with the general population. The baseline K10 score was the only significant predictor of distress at follow-up (r = 0.45, P < 0.001). The receiver operating characteristic curve shows that a cut-off of 14 on the K10 has suitable sensitivity (97%) and specificity (82%) for predicting ongoing psychological distress in women who miscarry. Conclusions: The K10 is effective in identifying women at risk for ensuring psychological symptoms following miscarriage.

Corneal confocal microscopy : a novel noninvasive test to diagnose and stratify the severity of human diabetic neuropathy

OBJECTIVE: The accurate quantification of human diabetic neuropathy is important to define at-risk patients, anticipate deterioration, and assess new therapies. ---------- RESEARCH DESIGN AND METHODS: A total of 101 diabetic patients and 17 age-matched control subjects underwent neurological evaluation, neurophysiology tests, quantitative sensory testing, and evaluation of corneal sensation and corneal nerve morphology using corneal confocal microscopy (CCM). ---------- RESULTS: Corneal sensation decreased significantly (P = 0.0001) with increasing neuropathic severity and correlated with the neuropathy disability score (NDS) (r = 0.441, P < 0.0001). Corneal nerve fiber density (NFD) (P < 0.0001), nerve fiber length (NFL), (P < 0.0001), and nerve branch density (NBD) (P < 0.0001) decreased significantly with increasing neuropathic severity and correlated with NDS (NFD r = −0.475, P < 0.0001; NBD r = −0.511, P < 0.0001; and NFL r = −0.581, P < 0.0001). NBD and NFL demonstrated a significant and progressive reduction with worsening heat pain thresholds (P = 0.01). Receiver operating characteristic curve analysis for the diagnosis of neuropathy (NDS >3) defined an NFD of <27.8/mm2 with a sensitivity of 0.82 (95% CI 0.68–0.92) and specificity of 0.52 (0.40–0.64) and for detecting patients at risk of foot ulceration (NDS >6) defined a NFD cutoff of <20.8/mm2 with a sensitivity of 0.71 (0.42–0.92) and specificity of 0.64 (0.54–0.74). ---------- CONCLUSIONS: CCM is a noninvasive clinical technique that may be used to detect early nerve damage and stratify diabetic patients with increasing neuropathic severity. Established diabetic neuropathy leads to pain and foot ulceration. Detecting neuropathy early may allow intervention with treatments to slow or reverse this condition (1). Recent studies suggested that small unmyelinated C-fibers are damaged early in diabetic neuropathy (2–4) but can only be detected using invasive procedures such as sural nerve biopsy (4,5) or skin-punch biopsy (6–8). Our studies have shown that corneal confocal microscopy (CCM) can identify early small nerve fiber damage and accurately quantify the severity of diabetic neuropathy (9–11). We have also shown that CCM relates to intraepidermal nerve fiber loss (12) and a reduction in corneal sensitivity (13) and detects early nerve fiber regeneration after pancreas transplantation (14). Recently we have also shown that CCM detects nerve fiber damage in patients with Fabry disease (15) and idiopathic small fiber neuropathy (16) when results of electrophysiology tests and quantitative sensory testing (QST) are normal. In this study we assessed corneal sensitivity and corneal nerve morphology using CCM in diabetic patients stratified for the severity of diabetic neuropathy using neurological evaluation, electrophysiology tests, and QST. This enabled us to compare CCM and corneal esthesiometry with established tests of diabetic neuropathy and define their sensitivity and specificity to detect diabetic patients with early neuropathy and those at risk of foot ulceration.

Analysis of acute-phase proteins, AHSG, C3, CLI, HP and SAA, reveals distinctive expression patterns associated with breast, colorectal and lung cancer

Early detection, clinical management and disease recurrence monitoring are critical areas in cancer treatment in which specific biomarker panels are likely to be very important in each of these key areas. We have previously demonstrated that levels of alpha-2-heremans-schmid-glycoprotein (AHSG), complement component C3 (C3), clusterin (CLI), haptoglobin (HP) and serum amyloid A (SAA) are significantly altered in serum from patients with squamous cell carcinoma of the lung. Here, we report the abundance levels for these proteins in serum samples from patients with advanced breast cancer, colorectal cancer (CRC) and lung cancer compared to healthy controls (age and gender matched) using commercially available enzyme-linked immunosorbent assay kits. Logistic regression (LR) models were fitted to the resulting data, and the classification ability of the proteins was evaluated using receiver-operating characteristic curve and leave-one-out cross-validation (LOOCV). The most accurate individual candidate biomarkers were C3 for breast cancer [area under the curve (AUC) = 0.89, LOOCV = 73%], CLI for CRC (AUC = 0.98, LOOCV = 90%), HP for small cell lung carcinoma (AUC = 0.97, LOOCV = 88%), C3 for lung adenocarcinoma (AUC = 0.94, LOOCV = 89%) and HP for squamous cell carcinoma of the lung (AUC = 0.94, LOOCV = 87%). The best dual combination of biomarkers using LR analysis were found to be AHSG + C3 (AUC = 0.91, LOOCV = 83%) for breast cancer, CLI + HP (AUC = 0.98, LOOCV = 92%) for CRC, C3 + SAA (AUC = 0.97, LOOCV = 91%) for small cell lung carcinoma and HP + SAA for both adenocarcinoma (AUC = 0.98, LOOCV = 96%) and squamous cell carcinoma of the lung (AUC = 0.98, LOOCV = 84%). The high AUC values reported here indicated that these candidate biomarkers have the potential to discriminate accurately between control and cancer groups both individually and in combination with other proteins. Copyright © 2011 UICC.

Development of the Rapid Assessment, Prioritisation and Referral Tool (RAPaRT) for multidisciplinary teams in emergency care settings

Objective This investigation utilised the expertise of allied members of multidisciplinary teams working in emergency care settings to develop and validate a Rapid Assessment Prioritisation and Referral Tool (RAPaRT). This instrument is intended for use among patients (with non-life threatening acuity) presenting to emergency care settings to indicate when referral to an allied member of the multidisciplinary team is warranted. Method This three stage instrument development and validation study included: a Delphi panel process to determine key criteria to guide instrument development and identify potential items to be carried forward for testing (stage 1); a prospective cohort of consecutive admissions (n=153) to investigate item sensitivity and specificity and retain only the most suitable items (stage 2); then final consultation with the Delphi panel to ensure the final instrument was clinically amenable (stage 3). Results 23 potential items were identified following stage 1. At the completion of item sensitivity and specificity analysis and in consultation with the Delphi panel, seven items were retained in the instrument. Area under the receiver operating characteristic curve was 0.803 for these seven items in predicting when a referral was warranted. Final consultation with the Delphi panel members also resulted in the addition of an open ended (eighth) item to allow description of any infrequent, but important, reason for referral. Conclusions The RAPaRT has demonstrated substantial promise as an efficient clinically amenable instrument to assist multidisciplinary teams in emergency care settings. Further research to investigate the wider implementation of the RAPaRT is warranted.

Comparison of intensity-based cut-points for the RT3 accelerometer in youth

OBJECTIVES: To compare the classification accuracy of previously published RT3 accelerometer cut-points for youth using energy expenditure, measured via portable indirect calorimetry, as a criterion measure. DESIGN: Cross-sectional cross-validation study. METHODS: 100 children (mean age 11.2±2.8 years, 61% male) completed 12 standardized activities trials (3 sedentary, 5 lifestyle and 4 ambulatory) while wearing an RT3 accelerometer. V˙O2 was measured concurrently using the Oxycon Mobile portable calorimeter. Cut-points by Vanhelst (VH), Rowlands (RW), Chu (CH), Kavouras (KV) and the RT3 manufacturer (RT3M) were used to classify PA intensity as sedentary (SED), light (LPA), moderate (MPA) or vigorous (VPA). Classification accuracy was evaluated using the area under the Receiver Operating Characteristic curve (ROC-AUC) and weighted Kappa (κ). RESULTS: For moderate-to-vigorous PA (MVPA), VH, KV and RW exhibited excellent accuracy classification (ROC-AUC≥0.90), while the CH and RT3M exhibited good classification accuracy (ROC-AUC>0.80). Classification accuracy for LPA was fair to poor (ROC-AUC<0.76). For SED, VH exhibited excellent classification accuracy (ROC-AUC>0.90), while RW, CH, and RT3M exhibited good classification accuracy (ROC-AUC>0.80). Kappa statistics ranged from 0.67 (VH) to 0.55 (CH). CONCLUSIONS: All cut-points provided acceptable classification accuracy for SED and MVPA, but limited accuracy for LPA. On the basis of classification accuracy over all four levels of intensity, the use of the VH cut-points is recommended.

Comparison of accelerometer cut points for predicting activity intensity in youth

The absence of comparative validity studies has prevented researchers from reaching consensus regarding the application of intensity-related accelerometer cut points for children and adolescents. PURPOSE This study aimed to evaluate the classification accuracy of five sets of independently developed ActiGraph cut points using energy expenditure, measured by indirect calorimetry, as a criterion reference standard. METHODS A total of 206 participants between the ages of 5 and 15 yr completed 12 standardized activity trials. Trials consisted of sedentary activities (lying down, writing, computer game), lifestyle activities (sweeping, laundry, throw and catch, aerobics, basketball), and ambulatory activities (comfortable walk, brisk walk, brisk treadmill walk, running). During each trial, participants wore an ActiGraph GT1M, and VO 2 was measured breath-by-breath using the Oxycon Mobile portable metabolic system. Physical activity intensity was estimated using five independently developed cut points: Freedson/Trost (FT), Puyau (PU), Treuth (TR), Mattocks (MT), and Evenson (EV). Classification accuracy was evaluated via weighted κ statistics and area under the receiver operating characteristic curve (ROC-AUC). RESULTS Across all four intensity levels, the EV (κ = 0.68) and FT (κ = 0.66) cut points exhibited significantly better agreement than TR (κ = 0.62), MT (κ = 0.54), and PU (κ = 0.36). The EV and FT cut points exhibited significantly better classification accuracy for moderate-to vigorous-intensity physical activity (ROC-AUC = 0.90) than TR, PU, or MT cut points (ROC-AUC = 0.77-0.85). Only the EV cut points provided acceptable classification accuracy for all four levels of physical activity intensity and performed well among children of all ages. The widely applied sedentary cut point of 100 counts per minute exhibited excellent classification accuracy (ROC-AUC = 0.90). CONCLUSIONS On the basis of these findings, we recommend that researchers use the EV ActiGraph cut points to estimate time spent in sedentary, light-, moderate-, and vigorous-intensity activity in children and adolescents. Copyright © 2011 by the American College of Sports Medicine.

Utility of assessing nerve morphology in central cornea versus whorl area for diagnosing diabetic peripheral neuropathy

Purpose To compare small nerve fiber damage in the central cornea and whorl area in participants with diabetic peripheral neuropathy (DPN) and to examine the accuracy of evaluating these 2 anatomical sites for the diagnosis of DPN. Methods A cohort of 187 participants (107 with type 1 diabetes and 80 controls) was enrolled. The neuropathy disability score (NDS) was used for the identification of DPN. The corneal nerve fiber length at the central cornea (CNFLcenter) and whorl (CNFLwhorl) was quantified using corneal confocal microscopy and a fully automated morphometric technique and compared according to the DPN status. Receiver operating characteristic analyses were used to compare the accuracy of the 2 corneal locations for the diagnosis of DPN. Results CNFLcenter and CNFLwhorl were able to differentiate all 3 groups (diabetic participants with and without DPN and controls) (P < 0.001). There was a weak but significant linear relationship for CNFLcenter and CNFLwhorl versus NDS (P < 0.001); however, the corneal location x NDS interaction was not statistically significant (P = 0.17). The area under the receiver operating characteristic curve was similar for CNFLcenter and CNFLwhorl (0.76 and 0.77, respectively, P = 0.98). The sensitivity and specificity of the cutoff points were 0.9 and 0.5 for CNFLcenter and 0.8 and 0.6 for CNFLwhorl. Conclusions Small nerve fiber pathology is comparable at the central and whorl anatomical sites of the cornea. Quantification of CNFL from the corneal center is as accurate as CNFL quantification of the whorl area for the diagnosis of DPN.

Small nerve fiber quantification in the diagnosis of diabetic sensorimotor polyneuropathy: Comparing corneal confocal microscopy with intraepidermal nerve fiber density

OBJECTIVE Quantitative assessment of small fiber damage is key to the early diagnosis and assessment of progression or regression of diabetic sensorimotor polyneuropathy (DSPN). Intraepidermal nerve fiber density (IENFD) is the current gold standard, but corneal confocal microscopy (CCM), an in vivo ophthalmic imaging modality, has the potential to be a noninvasive and objective image biomarker for identifying small fiber damage. The purpose of this study was to determine the diagnostic performance of CCM and IENFD by using the current guidelines as the reference standard. RESEARCH DESIGN AND METHODS Eighty-nine subjects (26 control subjects and 63 patients with type 1 diabetes), with and without DSPN, underwent a detailed assessment of neuropathy, including CCM and skin biopsy. RESULTS Manual and automated corneal nerve fiber density (CNFD) (P < 0.0001), branch density (CNBD) (P < 0.0001) and length (CNFL) (P < 0.0001), and IENFD (P < 0.001) were significantly reduced in patients with diabetes with DSPN compared with control subjects. The area under the receiver operating characteristic curve for identifying DSPN was 0.82 for manual CNFD, 0.80 for automated CNFD, and 0.66 for IENFD, which did not differ significantly (P = 0.14). CONCLUSIONS This study shows comparable diagnostic efficiency between CCM and IENFD, providing further support for the clinical utility of CCM as a surrogate end point for DSPN.

Diagnostic values for skin temperature assessment to detect diabetes-related foot complications

Background Skin temperature assessment is a promising modality for early detection of diabetic foot problems, but its diagnostic value has not been studied. Our aims were to investigate the diagnostic value of different cutoff skin temperature values for detecting diabetes-related foot complications such as ulceration, infection, and Charcot foot and to determine urgency of treatment in case of diagnosed infection or a red-hot swollen foot. Materials and Methods The plantar foot surfaces of 54 patients with diabetes visiting the outpatient foot clinic were imaged with an infrared camera. Nine patients had complications requiring immediate treatment, 25 patients had complications requiring non-immediate treatment, and 20 patients had no complications requiring treatment. Average pixel temperature was calculated for six predefined spots and for the whole foot. We calculated the area under the receiver operating characteristic curve for different cutoff skin temperature values using clinical assessment as reference and defined the sensitivity and specificity for the most optimal cutoff temperature value. Mean temperature difference between feet was analyzed using the Kruskal–Wallis tests. Results The most optimal cutoff skin temperature value for detection of diabetes-related foot complications was a 2.2°C difference between contralateral spots (sensitivity, 76%; specificity, 40%). The most optimal cutoff skin temperature value for determining urgency of treatment was a 1.35°C difference between the mean temperature of the left and right foot (sensitivity, 89%; specificity, 78%). Conclusions Detection of diabetes-related foot complications based on local skin temperature assessment is hindered by low diagnostic values. Mean temperature difference between two feet may be an adequate marker for determining urgency of treatment.

Evaluation of a 7-gene genetic profile for athletic endurance phenotype in Ironman championship triathletes

Polygenic profiling has been proposed for elite endurance performance, using an additive model determining the proportion of optimal alleles in endurance athletes. To investigate this model’s utility for elite triathletes, we genotyped seven polymorphisms previously associated with an endurance polygenic profile (ACE Ins/Del, ACTN3 Arg577Ter, AMPD1 Gln12Ter, CKMM 1170bp/985+185bp, HFE His63Asp, GDF8 Lys153Arg and PPARGC1A Gly482Ser) in a cohort of 196 elite athletes who participated in the 2008 Kona Ironman championship triathlon. Mean performance time (PT) was not significantly different in individual marker analysis. Age, sex, and continent of origin had a significant influence on PT and were adjusted for. Only the AMPD1 endurance-optimal Gln allele was found to be significantly associated with an improvement in PT (model p=5.79 x 10-17, AMPD1 genotype p=0.01). Individual genotypes were combined into a total genotype score (TGS); TGS distribution ranged from 28.6 to 92.9, concordant with prior studies in endurance athletes (mean±SD: 60.75±12.95). TGS distribution was shifted toward higher TGS in the top 10% of athletes, though the mean TGS was not significantly different (p=0.164) and not significantly associated with PT even when adjusted for age, sex, and origin. Receiver operating characteristic curve analysis determined that TGS alone could not significantly predict athlete finishing time with discriminating sensitivity and specificity for three outcomes (less than median PT, less than mean PT, or in the top 10%), though models with the age, sex, continent of origin, and either TGS or AMPD1 genotype could. These results suggest three things: that more sophisticated genetic models may be necessary to accurately predict athlete finishing time in endurance events; that non-genetic factors such as training are hugely influential and should be included in genetic analyses to prevent confounding; and that large collaborations may be necessary to obtain sufficient sample sizes for powerful and complex analyses of endurance performance.