The influence of experience and training in a group of novice observers: a jackknife alternative free-response receiver operating characteristic analysis

Purpose - The study evaluates the pre- and post-training lesion localisation ability of a group of novice observers. Parallels are drawn with the performance of inexperienced radiographers taking part in preliminary clinical evaluation (PCE) and ‘red-dot’ systems, operating within radiography practice. Materials and methods - Thirty-four novice observers searched 92 images for simulated lesions. Pre-training and post-training evaluations were completed following the free-response the receiver operating characteristic (FROC) method. Training consisted of observer performance methodology, the characteristics of the simulated lesions and information on lesion frequency. Jackknife alternative FROC (JAFROC) and highest rating inferred ROC analyses were performed to evaluate performance difference on lesion-based and case-based decisions. The significance level of the test was set at 0.05 to control the probability of Type I error. Results - JAFROC analysis (F(3,33) = 26.34, p < 0.0001) and highest-rating inferred ROC analysis (F(3,33) = 10.65, p = 0.0026) revealed a statistically significant difference in lesion detection performance. The JAFROC figure-of-merit was 0.563 (95% CI 0.512,0.614) pre-training and 0.677 (95% CI 0.639,0.715) post-training. Highest rating inferred ROC figure-of-merit was 0.728 (95% CI 0.701,0.755) pre-training and 0.772 (95% CI 0.750,0.793) post-training. Conclusions - This study has demonstrated that novice observer performance can improve significantly. This study design may have relevance in the assessment of inexperienced radiographers taking part in PCE or commenting scheme for trauma.

Avaliação do desempenho de indicadores com base na metodologia ROC (Receiver Operating Characteristic)

Dissertação de mestrado em Bioinformática

Receiver operating characteristic (ROC) curve for classification of18F-NaF uptake on PET/CT

Abstract Objective: To assess the cutoff values established by ROC curves to classify18F-NaF uptake as normal or malignant. Materials and Methods: PET/CT images were acquired 1 hour after administration of 185 MBq of18F-NaF. Volumes of interest (VOIs) were drawn on three regions of the skeleton as follows: proximal right humerus diaphysis (HD), proximal right femoral diaphysis (FD) and first vertebral body (VB1), in a total of 254 patients, totalling 762 VOIs. The uptake in the VOIs was classified as normal or malignant on the basis of the radiopharmaceutical distribution pattern and of the CT images. A total of 675 volumes were classified as normal and 52 were classified as malignant. Thirty-five VOIs classified as indeterminate or nonmalignant lesions were excluded from analysis. The standardized uptake value (SUV) measured on the VOIs were plotted on an ROC curve for each one of the three regions. The area under the ROC (AUC) as well as the best cutoff SUVs to classify the VOIs were calculated. The best cutoff values were established as the ones with higher result of the sum of sensitivity and specificity. Results: The AUCs were 0.933, 0.889 and 0.975 for UD, FD and VB1, respectively. The best SUV cutoffs were 9.0 (sensitivity: 73%; specificity: 99%), 8.4 (sensitivity: 79%; specificity: 94%) and 21.0 (sensitivity: 93%; specificity: 95%) for UD, FD and VB1, respectively. Conclusion: The best cutoff value varies according to bone region of analysis and it is not possible to establish one value for the whole body.

A nonparametric method of comparing the partial areas under two correlated receiver operating characteristic curves

A non-parametric method was developed and tested to compare the partial areas under two correlated Receiver Operating Characteristic curves. Based on the theory of generalized U-statistics the mathematical formulas have been derived for computing ROC area, and the variance and covariance between the portions of two ROC curves. A practical SAS application also has been developed to facilitate the calculations. The accuracy of the non-parametric method was evaluated by comparing it to other methods. By applying our method to the data from a published ROC analysis of CT image, our results are very close to theirs. A hypothetical example was used to demonstrate the effects of two crossed ROC curves. The two ROC areas are the same. However each portion of the area between two ROC curves were found to be significantly different by the partial ROC curve analysis. For computation of ROC curves with large scales, such as a logistic regression model, we applied our method to the breast cancer study with Medicare claims data. It yielded the same ROC area computation as the SAS Logistic procedure. Our method also provides an alternative to the global summary of ROC area comparison by directly comparing the true-positive rates for two regression models and by determining the range of false-positive values where the models differ. ^

Precision-recall operating characteristic (P-ROC) curves in imprecise environments

Traditionally, machine learning algorithms have been evaluated in applications where assumptions can be reliably made about class priors and/or misclassification costs. In this paper, we consider the case of imprecise environments, where little may be known about these factors and they may well vary significantly when the system is applied. Specifically, the use of precision-recall analysis is investigated and compared to the more well known performance measures such as error-rate and the receiver operating characteristic (ROC). We argue that while ROC analysis is invariant to variations in class priors, this invariance in fact hides an important factor of the evaluation in imprecise environments. Therefore, we develop a generalised precision-recall analysis methodology in which variation due to prior class probabilities is incorporated into a multi-way analysis of variance (ANOVA). The increased sensitivity and reliability of this approach is demonstrated in a remote sensing application.

Evaluation of non-contact sampling and detection of explosives using receiver operating characteristic curves

The growing need for fast sampling of explosives in high throughput areas has increased the demand for improved technology for the trace detection of illicit compounds. Detection of the volatiles associated with the presence of the illicit compounds offer a different approach for sensitive trace detection of these compounds without increasing the false positive alarm rate. This study evaluated the performance of non-contact sampling and detection systems using statistical analysis through the construction of Receiver Operating Characteristic (ROC) curves in real-world scenarios for the detection of volatiles in the headspace of smokeless powder, used as the model system for generalizing explosives detection. A novel sorbent coated disk coined planar solid phase microextraction (PSPME) was previously used for rapid, non-contact sampling of the headspace containers. The limits of detection for the PSPME coupled to IMS detection was determined to be 0.5-24 ng for vapor sampling of volatile chemical compounds associated with illicit compounds and demonstrated an extraction efficiency of three times greater than other commercially available substrates, retaining >50% of the analyte after 30 minutes sampling of an analyte spike in comparison to a non-detect for the unmodified filters. Both static and dynamic PSPME sampling was used coupled with two ion mobility spectrometer (IMS) detection systems in which 10-500 mg quantities of smokeless powders were detected within 5-10 minutes of static sampling and 1 minute of dynamic sampling time in 1-45 L closed systems, resulting in faster sampling and analysis times in comparison to conventional solid phase microextraction-gas chromatography-mass spectrometry (SPME-GC-MS) analysis. Similar real-world scenarios were sampled in low and high clutter environments with zero false positive rates. Excellent PSPME-IMS detection of the volatile analytes were visualized from the ROC curves, resulting with areas under the curves (AUC) of 0.85-1.0 and 0.81-1.0 for portable and bench-top IMS systems, respectively. Construction of ROC curves were also developed for SPME-GC-MS resulting with AUC of 0.95-1.0, comparable with PSPME-IMS detection. The PSPME-IMS technique provides less false positive results for non-contact vapor sampling, cutting the cost and providing an effective sampling and detection needed in high-throughput scenarios, resulting in similar performance in comparison to well-established techniques with the added advantage of fast detection in the field.

The Steady-State and Transient Characteristic Analysis of Thermal Fixing a Photorefractive Hologram Grating in Lithium Niobate

The transient interaction between a refraction index grating and light beams during simultaneous writing and thermal fixing of a photorefractive hologram is investigated. With a diffusion- and photovoltaic-dominated carrier transport mechanism and carrier thermal activation (temperature dependent) considered in Fe:LiNbO3 crystal, from the standpoint of field-material coupling, the theoretical thermal fixing time and the space-charge field buildup, spatial distribution, and temperature dependence are given numerically by combining the band transport model with mobile ions with the coupled-wave equation

Recursive hierarchic segmentation analysis of bone mineral density changes on digital panoramic images

Coordenação de Aperfeiçoamento de Pessoal de Nível Superior (CAPES)

Lesion detection performance: comparative analysis of low-dose CT data of the chest on two hybrid imaging systems

Incidental findings on low-dose CT images obtained during hybrid imaging are an increasing phenomenon as CT technology advances. Understanding the diagnostic value of incidental findings along with the technical limitations is important when reporting image results and recommending follow-up, which may result in an additional radiation dose from further diagnostic imaging and an increase in patient anxiety. This study assessed lesions incidentally detected on CT images acquired for attenuation correction on two SPECT/CT systems. Methods: An anthropomorphic chest phantom containing simulated lesions of varying size and density was imaged on an Infinia Hawkeye 4 and a Symbia T6 using the low-dose CT settings applied for attenuation correction acquisitions in myocardial perfusion imaging. Twenty-two interpreters assessed 46 images from each SPECT/CT system (15 normal images and 31 abnormal images; 41 lesions). Data were evaluated using a jackknife alternative free-response receiver-operating-characteristic analysis (JAFROC). Results: JAFROC analysis showed a significant difference (P < 0.0001) in lesion detection, with the figures of merit being 0.599 (95% confidence interval, 0.568, 0.631) and 0.810 (95% confidence interval, 0.781, 0.839) for the Infinia Hawkeye 4 and Symbia T6, respectively. Lesion detection on the Infinia Hawkeye 4 was generally limited to larger, higher-density lesions. The Symbia T6 allowed improved detection rates for midsized lesions and some lower-density lesions. However, interpreters struggled to detect small (5 mm) lesions on both image sets, irrespective of density. Conclusion: Lesion detection is more reliable on low-dose CT images from the Symbia T6 than from the Infinia Hawkeye 4. This phantom-based study gives an indication of potential lesion detection in the clinical context as shown by two commonly used SPECT/CT systems, which may assist the clinician in determining whether further diagnostic imaging is justified.

Evaluation Of Red Cell And Reticulocyte Parameters As Indicative Of Iron Deficiency In Patients With Anemia Of Chronic Disease.

The purpose of this study was to evaluate the effectiveness of mature red cell and reticulocyte parameters under three conditions: iron deficiency anemia, anemia of chronic disease, and anemia of chronic disease associated with absolute iron deficiency. Peripheral blood cells from 117 adult patients with anemia were classified according to iron status, and inflammatory activity, and the results of a hemoglobinopathy investigation as: iron deficiency anemia (n=42), anemia of chronic disease (n=28), anemia of chronic disease associated with iron deficiency anemia (n=22), and heterozygous β thalassemia (n=25). The percentage of microcytic red cells, hypochromic red cells, and levels of hemoglobin content in both reticulocytes and mature red cells were determined. Receiver operating characteristic analysis was used to evaluate the accuracy of the parameters in differentiating between the different types of anemia. There was no significant difference between the iron deficient group and anemia of chronic disease associated with absolute iron deficiency in respect to any parameter. The percentage of hypochromic red cells was the best parameter to discriminate anemia of chronic disease with and without absolute iron deficiency (area under curve=0.785; 95% confidence interval: 0.661-0.909, with sensitivity of 72.7%, and specificity of 70.4%; cut-off value 1.8%). The formula microcytic red cells minus hypochromic red cells was very accurate in differentiating iron deficiency anemia and heterozygous β thalassemia (area under curve=0.977; 95% confidence interval: 0.950-1.005; with sensitivity of 96.2%, and specificity of 92.7%; cut-off value 13.8). The indices related to red cells and reticulocytes have a moderate performance in identifying absolute iron deficiency in patients with anemia of chronic disease.

Limitations of the Mini-Mental State Examination for screening dementia in a community with low socioeconomic status

The Mini-Mental State Examination (MMSE) is the most widely used instrument for the screening of cognitive impairment worldwide, but its ability to produce valid estimates of dementia in populations of low socioeconomic status and minimal literacy skills has not been adequately established. The authors investigated the psychometric properties of the MMSE in a community-based sample of older Brazilians. Cross-sectional one-phase population-based study of all residents of pre-defined areas of the city of Sao Paulo, aged 65 years or over. The Brazilian version of the MMSE was compared with DSM-IV diagnosis of dementia assessed with a harmonized one-phase procedure developed by the 10/66 Dementia Research Group. Analyses were performed with 1,933 participants of the SPAH study. Receiver operating characteristic analysis showed that the MMSE cut-point of 14/15 was associated with 78.7% sensitivity and 77.8% specificity for the diagnosis of dementia amongst participants with no formal education, and the cut-point 17/18 with 91.9% sensitivity and 89.5% specificity for those with at least 1 year of formal education (areas under the curves 0.87 and 0.94, respectively; P = 0.03). Even with these best fitting cut-points, the MMSE estimate of the prevalence of dementia was four times higher than determined by the DSM-IV criteria. Education, age, sex and income influenced MMSE scores, independently of dementia caseness. The MMSE is an adequate tool for screening dementia in older adults with minimum literacy skills, but misclassification is unacceptably high for older adults who are illiterate, which has serious consequences for research and clinical practice in low and middle income countries, where the proportion of illiteracy among older adults is high.

Comparison of three antigenic extracts of Eurotium amstelodami in serological diagnosis of farmer's lung disease.

In France and Finland, farmer's lung disease (FLD), a hypersensitivity pneumonitis common in agricultural areas, is mainly caused by Eurotium species. The presence of antibodies in patients' serum is an important criterion for diagnosis. Our study aimed to improve the serological diagnosis of FLD by using common fungal particles that pollute the farm environment as antigens. Fungal particles of the Eurotium species were observed in handled hay. A strain of Eurotium amstelodami was grown in vitro using selected culture media; and antigen extracts from sexual (ascospores), asexual (conidia), and vegetative (hyphae) forms were made. Antigens were tested by enzyme-linked immunosorbent assay (ELISA), which was used to test for immunoglobulin G antibodies from the sera of 17 FLD patients, 40 healthy exposed farmers, and 20 nonexposed controls. The antigens were compared by receiver operating characteristic analysis, and a threshold was then established. The ascospores contained in asci enclosed within cleistothecia were present in 38% of the hay blades observed; conidial heads of aspergillus were less prevalent. The same protocol was followed to make the three antigen extracts. A comparison of the results for FLD patients and exposed controls showed the area under the curve to be 0.850 for the ascospore antigen, 0.731 for the conidia, and 0.690 for the hyphae. The cutoffs that we determined, with the standard deviation for measures being taken into account, showed 67% for sensitivity and 92% for specificity with the ascospore antigen. In conclusion, the serological diagnosis of FLD by ELISA was improved by the adjunction of ascospore antigen.

Symptoms Have Modest Accuracy in Detecting Endoscopic and Histologic Remission in Adults With Eosinophilic Esophagitis.

BACKGROUND & AIMS: It is not clear whether symptoms alone can be used to estimate the biologic activity of eosinophilic esophagitis (EoE). We aimed to evaluate whether symptoms can be used to identify patients with endoscopic and histologic features of remission. METHODS: Between April 2011 and June 2014, we performed a prospective, observational study and recruited 269 consecutive adults with EoE (67% male; median age, 39 years old) in Switzerland and the United States. Patients first completed the validated symptom-based EoE activity index patient-reported outcome instrument and then underwent esophagogastroduodenoscopy with esophageal biopsy collection. Endoscopic and histologic findings were evaluated with a validated grading system and standardized instrument, respectively. Clinical remission was defined as symptom score <20 (range, 0-100); histologic remission was defined as a peak count of <20 eosinophils/mm(2) in a high-power field (corresponds to approximately <5 eosinophils/median high-power field); and endoscopic remission as absence of white exudates, moderate or severe rings, strictures, or combination of furrows and edema. We used receiver operating characteristic analysis to determine the best symptom score cutoff values for detection of remission. RESULTS: Of the study subjects, 111 were in clinical remission (41.3%), 79 were in endoscopic remission (29.7%), and 75 were in histologic remission (27.9%). When the symptom score was used as a continuous variable, patients in endoscopic, histologic, and combined (endoscopic and histologic remission) remission were detected with area under the curve values of 0.67, 0.60, and 0.67, respectively. A symptom score of 20 identified patients in endoscopic remission with 65.1% accuracy and histologic remission with 62.1% accuracy; a symptom score of 15 identified patients with both types of remission with 67.7% accuracy. CONCLUSIONS: In patients with EoE, endoscopic or histologic remission can be identified with only modest accuracy based on symptoms alone. At any given time, physicians cannot rely on lack of symptoms to make assumptions about lack of biologic disease activity in adults with EoE. ClinicalTrials.gov, Number: NCT00939263.

Soft Computing Approach for Optimization of siRNA Efficiency Prediction for Post-transcriptional Gene Silencing

Post-transcriptional gene silencing by RNA interference is mediated by small interfering RNA called siRNA. This gene silencing mechanism can be exploited therapeutically to a wide variety of disease-associated targets, especially in AIDS, neurodegenerative diseases, cholesterol and cancer on mice with the hope of extending these approaches to treat humans. Over the recent past, a significant amount of work has been undertaken to understand the gene silencing mediated by exogenous siRNA. The design of efficient exogenous siRNA sequences is challenging because of many issues related to siRNA. While designing efficient siRNA, target mRNAs must be selected such that their corresponding siRNAs are likely to be efficient against that target and unlikely to accidentally silence other transcripts due to sequence similarity. So before doing gene silencing by siRNAs, it is essential to analyze their off-target effects in addition to their inhibition efficiency against a particular target. Hence designing exogenous siRNA with good knock-down efficiency and target specificity is an area of concern to be addressed. Some methods have been developed already by considering both inhibition efficiency and off-target possibility of siRNA against agene. Out of these methods, only a few have achieved good inhibition efficiency, specificity and sensitivity. The main focus of this thesis is to develop computational methods to optimize the efficiency of siRNA in terms of “inhibition capacity and off-target possibility” against target mRNAs with improved efficacy, which may be useful in the area of gene silencing and drug design for tumor development. This study aims to investigate the currently available siRNA prediction approaches and to devise a better computational approach to tackle the problem of siRNA efficacy by inhibition capacity and off-target possibility. The strength and limitations of the available approaches are investigated and taken into consideration for making improved solution. Thus the approaches proposed in this study extend some of the good scoring previous state of the art techniques by incorporating machine learning and statistical approaches and thermodynamic features like whole stacking energy to improve the prediction accuracy, inhibition efficiency, sensitivity and specificity. Here, we propose one Support Vector Machine (SVM) model, and two Artificial Neural Network (ANN) models for siRNA efficiency prediction. In SVM model, the classification property is used to classify whether the siRNA is efficient or inefficient in silencing a target gene. The first ANNmodel, named siRNA Designer, is used for optimizing the inhibition efficiency of siRNA against target genes. The second ANN model, named Optimized siRNA Designer, OpsiD, produces efficient siRNAs with high inhibition efficiency to degrade target genes with improved sensitivity-specificity, and identifies the off-target knockdown possibility of siRNA against non-target genes. The models are trained and tested against a large data set of siRNA sequences. The validations are conducted using Pearson Correlation Coefficient, Mathews Correlation Coefficient, Receiver Operating Characteristic analysis, Accuracy of prediction, Sensitivity and Specificity. It is found that the approach, OpsiD, is capable of predicting the inhibition capacity of siRNA against a target mRNA with improved results over the state of the art techniques. Also we are able to understand the influence of whole stacking energy on efficiency of siRNA. The model is further improved by including the ability to identify the “off-target possibility” of predicted siRNA on non-target genes. Thus the proposed model, OpsiD, can predict optimized siRNA by considering both “inhibition efficiency on target genes and off-target possibility on non-target genes”, with improved inhibition efficiency, specificity and sensitivity. Since we have taken efforts to optimize the siRNA efficacy in terms of “inhibition efficiency and offtarget possibility”, we hope that the risk of “off-target effect” while doing gene silencing in various bioinformatics fields can be overcome to a great extent. These findings may provide new insights into cancer diagnosis, prognosis and therapy by gene silencing. The approach may be found useful for designing exogenous siRNA for therapeutic applications and gene silencing techniques in different areas of bioinformatics.