Comparison of HRT-3 glaucoma probability score and subjective stereophotograph assessment for prediction of progression in glaucoma

PURPOSE. To assess whether baseline Glaucoma Probability Score (GPS; HRT-3; Heidelberg Engineering, Dossenheim, Germany) results are predictive of progression in patients with suspected glaucoma. The GPS is a new feature of the confocal scanning laser ophthalmoscope that generates an operator-independent, three-dimensional model of the optic nerve head and gives a score for the probability that this model is consistent with glaucomatous damage. METHODS. The study included 223 patients with suspected glaucoma during an average follow-up of 63.3 months. Included subjects had a suspect optic disc appearance and/or elevated intraocular pressure, but normal visual fields. Conversion was defined as development of either repeatable abnormal visual fields or glaucomatous deterioration in the appearance of the optic disc during the study period. The association between baseline GPS and conversion was investigated by Cox regression models. RESULTS. Fifty-four (24.2%) eyes converted. In multivariate models, both higher values of GPS global and subjective stereophotograph assessment ( larger cup-disc ratio and glaucomatous grading) were predictive of conversion: adjusted hazard ratios (95% CI): 1.31 (1.15 - 1.50) per 0.1 higher global GPS, 1.34 (1.12 - 1.62) per 0.1 higher CDR, and 2.34 (1.22 - 4.47) for abnormal grading, respectively. No significant differences ( P > 0.05 for all comparisons) were found between the c-index values ( equivalent to area under ROC curve) for the multivariate models (0.732, 0.705, and 0.699, respectively). CONCLUSIONS. GPS values were predictive of conversion in our population of patients with suspected glaucoma. Further, they performed as well as subjective assessment of the optic disc. These results suggest that GPS could potentially replace stereophotograph as a tool for estimating the likelihood of conversion to glaucoma.

Evaluating raw ensembles with the continuous ranked probability score

The continuous ranked probability score (CRPS) is a frequently used scoring rule. In contrast with many other scoring rules, the CRPS evaluates cumulative distribution functions. An ensemble of forecasts can easily be converted into a piecewise constant cumulative distribution function with steps at the ensemble members. This renders the CRPS a convenient scoring rule for the evaluation of ‘raw’ ensembles, obviating the need for sophisticated ensemble model output statistics or dressing methods prior to evaluation. In this article, a relation between the CRPS score and the quantile score is established. The evaluation of ‘raw’ ensembles using the CRPS is discussed in this light. It is shown that latent in this evaluation is an interpretation of the ensemble as quantiles but with non-uniform levels. This needs to be taken into account if the ensemble is evaluated further, for example with rank histograms.

Comparison of Quantitative Imaging Devices and Subjective Optic Nerve Head Assessment by General Ophthalmologists to Differentiate Normal From Glaucomatous Eyes

Purpose: To compare the ability of Subjective assessment of optic nerve head (ONH) and retinal nerve fiber layer (RNFL) by general ophthalmologists and by a glaucoma expert with objective measurements by optical coherence tomography (Stratus OCT, Carl Zeiss Meditec Inc), confocal scanning laser ophthalmoscope (HRT III; Heidelberg Engineering, Heidelberg. Germany), and scanning laser polarimetry (GDx enhanced corneal compensation; Carl Zeiss Meditec Inc, Dublin, CA) in discriminating glaucomatous and normal eyes. Methods: Sixty-one glaucomatous and 57 normal eyes or 118 subjects Were included in the study. Three independent general ophthalmologists and I glaucoma expert evaluated ONH stereo-photographs. Receiver operating characteristic curves were constructed for each imaging technique and sensitivity at fixed specificity was estimated. Comparisons or areas under these curves (aROCs) and agreement (k) were determined between stereophoto grading and best parameter from each technique. Results: Best parameter from each technique showed larger aROC (Stratus OCT RNFL 0.92; Stratus OCT ONH vertical integrated area = 0.86; Stratus OCT macular thickness = 0.82; GDx enhanced corneal compensation = 0.91, HRT3 global cup-to-disc ratio = 0.83; HRT3 glaucoma probability score numeric area score 0.83) compared with stereophotograph grading by general ophthalmologists (0.80) in separating glaucomatous and normal eyes. Glaucoma expert stereophoto grading provided equal or larger aROC (0.92) than best parameter of each computerized imaging device. Stereophoto evaluated by a glaucoma expert showed better agreement with best parameter of each quantitative imaging technique in classifying eyes either as glaucomatous or normal compared with stereophoto grading by general ophthalmologists, The combination Of Subjective assessment of the optic disc by general ophthalmologists with RNFL objective parameters improved identification of glaucoma patients in a larger proportion than the combination of these objective parameters with Subjective assessment of the optic disc by a glaucoma expert (29.5% vs. 19.7%, respectively). Conclusions: Diagnostic ability of all imaging techniques showed better performance than subjective assessment of the ONH by general ophthalmologists, but not by It glaucoma expert, Objective RNFL measurements may provide improvement in glaucoma detection when combined with subjective assessment of the optic disc by general ophthalmologists or by a glaucoma expert.

Influence of Clinically Invisible, but Optical Coherence Tomography Detected, Optic Disc Margin Anatomy on Neuroretinal Rim Evaluation

PURPOSE. We previously demonstrated that most eyes have regionally variable extensions of Bruch's membrane (BM) inside the clinically identified disc margin (DM) that are clinically and photographically invisible. We studied the impact of these findings on DM- and BM opening (BMO)-derived neuroretinal rim parameters. METHODS. Disc stereo-photography and spectral domain optical coherence tomography (SD-OCT, 24 radial B-scans centered on the optic nerve head) were performed on 30 glaucoma patients and 10 age-matched controls. Photographs were colocalized to SD-OCT data such that the DM and BMO could be visualized in each B-scan. Three parameters were computed: (1) DM-horizontal rim width (HRW), the distance between the DM and internal limiting membrane (ILM) along the DM reference plane; (2) BMO-HRW, the distance between BMO and ILM along the BMO reference plane; and (3) BMO-minimum rim width (MRW), the minimum distance between BMO and ILM. Rank-order correlations of sectors ranked by rim width and spatial concordance measured as angular distances between equivalently ranked sectors were derived. RESULTS. The average DM position was external to BMO in all quadrants, except inferotemporally. There were significant sectoral differences among all three rim parameters. DM- HRW and BMO-HRW sector ranks were better correlated (median rho = 0.84) than DM- HRW and BMO-MRW (median rho = 0.55), or BMO-HRW and BMO-MRW (median rho = 0.60) ranks. Sectors with the narrowest BMO-MRW were infrequently the same as those with the narrowest DM-HRW or BMO-HRW. CONCLUSIONS. BMO-MRW quantifies the neuroretinal rim from a true anatomical outer border and accounts for its variable trajectory at the point of measurement. (Invest Ophthalmol Vis Sci. 2012;53:1852-1860) DOI:10.1167/iovs.11-9309

Effect of Disease Severity and Optic Disc Size on Diagnostic Accuracy of RTVue Spectral Domain Optical Coherence Tomograph in Glaucoma

PURPOSE. To evaluate the effect of disease severity and optic disc size on the diagnostic accuracies of optic nerve head (ONH), retinal nerve fiber layer (RNFL), and macular parameters with RTVue (Optovue, Fremont, CA) spectral domain optical coherence tomography (SDOCT) in glaucoma. METHODS. 110 eyes of 62 normal subjects and 193 eyes of 136 glaucoma patients from the Diagnostic Innovations in Glaucoma Study underwent ONH, RNFL, and macular imaging with RTVue. Severity of glaucoma was based on visual field index (VFI) values from standard automated perimetry. Optic disc size was based on disc area measurement using the Heidelberg Retina Tomograph II (Heidelberg Engineering, Dossenheim, Germany). Influence of disease severity and disc size on the diagnostic accuracy of RTVue was evaluated by receiver operating characteristic (ROC) and logistic regression models. RESULTS. Areas under ROC curve (AUC) of all scanning areas increased (P < 0.05) as disease severity increased. For a VFI value of 99%, indicating early damage, AUCs for rim area, average RNLI thickness, and ganglion cell complex-root mean square were 0.693, 0.799, and 0.779, respectively. For a VFI of 70%, indicating severe damage, corresponding AUCs were 0.828, 0.985, and 0.992, respectively. Optic disc size did not influence the AUCs of any of the SDOCT scanning protocols of RTVue (P > 0.05). Sensitivity of the rim area increased and specificity decreased in large optic discs. CONCLUSIONS. Diagnostic accuracies of RTVue scanning protocols for glaucoma were significantly influenced by disease severity. Sensitivity of the rim area increased in large optic discs at the expense of specificity. (Invest Ophthalmol Vis Sci. 2011;92:1290-1296) DOI:10.1167/iovs.10-5516

Sequence conservation in Plasmodium falciparum alpha-helical coiled coil domains proposed for vaccine development.

BACKGROUND: The availability of the P. falciparum genome has led to novel ways to identify potential vaccine candidates. A new approach for antigen discovery based on the bioinformatic selection of heptad repeat motifs corresponding to alpha-helical coiled coil structures yielded promising results. To elucidate the question about the relationship between the coiled coil motifs and their sequence conservation, we have assessed the extent of polymorphism in putative alpha-helical coiled coil domains in culture strains, in natural populations and in the single nucleotide polymorphism data available at PlasmoDB. METHODOLOGY/PRINCIPAL FINDINGS: 14 alpha-helical coiled coil domains were selected based on preclinical experimental evaluation. They were tested by PCR amplification and sequencing of different P. falciparum culture strains and field isolates. We found that only 3 out of 14 alpha-helical coiled coils showed point mutations and/or length polymorphisms. Based on promising immunological results 5 of these peptides were selected for further analysis. Direct sequencing of field samples from Papua New Guinea and Tanzania showed that 3 out of these 5 peptides were completely conserved. An in silico analysis of polymorphism was performed for all 166 putative alpha-helical coiled coil domains originally identified in the P. falciparum genome. We found that 82% (137/166) of these peptides were conserved, and for one peptide only the detected SNPs decreased substantially the probability score for alpha-helical coiled coil formation. More SNPs were found in arrays of almost perfect tandem repeats. In summary, the coiled coil structure prediction was rarely modified by SNPs. The analysis revealed a number of peptides with strictly conserved alpha-helical coiled coil motifs. CONCLUSION/SIGNIFICANCE: We conclude that the selection of alpha-helical coiled coil structural motifs is a valuable approach to identify potential vaccine targets showing a high degree of conservation.

On reliability analysis of multi-categorical forecasts

Reliability analysis of probabilistic forecasts, in particular through the rank histogram or Talagrand diagram, is revisited. Two shortcomings are pointed out: Firstly, a uniform rank histogram is but a necessary condition for reliability. Secondly, if the forecast is assumed to be reliable, an indication is needed how far a histogram is expected to deviate from uniformity merely due to randomness. Concerning the first shortcoming, it is suggested that forecasts be grouped or stratified along suitable criteria, and that reliability is analyzed individually for each forecast stratum. A reliable forecast should have uniform histograms for all individual forecast strata, not only for all forecasts as a whole. As to the second shortcoming, instead of the observed frequencies, the probability of the observed frequency is plotted, providing and indication of the likelihood of the result under the hypothesis that the forecast is reliable. Furthermore, a Goodness-Of-Fit statistic is discussed which is essentially the reliability term of the Ignorance score. The discussed tools are applied to medium range forecasts for 2 m-temperature anomalies at several locations and lead times. The forecasts are stratified along the expected ranked probability score. Those forecasts which feature a high expected score turn out to be particularly unreliable.

How do I know if my forecasts are better? Using benchmarks in Hydrological ensemble prediction

The skill of a forecast can be assessed by comparing the relative proximity of both the forecast and a benchmark to the observations. Example benchmarks include climatology or a naïve forecast. Hydrological ensemble prediction systems (HEPS) are currently transforming the hydrological forecasting environment but in this new field there is little information to guide researchers and operational forecasters on how benchmarks can be best used to evaluate their probabilistic forecasts. In this study, it is identified that the forecast skill calculated can vary depending on the benchmark selected and that the selection of a benchmark for determining forecasting system skill is sensitive to a number of hydrological and system factors. A benchmark intercomparison experiment is then undertaken using the continuous ranked probability score (CRPS), a reference forecasting system and a suite of 23 different methods to derive benchmarks. The benchmarks are assessed within the operational set-up of the European Flood Awareness System (EFAS) to determine those that are ‘toughest to beat’ and so give the most robust discrimination of forecast skill, particularly for the spatial average fields that EFAS relies upon. Evaluating against an observed discharge proxy the benchmark that has most utility for EFAS and avoids the most naïve skill across different hydrological situations is found to be meteorological persistency. This benchmark uses the latest meteorological observations of precipitation and temperature to drive the hydrological model. Hydrological long term average benchmarks, which are currently used in EFAS, are very easily beaten by the forecasting system and the use of these produces much naïve skill. When decomposed into seasons, the advanced meteorological benchmarks, which make use of meteorological observations from the past 20 years at the same calendar date, have the most skill discrimination. They are also good at discriminating skill in low flows and for all catchment sizes. Simpler meteorological benchmarks are particularly useful for high flows. Recommendations for EFAS are to move to routine use of meteorological persistency, an advanced meteorological benchmark and a simple meteorological benchmark in order to provide a robust evaluation of forecast skill. This work provides the first comprehensive evidence on how benchmarks can be used in evaluation of skill in probabilistic hydrological forecasts and which benchmarks are most useful for skill discrimination and avoidance of naïve skill in a large scale HEPS. It is recommended that all HEPS use the evidence and methodology provided here to evaluate which benchmarks to employ; so forecasters can have trust in their skill evaluation and will have confidence that their forecasts are indeed better.

Towards improving the framework for probabilistic forecast evaluation

The evaluation of forecast performance plays a central role both in the interpretation and use of forecast systems and in their development. Different evaluation measures (scores) are available, often quantifying different characteristics of forecast performance. The properties of several proper scores for probabilistic forecast evaluation are contrasted and then used to interpret decadal probability hindcasts of global mean temperature. The Continuous Ranked Probability Score (CRPS), Proper Linear (PL) score, and IJ Good’s logarithmic score (also referred to as Ignorance) are compared; although information from all three may be useful, the logarithmic score has an immediate interpretation and is not insensitive to forecast busts. Neither CRPS nor PL is local; this is shown to produce counter intuitive evaluations by CRPS. Benchmark forecasts from empirical models like Dynamic Climatology place the scores in context. Comparing scores for forecast systems based on physical models (in this case HadCM3, from the CMIP5 decadal archive) against such benchmarks is more informative than internal comparison systems based on similar physical simulation models with each other. It is shown that a forecast system based on HadCM3 out performs Dynamic Climatology in decadal global mean temperature hindcasts; Dynamic Climatology previously outperformed a forecast system based upon HadGEM2 and reasons for these results are suggested. Forecasts of aggregate data (5-year means of global mean temperature) are, of course, narrower than forecasts of annual averages due to the suppression of variance; while the average “distance” between the forecasts and a target may be expected to decrease, little if any discernible improvement in probabilistic skill is achieved.

Bayesian Analysis for Nonlinear Regression Model Under Skewed Errors, with Application in Growth Curves

We have considered a Bayesian approach for the nonlinear regression model by replacing the normal distribution on the error term by some skewed distributions, which account for both skewness and heavy tails or skewness alone. The type of data considered in this paper concerns repeated measurements taken in time on a set of individuals. Such multiple observations on the same individual generally produce serially correlated outcomes. Thus, additionally, our model does allow for a correlation between observations made from the same individual. We have illustrated the procedure using a data set to study the growth curves of a clinic measurement of a group of pregnant women from an obstetrics clinic in Santiago, Chile. Parameter estimation and prediction were carried out using appropriate posterior simulation schemes based in Markov Chain Monte Carlo methods. Besides the deviance information criterion (DIC) and the conditional predictive ordinate (CPO), we suggest the use of proper scoring rules based on the posterior predictive distribution for comparing models. For our data set, all these criteria chose the skew-t model as the best model for the errors. These DIC and CPO criteria are also validated, for the model proposed here, through a simulation study. As a conclusion of this study, the DIC criterion is not trustful for this kind of complex model.

Rainfall spatial predictions: a two-part model and its assessment

Spatial prediction of hourly rainfall via radar calibration is addressed. The change of support problem (COSP), arising when the spatial supports of different data sources do not coincide, is faced in a non-Gaussian setting; in fact, hourly rainfall in Emilia-Romagna region, in Italy, is characterized by abundance of zero values and right-skeweness of the distribution of positive amounts. Rain gauge direct measurements on sparsely distributed locations and hourly cumulated radar grids are provided by the ARPA-SIMC Emilia-Romagna. We propose a three-stage Bayesian hierarchical model for radar calibration, exploiting rain gauges as reference measure. Rain probability and amounts are modeled via linear relationships with radar in the log scale; spatial correlated Gaussian effects capture the residual information. We employ a probit link for rainfall probability and Gamma distribution for rainfall positive amounts; the two steps are joined via a two-part semicontinuous model. Three model specifications differently addressing COSP are presented; in particular, a stochastic weighting of all radar pixels, driven by a latent Gaussian process defined on the grid, is employed. Estimation is performed via MCMC procedures implemented in C, linked to R software. Communication and evaluation of probabilistic, point and interval predictions is investigated. A non-randomized PIT histogram is proposed for correctly assessing calibration and coverage of two-part semicontinuous models. Predictions obtained with the different model specifications are evaluated via graphical tools (Reliability Plot, Sharpness Histogram, PIT Histogram, Brier Score Plot and Quantile Decomposition Plot), proper scoring rules (Brier Score, Continuous Rank Probability Score) and consistent scoring functions (Root Mean Square Error and Mean Absolute Error addressing the predictive mean and median, respectively). Calibration is reached and the inclusion of neighbouring information slightly improves predictions. All specifications outperform a benchmark model with incorrelated effects, confirming the relevance of spatial correlation for modeling rainfall probability and accumulation.

Adjusting fracture probability by trabecular bone score.

The aim of the present study was to determine the impact of trabecular bone score on the probability of fracture above that provided by the clinical risk factors utilized in FRAX. We performed a retrospective cohort study of 33,352 women aged 40-99 years from the province of Manitoba, Canada, with baseline measurements of lumbar spine trabecular bone score (TBS) and FRAX risk variables. The analysis was cohort-specific rather than based on the Canadian version of FRAX. The associations between trabecular bone score, the FRAX risk factors and the risk of fracture or death were examined using an extension of the Poisson regression model and used to calculate 10-year probabilities of fracture with and without TBS and to derive an algorithm to adjust fracture probability to take account of the independent contribution of TBS to fracture and mortality risk. During a mean follow-up of 4.7 years, 1754 women died and 1639 sustained one or more major osteoporotic fractures excluding hip fracture and 306 women sustained one or more hip fracture. When fully adjusted for FRAX risk variables, TBS remained a statistically significant predictor of major osteoporotic fractures excluding hip fracture (HR/SD 1.18, 95 % CI 1.12-1.24), death (HR/SD 1.20, 95 % CI 1.14-1.26) and hip fracture (HR/SD 1.23, 95 % CI 1.09-1.38). Models adjusting major osteoporotic fracture and hip fracture probability were derived, accounting for age and trabecular bone score with death considered as a competing event. Lumbar spine texture analysis using TBS is a risk factor for osteoporotic fracture and a risk factor for death. The predictive ability of TBS is independent of FRAX clinical risk factors and femoral neck BMD. Adjustment of fracture probability to take account of the independent contribution of TBS to fracture and mortality risk requires validation in independent cohorts.

Estimating reliability and resolution of probability forecasts through decomposition of the empirical score

References (20)Cited By (1)Export CitationAboutAbstract Proper scoring rules provide a useful means to evaluate probabilistic forecasts. Independent from scoring rules, it has been argued that reliability and resolution are desirable forecast attributes. The mathematical expectation value of the score allows for a decomposition into reliability and resolution related terms, demonstrating a relationship between scoring rules and reliability/resolution. A similar decomposition holds for the empirical (i.e. sample average) score over an archive of forecast–observation pairs. This empirical decomposition though provides a too optimistic estimate of the potential score (i.e. the optimum score which could be obtained through recalibration), showing that a forecast assessment based solely on the empirical resolution and reliability terms will be misleading. The differences between the theoretical and empirical decomposition are investigated, and specific recommendations are given how to obtain better estimators of reliability and resolution in the case of the Brier and Ignorance scoring rule.

Use of the probability of stone formation (PSF) score to assess stone forming risk and treatment response in a cohort of Brazilian stone formers

Introduction: The aim was to confirm that PSF (probability of stone formation) changed appropriately following medical therapy on recurrent stone formers.Materials and Methods: Data were collected on 26 Brazilian stone-formers. A baseline 24-hour urine collection was performed prior to treatment. Details of the medical treatment initiated for stone-disease were recorded. A PSF calculation was performed on the 24 hour urine sample using the 7 urinary parameters required: voided volume, oxalate, calcium, urate, pH, citrate and magnesium. A repeat 24-hour urine sample was performed for PSF calculation after treatment. Comparison was made between the PSF scores before and during treatment.Results: At baseline, 20 of the 26 patients (77%) had a high PSF score (> 0.5). Of the 26 patients, 17 (65%) showed an overall reduction in their PSF profiles with a medical treatment regimen. Eleven patients (42%) changed from a high risk (PSF > 0.5) to a low risk (PSF < 0.5) and 6 patients reduced their risk score but did not change risk category. Six (23%) patients remained in a high risk category (> 0.5) during both assessments.Conclusions: The PSF score reduced following medical treatment in the majority of patients in this cohort.

Impact of Atypical Retardation Patterns on Detection of Glaucoma Progression using the GDx with Variable Corneal Compensation

PURPOSE: To evaluate the impact of atypical retardation patterns (ARP) on detection of progressive retinal nerve fiber layer (RNFL) loss using scanning laser polarimetry with variable corneal compensation (VCC). DESIGN: Observational cohort study. METHODS: The study included 377 eyes of 221 patients with a median follow-up of 4.0 years. Images were obtained annually with the GDx VCC (Carl Zeiss Med, itec Inc, Dublin, California, USA), along with optic disc stereophotographs and standard automated perimetry (SAP) visual fields. Progression was determined by the Guided Progression Analysis software for SAP and by masked assessment of stereophotographs by expert graders. The typical scan score (TSS) was used to quantify the presence of ARPs on GDx VCC images. Random coefficients models were used to evaluate the relationship between ARP and RNFL thickness measurements over time. RESULTS: Thirty-eight eyes (10%) showed progression over time on visual fields, stereophotographs, or both. Changes in TSS scores from baseline were significantly associated with changes in RNFL thickness measurements in both progressing and nonprogressing eyes. Each I unit increase in TSS score was associated with a 0.19-mu m decrease in RNFL thickness measurement (P < .001) over time. CONCLUSIONS: ARPs had a significant effect on detection of progressive RNFL loss with the GDx VCC. Eyes with large amounts of atypical patterns, great fluctuations on these patterns over time, or both may show changes in measurements that can appear falsely as glaucomatous progression or can mask true changes in the RNFL. (Am J Ophthalmol 2009;148:155-163. (C) 2009 by Elsevier Inc. All rights reserved.)