Evaluating raw ensembles with the continuous ranked probability score

The continuous ranked probability score (CRPS) is a frequently used scoring rule. In contrast with many other scoring rules, the CRPS evaluates cumulative distribution functions. An ensemble of forecasts can easily be converted into a piecewise constant cumulative distribution function with steps at the ensemble members. This renders the CRPS a convenient scoring rule for the evaluation of ‘raw’ ensembles, obviating the need for sophisticated ensemble model output statistics or dressing methods prior to evaluation. In this article, a relation between the CRPS score and the quantile score is established. The evaluation of ‘raw’ ensembles using the CRPS is discussed in this light. It is shown that latent in this evaluation is an interpretation of the ensemble as quantiles but with non-uniform levels. This needs to be taken into account if the ensemble is evaluated further, for example with rank histograms.

Influence of Clinically Invisible, but Optical Coherence Tomography Detected, Optic Disc Margin Anatomy on Neuroretinal Rim Evaluation

PURPOSE. We previously demonstrated that most eyes have regionally variable extensions of Bruch's membrane (BM) inside the clinically identified disc margin (DM) that are clinically and photographically invisible. We studied the impact of these findings on DM- and BM opening (BMO)-derived neuroretinal rim parameters. METHODS. Disc stereo-photography and spectral domain optical coherence tomography (SD-OCT, 24 radial B-scans centered on the optic nerve head) were performed on 30 glaucoma patients and 10 age-matched controls. Photographs were colocalized to SD-OCT data such that the DM and BMO could be visualized in each B-scan. Three parameters were computed: (1) DM-horizontal rim width (HRW), the distance between the DM and internal limiting membrane (ILM) along the DM reference plane; (2) BMO-HRW, the distance between BMO and ILM along the BMO reference plane; and (3) BMO-minimum rim width (MRW), the minimum distance between BMO and ILM. Rank-order correlations of sectors ranked by rim width and spatial concordance measured as angular distances between equivalently ranked sectors were derived. RESULTS. The average DM position was external to BMO in all quadrants, except inferotemporally. There were significant sectoral differences among all three rim parameters. DM- HRW and BMO-HRW sector ranks were better correlated (median rho = 0.84) than DM- HRW and BMO-MRW (median rho = 0.55), or BMO-HRW and BMO-MRW (median rho = 0.60) ranks. Sectors with the narrowest BMO-MRW were infrequently the same as those with the narrowest DM-HRW or BMO-HRW. CONCLUSIONS. BMO-MRW quantifies the neuroretinal rim from a true anatomical outer border and accounts for its variable trajectory at the point of measurement. (Invest Ophthalmol Vis Sci. 2012;53:1852-1860) DOI:10.1167/iovs.11-9309

Developing the specifications of an open angle glaucoma screening intervention in the United Kingdom: a Delphi approach.

Abstract
BACKGROUND:
Glaucoma is a leading cause of blindness. Early detection is advocated but there is insufficient evidence from randomized controlled trials (RCTs) to inform health policy on population screening. Primarily, there is no agreed screening intervention. For a screening programme, agreement is required on the screening tests to be used, either individually or in combination, the person to deliver the test and the location where testing should take place. This study aimed to use ophthalmologists (who were experienced glaucoma subspecialists), optometrists, ophthalmic nurses and patients to develop a reduced set of potential screening tests and testing arrangements that could then be explored in depth in a further study of their feasibility for evaluation in a glaucoma screening RCT.
METHODS:
A two-round Delphi survey involving 38 participants was conducted. Materials were developed from a prior evidence synthesis. For round one, after some initial priming questions in four domains, specialists were asked to nominate three screening interventions, the intervention being a combination of the four domains; target population, (age and higher risk groups), site, screening test and test operator (provider). More than 250 screening interventions were identified. For round two, responses were condensed into 72 interventions and each was rated by participants on a 0-10 scale in terms of feasibility.
RESULTS:
Using a cut-off of a median rating of feasibility of =5.5 as evidence of agreement of intervention feasibility, six interventions were identified from round 2. These were initiating screening at age 50, with a combination of two or three screening tests (varying combinations of tonometry/measures of visual function/optic nerve damage) organized in a community setting with an ophthalmic trained technical assistant delivering the tests. An alternative intervention was a 'glaucoma risk score' ascertained by questionnaire. The advisory panel recommended that further exploration of the feasibility of screening higher risk populations and detailed specification of the screening tests was required.
CONCLUSIONS:
With systematic use of expert opinions, a shortlist of potential screening interventions was identified. Views of users, service providers and cost-effectiveness modeling are now required to identify a feasible intervention to evaluate in a future glaucoma screening trial.

On reliability analysis of multi-categorical forecasts

Reliability analysis of probabilistic forecasts, in particular through the rank histogram or Talagrand diagram, is revisited. Two shortcomings are pointed out: Firstly, a uniform rank histogram is but a necessary condition for reliability. Secondly, if the forecast is assumed to be reliable, an indication is needed how far a histogram is expected to deviate from uniformity merely due to randomness. Concerning the first shortcoming, it is suggested that forecasts be grouped or stratified along suitable criteria, and that reliability is analyzed individually for each forecast stratum. A reliable forecast should have uniform histograms for all individual forecast strata, not only for all forecasts as a whole. As to the second shortcoming, instead of the observed frequencies, the probability of the observed frequency is plotted, providing and indication of the likelihood of the result under the hypothesis that the forecast is reliable. Furthermore, a Goodness-Of-Fit statistic is discussed which is essentially the reliability term of the Ignorance score. The discussed tools are applied to medium range forecasts for 2 m-temperature anomalies at several locations and lead times. The forecasts are stratified along the expected ranked probability score. Those forecasts which feature a high expected score turn out to be particularly unreliable.

How do I know if my forecasts are better? Using benchmarks in Hydrological ensemble prediction

The skill of a forecast can be assessed by comparing the relative proximity of both the forecast and a benchmark to the observations. Example benchmarks include climatology or a naïve forecast. Hydrological ensemble prediction systems (HEPS) are currently transforming the hydrological forecasting environment but in this new field there is little information to guide researchers and operational forecasters on how benchmarks can be best used to evaluate their probabilistic forecasts. In this study, it is identified that the forecast skill calculated can vary depending on the benchmark selected and that the selection of a benchmark for determining forecasting system skill is sensitive to a number of hydrological and system factors. A benchmark intercomparison experiment is then undertaken using the continuous ranked probability score (CRPS), a reference forecasting system and a suite of 23 different methods to derive benchmarks. The benchmarks are assessed within the operational set-up of the European Flood Awareness System (EFAS) to determine those that are ‘toughest to beat’ and so give the most robust discrimination of forecast skill, particularly for the spatial average fields that EFAS relies upon. Evaluating against an observed discharge proxy the benchmark that has most utility for EFAS and avoids the most naïve skill across different hydrological situations is found to be meteorological persistency. This benchmark uses the latest meteorological observations of precipitation and temperature to drive the hydrological model. Hydrological long term average benchmarks, which are currently used in EFAS, are very easily beaten by the forecasting system and the use of these produces much naïve skill. When decomposed into seasons, the advanced meteorological benchmarks, which make use of meteorological observations from the past 20 years at the same calendar date, have the most skill discrimination. They are also good at discriminating skill in low flows and for all catchment sizes. Simpler meteorological benchmarks are particularly useful for high flows. Recommendations for EFAS are to move to routine use of meteorological persistency, an advanced meteorological benchmark and a simple meteorological benchmark in order to provide a robust evaluation of forecast skill. This work provides the first comprehensive evidence on how benchmarks can be used in evaluation of skill in probabilistic hydrological forecasts and which benchmarks are most useful for skill discrimination and avoidance of naïve skill in a large scale HEPS. It is recommended that all HEPS use the evidence and methodology provided here to evaluate which benchmarks to employ; so forecasters can have trust in their skill evaluation and will have confidence that their forecasts are indeed better.

Towards improving the framework for probabilistic forecast evaluation

The evaluation of forecast performance plays a central role both in the interpretation and use of forecast systems and in their development. Different evaluation measures (scores) are available, often quantifying different characteristics of forecast performance. The properties of several proper scores for probabilistic forecast evaluation are contrasted and then used to interpret decadal probability hindcasts of global mean temperature. The Continuous Ranked Probability Score (CRPS), Proper Linear (PL) score, and IJ Good’s logarithmic score (also referred to as Ignorance) are compared; although information from all three may be useful, the logarithmic score has an immediate interpretation and is not insensitive to forecast busts. Neither CRPS nor PL is local; this is shown to produce counter intuitive evaluations by CRPS. Benchmark forecasts from empirical models like Dynamic Climatology place the scores in context. Comparing scores for forecast systems based on physical models (in this case HadCM3, from the CMIP5 decadal archive) against such benchmarks is more informative than internal comparison systems based on similar physical simulation models with each other. It is shown that a forecast system based on HadCM3 out performs Dynamic Climatology in decadal global mean temperature hindcasts; Dynamic Climatology previously outperformed a forecast system based upon HadGEM2 and reasons for these results are suggested. Forecasts of aggregate data (5-year means of global mean temperature) are, of course, narrower than forecasts of annual averages due to the suppression of variance; while the average “distance” between the forecasts and a target may be expected to decrease, little if any discernible improvement in probabilistic skill is achieved.

Bayesian Analysis for Nonlinear Regression Model Under Skewed Errors, with Application in Growth Curves

We have considered a Bayesian approach for the nonlinear regression model by replacing the normal distribution on the error term by some skewed distributions, which account for both skewness and heavy tails or skewness alone. The type of data considered in this paper concerns repeated measurements taken in time on a set of individuals. Such multiple observations on the same individual generally produce serially correlated outcomes. Thus, additionally, our model does allow for a correlation between observations made from the same individual. We have illustrated the procedure using a data set to study the growth curves of a clinic measurement of a group of pregnant women from an obstetrics clinic in Santiago, Chile. Parameter estimation and prediction were carried out using appropriate posterior simulation schemes based in Markov Chain Monte Carlo methods. Besides the deviance information criterion (DIC) and the conditional predictive ordinate (CPO), we suggest the use of proper scoring rules based on the posterior predictive distribution for comparing models. For our data set, all these criteria chose the skew-t model as the best model for the errors. These DIC and CPO criteria are also validated, for the model proposed here, through a simulation study. As a conclusion of this study, the DIC criterion is not trustful for this kind of complex model.

An inclusion complex of β-Cyclodextrin-L-Phenylalanine : 1H NMR and molecular docking studies

An inclusion host-guest complex between β-cyclodextrin (β-CD) and L-phenylalanine (LPhe) was investigated using ¹H nuclear magnetic resonance spectroscopy and molecular docking techniques. ¹H chemical shift changes of β-CD were used to calculate the stability constant (K_stb) of the complex. On the basis of the Hildebrand-Benesi method, the K_stb of the 1:1 complex in D₂O solution at 300 K, pD 7.6 was of 25.5 M^-1, implying a fast intermolecular exchange rate process. Interestingly, docking simulation indicates the toroidal space can be occupied by L-Phe with two favorable arrangements. For the predicted model with the higher probability score, the L-Phe aromatic ring is facing to the secondary hydroxyl groups of β-CD. Results from NMR and docking simulation are in good agreement with the x-ray structures of β-CD/L-phenylalanine derivatives.

Rainfall spatial predictions: a two-part model and its assessment

Spatial prediction of hourly rainfall via radar calibration is addressed. The change of support problem (COSP), arising when the spatial supports of different data sources do not coincide, is faced in a non-Gaussian setting; in fact, hourly rainfall in Emilia-Romagna region, in Italy, is characterized by abundance of zero values and right-skeweness of the distribution of positive amounts. Rain gauge direct measurements on sparsely distributed locations and hourly cumulated radar grids are provided by the ARPA-SIMC Emilia-Romagna. We propose a three-stage Bayesian hierarchical model for radar calibration, exploiting rain gauges as reference measure. Rain probability and amounts are modeled via linear relationships with radar in the log scale; spatial correlated Gaussian effects capture the residual information. We employ a probit link for rainfall probability and Gamma distribution for rainfall positive amounts; the two steps are joined via a two-part semicontinuous model. Three model specifications differently addressing COSP are presented; in particular, a stochastic weighting of all radar pixels, driven by a latent Gaussian process defined on the grid, is employed. Estimation is performed via MCMC procedures implemented in C, linked to R software. Communication and evaluation of probabilistic, point and interval predictions is investigated. A non-randomized PIT histogram is proposed for correctly assessing calibration and coverage of two-part semicontinuous models. Predictions obtained with the different model specifications are evaluated via graphical tools (Reliability Plot, Sharpness Histogram, PIT Histogram, Brier Score Plot and Quantile Decomposition Plot), proper scoring rules (Brier Score, Continuous Rank Probability Score) and consistent scoring functions (Root Mean Square Error and Mean Absolute Error addressing the predictive mean and median, respectively). Calibration is reached and the inclusion of neighbouring information slightly improves predictions. All specifications outperform a benchmark model with incorrelated effects, confirming the relevance of spatial correlation for modeling rainfall probability and accumulation.

Estimating reliability and resolution of probability forecasts through decomposition of the empirical score

References (20)Cited By (1)Export CitationAboutAbstract Proper scoring rules provide a useful means to evaluate probabilistic forecasts. Independent from scoring rules, it has been argued that reliability and resolution are desirable forecast attributes. The mathematical expectation value of the score allows for a decomposition into reliability and resolution related terms, demonstrating a relationship between scoring rules and reliability/resolution. A similar decomposition holds for the empirical (i.e. sample average) score over an archive of forecast–observation pairs. This empirical decomposition though provides a too optimistic estimate of the potential score (i.e. the optimum score which could be obtained through recalibration), showing that a forecast assessment based solely on the empirical resolution and reliability terms will be misleading. The differences between the theoretical and empirical decomposition are investigated, and specific recommendations are given how to obtain better estimators of reliability and resolution in the case of the Brier and Ignorance scoring rule.

Use of the probability of stone formation (PSF) score to assess stone forming risk and treatment response in a cohort of Brazilian stone formers

Introduction: The aim was to confirm that PSF (probability of stone formation) changed appropriately following medical therapy on recurrent stone formers.Materials and Methods: Data were collected on 26 Brazilian stone-formers. A baseline 24-hour urine collection was performed prior to treatment. Details of the medical treatment initiated for stone-disease were recorded. A PSF calculation was performed on the 24 hour urine sample using the 7 urinary parameters required: voided volume, oxalate, calcium, urate, pH, citrate and magnesium. A repeat 24-hour urine sample was performed for PSF calculation after treatment. Comparison was made between the PSF scores before and during treatment.Results: At baseline, 20 of the 26 patients (77%) had a high PSF score (> 0.5). Of the 26 patients, 17 (65%) showed an overall reduction in their PSF profiles with a medical treatment regimen. Eleven patients (42%) changed from a high risk (PSF > 0.5) to a low risk (PSF < 0.5) and 6 patients reduced their risk score but did not change risk category. Six (23%) patients remained in a high risk category (> 0.5) during both assessments.Conclusions: The PSF score reduced following medical treatment in the majority of patients in this cohort.

Trauma and Injury Severity Score (TRISS) coefficients 2009 revision

Background: Currently used Trauma and Injury Severity Score (TRISS) coefficients, which measure probability of survival (Ps), were derived from the Major Trauma Outcome Study (MTOS) in 1995 and are now unlikely to be optimal. This study aims to estimate new TRISS coefficients using a contemporary database of injured patients presenting to emergency departments in the United States; and to compare these against the MTOS coefficients.---------- Methods: Data were obtained from the National Trauma Data Bank (NTDB) and the NTDB National Sample Project (NSP). TRISS coefficients were estimated using logistic regression. Separate coefficients were derived from complete case and multistage multiple imputation analyses for each NTDB and NSP dataset. Associated Ps over Injury Severity Score values were graphed and compared by age (adult ≥ 15 years; pediatric < 15 years) and injury mechanism (blunt; penetrating) groups. Area under the Receiver Operating Characteristic curves was used to assess coefficients’ predictive performance.---------- Results: Overall 1,072,033 NTDB and 1,278,563 weighted NSP injury events were included, compared with 23,177 used in the original MTOS analyses. Large differences were seen between results from complete case and imputed analyses. For blunt mechanism and adult penetrating mechanism injuries, there were similarities between coefficients estimated on imputed samples, and marked divergences between associated Ps estimated and those from the MTOS. However, negligible differences existed between area under the receiver operating characteristic curves estimates because the overwhelming majority of patients had minor trauma and survived. For pediatric penetrating mechanism injuries, variability in coefficients was large and Ps estimates unreliable.---------- Conclusions: Imputed NTDB coefficients are recommended as the TRISS coefficients 2009 revision for blunt mechanism and adult penetrating mechanism injuries. Coefficients for pediatric penetrating mechanism injuries could not be reliably estimated.

Functional status, postural stability and falls among older adults with glaucoma

Visual impairment is an important contributing factor in falls among older adults, which is one of the leading causes of injury and injury-related death in this population. Visual impairment is also associated with greater disability among older adults, including poorer health-related quality of life, increased frailty and reduced postural stability. The majority of this evidence, however, is based on measures of central visual function, rather than peripheral visual function. As such, there is comparatively limited research on the associations between peripheral visual function, disability and falls, and even fewer studies involving older adults with specific diseases which affect peripheral visual function, the most common of which is glaucoma. Glaucoma is one of the leading causes of irreversible vision loss among older adults, affecting around 3 per cent of adults aged over 60 years. The condition is characterised by retinal nerve fibre loss, primarily affecting peripheral visual function. Importantly, the number of older adults with glaucomatous visual impairment is projected to increase as the ageing population grows. The first component of the thesis examined the cross-sectional association between glaucomatous visual impairment and health-related quality of life (Study 1a), functional status (Study 1b) and postural stability (Study 1c) among older adults. A cohort of 74 community-dwelling adults with glaucoma (mean age 74.2 ± 5.9 years) was recruited and completed a baseline assessment. A number of visual function measures was assessed, including central visual function (visual acuity and contrast sensitivity), motion sensitivity, retinal nerve fibre analysis and monocular and binocular visual field measures (monocular 24-2 and binocular integrated visual fields (IVF): IVF-60 and IVF-120). The analyses focused on the associations between the outcomes measures and severity and location of visual field loss, as this is the primary visual function affected by glaucoma. In Study 1a, we examined the association between visual field loss and health-related quality of life, measured by the Short Form 36-item Health Survey (SF-36). Greater binocular visual field loss, on both IVF measures, was associated with lower SF-36 physical component scores, adjusted for age and gender (Pearson's r =|0.32| to |0.36|, p<0.001). Furthermore, inferior visual field loss was more strongly associated with the SF-36 physical component than superior field loss. No association was found between visual field loss and SF-36 mental component scores. The association between visual field loss and functional status was examined in Study 1b. Functional status outcomes measures included a physical activity questionnaire (Physical Activity Scale for the Elderly, PASE), performance tests (six-minute walk test, timed up and go test and lower leg strength) and an overall functional status score. Significant, but weak, correlations were found between binocular visual field loss and PASE and overall functional status scores, adjusted for age and gender (Pearson's r =|0.24| to |0.33|, p<0.05). Greater inferior visual field loss, independent of superior visual field loss, was significantly associated with poorer physical performance results and lower overall functional status scores. In Study 1c, we examined the association between visual field loss and postural stability, using a swaymeter device which recorded body movement during four conditions: eyes open and closed, on a firm and foam surface. Greater binocular visual field loss was associated with increased postural sway, both on firm and foam surfaces, independent of age and gender (Pearson’s r =|0.44| to |0.46|, p <0.001). Furthermore, inferior visual field was a stronger contributor to postural stability, more so than the superior visual field, particularly on the foam condition with the eyes open. Greater visual field loss was associated with a reduction in the visual contribution to postural sway, which underlies the observed association with postural sway. The second component of the thesis examined the association between severity and location of visual field loss and falls during a 12-month longitudinal follow-up. The number of falls was assessed prospectively using monthly fall calendars. Of the 71 participants who successfully completed the follow up (mean age 73.9 ± 5.7 years), 44% reported one or more falls, and around 20% reported two or more falls. After adjusting for age and gender, every 10 points missed on the IVF-120 increased the rate of falls by 25% (rate ratio 1.25, 95% confidence interval 1.08 - 1.44) or every 5dB reduction in IVF-60 increased the rate of falls by 47% (rate ratio 1.47, 95% confidence interval 1.16 - 1.87). Inferior visual field loss was a significant predictor of falls, more so than superior field loss, highlighting the importance of the inferior visual field area in safe and efficient navigation. Further analyses indicated that postural stability, more so than functional status, may be a potential mediating factor in the relationship between visual field loss and falls. Future research is required to confirm this causal pathway. In addition, the use of topical beta-blocker medications was not associated with an increased rate of falls in this cohort, compared with the use of other topical anti-glaucoma medications. In summary, greater binocular visual field loss among older adults with glaucoma was associated with poorer health-related quality of life in the physical domain, reduced functional status, greater postural instability and higher rates of falling. When the location of visual field loss was examined, inferior visual field loss was consistently more strongly associated with these outcomes than superior visual field loss. Insights gained from this research improve our understanding of the association between glaucomatous visual field loss and disability, and its link with falls among older adults. The clinical implications of this research include the need to include visual field screening in falls risk assessments among older adults and to raise awareness of these findings to eye care practitioners and adults with glaucoma. The findings also assist in developing further research to examine strategies to reduce disability and prevent falls among older adults with glaucoma to promote healthy ageing and independence for these individuals.

Noise robust voice activity detection using normal probability testing and time-domain histogram analysis

This paper presents a method of voice activity detection (VAD) suitable for high noise scenarios, based on the fusion of two complementary systems. The first system uses a proposed non-Gaussianity score (NGS) feature based on normal probability testing. The second system employs a histogram distance score (HDS) feature that detects changes in the signal through conducting a template-based similarity measure between adjacent frames. The decision outputs by the two systems are then merged using an open-by-reconstruction fusion stage. Accuracy of the proposed method was compared to several baseline VAD methods on a database created using real recordings of a variety of high-noise environments.

Inferior visual field reductions are associated with poorer functional status among older adults with glaucoma

Purpose: To examine the relationship between visual impairment and functional status in a community-dwelling sample of older adults with glaucoma. Methods: This study included 74 community-dwelling older adults with open-angle glaucoma (aged 74 ± 6 years). Assessment of central vision included high-contrast visual acuity and Pelli-Robson contrast sensitivity. Binocular integrated visual fields were derived from merged monocular Humphrey Field Analyser visual field plots. Functional status outcome measures included physical performance tests (6-min walk test, timed up and go test and lower limb strength), a physical activity questionnaire (Physical Activity Scale for the Elderly) and an overall functional status score. Correlation and linear regression analyses, adjusting for age and gender, examined the association between visual impairment and functional status outcomes. Results: Greater levels of visual impairment were significantly associated with lower levels of functional status among community-dwelling older adults with glaucoma, independent of age and gender. Specifically, lower levels of visual function were associated with slower timed up and go performance, weaker lower limb strength, lower self-reported physical activity, and lower overall functional status scores. Of the components of vision examined, the inferior visual field and contrast factors were the strongest predictors of these functional outcomes, whereas the superior visual field factor was not related to functional status. Conclusions: Greater visual impairment, particularly in the inferior visual field and loss of contrast sensitivity, was associated with poorer functional status among older adults with glaucoma. The findings of this study highlight the potential links between visual impairment and the onset of functional decline. Interventions which promote physical activity among older adults with glaucoma may assist in preventing functional decline, frailty and falls, and improve overall health and well-being.