Estimating reliability and resolution of probability forecasts through decomposition of the empirical score

References (20)Cited By (1)Export CitationAboutAbstract Proper scoring rules provide a useful means to evaluate probabilistic forecasts. Independent from scoring rules, it has been argued that reliability and resolution are desirable forecast attributes. The mathematical expectation value of the score allows for a decomposition into reliability and resolution related terms, demonstrating a relationship between scoring rules and reliability/resolution. A similar decomposition holds for the empirical (i.e. sample average) score over an archive of forecast–observation pairs. This empirical decomposition though provides a too optimistic estimate of the potential score (i.e. the optimum score which could be obtained through recalibration), showing that a forecast assessment based solely on the empirical resolution and reliability terms will be misleading. The differences between the theoretical and empirical decomposition are investigated, and specific recommendations are given how to obtain better estimators of reliability and resolution in the case of the Brier and Ignorance scoring rule.

Modelling the effect of fatty acids in odour development of cooked meat in vitro: part I - sensory perception

We have investigated the contribution of muscle components to the development of cooked meat odour in an aqueous model system using trained taste panels. Reaction mixtures were prepared with oleic, linoleic and linolenic acids with or without cysteine and ribose in a buffer with or without ferrous sulphate. Odour profiles were assessed and triangular tests were used to determine the ability of panellists to discriminate between mixtures. The presence of sugar and amino acid was highly detectable by panellists independently of the fatty acid considered (P < 0.001). However, the presence of C18:3 made differences. more obvious between mixtures than the presence of C18:1 or C18:2. `Meaty' notes were only associated with cysteine and ribose. `Fishy' notes were only apparent in C18:3 mixtures with or without sugar and amino acid, although the presence of cysteine and ribose decreased the perception. The addition of Fe+ +, a pro-oxidant present in the muscle, produced a reduction in the score of the attributes although the pattern was the same as when Fe was not used in the mixtures. Only `fishy' notes that were exclusively perceived in C18:3 mixtures showed a higher score in the presence of iron. Iron also produced a better discrimination in C18:3 mixtures, which were closely related to `grassy' notes in the presence of cysteine and ribose. (C) 2002 Published by Elsevier Science Ltd.

Equitability revisited: why the “equitable threat score” is not equitable

In the forecasting of binary events, verification measures that are “equitable” were defined by Gandin and Murphy to satisfy two requirements: 1) they award all random forecasting systems, including those that always issue the same forecast, the same expected score (typically zero), and 2) they are expressible as the linear weighted sum of the elements of the contingency table, where the weights are independent of the entries in the table, apart from the base rate. The authors demonstrate that the widely used “equitable threat score” (ETS), as well as numerous others, satisfies neither of these requirements and only satisfies the first requirement in the limit of an infinite sample size. Such measures are referred to as “asymptotically equitable.” In the case of ETS, the expected score of a random forecasting system is always positive and only falls below 0.01 when the number of samples is greater than around 30. Two other asymptotically equitable measures are the odds ratio skill score and the symmetric extreme dependency score, which are more strongly inequitable than ETS, particularly for rare events; for example, when the base rate is 2% and the sample size is 1000, random but unbiased forecasting systems yield an expected score of around −0.5, reducing in magnitude to −0.01 or smaller only for sample sizes exceeding 25 000. This presents a problem since these nonlinear measures have other desirable properties, in particular being reliable indicators of skill for rare events (provided that the sample size is large enough). A potential way to reconcile these properties with equitability is to recognize that Gandin and Murphy’s two requirements are independent, and the second can be safely discarded without losing the key advantages of equitability that are embodied in the first. This enables inequitable and asymptotically equitable measures to be scaled to make them equitable, while retaining their nonlinearity and other properties such as being reliable indicators of skill for rare events. It also opens up the possibility of designing new equitable verification measures.

Pronoun processing in Broca's aphasia: discourse-syntax effects in ambiguous anaphora resolution

Background: The interpretation of ambiguous subject pronouns in a null subject language, like Greek, requires that one possesses grammatical knowledge of the two subject pronominal forms, i.e., null and overt, and that discourse constraints regulating the distribution of the two pronouns in context are respected. Aims: We investigated whether the topic-shift feature encoded in overt subject pronouns would exert similar interpretive effects in a group of seven participants with Broca’s aphasia and a group of language-unimpaired adults during online processing of null and overt subject pronouns in referentially ambiguous contexts. Method & Procedures: An offline picture–sentence matching task was initially administered to investigate whether the participants with Broca’s aphasia had access to the gender and number features of clitic pronouns. An online self-paced listening picture-verification task was subsequently administered to examine how the aphasic individuals resolve pronoun ambiguities in contexts with either null or overt subject pronouns and how their performance compares to that of language-unimpaired adults. Outcomes & Results: Results demonstrate that the Broca group, along with controls, had intact access to the morphosyntactic features of clitic pronouns. However, the aphasic individuals showed decreased preference for non-salient antecedents in object position during the online resolution of ambiguous overt subject pronouns and preferred to pick the subject antecedent instead. Conclusions: Broca’s aphasic participants’ parsing decisions in the online task reflect their difficulty with establishing topic-shifted interpretations of the ambiguous overt subject pronouns. The presence of a local topic-shift effect in the immediate temporal vicinity of the overt pronoun suggests that sensitivity to the marked informational status of overt pronouns is preserved in the aphasic individuals, yet, it is blocked under conditions of global sentential processing.

How do I know if my forecasts are better? Using benchmarks in Hydrological ensemble prediction

The skill of a forecast can be assessed by comparing the relative proximity of both the forecast and a benchmark to the observations. Example benchmarks include climatology or a naïve forecast. Hydrological ensemble prediction systems (HEPS) are currently transforming the hydrological forecasting environment but in this new field there is little information to guide researchers and operational forecasters on how benchmarks can be best used to evaluate their probabilistic forecasts. In this study, it is identified that the forecast skill calculated can vary depending on the benchmark selected and that the selection of a benchmark for determining forecasting system skill is sensitive to a number of hydrological and system factors. A benchmark intercomparison experiment is then undertaken using the continuous ranked probability score (CRPS), a reference forecasting system and a suite of 23 different methods to derive benchmarks. The benchmarks are assessed within the operational set-up of the European Flood Awareness System (EFAS) to determine those that are ‘toughest to beat’ and so give the most robust discrimination of forecast skill, particularly for the spatial average fields that EFAS relies upon. Evaluating against an observed discharge proxy the benchmark that has most utility for EFAS and avoids the most naïve skill across different hydrological situations is found to be meteorological persistency. This benchmark uses the latest meteorological observations of precipitation and temperature to drive the hydrological model. Hydrological long term average benchmarks, which are currently used in EFAS, are very easily beaten by the forecasting system and the use of these produces much naïve skill. When decomposed into seasons, the advanced meteorological benchmarks, which make use of meteorological observations from the past 20 years at the same calendar date, have the most skill discrimination. They are also good at discriminating skill in low flows and for all catchment sizes. Simpler meteorological benchmarks are particularly useful for high flows. Recommendations for EFAS are to move to routine use of meteorological persistency, an advanced meteorological benchmark and a simple meteorological benchmark in order to provide a robust evaluation of forecast skill. This work provides the first comprehensive evidence on how benchmarks can be used in evaluation of skill in probabilistic hydrological forecasts and which benchmarks are most useful for skill discrimination and avoidance of naïve skill in a large scale HEPS. It is recommended that all HEPS use the evidence and methodology provided here to evaluate which benchmarks to employ; so forecasters can have trust in their skill evaluation and will have confidence that their forecasts are indeed better.