Estimating reliability and resolution of probability forecasts through decomposition of the empirical score

References (20)Cited By (1)Export CitationAboutAbstract Proper scoring rules provide a useful means to evaluate probabilistic forecasts. Independent from scoring rules, it has been argued that reliability and resolution are desirable forecast attributes. The mathematical expectation value of the score allows for a decomposition into reliability and resolution related terms, demonstrating a relationship between scoring rules and reliability/resolution. A similar decomposition holds for the empirical (i.e. sample average) score over an archive of forecast–observation pairs. This empirical decomposition though provides a too optimistic estimate of the potential score (i.e. the optimum score which could be obtained through recalibration), showing that a forecast assessment based solely on the empirical resolution and reliability terms will be misleading. The differences between the theoretical and empirical decomposition are investigated, and specific recommendations are given how to obtain better estimators of reliability and resolution in the case of the Brier and Ignorance scoring rule.

Evaluating raw ensembles with the continuous ranked probability score

The continuous ranked probability score (CRPS) is a frequently used scoring rule. In contrast with many other scoring rules, the CRPS evaluates cumulative distribution functions. An ensemble of forecasts can easily be converted into a piecewise constant cumulative distribution function with steps at the ensemble members. This renders the CRPS a convenient scoring rule for the evaluation of ‘raw’ ensembles, obviating the need for sophisticated ensemble model output statistics or dressing methods prior to evaluation. In this article, a relation between the CRPS score and the quantile score is established. The evaluation of ‘raw’ ensembles using the CRPS is discussed in this light. It is shown that latent in this evaluation is an interpretation of the ensemble as quantiles but with non-uniform levels. This needs to be taken into account if the ensemble is evaluated further, for example with rank histograms.

Probability of survival in a random exchange economy with dependent agents

In this paper I analyze the general equilibrium in a random Walrasian economy. Dependence among agents is introduced in the form of dependency neighborhoods. Under the uncertainty, an agent may fail to survive due to a meager endowment in a particular state (direct effect), as well as due to unfavorable equilibrium price system at which the value of the endowment falls short of the minimum needed for survival (indirect terms-of-trade effect). To illustrate the main result I compute the stochastic limit of equilibrium price and probability of survival of an agent in a large Cobb-Douglas economy.

On the seasonal onset of polar mesospheric clouds and the breakdown of the stratospheric polar vortex in the Southern Hemisphere

Southern Hemisphere (SH) polar mesospheric clouds (PMCs), also known as noctilucent clouds, have been observed to be more variable and, in general, dimmer than their Northern Hemisphere (NH) counterparts. The precise cause of these hemispheric differences is not well understood. This paper focuses on one aspect of the hemispheric differences: the timing of the PMC season onset. Observations from the Aeronomy of Ice in the Mesosphere satellite indicate that in recent years the date on which the PMC season begins varies much more in the SH than in the NH. Using the Canadian Middle Atmosphere Model, we show that the generation of sufficiently low temperatures necessary for cloud formation in the SH summer polar mesosphere is perturbed by year‐to‐year variations in the timing of the late‐spring breakdown of the SH stratospheric polar vortex. These stratospheric variations, which persist until the end of December, influence the propagation of gravity waves up to the mesosphere. This adds a stratospheric control to the temperatures in the polar mesopause region during early summer, which causes the onset of PMCs to vary from one year to another. This effect is much stronger in the SH than in the NH because the breakdown of the polar vortex occurs much later in the SH, closer in time to the PMC season.

Sparse probability density function estimation using the minimum integrated square error

We develop a new sparse kernel density estimator using a forward constrained regression framework, within which the nonnegative and summing-to-unity constraints of the mixing weights can easily be satisfied. Our main contribution is to derive a recursive algorithm to select significant kernels one at time based on the minimum integrated square error (MISE) criterion for both the selection of kernels and the estimation of mixing weights. The proposed approach is simple to implement and the associated computational cost is very low. Specifically, the complexity of our algorithm is in the order of the number of training data N, which is much lower than the order of N2 offered by the best existing sparse kernel density estimators. Numerical examples are employed to demonstrate that the proposed approach is effective in constructing sparse kernel density estimators with comparable accuracy to those of the classical Parzen window estimate and other existing sparse kernel density estimators.

The probability of sale and price premiums in withdrawn auctioned properties

This paper examines the impact of the auction process of residential properties that whilst unsuccessful at auction sold subsequently. The empirical analysis considers both the probability of sale and the premium of the subsequent sale price over the guide price, reserve and opening bid. The findings highlight that the final achieved sale price is influenced by key price variables revealed both prior to and during the auction itself. Factors such as auction participation, the number of individual bidders and the number of bids are significant in a number of the alternative specifications.

Forecast encompassing tests and probability forecasts

We consider tests of forecast encompassing for probability forecasts, for both quadratic and logarithmic scoring rules. We propose test statistics for the null of forecast encompassing, present the limiting distributions of the test statistics, and investigate the impact of estimating the forecasting models' parameters on these distributions. The small-sample performance is investigated, in terms of small numbers of forecasts and model estimation sample sizes. We show the usefulness of the tests for the evaluation of recession probability forecasts from logit models with different leading indicators as explanatory variables, and for evaluating survey-based probability forecasts.

A fast algorithm for sparse probability density function construction

A new sparse kernel density estimator is introduced. Our main contribution is to develop a recursive algorithm for the selection of significant kernels one at time using the minimum integrated square error (MISE) criterion for both kernel selection. The proposed approach is simple to implement and the associated computational cost is very low. Numerical examples are employed to demonstrate that the proposed approach is effective in constructing sparse kernel density estimators with competitive accuracy to existing kernel density estimators.

Evaluating the Survey of Professional Forecasters probability distributions of expected inflation based on derived event probability forecasts

Techniques are proposed for evaluating forecast probabilities of events. The tools are especially useful when, as in the case of the Survey of Professional Forecasters (SPF) expected probability distributions of inflation, recourse cannot be made to the method of construction in the evaluation of the forecasts. The tests of efficiency and conditional efficiency are applied to the forecast probabilities of events of interest derived from the SPF distributions, and supplement a whole-density evaluation of the SPF distributions based on the probability integral transform approach.

Combining probability forecasts

We consider different methods for combining probability forecasts. In empirical exercises, the data generating process of the forecasts and the event being forecast is not known, and therefore the optimal form of combination will also be unknown. We consider the properties of various combination schemes for a number of plausible data generating processes, and indicate which types of combinations are likely to be useful. We also show that whether forecast encompassing is found to hold between two rival sets of forecasts or not may depend on the type of combination adopted. The relative performances of the different combination methods are illustrated, with an application to predicting recession probabilities using leading indicators.

Probability distributions or point predictions? Survey forecasts of US output growth and inflation

We consider whether survey respondents’ probability distributions, reported as histograms, provide reliable and coherent point predictions, when viewed through the lens of a Bayesian learning model. We argue that a role remains for eliciting directly-reported point predictions in surveys of professional forecasters.

Intra-crystalline protein diagenesis (IcPD) in Patella vulgata. Part II: Breakdown and temperature sensitivity

Artificial diagenesis of the intra-crystalline proteins isolated from Patella vulgata was induced by isothermal heating at 140 °C, 110 °C and 80 °C. Protein breakdown was quantified for multiple amino acids, measuring the extent of peptide bond hydrolysis, amino acid racemisation and decomposition. The patterns of diagenesis are complex; therefore the kinetic parameters of the main reactions were estimated by two different methods: 1) a well-established approach based on fitting mathematical expressions to the experimental data, e.g. first-order rate equations for hydrolysis and power-transformed first-order rate equations for racemisation; and 2) an alternative model-free approach, which was developed by estimating a “scaling” factor for the independent variable (time) which produces the best alignment of the experimental data. This method allows the calculation of the relative reaction rates for the different temperatures of isothermal heating. High-temperature data were compared with the extent of degradation detected in sub-fossil Patella specimens of known age, and we evaluated the ability of kinetic experiments to mimic diagenesis at burial temperature. The results highlighted a difference between patterns of degradation at low and high temperature and therefore we recommend caution for the extrapolation of protein breakdown rates to low burial temperatures for geochronological purposes when relying solely on kinetic data.

Occurrence probability, width and number of steps of cusp precipitation for fully pulsed reconnection at the dayside magnetopause

We discuss the characteristics of magnetosheath plasma precipitation in the “cusp” ionosphere for when the reconnection at the dayside magnetopause takes place only in a series of pulses. It is shown that even in this special case, the low-altitude cusp precipitation is continuous, unless the intervals between the pulses are longer than observed intervals between magnetopause flux transfer event (FTE) signatures. We use FTE observation statistics to predict, for this case of entirely pulsed reconnection, the occurrence frequency, the distribution of latitudinal widths, and the number of ion dispersion steps of the cusp precipitation for a variety of locations of the reconnection site and a range of values of the local de-Hoffman Teller velocity. It is found that the cusp occurrence frequency is comparable with observed values for virtually all possible locations of the reconnection site. The distribution of cusp width is also comparable with observations and is shown to be largely dependent on the distribution of the mean reconnection rate, but pulsing the reconnection does very slightly increase the width of that distribution compared with the steady state case. We conclude that neither cusp occurrence probability nor width can be used to evaluate the relative occurrence of reconnection behaviors that are entirely pulsed, pulsed but continuous and quasi-steady. We show that the best test of the relative frequency of these three types of reconnection is to survey the distribution of steps in the cusp ion dispersion characteristics.