Contrasting probabilistic scoring rules

There are several scoring rules that one can choose from in order to score probabilistic forecasting models or estimate model parameters. Whilst it is generally agreed that proper scoring rules are preferable, there is no clear criterion for preferring one proper scoring rule above another. This manuscript compares and contrasts some commonly used proper scoring rules and provides guidance on scoring rule selection. In particular, it is shown that the logarithmic scoring rule prefers erring with more uncertainty, the spherical scoring rule prefers erring with lower uncertainty, whereas the other scoring rules are indifferent to either option.

Scoring rules and abnormally low bids criteria in construction tenders: a taxonomic review

In the global construction context, the best value or most economically advantageous tender is becoming a widespread approach for contractor selection, as an alternative to other traditional awarding criteria such as the lowest price. In these multi-attribute tenders, the owner or auctioneer solicits proposals containing both a price bid and additional technical features. Once the proposals are received, each bidder’s price bid is given an economic score according to a scoring rule, generally called an economic scoring formula (ESF) and a technical score according to pre-specified criteria. Eventually, the contract is awarded to the bidder with the highest weighted overall score (economic + technical). However, economic scoring formula selection by auctioneers is invariably and paradoxically a highly intuitive process in practice, involving few theoretical or empirical considerations, despite having been considered traditionally and mistakenly as objective, due to its mathematical nature. This paper provides a taxonomic classification of a wide variety of ESFs and abnormally low bids criteria (ALBC) gathered in several countries with different tendering approaches. Practical implications concern the optimal design of price scoring rules in construction contract tenders, as well as future analyses of the effects of the ESF and ALBC on competitive bidding behaviour.

Scoring rules and competitive behavior in best value construction auctions

This paper examines the extent to which engineers can influence the competitive behavior of bidders in Best Value or multi-attribute construction auctions, where both the (dollar) bid and technical non-price criteria are scored according to a scoring rule. From a sample of Spanish construction auctions with a variety of bid scoring rules, it is found that bidders are influenced by the auction rules in significant and predictable ways. The bid score weighting, bid scoring formula and abnormally low bid criterion are variables likely to influence the competitiveness of bidders in terms of both their aggressive/conservative bidding and concentration/dispersion of bids. Revealing the influence of the bid scoring rules and their magnitude on bidders’ competitive behavior opens the door for the engineer to condition bidder competitive behavior in such a way as to provide the balance needed to achieve the owner’s desired strategic outcomes.

Reliability, sufficiency, and the decomposition of proper scores

Scoring rules are an important tool for evaluating the performance of probabilistic forecasting schemes. A scoring rule is called strictly proper if its expectation is optimal if and only if the forecast probability represents the true distribution of the target. In the binary case, strictly proper scoring rules allow for a decomposition into terms related to the resolution and the reliability of a forecast. This fact is particularly well known for the Brier Score. In this article, this result is extended to forecasts for finite-valued targets. Both resolution and reliability are shown to have a positive effect on the score. It is demonstrated that resolution and reliability are directly related to forecast attributes that are desirable on grounds independent of the notion of scores. This finding can be considered an epistemological justification of measuring forecast quality by proper scoring rules. A link is provided to the original work of DeGroot and Fienberg, extending their concepts of sufficiency and refinement. The relation to the conjectured sharpness principle of Gneiting, et al., is elucidated.

Estimating reliability and resolution of probability forecasts through decomposition of the empirical score

References (20)Cited By (1)Export CitationAboutAbstract Proper scoring rules provide a useful means to evaluate probabilistic forecasts. Independent from scoring rules, it has been argued that reliability and resolution are desirable forecast attributes. The mathematical expectation value of the score allows for a decomposition into reliability and resolution related terms, demonstrating a relationship between scoring rules and reliability/resolution. A similar decomposition holds for the empirical (i.e. sample average) score over an archive of forecast–observation pairs. This empirical decomposition though provides a too optimistic estimate of the potential score (i.e. the optimum score which could be obtained through recalibration), showing that a forecast assessment based solely on the empirical resolution and reliability terms will be misleading. The differences between the theoretical and empirical decomposition are investigated, and specific recommendations are given how to obtain better estimators of reliability and resolution in the case of the Brier and Ignorance scoring rule.

Evaluating raw ensembles with the continuous ranked probability score

The continuous ranked probability score (CRPS) is a frequently used scoring rule. In contrast with many other scoring rules, the CRPS evaluates cumulative distribution functions. An ensemble of forecasts can easily be converted into a piecewise constant cumulative distribution function with steps at the ensemble members. This renders the CRPS a convenient scoring rule for the evaluation of ‘raw’ ensembles, obviating the need for sophisticated ensemble model output statistics or dressing methods prior to evaluation. In this article, a relation between the CRPS score and the quantile score is established. The evaluation of ‘raw’ ensembles using the CRPS is discussed in this light. It is shown that latent in this evaluation is an interpretation of the ensemble as quantiles but with non-uniform levels. This needs to be taken into account if the ensemble is evaluated further, for example with rank histograms.

Forecast encompassing tests and probability forecasts

We consider tests of forecast encompassing for probability forecasts, for both quadratic and logarithmic scoring rules. We propose test statistics for the null of forecast encompassing, present the limiting distributions of the test statistics, and investigate the impact of estimating the forecasting models' parameters on these distributions. The small-sample performance is investigated, in terms of small numbers of forecasts and model estimation sample sizes. We show the usefulness of the tests for the evaluation of recession probability forecasts from logit models with different leading indicators as explanatory variables, and for evaluating survey-based probability forecasts.

Selection rules for rovibronic transitions in symmetric top molecules

A simple diagrammatic rule is presented for determining the rotational selection rules governing transitions between any pair of vibronic states in electric dipole spectra of symmetric top molecules. The rule is useful in cases where degenerate vibronic levels with first-order Coriolis splittings occur, because it gives immediately the selection rule for the (+l) and (-l) components in any degenerate state. The rule is also helpful in determining the symmetry species and the effective zeta constants in overtone and combination levels involving degenerate vibrations. Particular attention is devoted to the conventions concerning the signs of zeta constants.

Raman selection rules for vibration-rotation transitions in symmetric top molecules

Symmetry restrictions on Raman selection rules can be obtained, quite generally, by considering a Raman allowed transition as the result of two successive dipole allowed transitions, and imposing the usual symmetry restrictions on the dipole transitions. This leads to the same results as the more familiar polarizability theory, but the vibration-rotation selection rules are easier to obtain by this argument. The selection rules for symmetric top molecules involving the (+l) and (-l) components of a degenerate vibrational level with first-order Coriolis splitting are derived in this paper. It is shown that these selection rules depend on the order of the highest-fold symmetry axis Cn, being different for molecules with n=3, n=4, or n ≧ 5; moreover the selection rules are different again for molecules belonging to the point groups Dnd with n even, and Sm with 1/2m even, for which the highest-fold symmetry axes Cn and Sm are related by m=2n. Finally it is shown that an apparent anomaly between the observed Raman and infra-red vibration-rotation spectra of the allene molecule is resolved when the correct selection rules are used, and a value for the A rotational constant of allene is derived without making use of the zeta sum rule.

Infrared and Raman selection rules for cyclopentane

Infra-red and Raman selection rules are obtained for the cyclopentane molecule, on the assumption that it has a free pseudo-rotation with a large potential hump at the D5h configuration. The selection rules obtained, which concern the vibrational, pseudo-rotational, and rotational quantum numbers, are summarized in tables 1, 2 and 3.

Assembly rules for protein networks derived from phylogenetic-statistical analysis of whole genomes

Background: We report an analysis of a protein network of functionally linked proteins, identified from a phylogenetic statistical analysis of complete eukaryotic genomes. Phylogenetic methods identify pairs of proteins that co-evolve on a phylogenetic tree, and have been shown to have a high probability of correctly identifying known functional links. Results: The eukaryotic correlated evolution network we derive displays the familiar power law scaling of connectivity. We introduce the use of explicit phylogenetic methods to reconstruct the ancestral presence or absence of proteins at the interior nodes of a phylogeny of eukaryote species. We find that the connectivity distribution of proteins at the point they arise on the tree and join the network follows a power law, as does the connectivity distribution of proteins at the time they are lost from the network. Proteins resident in the network acquire connections over time, but we find no evidence that 'preferential attachment' - the phenomenon of newly acquired connections in the network being more likely to be made to proteins with large numbers of connections - influences the network structure. We derive a 'variable rate of attachment' model in which proteins vary in their propensity to form network interactions independently of how many connections they have or of the total number of connections in the network, and show how this model can produce apparent power-law scaling without preferential attachment. Conclusion: A few simple rules can explain the topological structure and evolutionary changes to protein-interaction networks: most change is concentrated in satellite proteins of low connectivity and small phenotypic effect, and proteins differ in their propensity to form attachments. Given these rules of assembly, power law scaled networks naturally emerge from simple principles of selection, yielding protein interaction networks that retain a high-degree of robustness on short time scales and evolvability on longer evolutionary time scales.