Estimating reliability and resolution of probability forecasts through decomposition of the empirical score

References (20)Cited By (1)Export CitationAboutAbstract Proper scoring rules provide a useful means to evaluate probabilistic forecasts. Independent from scoring rules, it has been argued that reliability and resolution are desirable forecast attributes. The mathematical expectation value of the score allows for a decomposition into reliability and resolution related terms, demonstrating a relationship between scoring rules and reliability/resolution. A similar decomposition holds for the empirical (i.e. sample average) score over an archive of forecast–observation pairs. This empirical decomposition though provides a too optimistic estimate of the potential score (i.e. the optimum score which could be obtained through recalibration), showing that a forecast assessment based solely on the empirical resolution and reliability terms will be misleading. The differences between the theoretical and empirical decomposition are investigated, and specific recommendations are given how to obtain better estimators of reliability and resolution in the case of the Brier and Ignorance scoring rule.

Curve crossing for the reflected Levy process at zero and infinity

Let $R_{t}=\sup_{0\leq s\leq t}X_{s}-X_{t}$ be a Levy process reflected in its maximum. We give necessary and sufficient conditions for finiteness of passage times above power law boundaries at infinity. Information as to when the expected passage time for $R_{t}$ is finite, is given. We also discuss the almost sure finiteness of $\limsup_{t\to 0}R_{t}/t^{\kappa}$, for each $\kappa\geq 0$.

Reliability of regional climate model trends

A necessary condition for a good probabilistic forecast is that the forecast system is shown to be reliable: forecast probabilities should equal observed probabilities verified over a large number of cases. As climate change trends are now emerging from the natural variability, we can apply this concept to climate predictions and compute the reliability of simulated local and regional temperature and precipitation trends (1950–2011) in a recent multi-model ensemble of climate model simulations prepared for the Intergovernmental Panel on Climate Change (IPCC) fifth assessment report (AR5). With only a single verification time, the verification is over the spatial dimension. The local temperature trends appear to be reliable. However, when the global mean climate response is factored out, the ensemble is overconfident: the observed trend is outside the range of modelled trends in many more regions than would be expected by the model estimate of natural variability and model spread. Precipitation trends are overconfident for all trend definitions. This implies that for near-term local climate forecasts the CMIP5 ensemble cannot simply be used as a reliable probabilistic forecast.

The Carnegie curve

The Earth’s fair weather atmospheric electric field shows, in clean air, an average daily variation which follows universal time, globally independent of the measurement position. This single diurnal cycle variation (maximum around 19UT and minimum around 03UT) is widely known as the Carnegie curve, after the geophysical survey vessel of the Carnegie Institution of Washington on which the original measurement campaigns demonstrating the universal time variation were undertaken. The Carnegie curve’s enduring importance is in providing a reference variation against which atmospheric electricity measurements are still compared; it is believed to originate from regular daily variations in atmospheric electrification associated with the different global disturbed weather regions. Details of the instrumentation, measurement principles and data obtained on the Carnegie’s seventh and final cruise are reviewed here, also deriving new harmonic coefficients allowing calculation of the Carnegie curve for different seasons. The additional harmonic analysis now identifies changes in the phasing of the maximum and minimum in the Carnegie curve, which shows a systematic seasonal variation, linked to the solstices and equinoxes, respectively.

The M curve and the performance of Spanish international new ventures

In this paper we address three challenges. First, we discuss how international new ventures (INVs) are probably not explained by the Uppsala model as there is no time for learning about foreign markets in newly born and small firms. Only in the longer term can INVs develop experiential learning to overcome the liability of foreignness as they expand abroad. Second, we advance theoretically on previous research demonstrating that the multinationality−performance relationship of INVs follows a traditional S-shaped relationship, but they first experience a ‘born global illusion’ which leads to a non-traditional M curve. Third, using a panel data analysis for the period 1994–2008 we find empirically that Spanish INVs follow an inverted U curve in the very short term, where no learning takes place, but that experience gained over time yields an M-curve relationship once learning takes place.

Developing a reliable strategy to infer the effective soil hydraulic properties from field evaporation experiments for agro-hydrological models

The Richards equation has been widely used for simulating soil water movement. However, the take-up of agro-hydrological models using the basic theory of soil water flow for optimizing irrigation, fertilizer and pesticide practices is still low. This is partly due to the difficulties in obtaining accurate values for soil hydraulic properties at a field scale. Here, we use an inverse technique to deduce the effective soil hydraulic properties, based on measuring the changes in the distribution of soil water with depth in a fallow field over a long period, subject to natural rainfall and evaporation using a robust micro Genetic Algorithm. A new optimized function was constructed from the soil water contents at different depths, and the soil water at field capacity. The deduced soil water retention curve was approximately parallel but higher than that derived from published pedo-tranfer functions for a given soil pressure head. The water contents calculated from the deduced soil hydraulic properties were in good agreement with the measured values. The reliability of the deduced soil hydraulic properties was tested in reproducing data measured from an independent experiment on the same soil cropped with leek. The calculation of root water uptake took account for both soil water potential and root density distribution. Results show that the predictions of soil water contents at various depths agree fairly well with the measurements, indicating that the inverse analysis is an effective and reliable approach to estimate soil hydraulic properties, and thus permits the simulation of soil water dynamics in both cropped and fallow soils in the field accurately. (C) 2009 Elsevier B.V. All rights reserved.

The end of the opportunism vs trust debate: bounded reliability as a new envelope concept in research on MNE governance

Modern transaction cost economics (TCE) thinking has developed into a key intellectual foundation of international business (IB) research, but the Williamsonian version has faced substantial criticism for adopting the behavioral assumption of opportunism. In this paper we assess both the opportunism concept and existing alternatives such as trust within the context of IB research, especially work on multinational enterprise (MNE) governance. Case analyses of nine global MNEs illustrate an alternative to the opportunism assumption that captures more fully the mechanisms underlying failed commitments inside the MNE. As a substitute for the often-criticized assumption of opportunism, we propose the envelope concept of bounded reliability (BRel), an assumption that represents more accurately and more completely the reasons for failed commitments, without invalidating the other critical assumption in conventional TCE (and internalization theory) thinking, namely the widely accepted envelope concept of bounded rationality (BRat). Bounded reliability as an envelope concept includes two main components, within the context of global MNE management: opportunism as intentional deceit, and benevolent preference reversal. The implications for IB research of adopting the bounded reliability concept are far reaching, as this concept may increase the legitimacy of comparative institutional analysis in the social sciences.

Examining reliability of seasonal to decadal sea surface temperature forecasts: the role of ensemble dispersion

Useful probabilistic climate forecasts on decadal timescales should be reliable (i.e. forecast probabilities match the observed relative frequencies) but this is seldom examined. This paper assesses a necessary condition for reliability, that the ratio of ensemble spread to forecast error being close to one, for seasonal to decadal sea surface temperature retrospective forecasts from the Met Office Decadal Prediction System (DePreSys). Factors which may affect reliability are diagnosed by comparing this spread-error ratio for an initial condition ensemble and two perturbed physics ensembles for initialized and uninitialized predictions. At lead times less than 2 years, the initialized ensembles tend to be under-dispersed, and hence produce overconfident and hence unreliable forecasts. For longer lead times, all three ensembles are predominantly over-dispersed. Such over-dispersion is primarily related to excessive inter-annual variability in the climate model. These findings highlight the need to carefully evaluate simulated variability in seasonal and decadal prediction systems.Useful probabilistic climate forecasts on decadal timescales should be reliable (i.e. forecast probabilities match the observed relative frequencies) but this is seldom examined. This paper assesses a necessary condition for reliability, that the ratio of ensemble spread to forecast error being close to one, for seasonal to decadal sea surface temperature retrospective forecasts from the Met Office Decadal Prediction System (DePreSys). Factors which may affect reliability are diagnosed by comparing this spread-error ratio for an initial condition ensemble and two perturbed physics ensembles for initialized and uninitialized predictions. At lead times less than 2 years, the initialized ensembles tend to be under-dispersed, and hence produce overconfident and hence unreliable forecasts. For longer lead times, all three ensembles are predominantly over-dispersed. Such over-dispersion is primarily related to excessive inter-annual variability in the climate model. These findings highlight the need to carefully evaluate simulated variability in seasonal and decadal prediction systems.

Asymmetric output gap effects in Phillips Curve and mark-up pricing models: evidence for the US and the UK

A number of studies have found an asymmetric response of consumer price index inflation to the output gap in the US in simple Phillips curve models. We consider whether there are similar asymmetries in mark-up pricing models, that is, whether the mark-up over producers' costs also depends upon the sign of the (adjusted) output gap. The robustness of our findings to the price series is assessed, and also whether price-output responses in the UK are asymmetric.

Dependence of sea ice yield-curve shape on ice thickness

In this note, the authors discuss the contribution that frictional sliding of ice floes (or floe aggregates) past each other and pressure ridging make to the plastic yield curve of sea ice. Using results from a previous study that explicitly modeled the amount of sliding and ridging that occurs for a given global strain rate, it is noted that the relative contribution of sliding and ridging to ice stress depends upon ice thickness. The implication is that the shape and size of the plastic yield curve is dependent upon ice thickness. The yield-curve shape dependence is in addition to plastic hardening/weakening that relates the size of the yield curve to ice thickness. In most sea ice dynamics models the yield-curve shape is taken to be independent of ice thickness. The authors show that the change of the yield curve due to a change in the ice thickness can be taken into account by a weighted sum of two thickness-independent rheologies describing ridging and sliding effects separately. It would be straightforward to implement the thickness-dependent yield-curve shape described here into sea ice models used for global or regional ice prediction.

Examining reliability of seasonal to decadal sea surface temperature forecasts: the role of ensemble dispersion

Useful probabilistic climate forecasts on decadal timescales should be reliable (i.e. forecast probabilities match the observed relative frequencies) but this is seldom examined. This paper assesses a necessary condition for reliability, that the ratio of ensemble spread to forecast error being close to one, for seasonal to decadal sea surface temperature retrospective forecasts from the Met Office Decadal Prediction System (DePreSys). Factors which may affect reliability are diagnosed by comparing this spread-error ratio for an initial condition ensemble and two perturbed physics ensembles for initialized and uninitialized predictions. At lead times less than 2 years, the initialized ensembles tend to be under-dispersed, and hence produce overconfident and hence unreliable forecasts. For longer lead times, all three ensembles are predominantly over-dispersed. Such over-dispersion is primarily related to excessive inter-annual variability in the climate model. These findings highlight the need to carefully evaluate simulated variability in seasonal and decadal prediction systems.Useful probabilistic climate forecasts on decadal timescales should be reliable (i.e. forecast probabilities match the observed relative frequencies) but this is seldom examined. This paper assesses a necessary condition for reliability, that the ratio of ensemble spread to forecast error being close to one, for seasonal to decadal sea surface temperature retrospective forecasts from the Met Office Decadal Prediction System (DePreSys). Factors which may affect reliability are diagnosed by comparing this spread-error ratio for an initial condition ensemble and two perturbed physics ensembles for initialized and uninitialized predictions. At lead times less than 2 years, the initialized ensembles tend to be under-dispersed, and hence produce overconfident and hence unreliable forecasts. For longer lead times, all three ensembles are predominantly over-dispersed. Such over-dispersion is primarily related to excessive inter-annual variability in the climate model. These findings highlight the need to carefully evaluate simulated variability in seasonal and decadal prediction systems.

What will be the risk-free rate and benchmark yield curve following European monetary union?

Using a linear factor model, we study the behaviour of French, Germany, Italian and British sovereign yield curves in the run up to EMU. This allows us to determine which of these yield curves might best approximate a benchmark yield curve post EMU. We find that the best approximation for the risk free yield is the UK three month T-bill yield, followed by the German three month T-bill yield. As no one sovereign yield curve dominates all others, we find that a composite yield curve, consisting of French, Italian and UK bonds at different maturity points along the yield curve should be the benchmark post EMU.

Reliability comparison of transmit/receive diversity and error control coding in low-power medium access control protocols

Low-power medium access control (MAC) protocols used for communication of energy constraint wireless embedded devices do not cope well with situations where transmission channels are highly erroneous. Existing MAC protocols discard corrupted messages which lead to costly retransmissions. To improve transmission performance, it is possible to include an error correction scheme and transmit/receive diversity. It is possible to add redundant information to transmitted packets in order to recover data from corrupted packets. It is also possible to make use of transmit/receive diversity via multiple antennas to improve error resiliency of transmissions. Both schemes may be used in conjunction to further improve the performance. In this study, the authors show how an error correction scheme and transmit/receive diversity can be integrated in low-power MAC protocols. Furthermore, the authors investigate the achievable performance gains of both methods. This is important as both methods have associated costs (processing requirements; additional antennas and power) and for a given communication situation it must be decided which methods should be employed. The authors’ results show that, in many practical situations, error control coding outperforms transmission diversity; however, if very high reliability is required, it is useful to employ both schemes together.

Improving transmission reliability of low-power medium access control protocols using average diversity combining

Embedded computer systems equipped with wireless communication transceivers are nowadays used in a vast number of application scenarios. Energy consumption is important in many of these scenarios, as systems are battery operated and long maintenance-free operation is required. To achieve this goal, embedded systems employ low-power communication transceivers and protocols. However, currently used protocols cannot operate efficiently when communication channels are highly erroneous. In this study, we show how average diversity combining (ADC) can be used in state-of-the-art low-power communication protocols. This novel approach improves transmission reliability and in consequence energy consumption and transmission latency in the presence of erroneous channels. Using a testbed, we show that highly erroneous channels are indeed a common occurrence in situations, where low-power systems are used and we demonstrate that ADC improves low-power communication dramatically.