Evaluating the Bank of England density forecasts of inflation

We consider evaluating the UK Monetary Policy Committee's inflation density forecasts using probability integral transform goodness-of-fit tests. These tests evaluate the whole forecast density. We also consider whether the probabilities assigned to inflation being in certain ranges are well calibrated, where the ranges are chosen to be those of particular relevance to the MPC, given its remit of maintaining inflation rates in a band around per annum. Finally, we discuss the decision-based approach to forecast evaluation in relation to the MPC forecasts

Evaluating the Survey of Professional Forecasters probability distributions of expected inflation based on derived event probability forecasts

Techniques are proposed for evaluating forecast probabilities of events. The tools are especially useful when, as in the case of the Survey of Professional Forecasters (SPF) expected probability distributions of inflation, recourse cannot be made to the method of construction in the evaluation of the forecasts. The tests of efficiency and conditional efficiency are applied to the forecast probabilities of events of interest derived from the SPF distributions, and supplement a whole-density evaluation of the SPF distributions based on the probability integral transform approach.

Quantile forecasts of daily exchange rate returns from forecasts of realized volatility

Quantile forecasts are central to risk management decisions because of the widespread use of Value-at-Risk. A quantile forecast is the product of two factors: the model used to forecast volatility, and the method of computing quantiles from the volatility forecasts. In this paper we calculate and evaluate quantile forecasts of the daily exchange rate returns of five currencies. The forecasting models that have been used in recent analyses of the predictability of daily realized volatility permit a comparison of the predictive power of different measures of intraday variation and intraday returns in forecasting exchange rate variability. The methods of computing quantile forecasts include making distributional assumptions for future daily returns as well as using the empirical distribution of predicted standardized returns with both rolling and recursive samples. Our main findings are that the Heterogenous Autoregressive model provides more accurate volatility and quantile forecasts for currencies which experience shifts in volatility, such as the Canadian dollar, and that the use of the empirical distribution to calculate quantiles can improve forecasts when there are shifts

Forecasting US output growth with non-linear models in the presence of data uncertainty

We consider the impact of data revisions on the forecast performance of a SETAR regime-switching model of U.S. output growth. The impact of data uncertainty in real-time forecasting will affect a model's forecast performance via the effect on the model parameter estimates as well as via the forecast being conditioned on data measured with error. We find that benchmark revisions do affect the performance of the non-linear model of the growth rate, and that the performance relative to a linear comparator deteriorates in real-time compared to a pseudo out-of-sample forecasting exercise.

Probability distributions or point predictions? Survey forecasts of US output growth and inflation

We consider whether survey respondents’ probability distributions, reported as histograms, provide reliable and coherent point predictions, when viewed through the lens of a Bayesian learning model. We argue that a role remains for eliciting directly-reported point predictions in surveys of professional forecasters.

Direct soil moisture controls of future global soil carbon changes: An important source of uncertainty

The nature of the climate–carbon cycle feedback depends critically on the response of soil carbon to climate, including changes in moisture. However, soil moisture–carbon feedback responses have not been investigated thoroughly. Uncertainty in the response of soil carbon to soil moisture changes could arise from uncertainty in the relationship between soil moisture and heterotrophic respiration. We used twelve soil moisture–respiration functions (SMRFs) with a soil carbon model (RothC) and data from a coupled climate–carbon cycle general circulation model to investigate the impact of direct heterotrophic respiration dependence on soil moisture on the climate carbon cycle feedback. Global changes in soil moisture acted to oppose temperature‐driven decreases in soil carbon and hence tended to increase soil carbon storage. We found considerable uncertainty in soil carbon changes due to the response of soil respiration to soil moisture. The use of different SMRFs resulted in both large losses and small gains in future global soil carbon stocks, whether considering all climate forcings or only moisture changes. Regionally, the greatest range in soil carbon changes across SMRFs was found where the largest soil carbon changes occurred. Further research is needed to constrain the soil moisture–respiration relationship and thus reduce uncertainty in climate–carbon cycle feedbacks. There may also be considerable uncertainty in the regional responses of soil carbon to soil moisture changes since climate model predictions of regional soil moisture changes are less coherent than temperature changes.

Examining reliability of seasonal to decadal sea surface temperature forecasts: the role of ensemble dispersion

Useful probabilistic climate forecasts on decadal timescales should be reliable (i.e. forecast probabilities match the observed relative frequencies) but this is seldom examined. This paper assesses a necessary condition for reliability, that the ratio of ensemble spread to forecast error being close to one, for seasonal to decadal sea surface temperature retrospective forecasts from the Met Office Decadal Prediction System (DePreSys). Factors which may affect reliability are diagnosed by comparing this spread-error ratio for an initial condition ensemble and two perturbed physics ensembles for initialized and uninitialized predictions. At lead times less than 2 years, the initialized ensembles tend to be under-dispersed, and hence produce overconfident and hence unreliable forecasts. For longer lead times, all three ensembles are predominantly over-dispersed. Such over-dispersion is primarily related to excessive inter-annual variability in the climate model. These findings highlight the need to carefully evaluate simulated variability in seasonal and decadal prediction systems.Useful probabilistic climate forecasts on decadal timescales should be reliable (i.e. forecast probabilities match the observed relative frequencies) but this is seldom examined. This paper assesses a necessary condition for reliability, that the ratio of ensemble spread to forecast error being close to one, for seasonal to decadal sea surface temperature retrospective forecasts from the Met Office Decadal Prediction System (DePreSys). Factors which may affect reliability are diagnosed by comparing this spread-error ratio for an initial condition ensemble and two perturbed physics ensembles for initialized and uninitialized predictions. At lead times less than 2 years, the initialized ensembles tend to be under-dispersed, and hence produce overconfident and hence unreliable forecasts. For longer lead times, all three ensembles are predominantly over-dispersed. Such over-dispersion is primarily related to excessive inter-annual variability in the climate model. These findings highlight the need to carefully evaluate simulated variability in seasonal and decadal prediction systems.

Predictability of the western North Pacific summer climate associated with different ENSO phases by ENSEMBLES multi-model seasonal forecasts

Predictability of the western North Pacific (WNP) summer climate associated with different El Niño–Southern Oscillation (ENSO) phases is investigated in this study based on the 1-month lead retrospective forecasts of five state-of-the-art coupled models from ENSEMBLES. During the period from 1960 to 2005, the models well capture the WNP summer climate anomalies during most of years in different ENSO phases except the La Niña decaying summers. In the El Niño developing, El Niño decaying and La Niña developing summers, the prediction skills are high for the WNP summer monsoon index (WNPMI), with the prediction correlation larger than 0.7. The high prediction skills of the lower-tropospheric circulation during these phases are found mainly over the tropical western Pacific Ocean, South China Sea and subtropical WNP. These good predictions correspond well to their close teleconnection with ENSO and the high prediction skills of tropical SSTs. By contrast, for the La Niña decaying summers, the prediction skills are considerably low with the prediction correlation for the WNPMI near to zero and low prediction skills around the Philippines and subtropical WNP. These poor predictions relate to the weak summer anomalies of the WNPMI during the La Niña decaying years and no significant connections between the WNP lower-tropospheric circulation anomalies and the SSTs over the tropical central and eastern Pacific Ocean in observations. However, the models tend to predict an apparent anomalous cyclone over the WNP during the La Niña decaying years, indicating a linearity of the circulation response over WNP in the models prediction in comparison with that during the El Niño decaying years which differs from observations. In addition, the models show considerable capability in describing the WNP summer anomalies during the ENSO neutral summers. These anomalies are related to the positive feedback between the WNP lower-tropospheric circulation and the local SSTs. The models can capture this positive feedback but with some uncertainties from different ensemble members during the ENSO neutral summers.

Northern winter climate change: assessment of uncertainty in CMIP5 projections related to stratosphere–troposphere coupling

Future changes in the stratospheric circulation could have an important impact on Northern winter tropospheric climate change, given that sea level pressure (SLP) responds not only to tropospheric circulation variations but also to vertically coherent variations in troposphere-stratosphere circulation. Here we assess Northern winter stratospheric change and its potential to influence surface climate change in the Coupled Model Intercomparison Project – phase 5 (CMIP5) multi-model ensemble. In the stratosphere at high latitudes, an easterly change in zonally averaged zonal wind is found for the majority of the CMIP5 models, under the Representative Concentration Pathway 8.5 scenario. Comparable results are also found in the 1% CO2 increase per year projections, indicating that the stratospheric easterly change is common feature in future climate projections. This stratospheric wind change, however, shows a significant spread among the models. By using linear regression, we quantify the impact of tropical upper troposphere warming, polar amplification and the stratospheric wind change on SLP. We find that the inter-model spread in stratospheric wind change contributes substantially to the inter-model spread in Arctic SLP change. The role of the stratosphere in determining part of the spread in SLP change is supported by the fact that the SLP change lags the stratospheric zonally averaged wind change. Taken together, these findings provide further support for the importance of simulating the coupling between the stratosphere and the troposphere, to narrow the uncertainty in the future projection of tropospheric circulation changes.

Verification of European subseasonal wind speed forecasts

Analysis of the forecasts and hindcasts from the ECMWF 32-day forecast model reveals that there is statistically significant skill in predicting weekly mean wind speeds over areas of Europe at lead times of at least 14–20 days. Previous research on wind speed predictability has focused on the short- to medium-range time scales, typically finding that forecasts lose all skill by the later part of the medium-range forecast. To the authors’ knowledge, this research is the first to look beyond the medium-range time scale by taking weekly mean wind speeds, instead of averages at hourly or daily resolution, for the ECMWF monthly forecasting system. It is shown that the operational forecasts have high levels of correlation (~0.6) between the forecasts and observations over the winters of 2008–12 for some areas of Europe. Hindcasts covering 20 winters show a more modest level of correlation but are still skillful. Additional analysis examines the probabilistic skill for the United Kingdom with the application of wind power forecasting in mind. It is also shown that there is forecast “value” for end users (operating in a simple cost/loss ratio decision-making framework). End users that are sensitive to winter wind speed variability over the United Kingdom, Germany, and some other areas of Europe should therefore consider forecasts beyond the medium-range time scale as it is clear there is useful information contained within the forecast.

Review: the predictability of the extra-tropical stratosphere on monthly timescales and its impact on the skill of tropospheric forecasts

Extreme variability of the winter- and spring-time stratospheric polar vortex has been shown to affect extratropical tropospheric weather. Therefore, reducing stratospheric forecast error may be one way to improve the skill of tropospheric weather forecasts. In this review, the basis for this idea is examined. A range of studies of different stratospheric extreme vortex events shows that they can be skilfully forecasted beyond five days and into the sub-seasonal range (0-30 days) in some cases. Separate studies show that typical errors in forecasting a stratospheric extreme vortex event can alter tropospheric forecasts skill by 5-7% in the extratropics on sub-seasonal timescales. Thus understanding what limits stratospheric predictability is of significant interest to operational forecasting centres. Both limitations in forecasting tropospheric planetary waves and stratospheric model biases have been shown to be important in this context.

Atmospheric circulation as a source of uncertainty in climate change projections

The evidence for anthropogenic climate change continues to strengthen, and concerns about severe weather events are increasing. As a result, scientific interest is rapidly shifting from detection and attribution of global climate change to prediction of its impacts at the regional scale. However, nearly everything we have any confidence in when it comes to climate change is related to global patterns of surface temperature, which are primarily controlled by thermodynamics. In contrast, we have much less confidence in atmospheric circulation aspects of climate change, which are primarily controlled by dynamics and exert a strong control on regional climate. Model projections of circulation-related fields, including precipitation, show a wide range of possible outcomes, even on centennial timescales. Sources of uncertainty include low-frequency chaotic variability and the sensitivity to model error of the circulation response to climate forcing. As the circulation response to external forcing appears to project strongly onto existing patterns of variability, knowledge of errors in the dynamics of variability may provide some constraints on model projections. Nevertheless, higher scientific confidence in circulation-related aspects of climate change will be difficult to obtain. For effective decision-making, it is necessary to move to a more explicitly probabilistic, risk-based approach.

Effect of uncertainty in surface mass balance--elevation feedback on projections of the future sea level contribution of the Greenland ice sheet

We apply a new parameterisation of the Greenland ice sheet (GrIS) feedback between surface mass balance (SMB: the sum of surface accumulation and surface ablation) and surface elevation in the MAR regional climate model (Edwards et al., 2014) to projections of future climate change using five ice sheet models (ISMs). The MAR (Modèle Atmosphérique Régional: Fettweis, 2007) climate projections are for 2000–2199, forced by the ECHAM5 and HadCM3 global climate models (GCMs) under the SRES A1B emissions scenario. The additional sea level contribution due to the SMB– elevation feedback averaged over five ISM projections for ECHAM5 and three for HadCM3 is 4.3% (best estimate; 95% credibility interval 1.8–6.9 %) at 2100, and 9.6% (best estimate; 95% credibility interval 3.6–16.0 %) at 2200. In all results the elevation feedback is significantly positive, amplifying the GrIS sea level contribution relative to the MAR projections in which the ice sheet topography is fixed: the lower bounds of our 95% credibility intervals (CIs) for sea level contributions are larger than the “no feedback” case for all ISMs and GCMs. Our method is novel in sea level projections because we propagate three types of modelling uncertainty – GCM and ISM structural uncertainties, and elevation feedback parameterisation uncertainty – along the causal chain, from SRES scenario to sea level, within a coherent experimental design and statistical framework. The relative contributions to uncertainty depend on the timescale of interest. At 2100, the GCM uncertainty is largest, but by 2200 both the ISM and parameterisation uncertainties are larger. We also perform a perturbed parameter ensemble with one ISM to estimate the shape of the projected sea level probability distribution; our results indicate that the probability density is slightly skewed towards higher sea level contributions.

The impact of uncertain precipitation data on insurance loss estimates using a flood catastrophe model

Catastrophe risk models used by the insurance industry are likely subject to significant uncertainty, but due to their proprietary nature and strict licensing conditions they are not available for experimentation. In addition, even if such experiments were conducted, these would not be repeatable by other researchers because commercial confidentiality issues prevent the details of proprietary catastrophe model structures from being described in public domain documents. However, such experimentation is urgently required to improve decision making in both insurance and reinsurance markets. In this paper we therefore construct our own catastrophe risk model for flooding in Dublin, Ireland, in order to assess the impact of typical precipitation data uncertainty on loss predictions. As we consider only a city region rather than a whole territory and have access to detailed data and computing resources typically unavailable to industry modellers, our model is significantly more detailed than most commercial products. The model consists of four components, a stochastic rainfall module, a hydrological and hydraulic flood hazard module, a vulnerability module, and a financial loss module. Using these we undertake a series of simulations to test the impact of driving the stochastic event generator with four different rainfall data sets: ground gauge data, gauge-corrected rainfall radar, meteorological reanalysis data (European Centre for Medium-Range Weather Forecasts Reanalysis-Interim; ERA-Interim) and a satellite rainfall product (The Climate Prediction Center morphing method; CMORPH). Catastrophe models are unusual because they use the upper three components of the modelling chain to generate a large synthetic database of unobserved and severe loss-driving events for which estimated losses are calculated. We find the loss estimates to be more sensitive to uncertainties propagated from the driving precipitation data sets than to other uncertainties in the hazard and vulnerability modules, suggesting that the range of uncertainty within catastrophe model structures may be greater than commonly believed.