Finite sample weighting of recursive forecast errors

This paper proposes and tests a new framework for weighting recursive out-of-sample prediction errors according to their corresponding levels of in-sample estimation uncertainty. In essence, we show how to use the maximum possible amount of information from the sample in the evaluation of the prediction accuracy, by commencing the forecasts at the earliest opportunity and weighting the prediction errors. Via a Monte Carlo study, we demonstrate that the proposed framework selects the correct model from a set of candidate models considerably more often than the existing standard approach when only a small sample is available. We also show that the proposed weighting approaches result in tests of equal predictive accuracy that have much better sizes than the standard approach. An application to an exchange rate dataset highlights relevant differences in the results of tests of predictive accuracy based on the standard approach versus the framework proposed in this paper.

How is the balance of a forecast ensemble affected by adaptive and non-adaptive localization schemes?

This paper investigates the effect on balance of a number of Schur product-type localization schemes which have been designed with the primary function of reducing spurious far-field correlations in forecast error statistics. The localization schemes studied comprise a non-adaptive scheme (where the moderation matrix is decomposed in a spectral basis), and two adaptive schemes, namely a simplified version of SENCORP (Smoothed ENsemble COrrelations Raised to a Power) and ECO-RAP (Ensemble COrrelations Raised to A Power). The paper shows, we believe for the first time, how the degree of balance (geostrophic and hydrostatic) implied by the error covariance matrices localized by these schemes can be diagnosed. Here it is considered that an effective localization scheme is one that reduces spurious correlations adequately but also minimizes disruption of balance (where the 'correct' degree of balance or imbalance is assumed to be possessed by the unlocalized ensemble). By varying free parameters that describe each scheme (e.g. the degree of truncation in the schemes that use the spectral basis, the 'order' of each scheme, and the degree of ensemble smoothing), it is found that a particular configuration of the ECO-RAP scheme is best suited to the convective-scale system studied. According to our diagnostics this ECO-RAP configuration still weakens geostrophic and hydrostatic balance, but overall this is less so than for other schemes.

Towards improving the framework for probabilistic forecast evaluation

The evaluation of forecast performance plays a central role both in the interpretation and use of forecast systems and in their development. Different evaluation measures (scores) are available, often quantifying different characteristics of forecast performance. The properties of several proper scores for probabilistic forecast evaluation are contrasted and then used to interpret decadal probability hindcasts of global mean temperature. The Continuous Ranked Probability Score (CRPS), Proper Linear (PL) score, and IJ Good’s logarithmic score (also referred to as Ignorance) are compared; although information from all three may be useful, the logarithmic score has an immediate interpretation and is not insensitive to forecast busts. Neither CRPS nor PL is local; this is shown to produce counter intuitive evaluations by CRPS. Benchmark forecasts from empirical models like Dynamic Climatology place the scores in context. Comparing scores for forecast systems based on physical models (in this case HadCM3, from the CMIP5 decadal archive) against such benchmarks is more informative than internal comparison systems based on similar physical simulation models with each other. It is shown that a forecast system based on HadCM3 out performs Dynamic Climatology in decadal global mean temperature hindcasts; Dynamic Climatology previously outperformed a forecast system based upon HadGEM2 and reasons for these results are suggested. Forecasts of aggregate data (5-year means of global mean temperature) are, of course, narrower than forecasts of annual averages due to the suppression of variance; while the average “distance” between the forecasts and a target may be expected to decrease, little if any discernible improvement in probabilistic skill is achieved.

Atmospheric and oceanic contributions to irreducible forecast uncertainty of Arctic surface climate

Uncertainty of Arctic seasonal to interannual predictions arising from model errors and initial state uncertainty has been widely discussed in the literature, whereas the irreducible forecast uncertainty (IFU) arising from the chaoticity of the climate system has received less attention. However, IFU provides important insights into the mechanisms through which predictability is lost, and hence can inform prioritization of model development and observations deployment. Here, we characterize how internal oceanic and surface atmospheric heat fluxes contribute to IFU of Arctic sea ice and upper ocean heat content in an Earth system model by analyzing a set of idealized ensemble prediction experiments. We find that atmospheric and oceanic heat flux are often equally important for driving unpredictable Arctic-wide changes in sea ice and surface water temperatures, and hence contribute equally to IFU. Atmospheric surface heat flux tends to dominate Arctic-wide changes for lead times of up to a year, whereas oceanic heat flux tends to dominate regionally and on interannual time scales. There is in general a strong negative covariance between surface heat flux and ocean vertical heat flux at depth, and anomalies of lateral ocean heat transport are wind-driven, which suggests that the unpredictable oceanic heat flux variability is mainly forced by the atmosphere. These results are qualitatively robust across different initial states, but substantial variations in the amplitude of IFU exist. We conclude that both atmospheric variability and the initial state of the upper ocean are key ingredients for predictions of Arctic surface climate on seasonal to interannual time scales.

Enhanced long-range forecast skill in boreal winter following stratospheric strong vortex conditions

There has been a great deal of recent interest in producing weather forecasts on the 2–6 week sub-seasonal timescale, which bridges the gap between medium-range (0–10 day) and seasonal (3–6 month) forecasts. While much of this interest is focused on the potential applications of skilful forecasts on the sub-seasonal range, understanding the potential sources of sub-seasonal forecast skill is a challenging and interesting problem, particularly because of the likely state-dependence of this skill (Hudson et al 2011). One such potential source of state-dependent skill for the Northern Hemisphere in winter is the occurrence of stratospheric sudden warming (SSW) events (Sigmond et al 2013). Here we show, by analysing a set of sub-seasonal hindcasts, that there is enhanced predictability of surface circulation not only when the stratospheric vortex is anomalously weak following SSWs but also when the vortex is extremely strong. Sub-seasonal forecasts initialized during strong vortex events are able to successfully capture the associated surface temperature and circulation anomalies. This results in an enhancement of Northern annular mode forecast skill compared to forecasts initialized during the cases when the stratospheric state is close to climatology. We demonstrate that the enhancement of skill for forecasts initialized during periods of strong vortex conditions is comparable to that achieved for forecasts initialized during weak events. This result indicates that additional confidence can be placed in sub-seasonal forecasts when the stratospheric polar vortex is significantly disturbed from its normal state.

The Climate-system Historical Forecast Project: do stratosphere-resolving models make better seasonal climate predictions in boreal winter?

Using an international, multi-model suite of historical forecasts from the World Climate Research Programme (WCRP) Climate-system Historical Forecast Project (CHFP), we compare the seasonal prediction skill in boreal wintertime between models that resolve the stratosphere and its dynamics (“high-top”) and models that do not (“low-top”). We evaluate hindcasts that are initialized in November, and examine the model biases in the stratosphere and how they relate to boreal wintertime (Dec-Mar) seasonal forecast skill. We are unable to detect more skill in the high-top ensemble-mean than the low-top ensemble-mean in forecasting the wintertime North Atlantic Oscillation, but model performance varies widely. Increasing the ensemble size clearly increases the skill for a given model. We then examine two major processes involving stratosphere-troposphere interactions (the El Niño-Southern Oscillation/ENSO and the Quasi-biennial Oscillation/QBO) and how they relate to predictive skill on intra-seasonal to seasonal timescales, particularly over the North Atlantic and Eurasia regions. High-top models tend to have a more realistic stratospheric response to El Niño and the QBO compared to low-top models. Enhanced conditional wintertime skill over high-latitudes and the North Atlantic region during winters with El Niño conditions suggests a possible role for a stratospheric pathway.

Willingness-to-pay for a probabilistic flood forecast: a risk-based decision-making game

Probabilistic hydro-meteorological forecasts have over the last decades been used more frequently to communicate forecastuncertainty. This uncertainty is twofold, as it constitutes both an added value and a challenge for the forecaster and the user of the forecasts. Many authors have demonstrated the added (economic) value of probabilistic over deterministic forecasts across the water sector (e.g. flood protection, hydroelectric power management and navigation). However, the richness of the information is also a source of challenges for operational uses, due partially to the difficulty to transform the probability of occurrence of an event into a binary decision. This paper presents the results of a risk-based decision-making game on the topic of flood protection mitigation, called “How much are you prepared to pay for a forecast?”. The game was played at several workshops in 2015, which were attended by operational forecasters and academics working in the field of hydrometeorology. The aim of this game was to better understand the role of probabilistic forecasts in decision-making processes and their perceived value by decision-makers. Based on the participants’ willingness-to-pay for a forecast, the results of the game show that the value (or the usefulness) of a forecast depends on several factors, including the way users perceive the quality of their forecasts and link it to the perception of their own performances as decision-makers.

Measuring expectations: forecast vs. ideal expectations. Does it really matter?

Consumer’s participation in service delivery is so central to cognition that it affects consumer’s quality evaluations. The study presented in this paper investigates the ways that visitor expectations change as a result of first hand experience with a service in the context of a major art exhibition. The research design allowed for two operational definitions of expectations, namely forecast and ideal expectations, in order to investigate differences between respondents’ pre and post experiences with a service. A total of 550 respondent visitors were interviewed during a major art exhibition, using two questionnaires delivered to two sub samples of respondents. The primary questionnaire was designed to capture recalled expectations after visitation while the parallel questionnaire captured forecast expectations prior to visitation and perceptions in the post experience phase. The findings suggest that forecast expectations were different to ideal expectations in both qualitative and quantitative ways and that these differences had important implications for perceptions of service quality. These differences can be explained, at least in part, by the way that expectations are formed and by the way that expectations are shaped by the actual visitation experience. For market researchers, the question of when and how to measure expectations has important implications for research design.

The forecast quality of CBOE implied volatility indexes

We examine the forecast quality of Chicago Board Options Exchange (CBOE) implied volatility indexes based on the Nasdaq 100 and Standard and Poor's 100 and 500 stock indexes. We find that the forecast quality of CBOE implied volatilities for the S&P 100 (VXO) and S&P 500 (VIX) has improved since 1995. Implied volatilities for the Nasdaq 100 (VXN) appear to provide even higher quality forecasts of future volatility. We further find that attenuation biases induced by the econometric problem of errors in variables appear to have largely disappeared from CBOE volatility index data since 1995.

Evaluation and comparison of type reduction algorithms from a forecast accuracy perspective

A variety of type reduction (TR) algorithms have been proposed for interval type-2 fuzzy logic systems (IT2 FLSs). The focus of existing literature is mainly on computational requirements of TR algorithm. Often researchers give more rewards to computationally less expensive TR algorithms. This paper evaluates and compares five frequently used TR algorithms from a forecasting performance perspective. Algorithms are judged based on the generalization power of IT2 FLS models developed using them. Four synthetic and real world case studies with different levels of uncertainty are considered to examine effects of TR algorithms on forecasts accuracies. It is found that Coupland-Jonh TR algorithm leads to models with a better forecasting performance. However, there is no clear relationship between the width of the type reduced set and TR algorithm.

State dependent asymmetric loss and the consensus forecast of real U.S. GDP growth

It has been well documented that the consensus forecast from surveys of professional forecasters shows a bias that varies over time. In this paper, we examine whether this bias may be due to forecasters having an asymmetric loss function. In contrast to previous research, we account for the time variation in the bias by making the loss function depend on the state of the economy. The asymmetry parameter in the loss function is specified to depend on set state variables which may cause forecaster to intentionally bias their forecasts. We consider both the Lin–Ex and asymmetric power loss functions. For the commonly used Lin–Ex and Lin–Lin loss functions, we show the model can be easily estimated by least squares. We apply our methodology to the consensus forecast of real U.S. GDP growth from the Survey of Professional Forecasters. We find that forecast uncertainty has an asymmetric effect on the asymmetry parameter in the loss function dependent upon whether the economy is in expansion or contraction. When the economy is in expansion, forecaster uncertainty is related to an overprediction in the median forecast of real GDP growth. In contrast, when the economy is in contraction, forecaster uncertainty is related to an underprediction in the median forecast of real GDP growth. Our results are robust to the particular loss function that is employed in the analysis.

Aggregation of Pi-based forecast to enhance prediction accuracy

In contrast to point forecast, prediction interval-based neural network offers itself as an effective tool to quantify the uncertainty and disturbances that associated with process data. However, single best neural network (NN) does not always guarantee to predict better quality of forecast for different data sets or a whole range of data set. Literature reported that ensemble of NNs using forecast combination produces stable and consistence forecast than single best NN. In this work, a NNs ensemble procedure is introduced to construct better quality of Pis. Weighted averaging forecasts combination mechanism is employed to combine the Pi-based forecast. As the key contribution of this paper, a new Pi-based cost function is proposed to optimize the individual weights for NN in combination process. An optimization algorithm, named simulated annealing (SA) is used to minimize the PI-based cost function. Finally, the proposed method is examined in two different case studies and compared the results with the individual best NNs and available simple averaging Pis aggregating method. Simulation results demonstrated that the proposed method improved the quality of Pis than individual best NNs and simple averaging ensemble method.

A novel fuzzy multi-objective framework to construct optimal prediction intervals for wind power forecast

The forecasting behavior of the high volatile and unpredictable wind power energy has always been a challenging issue in the power engineering area. In this regard, this paper proposes a new multi-objective framework based on fuzzy idea to construct optimal prediction intervals (Pis) to forecast wind power generation more sufficiently. The proposed method makes it possible to satisfy both the PI coverage probability (PICP) and PI normalized average width (PINAW), simultaneously. In order to model the stochastic and nonlinear behavior of the wind power samples, the idea of lower upper bound estimation (LUBE) method is used here. Regarding the optimization tool, an improved version of particle swam optimization (PSO) is proposed. In order to see the feasibility and satisfying performance of the proposed method, the practical data of a wind farm in Australia is used as the case study.

A panel data approach to economic forecasting: the bias-corrected average forecast

In this paper, we propose a novel approach to econometric forecasting of stationary and ergodic time series within a panel-data framework. Our key element is to employ the (feasible) bias-corrected average forecast. Using panel-data sequential asymptotics we show that it is potentially superior to other techniques in several contexts. In particular, it is asymptotically equivalent to the conditional expectation, i.e., has an optimal limiting mean-squared error. We also develop a zeromean test for the average bias and discuss the forecast-combination puzzle in small and large samples. Monte-Carlo simulations are conducted to evaluate the performance of the feasible bias-corrected average forecast in finite samples. An empirical exercise based upon data from a well known survey is also presented. Overall, theoretical and empirical results show promise for the feasible bias-corrected average forecast.