Finite sample weighting of recursive forecast errors

This paper proposes and tests a new framework for weighting recursive out-of-sample prediction errors according to their corresponding levels of in-sample estimation uncertainty. In essence, we show how to use the maximum possible amount of information from the sample in the evaluation of the prediction accuracy, by commencing the forecasts at the earliest opportunity and weighting the prediction errors. Via a Monte Carlo study, we demonstrate that the proposed framework selects the correct model from a set of candidate models considerably more often than the existing standard approach when only a small sample is available. We also show that the proposed weighting approaches result in tests of equal predictive accuracy that have much better sizes than the standard approach. An application to an exchange rate dataset highlights relevant differences in the results of tests of predictive accuracy based on the standard approach versus the framework proposed in this paper.

How is the balance of a forecast ensemble affected by adaptive and non-adaptive localization schemes?

This paper investigates the effect on balance of a number of Schur product-type localization schemes which have been designed with the primary function of reducing spurious far-field correlations in forecast error statistics. The localization schemes studied comprise a non-adaptive scheme (where the moderation matrix is decomposed in a spectral basis), and two adaptive schemes, namely a simplified version of SENCORP (Smoothed ENsemble COrrelations Raised to a Power) and ECO-RAP (Ensemble COrrelations Raised to A Power). The paper shows, we believe for the first time, how the degree of balance (geostrophic and hydrostatic) implied by the error covariance matrices localized by these schemes can be diagnosed. Here it is considered that an effective localization scheme is one that reduces spurious correlations adequately but also minimizes disruption of balance (where the 'correct' degree of balance or imbalance is assumed to be possessed by the unlocalized ensemble). By varying free parameters that describe each scheme (e.g. the degree of truncation in the schemes that use the spectral basis, the 'order' of each scheme, and the degree of ensemble smoothing), it is found that a particular configuration of the ECO-RAP scheme is best suited to the convective-scale system studied. According to our diagnostics this ECO-RAP configuration still weakens geostrophic and hydrostatic balance, but overall this is less so than for other schemes.

Towards improving the framework for probabilistic forecast evaluation

The evaluation of forecast performance plays a central role both in the interpretation and use of forecast systems and in their development. Different evaluation measures (scores) are available, often quantifying different characteristics of forecast performance. The properties of several proper scores for probabilistic forecast evaluation are contrasted and then used to interpret decadal probability hindcasts of global mean temperature. The Continuous Ranked Probability Score (CRPS), Proper Linear (PL) score, and IJ Good’s logarithmic score (also referred to as Ignorance) are compared; although information from all three may be useful, the logarithmic score has an immediate interpretation and is not insensitive to forecast busts. Neither CRPS nor PL is local; this is shown to produce counter intuitive evaluations by CRPS. Benchmark forecasts from empirical models like Dynamic Climatology place the scores in context. Comparing scores for forecast systems based on physical models (in this case HadCM3, from the CMIP5 decadal archive) against such benchmarks is more informative than internal comparison systems based on similar physical simulation models with each other. It is shown that a forecast system based on HadCM3 out performs Dynamic Climatology in decadal global mean temperature hindcasts; Dynamic Climatology previously outperformed a forecast system based upon HadGEM2 and reasons for these results are suggested. Forecasts of aggregate data (5-year means of global mean temperature) are, of course, narrower than forecasts of annual averages due to the suppression of variance; while the average “distance” between the forecasts and a target may be expected to decrease, little if any discernible improvement in probabilistic skill is achieved.

Atmospheric and oceanic contributions to irreducible forecast uncertainty of Arctic surface climate

Uncertainty of Arctic seasonal to interannual predictions arising from model errors and initial state uncertainty has been widely discussed in the literature, whereas the irreducible forecast uncertainty (IFU) arising from the chaoticity of the climate system has received less attention. However, IFU provides important insights into the mechanisms through which predictability is lost, and hence can inform prioritization of model development and observations deployment. Here, we characterize how internal oceanic and surface atmospheric heat fluxes contribute to IFU of Arctic sea ice and upper ocean heat content in an Earth system model by analyzing a set of idealized ensemble prediction experiments. We find that atmospheric and oceanic heat flux are often equally important for driving unpredictable Arctic-wide changes in sea ice and surface water temperatures, and hence contribute equally to IFU. Atmospheric surface heat flux tends to dominate Arctic-wide changes for lead times of up to a year, whereas oceanic heat flux tends to dominate regionally and on interannual time scales. There is in general a strong negative covariance between surface heat flux and ocean vertical heat flux at depth, and anomalies of lateral ocean heat transport are wind-driven, which suggests that the unpredictable oceanic heat flux variability is mainly forced by the atmosphere. These results are qualitatively robust across different initial states, but substantial variations in the amplitude of IFU exist. We conclude that both atmospheric variability and the initial state of the upper ocean are key ingredients for predictions of Arctic surface climate on seasonal to interannual time scales.

Enhanced long-range forecast skill in boreal winter following stratospheric strong vortex conditions

There has been a great deal of recent interest in producing weather forecasts on the 2–6 week sub-seasonal timescale, which bridges the gap between medium-range (0–10 day) and seasonal (3–6 month) forecasts. While much of this interest is focused on the potential applications of skilful forecasts on the sub-seasonal range, understanding the potential sources of sub-seasonal forecast skill is a challenging and interesting problem, particularly because of the likely state-dependence of this skill (Hudson et al 2011). One such potential source of state-dependent skill for the Northern Hemisphere in winter is the occurrence of stratospheric sudden warming (SSW) events (Sigmond et al 2013). Here we show, by analysing a set of sub-seasonal hindcasts, that there is enhanced predictability of surface circulation not only when the stratospheric vortex is anomalously weak following SSWs but also when the vortex is extremely strong. Sub-seasonal forecasts initialized during strong vortex events are able to successfully capture the associated surface temperature and circulation anomalies. This results in an enhancement of Northern annular mode forecast skill compared to forecasts initialized during the cases when the stratospheric state is close to climatology. We demonstrate that the enhancement of skill for forecasts initialized during periods of strong vortex conditions is comparable to that achieved for forecasts initialized during weak events. This result indicates that additional confidence can be placed in sub-seasonal forecasts when the stratospheric polar vortex is significantly disturbed from its normal state.

The Climate-system Historical Forecast Project: do stratosphere-resolving models make better seasonal climate predictions in boreal winter?

Using an international, multi-model suite of historical forecasts from the World Climate Research Programme (WCRP) Climate-system Historical Forecast Project (CHFP), we compare the seasonal prediction skill in boreal wintertime between models that resolve the stratosphere and its dynamics (“high-top”) and models that do not (“low-top”). We evaluate hindcasts that are initialized in November, and examine the model biases in the stratosphere and how they relate to boreal wintertime (Dec-Mar) seasonal forecast skill. We are unable to detect more skill in the high-top ensemble-mean than the low-top ensemble-mean in forecasting the wintertime North Atlantic Oscillation, but model performance varies widely. Increasing the ensemble size clearly increases the skill for a given model. We then examine two major processes involving stratosphere-troposphere interactions (the El Niño-Southern Oscillation/ENSO and the Quasi-biennial Oscillation/QBO) and how they relate to predictive skill on intra-seasonal to seasonal timescales, particularly over the North Atlantic and Eurasia regions. High-top models tend to have a more realistic stratospheric response to El Niño and the QBO compared to low-top models. Enhanced conditional wintertime skill over high-latitudes and the North Atlantic region during winters with El Niño conditions suggests a possible role for a stratospheric pathway.

Willingness-to-pay for a probabilistic flood forecast: a risk-based decision-making game

Probabilistic hydro-meteorological forecasts have over the last decades been used more frequently to communicate forecastuncertainty. This uncertainty is twofold, as it constitutes both an added value and a challenge for the forecaster and the user of the forecasts. Many authors have demonstrated the added (economic) value of probabilistic over deterministic forecasts across the water sector (e.g. flood protection, hydroelectric power management and navigation). However, the richness of the information is also a source of challenges for operational uses, due partially to the difficulty to transform the probability of occurrence of an event into a binary decision. This paper presents the results of a risk-based decision-making game on the topic of flood protection mitigation, called “How much are you prepared to pay for a forecast?”. The game was played at several workshops in 2015, which were attended by operational forecasters and academics working in the field of hydrometeorology. The aim of this game was to better understand the role of probabilistic forecasts in decision-making processes and their perceived value by decision-makers. Based on the participants’ willingness-to-pay for a forecast, the results of the game show that the value (or the usefulness) of a forecast depends on several factors, including the way users perceive the quality of their forecasts and link it to the perception of their own performances as decision-makers.

A note on atmospheric predictability

Using the method of Lorenz (1982), we have estimated the predictability of a recent version of the European Center for Medium-Range Weather Forecasting (ECMWF) model using two different estimates of the initial error corresponding to 6- and 24-hr forecast errors, respectively. For a 6-hr forecast error of the extratropical 500-hPa geopotential height field, a potential increase in forecast skill by more than 3 d is suggested, indicating a further increase in predictability by another 1.5 d compared to the use of a 24-hr forecast error. This is due to a smaller initial error and to an initial error reduction resulting in a smaller averaged growth rate for the whole 7-d forecast. A similar assessment for the tropics using the wind vector fields at 850 and 250 hPa suggests a huge potential improvement with a 7-d forecast providing the same skill as a 1-d forecast now. A contributing factor to the increase in the estimate of predictability is the apparent slow increase of error during the early part of the forecast.

On the impact of humidity observations in numerical weather prediction

The impact of humidity observations on forecast skill is explored by producing a series of global forecasts using initial data derived from the ERA-40 reanalyses system, in which all humidity data have been removed during the data assimilation. The new forecasts have been compared with the original ERA-40 analyses and forecasts made from them. Both sets of forecasts show virtually identical prediction skill in the extratropics and the tropics. Differences between the forecasts are small and undergo characteristic amplification rate. There are larger differences in temperature and geopotential in the tropics but the differences are small-scale and unstructured and have no noticeable effect on the skill of the wind forecasts. The results highlight the current very limited impact of the humidity observations, used to produce the initial state, on the forecasts.

The predictability of extratropical storm tracks and the sensitivity of their prediction to the observing system

A new method for assessing forecast skill and predictability that involves the identification and tracking of extratropical cyclones has been developed and implemented to obtain detailed information about the prediction of cyclones that cannot be obtained from more conventional analysis methodologies. The cyclones were identified and tracked along the forecast trajectories, and statistics were generated to determine the rate at which the position and intensity of the forecasted storms diverge from the analyzed tracks as a function of forecast lead time. The results show a higher level of skill in predicting the position of extratropical cyclones than the intensity. They also show that there is potential to improve the skill in predicting the position by 1 - 1.5 days and the intensity by 2 - 3 days, via improvements to the forecast model. Further analysis shows that forecasted storms move at a slower speed than analyzed storms on average and that there is a larger error in the predicted amplitudes of intense storms than the weaker storms. The results also show that some storms can be predicted up to 3 days before they are identified as an 850-hPa vorticity center in the analyses. In general, the results show a higher level of skill in the Northern Hemisphere (NH) than the Southern Hemisphere (SH); however, the rapid growth of NH winter storms is not very well predicted. The impact that observations of different types have on the prediction of the extratropical cyclones has also been explored, using forecasts integrated from analyses that were constructed from reduced observing systems. A terrestrial, satellite, and surface-based system were investigated and the results showed that the predictive skill of the terrestrial system was superior to the satellite system in the NH. Further analysis showed that the satellite system was not very good at predicting the growth of the storms. In the SH the terrestrial system has significantly less skill than the satellite system, highlighting the dominance of satellite observations in this hemisphere. The surface system has very poor predictive skill in both hemispheres.

The prediction of extratropical storm tracks by the ECMWF and NCEP ensemble prediction systems

The prediction of extratropical cyclones by the European Centre for Medium Range Weather Forecasts (ECMWF) and the National Centers for Environmental Prediction (NCEP) Ensemble Prediction Systems (EPS) has been investigated using an objective feature tracking methodology to identify and track the cyclones along the forecast trajectories. Overall the results show that the ECMWF EPS has a slightly higher level of skill than the NCEP EPS in the northern hemisphere (NH). However in the southern hemisphere (SH), NCEP has higher predictive skill than ECMWF for the intensity of the cyclones. The results from both EPS indicate a higher level of predictive skill for the position of extratropical cyclones than their intensity and show that there is a larger spread in intensity than position. Further analysis shows that the predicted propagation speed of cyclones is generally too slow for the ECMWF EPS and show a slight bias for the intensity of the cyclones to be overpredicted. This is also true for the NCEP EPS in the SH. For the NCEP EPS in the NH the intensity of the cyclones is underpredicted. There is small bias in both the EPS for the cyclones to be displaced towards the poles. For each ensemble forecast of each cyclone, the predictive skill of the ensemble member that best predicts the cyclones position and intensity was computed. The results are very encouraging showing that the predictive skill of the best ensemble member is significantly higher than that of the control forecast in terms of both the position and intensity of the cyclones. The prediction of cyclones before they are identified as 850 hPa vorticity centers in the analysis cycle was also considered. It is shown that an indication of extratropical cyclones can be given by at least 1 ensemble member 7 days before they are identified in the analysis. Further analysis of the ECMWF EPS shows that the ensemble mean has a higher level of skill than the control forecast, particularly for the intensity of the cyclones, 2 from day 3 of the forecast. There is a higher level of skill in the NH than the SH and the spread in the SH is correspondingly larger. The difference between the ensemble mean and spread is very small for the position of the cyclones, but the spread of the ensemble is smaller than the ensemble mean error for the intensity of the cyclones in both hemispheres. Results also show that the ECMWF control forecast has ½ to 1 day more skill than the perturbed members, for both the position and intensity of the cyclones, throughout the forecast.

Storm tracking and ensemble prediction

The prediction of extratropical cyclones by the European Centre for Medium Range Weather Forecasts (ECMWF) and the National Centers for Environmental Prediction (NCEP) Ensemble Prediction Systems (EPS) is investigated using a storm-tracking forecast verifica-tion methodology. The cyclones are identified and tracked along the forecast trajectories so that statistics can be generated to determine the rate at which the position and intensity of the forecasted cyclones diverge from the corresponding analysed cyclones with forecast time. Overall the ECMWF EPS has a slightly higher level of performance than the NCEP EPS. However, in the southern hemisphere the NCEP EPS has a slightly higher level of skill for the intensity of the storms. The results from both EPS indicate a higher level of predictive skill for the position of extratropical cyclones than their intensity and show that there is a larger spread in intensity than position. The results also illustrate several benefits an EPS can offer over a deterministic forecast.

Improved simulation of water vapour and clear-sky radiation using 24 hour forecasts from ERA40

Monthly mean water vapour and clear-sky radiation extracted from the European Centre for Medium Range Weather Forecasts 40-year reanalysis (ERA40) forecasts are assessed using satellite observations and additional reanalysis data. There is a marked improvement in the interannual variability of column-integrated water vapour (CWV) over the oceans when using the 24-hour forecasts compared with the standard 6-hour forecasts products. The spatial distribution of CWV are well simulated by the 6-hour forecasts; using the 24-hour forecasts does not degrade this simulation substantially and in many cases improves on the quality. There is also an improved simulation of clear-sky radiation from the 24-hour forecasts compared with the 6-hour forecasts based on comparison with satellite observations and empirical estimates. Further work is required to assess the quality of water vapour simulation by reanalyses over land regions. Over the oceans, it is recommended that 24-hour forecasts of CWV and clear-sky radiation are used in preference to the standard 6-hour forecast products from ERA40

Life-cycle simulations of shallow frontal waves and the impact of deformation strain

The life-cycle of shallow frontal waves and the impact of deformation strain on their development is investigated using the idealised version of the Met Office non-hydrostatic Unified Model which includes the same physics and dynamics as the operational forecast model. Frontal wave development occurs in two stages; first, a deformation strain is applied to a front and a positive potential vorticity (PV) strip forms, generated by latent heat release in the frontal updraft; second, as the deformation strain is reduced the PV strip breaks up into individual anomalies. The circulations associated with the PV anomalies cause shallow frontal waves to form. The structure of the simulated frontal waves is consistent with the conceptual model of a frontal cyclone. Deeper frontal waves are simulated if the stability of the atmosphere is reduced. Deformation strain rates of different strengths are applied to the PV strip to determine whether a deformation strain threshold exists above which frontal wave development is suppressed. An objective method of frontal wave activity is defined and frontal wave development was found to be suppressed by deformation strain rates $\ge 0.4\times10^{-5}\mbox{s}^{-1}$. This value compares well with observed deformation strain rate thresholds and the analytical solution for the minimum deformation strain rate needed to suppress barotropic frontal wave development. The deformation strain rate threshold is dependent on the strength of the PV strip with strong PV strips able to overcome stronger deformation strain rates (leading to frontal wave development) than weaker PV strips.

A high-order arbitrarily unstructured finite-volume model of the global atmosphere: tests solving the shallow-water equations

Simulations of the global atmosphere for weather and climate forecasting require fast and accurate solutions and so operational models use high-order finite differences on regular structured grids. This precludes the use of local refinement; techniques allowing local refinement are either expensive (eg. high-order finite element techniques) or have reduced accuracy at changes in resolution (eg. unstructured finite-volume with linear differencing). We present solutions of the shallow-water equations for westerly flow over a mid-latitude mountain from a finite-volume model written using OpenFOAM. A second/third-order accurate differencing scheme is applied on arbitrarily unstructured meshes made up of various shapes and refinement patterns. The results are as accurate as equivalent resolution spectral methods. Using lower order differencing reduces accuracy at a refinement pattern which allows errors from refinement of the mountain to accumulate and reduces the global accuracy over a 15 day simulation. We have therefore introduced a scheme which fits a 2D cubic polynomial approximately on a stencil around each cell. Using this scheme means that refinement of the mountain improves the accuracy after a 15 day simulation. This is a more severe test of local mesh refinement for global simulations than has been presented but a realistic test if these techniques are to be used operationally. These efficient, high-order schemes may make it possible for local mesh refinement to be used by weather and climate forecast models.