Forecast convergence score: a forecaster's approach to analysing hydro-meteorological forecast systems.

In this paper the properties of a hydro-meteorological forecasting system for forecasting river flows have been analysed using a probabilistic forecast convergence score (FCS). The focus on fixed event forecasts provides a forecaster's approach to system behaviour and adds an important perspective to the suite of forecast verification tools commonly used in this field. A low FCS indicates a more consistent forecast. It can be demonstrated that the FCS annual maximum decreases over the last 10 years. With lead time, the FCS of the ensemble forecast decreases whereas the control and high resolution forecast increase. The FCS is influenced by the lead time, threshold and catchment size and location. It indicates that one should use seasonality based decision rules to issue flood warnings.

Proceedings of the 1998 Science Board Symposium on The Impacts of the 1997/98 El Niño Event on the North Pacific Ocean and Its Marginal Seas

Preface [pdf, 0.01 Mb] James J. O'Brien The big picture - The ENSO of 1997-98 [pdf, 0.01 Mb] James E. Overland, Nicholas A. Bond & Jennifer Miletta Adams Atmospheric anomalies in 1997: Links to ENSO? [pdf, 0.54 Mb] Vladimir I. Ponomarev, Olga Trusenkova, Serge Trousenkov, Dmitry Kaplunenko, Elena Ustinova & Antonina Polyakova The ENSO signal in the northwest Pacific [pdf, 0.47 Mb] Robert L. Smith, A. Huyer, P.M. Kosro & J.A. Barth Observations of El Niño off Oregon: July 1997 to present (October 1998) [pdf, 1.31 Mb] Patrica A. Wheeler & Jon Hill Biological effects of the 1997-1998 El Niño event off Oregon: Nutrient and chlorophyll distributions [pdf, 1.13 Mb] William T. Peterson Hydrography and zooplankton off the central Oregon coast during the 1997-1998 El Niño event [pdf, 0.26 Mb] William Crawford, Josef Cherniawsky, Michael Foreman & Peter Chandler El Niño sea level signal along the west coast of Canada [pdf, 1.25 Mb] Howard J. Freeland & Rick Thomson The El Niño signal along the west coast of Canada - temperature, salinity and velocity [pdf, 0.49 Mb] Frank A. Whitney, David L. Mackas, David W. Welch & Marie Robert Impact of the 1990s El Niños on nutrient supply and productivity of Gulf of Alaska waters [pdf, 0.06 Mb] Craig McNeil, David Farmer & Mark Trevorrow Dissolved gas measurements at Stn. P4 during the 97-98 El Niño [pdf, 0.13 Mb] Kristen L.D. Milligan, Colin D. Levings & Robert E. DeWreede Data compilation and preliminary time series analysis of abundance of a dominant intertidal kelp species in relation to the 1997/1998 El Niño event [pdf, 0.05 Mb] S.M. McKinnell, C.C. Wood, M. Lapointe, J.C. Woodey, K.E. Kostow, J. Nelson & K.D. Hyatt Reviewing the evidence that adult sockeye salmon strayed from the Fraser River and spawned in other rivers in 1997 [pdf,0.03 Mb] G.A. McFarlane & R.J. Beamish Sardines return to British Columbia waters [pdf, 0.34 Mb] Ken H. Morgan Impact of the 1997/98 El Niño on seabirds of the northeast Pacific [pdf, 0.06 Mb] Thomas C. Royer & Thomas Weingartner Coastal hydrographic responses in the northern Gulf of Alaska to the 1997-98 ENSO event [pdf, 0.76 Mb] John F. Piatt, Gary Drew, Thomas Van Pelt, Alisa Abookire, April Nielsen, Mike Shultz & Alexander Kitaysky Biological effects of the 1997/98 ENSO in Cook Inlet, Alaska [pdf, 0.22 Mb] H.J. Niebauer The 1997-98 El Niño in the Bering Sea as compared with previous ENSO events and the "regime shift" of the late 1970s [pdf, 0.10 Mb] A.S. Krovnin, G.P. Nanyushin, M.Yu. Kruzhalov, G.V. Khen, M.A. Bogdanov, E.I. Ustinova, V.V. Maslennikov, A.M. Orlov, B.N. Kotenev, V.V. Bulanov & G.P. Muriy The state of the Far East seas during the 1997/98 El Niño event [pdf, 0.15 Mb] Stacy Smith & Susan Henrichs Phytoplankton collected by a time-series sediment trap deployed in the southeast Bering Sea during 1997 [pdf, 0.21 Mb] Cynthia T. Tynan Redistributions of cetaceans in the southeast Bering Sea relative to anomalous oceanographic conditions during the 1997 El Niño [pdf, 0.02 Mb] Akihiko Yatsu, Junta Mori, Hiroyuki Tanaka, Tomowo Watanabe, Kazuya Nagasawa, Yikimasa Ishida, Toshimi Meguro, Yoshihiko Kamei & Yasunori Sakurai Stock abundance and size compositions of the neon flying squid in the central North Pacific Ocean during 1979-1998 [pdf, 0.11 Mb] O.B. Feschenko A new point of view concerning the El Niño mechanism [pdf, 0.01 Mb] Nathan Mantua 97/98 Ocean climate variability in the northeast Pacific: How much blame does El Niño deserve? [pdf, 0.01 Mb] Vadim P. Pavlychev Sharp changes of hydrometeorological conditions in the northwestern Pacific during the 1997/1998 El Niño event [pdf, 0.01 Mb] Jingyi Wang Predictability and forecast verification of El Niño events [pdf, 0.01 Mb] (Document contains 110 pages)

TIGGE: Comparison of the prediction of northern hemisphere extratropical cyclones by different ensemble prediction systems

The Observing System Research and Predictability Experiment (THORPEX) Interactive Grand Global Ensemble (TIGGE) is a World Weather Research Programme project. One of its main objectives is to enhance collaboration on the development of ensemble prediction between operational centers and universities by increasing the availability of ensemble prediction system (EPS) data for research. This study analyzes the prediction of Northern Hemisphere extratropical cyclones by nine different EPSs archived as part of the TIGGE project for the 6-month time period of 1 February 2008–31 July 2008, which included a sample of 774 cyclones. An objective feature tracking method has been used to identify and track the cyclones along the forecast trajectories. Forecast verification statistics have then been produced [using the European Centre for Medium-Range Weather Forecasts (ECMWF) operational analysis as the truth] for cyclone position, intensity, and propagation speed, showing large differences between the different EPSs. The results show that the ECMWF ensemble mean and control have the highest level of skill for all cyclone properties. The Japanese Meteorological Administration (JMA), the National Centers for Environmental Prediction (NCEP), the Met Office (UKMO), and the Canadian Meteorological Centre (CMC) have 1 day less skill for the position of cyclones throughout the forecast range. The relative performance of the different EPSs remains the same for cyclone intensity except for NCEP, which has larger errors than for position. NCEP, the Centro de Previsão de Tempo e Estudos Climáticos (CPTEC), and the Australian Bureau of Meteorology (BoM) all have faster intensity error growth in the earlier part of the forecast. They are also very underdispersive and significantly underpredict intensities, perhaps due to the comparatively low spatial resolutions of these EPSs not being able to accurately model the tilted structure essential to cyclone growth and decay. There is very little difference between the levels of skill of the ensemble mean and control for cyclone position, but the ensemble mean provides an advantage over the control for all EPSs except CPTEC in cyclone intensity and there is an advantage for propagation speed for all EPSs. ECMWF and JMA have an excellent spread–skill relationship for cyclone position. The EPSs are all much more underdispersive for cyclone intensity and propagation speed than for position, with ECMWF and CMC performing best for intensity and CMC performing best for propagation speed. ECMWF is the only EPS to consistently overpredict cyclone intensity, although the bias is small. BoM, NCEP, UKMO, and CPTEC significantly underpredict intensity and, interestingly, all the EPSs underpredict the propagation speed, that is, the cyclones move too slowly on average in all EPSs.

A new error measure for forecasts of household-level, high resolution electrical energy consumption

As low carbon technologies become more pervasive, distribution network operators are looking to support the expected changes in the demands on the low voltage networks through the smarter control of storage devices. Accurate forecasts of demand at the single household-level, or of small aggregations of households, can improve the peak demand reduction brought about through such devices by helping to plan the appropriate charging and discharging cycles. However, before such methods can be developed, validation measures are required which can assess the accuracy and usefulness of forecasts of volatile and noisy household-level demand. In this paper we introduce a new forecast verification error measure that reduces the so called “double penalty” effect, incurred by forecasts whose features are displaced in space or time, compared to traditional point-wise metrics, such as Mean Absolute Error and p-norms in general. The measure that we propose is based on finding a restricted permutation of the original forecast that minimises the point wise error, according to a given metric. We illustrate the advantages of our error measure using half-hourly domestic household electrical energy usage data recorded by smart meters and discuss the effect of the permutation restriction.

Economic Bias of Weather Forecasting: A Spatial Modeling Approach

The value of accurate weather forecast information is substantial. In this paper we examine competition among forecast providers and its implications for the quality of forecasts. A simple economic model shows that an economic bias geographical inequality in forecast accuracy arises due to the extent of the market. Using the unique data on daily high temperature forecasts for 704 U.S. cities, we find that forecast accuracy increases with population and income. Furthermore, the economic bias gets larger when the day of forecasting is closer to the target day; i.e. when people are more concerned about the quality of forecasts. The results hold even after we control for location-specific heterogeneity and difficulty of forecasting.

Coherence and Elicitability

The risk of a financial position is usually summarized by a risk measure. As this risk measure has to be estimated from historical data, it is important to be able to verify and compare competing estimation procedures. In statistical decision theory, risk measures for which such verification and comparison is possible, are called elicitable. It is known that quantile-based risk measures such as value at risk are elicitable. In this paper, the existing result of the nonelicitability of expected shortfall is extended to all law-invariant spectral risk measures unless they reduce to minus the expected value. Hence, it is unclear how to perform forecast verification or comparison. However, the class of elicitable law-invariant coherent risk measures does not reduce to minus the expected value. We show that it consists of certain expectiles.

Verification of cloud-fraction forecasts

Cloud radar and lidar can be used to evaluate the skill of numerical weather prediction models in forecasting the timing and placement of clouds, but care must be taken in choosing the appropriate metric of skill to use due to the non- Gaussian nature of cloud-fraction distributions. We compare the properties of a number of different verification measures and conclude that of existing measures the Log of Odds Ratio is the most suitable for cloud fraction. We also propose a new measure, the Symmetric Extreme Dependency Score, which has very attractive properties, being equitable (for large samples), difficult to hedge and independent of the frequency of occurrence of the quantity being verified. We then use data from five European ground-based sites and seven forecast models, processed using the ‘Cloudnet’ analysis system, to investigate the dependence of forecast skill on cloud fraction threshold (for binary skill scores), height, horizontal scale and (for the Met Office and German Weather Service models) forecast lead time. The models are found to be least skillful at predicting the timing and placement of boundary-layer clouds and most skilful at predicting mid-level clouds, although in the latter case they tend to underestimate mean cloud fraction when cloud is present. It is found that skill decreases approximately inverse-exponentially with forecast lead time, enabling a forecast ‘half-life’ to be estimated. When considering the skill of instantaneous model snapshots, we find typical values ranging between 2.5 and 4.5 days. Copyright c 2009 Royal Meteorological Society

A verification framework for interannual-to-decadal predictions experiments

Decadal predictions have a high profile in the climate science community and beyond, yet very little is known about their skill. Nor is there any agreed protocol for estimating their skill. This paper proposes a sound and coordinated framework for verification of decadal hindcast experiments. The framework is illustrated for decadal hindcasts tailored to meet the requirements and specifications of CMIP5 (Coupled Model Intercomparison Project phase 5). The chosen metrics address key questions about the information content in initialized decadal hindcasts. These questions are: (1) Do the initial conditions in the hindcasts lead to more accurate predictions of the climate, compared to un-initialized climate change projections? and (2) Is the prediction model’s ensemble spread an appropriate representation of forecast uncertainty on average? The first question is addressed through deterministic metrics that compare the initialized and uninitialized hindcasts. The second question is addressed through a probabilistic metric applied to the initialized hindcasts and comparing different ways to ascribe forecast uncertainty. Verification is advocated at smoothed regional scales that can illuminate broad areas of predictability, as well as at the grid scale, since many users of the decadal prediction experiments who feed the climate data into applications or decision models will use the data at grid scale, or downscale it to even higher resolution. An overall statement on skill of CMIP5 decadal hindcasts is not the aim of this paper. The results presented are only illustrative of the framework, which would enable such studies. However, broad conclusions that are beginning to emerge from the CMIP5 results include (1) Most predictability at the interannual-to-decadal scale, relative to climatological averages, comes from external forcing, particularly for temperature; (2) though moderate, additional skill is added by the initial conditions over what is imparted by external forcing alone; however, the impact of initialization may result in overall worse predictions in some regions than provided by uninitialized climate change projections; (3) limited hindcast records and the dearth of climate-quality observational data impede our ability to quantify expected skill as well as model biases; and (4) as is common to seasonal-to-interannual model predictions, the spread of the ensemble members is not necessarily a good representation of forecast uncertainty. The authors recommend that this framework be adopted to serve as a starting point to compare prediction quality across prediction systems. The framework can provide a baseline against which future improvements can be quantified. The framework also provides guidance on the use of these model predictions, which differ in fundamental ways from the climate change projections that much of the community has become familiar with, including adjustment of mean and conditional biases, and consideration of how to best approach forecast uncertainty.

Improving the evaluation of hydrological multi-model forecast performance in the Upper Danube Catchment

Medium range flood forecasting activities, driven by various meteorological forecasts ranging from high resolution deterministic forecasts to low spatial resolution ensemble prediction systems, share a major challenge in the appropriateness and design of performance measures. In this paper possible limitations of some traditional hydrological and meteorological prediction quality and verification measures are identified. Some simple modifications are applied in order to circumvent the problem of the autocorrelation dominating river discharge time-series and in order to create a benchmark model enabling the decision makers to evaluate the forecast quality and the model quality. Although the performance period is quite short the advantage of a simple cost-loss function as a measure of forecast quality can be demonstrated.

Verification of European subseasonal wind speed forecasts

Analysis of the forecasts and hindcasts from the ECMWF 32-day forecast model reveals that there is statistically significant skill in predicting weekly mean wind speeds over areas of Europe at lead times of at least 14–20 days. Previous research on wind speed predictability has focused on the short- to medium-range time scales, typically finding that forecasts lose all skill by the later part of the medium-range forecast. To the authors’ knowledge, this research is the first to look beyond the medium-range time scale by taking weekly mean wind speeds, instead of averages at hourly or daily resolution, for the ECMWF monthly forecasting system. It is shown that the operational forecasts have high levels of correlation (~0.6) between the forecasts and observations over the winters of 2008–12 for some areas of Europe. Hindcasts covering 20 winters show a more modest level of correlation but are still skillful. Additional analysis examines the probabilistic skill for the United Kingdom with the application of wind power forecasting in mind. It is also shown that there is forecast “value” for end users (operating in a simple cost/loss ratio decision-making framework). End users that are sensitive to winter wind speed variability over the United Kingdom, Germany, and some other areas of Europe should therefore consider forecasts beyond the medium-range time scale as it is clear there is useful information contained within the forecast.