A verification framework for interannual-to-decadal predictions experiments

Decadal predictions have a high profile in the climate science community and beyond, yet very little is known about their skill. Nor is there any agreed protocol for estimating their skill. This paper proposes a sound and coordinated framework for verification of decadal hindcast experiments. The framework is illustrated for decadal hindcasts tailored to meet the requirements and specifications of CMIP5 (Coupled Model Intercomparison Project phase 5). The chosen metrics address key questions about the information content in initialized decadal hindcasts. These questions are: (1) Do the initial conditions in the hindcasts lead to more accurate predictions of the climate, compared to un-initialized climate change projections? and (2) Is the prediction model’s ensemble spread an appropriate representation of forecast uncertainty on average? The first question is addressed through deterministic metrics that compare the initialized and uninitialized hindcasts. The second question is addressed through a probabilistic metric applied to the initialized hindcasts and comparing different ways to ascribe forecast uncertainty. Verification is advocated at smoothed regional scales that can illuminate broad areas of predictability, as well as at the grid scale, since many users of the decadal prediction experiments who feed the climate data into applications or decision models will use the data at grid scale, or downscale it to even higher resolution. An overall statement on skill of CMIP5 decadal hindcasts is not the aim of this paper. The results presented are only illustrative of the framework, which would enable such studies. However, broad conclusions that are beginning to emerge from the CMIP5 results include (1) Most predictability at the interannual-to-decadal scale, relative to climatological averages, comes from external forcing, particularly for temperature; (2) though moderate, additional skill is added by the initial conditions over what is imparted by external forcing alone; however, the impact of initialization may result in overall worse predictions in some regions than provided by uninitialized climate change projections; (3) limited hindcast records and the dearth of climate-quality observational data impede our ability to quantify expected skill as well as model biases; and (4) as is common to seasonal-to-interannual model predictions, the spread of the ensemble members is not necessarily a good representation of forecast uncertainty. The authors recommend that this framework be adopted to serve as a starting point to compare prediction quality across prediction systems. The framework can provide a baseline against which future improvements can be quantified. The framework also provides guidance on the use of these model predictions, which differ in fundamental ways from the climate change projections that much of the community has become familiar with, including adjustment of mean and conditional biases, and consideration of how to best approach forecast uncertainty.

Scale-selective verification of rainfall accumulations from high-resolution forecasts of convective events

The development of NWP models with grid spacing down to 1 km should produce more realistic forecasts of convective storms. However, greater realism does not necessarily mean more accurate precipitation forecasts. The rapid growth of errors on small scales in conjunction with preexisting errors on larger scales may limit the usefulness of such models. The purpose of this paper is to examine whether improved model resolution alone is able to produce more skillful precipitation forecasts on useful scales, and how the skill varies with spatial scale. A verification method will be described in which skill is determined from a comparison of rainfall forecasts with radar using fractional coverage over different sized areas. The Met Office Unified Model was run with grid spacings of 12, 4, and 1 km for 10 days in which convection occurred during the summers of 2003 and 2004. All forecasts were run from 12-km initial states for a clean comparison. The results show that the 1-km model was the most skillful over all but the smallest scales (approximately <10–15 km). A measure of acceptable skill was defined; this was attained by the 1-km model at scales around 40–70 km, some 10–20 km less than that of the 12-km model. The biggest improvement occurred for heavier, more localized rain, despite it being more difficult to predict. The 4-km model did not improve much on the 12-km model because of the difficulties of representing convection at that resolution, which was accentuated by the spinup from 12-km fields.

Assessing the spatial and temporal variation in the skill of precipitation forecasts from an NWP model

It is becoming increasingly important to be able to verify the spatial accuracy of precipitation forecasts, especially with the advent of high-resolution numerical weather prediction (NWP) models. In this article, the fractions skill score (FSS) approach has been used to perform a scale-selective evaluation of precipitation forecasts during 2003 from the Met Office mesoscale model (12 km grid length). The investigation shows how skill varies with spatial scale, the scales over which the data assimilation (DA) adds most skill, and how the loss of that skill is dependent on both the spatial scale and the rainfall coverage being examined. Although these results come from a specific model, they demonstrate how this verification approach can provide a quantitative assessment of the spatial behaviour of new finer-resolution models and DA techniques.

Quantitative genetic divergence and standing genetic (co)variance in thermal reaction norms along latitude

Although the potential to adapt to warmer climate is constrained by genetic trade-offs, our understanding of how selection and mutation shape genetic (co)variances in thermal reaction norms is poor. Using 71 isofemale lines of the fly Sepsis punctum, originating from northern, central, and southern European climates, we tested for divergence in juvenile development rate across latitude at five experimental temperatures. To investigate effects of evolutionary history in different climates on standing genetic variation in reaction norms, we further compared genetic (co)variances between regions. Flies were reared on either high or low food resources to explore the role of energy acquisition in determining genetic trade-offs between different temperatures. Although the latter had only weak effects on the strength and sign of genetic correlations, genetic architecture differed significantly between climatic regions, implying that evolution of reaction norms proceeds via different trajectories at high latitude versus low latitude in this system. Accordingly, regional genetic architecture was correlated to region-specific differentiation. Moreover, hot development temperatures were associated with low genetic variance and stronger genetic correlations compared to cooler temperatures. We discuss the evolutionary potential of thermal reaction norms in light of their underlying genetic architectures, evolutionary histories, and the materialization of trade-offs in natural environments.

Quantitative estimates of glacial and Holocene temperature and precipitation change in lowland Amazonian Bolivia

Quantitative estimates of temperature and precipitation change during the late Pleistocene and Holocene have been difficult to obtain for much of the lowland Neotropics. Using two published lacustrine pollen records and a climate-vegetation model based on the modern abundance distributions of 154 Neotropical plant families, we demonstrate how family-level counts of fossil pollen can be used to quantitatively reconstruct tropical paleoclimate and provide needed information on historic patterns of climatic change. With this family-level analysis, we show that one area of the lowland tropics, northeastern Bolivia, experienced cooling (1–3 °C) and drying (400 mm/yr), relative to present, during the late Pleistocene (50,000–12,000 calendar years before present [cal. yr B.P.]). Immediately prior to the Last Glacial Maximum (LGM, ca. 21,000 cal. yr B.P.), we observe a distinct transition from cooler temperatures and variable precipitation to a period of warmer temperatures and relative dryness that extends to the middle Holocene (5000–3000 cal. yr B.P.). This prolonged reduction in precipitation occurs against the backdrop of increasing atmospheric CO2 concentrations, indicating that the presence of mixed savanna and dry-forest communities in northeastern Bolivia durng the LGM was not solely the result of low CO2 levels, as suggested previously, but also lower precipitation. The results of our analysis demonstrate the potential for using the distribution and abundance structure of modern Neotropical plant families to infer paleoclimate from the fossil pollen record.

A global quantitative synthesis of local and landscape effects on wild bee pollinators in agroecosystems

Bees provide essential pollination services that are potentially affected both by local farm management and the surrounding landscape. To better understand these different factors, we modelled the relative effects of landscape composition (nesting and floral resources within foraging distances), landscape configuration (patch shape, interpatch connectivity and habitat aggregation) and farm management (organic vs. conventional and local-scale field diversity), and their interactions, on wild bee abundance and richness for 39 crop systems globally. Bee abundance and richness were higher in diversified and organic fields and in landscapes comprising more high-quality habitats; bee richness on conventional fields with low diversity benefited most from high-quality surrounding land cover. Landscape configuration effects were weak. Bee responses varied slightly by biome. Our synthesis reveals that pollinator persistence will depend on both the maintenance of high-quality habitats around farms and on local management practices that may offset impacts of intensive monoculture agriculture.

The dependent variable problem in quantitative studies of Active Labour Market Programmes: uncovering hidden dynamics?

The question of what explains variation in expenditures on Active Labour Market Programs (ALMPs) has attracted significant scholarship in recent years. Significant insights have been gained with respect to the role of employers, unions and dual labour markets, openness, and partisanship. However, there remain significant disagreements with respects to key explanatory variables such the role of unions or the impact of partisanship. Qualitative studies have shown that there are both good conceptual reasons as well as historical evidence that different ALMPs are driven by different dynamics. There is little reason to believe that vastly different programs such as training and employment subsidies are driven by similar structural, interest group or indeed partisan dynamics. The question is therefore whether different ALMPs have the same correlation with different key explanatory variables identified in the literature? Using regression analysis, this paper shows that the explanatory variables identified by the literature have different relation to distinct ALMPs. This refinement adds significant analytical value and shows that disagreements are at least partly due to a dependent variable problem of ‘over-aggregation’.

Rapid qualitative and quantitative analysis of proanthocyanidin oligomers and polymers by UPLC-MS/MS.

This paper presents the development of a rapid method with ultraperformance liquid chromatography–tandem mass spectrometry (UPLC-MS/MS) for the qualitative and quantitative analyses of plant proanthocyanidins directly from crude plant extracts. The method utilizes a range of cone voltages to achieve the depolymerization step in the ion source of both smaller oligomers and larger polymers. The formed depolymerization products are further fragmented in the collision cell to enable their selective detection. This UPLC-MS/MS method is able to separately quantitate the terminal and extension units of the most common proanthocyanidin subclasses, that is, procyanidins and prodelphinidins. The resulting data enable (1) quantitation of the total proanthocyanidin content, (2) quantitation of total procyanidins and prodelphinidins including the procyanidin/prodelphinidin ratio, (3) estimation of the mean degree of polymerization for the oligomers and polymers, and (4) estimation of how the different procyanidin and prodelphinidin types are distributed along the chromatographic hump typically produced by large proanthocyanidins. All of this is achieved within the 10 min period of analysis, which makes the presented method a significant addition to the chemistry tools currently available for the qualitative and quantitative analyses of complex proanthocyanidin mixtures from plant extracts.

Quantitative ventilation assessments of idealized urban canopy layers with various urban layouts and the same building packing density

This paper investigates urban canopy layers (UCL) ventilation under neutral atmospheric condition with the same building area density (λp=0.25) and frontal area density (λf=0.25) but various urban sizes, building height variations, overall urban forms and wind directions. Turbulent airflows are first predicted by CFD simulations with standard k-ε model evaluated by wind tunnel data. Then air change rates per hour (ACH) and canopy purging flow rate (PFR) are numerically analyzed to quantify the rate of air exchange and the net ventilation capacity induced by mean flows and turbulence. With a parallel approaching wind (θ=0o), the velocity ratio first decreases in the adjustment region, followed by the fully-developed region where the flow reaches a balance. Although the flow quantities macroscopically keep constant, however ACH decreases and overall UCL ventilation becomes worse if urban size rises from 390m to 5km. Theoretically if urban size is infinite, ACH may reach a minimum value depending on local roof ventilation, and it rises from 1.7 to 7.5 if the standard deviation of building height variations increases (0% to 83.3%). Overall UCL ventilation capacity (PFR) with a square overall urban form (Lx=Ly=390m) is better as θ=0o than oblique winds (θ=15o, 30o, 45o), and it exceeds that of a staggered urban form under all wind directions (θ=0o to 45o), but is less than that of a rectangular urban form (Lx=570m, Ly=270m) under most wind directions (θ=30o to 90o). Further investigations are still required to quantify the net ventilation efficiency induced by mean flows and turbulence.

Verification of European subseasonal wind speed forecasts

Analysis of the forecasts and hindcasts from the ECMWF 32-day forecast model reveals that there is statistically significant skill in predicting weekly mean wind speeds over areas of Europe at lead times of at least 14–20 days. Previous research on wind speed predictability has focused on the short- to medium-range time scales, typically finding that forecasts lose all skill by the later part of the medium-range forecast. To the authors’ knowledge, this research is the first to look beyond the medium-range time scale by taking weekly mean wind speeds, instead of averages at hourly or daily resolution, for the ECMWF monthly forecasting system. It is shown that the operational forecasts have high levels of correlation (~0.6) between the forecasts and observations over the winters of 2008–12 for some areas of Europe. Hindcasts covering 20 winters show a more modest level of correlation but are still skillful. Additional analysis examines the probabilistic skill for the United Kingdom with the application of wind power forecasting in mind. It is also shown that there is forecast “value” for end users (operating in a simple cost/loss ratio decision-making framework). End users that are sensitive to winter wind speed variability over the United Kingdom, Germany, and some other areas of Europe should therefore consider forecasts beyond the medium-range time scale as it is clear there is useful information contained within the forecast.

Interpolation of Hilbert and Sobolev spaces: quantitative estimates and counterexamples

This paper provides an overview of interpolation of Banach and Hilbert spaces, with a focus on establishing when equivalence of norms is in fact equality of norms in the key results of the theory. (In brief, our conclusion for the Hilbert space case is that, with the right normalisations, all the key results hold with equality of norms.) In the final section we apply the Hilbert space results to the Sobolev spaces Hs(Ω) and tildeHs(Ω), for s in R and an open Ω in R^n. We exhibit examples in one and two dimensions of sets Ω for which these scales of Sobolev spaces are not interpolation scales. In the cases when they are interpolation scales (in particular, if Ω is Lipschitz) we exhibit examples that show that, in general, the interpolation norm does not coincide with the intrinsic Sobolev norm and, in fact, the ratio of these two norms can be arbitrarily large.

Quantitative single-molecule localization microscopy combined with rule-based modeling reveals ligand-induced TNF-R1 reorganization toward higher-order oligomers

We report on the assembly of tumor necrosis factor receptor 1 (TNF-R1) prior to ligand activation and its ligand-induced reorganization at the cell membrane. We apply single-molecule localization microscopy to obtain quantitative information on receptor cluster sizes and copy numbers. Our data suggest a dimeric pre-assembly of TNF-R1, as well as receptor reorganization toward higher oligomeric states with stable populations comprising three to six TNF-R1. Our experimental results directly serve as input parameters for computational modeling of the ligand-receptor interaction. Simulations corroborate the experimental finding of higher-order oligomeric states. This work is a first demonstration how quantitative, super-resolution and advanced microscopy can be used for systems biology approaches at the single-molecule and single-cell level.

A new method for the characterisation and verification of local spatial predictability for convective scale ensembles

The use of kilometre-scale ensembles in operational forecasting provides new challenges for forecast interpretation and evaluation to account for uncertainty on the convective scale. A new neighbourhood based method is presented for evaluating and characterising the local predictability variations from convective scale ensembles. Spatial scales over which ensemble forecasts agree (agreement scales, S^A) are calculated at each grid point ij, providing a map of the spatial agreement between forecasts. By comparing the average agreement scale obtained from ensemble member pairs (S^A(mm)_ij), with that between members and radar observations (S^A(mo)_ij), this approach allows the location-dependent spatial spread-skill relationship of the ensemble to be assessed. The properties of the agreement scales are demonstrated using an idealised experiment. To demonstrate the methods in an operational context the S^A(mm)_ij and S^A(mo)_ij are calculated for six convective cases run with the Met Office UK Ensemble Prediction System. The S^A(mm)_ij highlight predictability differences between cases, which can be linked to physical processes. Maps of S^A(mm)_ij are found to summarise the spatial predictability in a compact and physically meaningful manner that is useful for forecasting and for model interpretation. Comparison of S^A(mm)_ij and S^A(mo)_ij demonstrates the case-by-case and temporal variability of the spatial spread-skill, which can again be linked to physical processes.