Inferential, non-parametric statistics to assess quality of probabilistic forecast systems

Many statistical forecast systems are available to interested users. In order to be useful for decision-making, these systems must be based on evidence of underlying mechanisms. Once causal connections between the mechanism and their statistical manifestation have been firmly established, the forecasts must also provide some quantitative evidence of `quality’. However, the quality of statistical climate forecast systems (forecast quality) is an ill-defined and frequently misunderstood property. Often, providers and users of such forecast systems are unclear about what ‘quality’ entails and how to measure it, leading to confusion and misinformation. Here we present a generic framework to quantify aspects of forecast quality using an inferential approach to calculate nominal significance levels (p-values) that can be obtained either by directly applying non-parametric statistical tests such as Kruskal-Wallis (KW) or Kolmogorov-Smirnov (KS) or by using Monte-Carlo methods (in the case of forecast skill scores). Once converted to p-values, these forecast quality measures provide a means to objectively evaluate and compare temporal and spatial patterns of forecast quality across datasets and forecast systems. Our analysis demonstrates the importance of providing p-values rather than adopting some arbitrarily chosen significance levels such as p < 0.05 or p < 0.01, which is still common practice. This is illustrated by applying non-parametric tests (such as KW and KS) and skill scoring methods (LEPS and RPSS) to the 5-phase Southern Oscillation Index classification system using historical rainfall data from Australia, The Republic of South Africa and India. The selection of quality measures is solely based on their common use and does not constitute endorsement. We found that non-parametric statistical tests can be adequate proxies for skill measures such as LEPS or RPSS. The framework can be implemented anywhere, regardless of dataset, forecast system or quality measure. Eventually such inferential evidence should be complimented by descriptive statistical methods in order to fully assist in operational risk management.

Patterns of seed bank and size asymmetry of plant growth across varying sites in the invasive Lantana camara L. (Verbenaceae)

Lantana camara L. (Verbenaceae) is a weed of great significance in Australia and worldwide, but little is known about connections among components of its life history. We document over a 3-year period, the links between L. camara seed-bank dynamics and its above-ground growth, including size asymmetry in four land-use types (a farm, a hoop pine plantation and two open eucalypt forests) invaded by the weed near Brisbane, Queensland Australia. Seed-bank populations varied appreciably across sites and in response to rainfall and control measures, and they were higher (~1,000 seeds/m2) when annual rainfall was 15-30 % below the long-term yearly average. Fire reduced seed-bank populations but not the proportion germinating (6-8 %). Nearly a quarter of fresh seeds remain germinable after 3 years of soil burial. For small seedlings (<10 cm high), the expected trade-offs in two life-history traits-survival and growth-did not apply; rather the observed positive association between these two traits, coupled with a persistent seed-bank population could contribute to the invasiveness of the plant. Relationships between absolute growth rate and initial plant size (crown volume) were positively linear, suggesting that most populations are still at varying stages of the exponential phase of the sigmoid growth; this trend also suggests that at most sites and despite increasing stand density and limiting environmental resources of light and soil moisture, lantana growth is inversely size asymmetric. From the observed changes in measures of plant size inequality, asymmetric competition appeared limited in all the infestations surveyed. © 2013 Crown Copyright as represented by: Department of Agriculture, Fisheries and Forestry, Australia.