Inferential, non-parametric statistics to assess quality of probabilistic forecast systems

Many statistical forecast systems are available to interested users. In order to be useful for decision-making, these systems must be based on evidence of underlying mechanisms. Once causal connections between the mechanism and their statistical manifestation have been firmly established, the forecasts must also provide some quantitative evidence of `quality’. However, the quality of statistical climate forecast systems (forecast quality) is an ill-defined and frequently misunderstood property. Often, providers and users of such forecast systems are unclear about what ‘quality’ entails and how to measure it, leading to confusion and misinformation. Here we present a generic framework to quantify aspects of forecast quality using an inferential approach to calculate nominal significance levels (p-values) that can be obtained either by directly applying non-parametric statistical tests such as Kruskal-Wallis (KW) or Kolmogorov-Smirnov (KS) or by using Monte-Carlo methods (in the case of forecast skill scores). Once converted to p-values, these forecast quality measures provide a means to objectively evaluate and compare temporal and spatial patterns of forecast quality across datasets and forecast systems. Our analysis demonstrates the importance of providing p-values rather than adopting some arbitrarily chosen significance levels such as p < 0.05 or p < 0.01, which is still common practice. This is illustrated by applying non-parametric tests (such as KW and KS) and skill scoring methods (LEPS and RPSS) to the 5-phase Southern Oscillation Index classification system using historical rainfall data from Australia, The Republic of South Africa and India. The selection of quality measures is solely based on their common use and does not constitute endorsement. We found that non-parametric statistical tests can be adequate proxies for skill measures such as LEPS or RPSS. The framework can be implemented anywhere, regardless of dataset, forecast system or quality measure. Eventually such inferential evidence should be complimented by descriptive statistical methods in order to fully assist in operational risk management.

Potential for using through-boring to improve groundline treatment of Australian wood species: A preliminary study.

An aging electricity distribution system and reduced availability of naturally durable tropical hardwoods in Australia will combine in the next decade to produce a major shortage of poles. One approach to mitigating this shortage is to utilize lower durability species and improve the penetration of preservatives into the refractory heartwood by introducing additional pretreatment processes. A potential method for improving preservative penetration in the critical ground-line zone is through-boring. This process, in which holes are drilled through the pole perpendicular to the grain in the ground-line zone, is widely used in the western United States for treatment of Douglas-fir and may be Suitable for many Australian wood species. The potential for improving heartwood penetration in eucalypts with alkaline-copper-quaternary (ACQ) compound was assessed on heartwood specimens from four species (Eucalyptus cloeziana F.Muell., E. grandis W.Hill ex Maiden, E. obliqua L'Her. and E. pellita F.Muell.) and Lophostemon confertus (R.Br.) Peter G.Wilson & J.T.Wateril). Longitudinal ACQ penetration was extremely shallow in L. confertus and only slightly better in E. cloeziana. Longitudinal penetration was good in both E. obliqua and E. pellita, although there was some variation in treatment results with length of pressure period. The results suggest that through-boring might be a reasonable approach for achieving heartwood penetration in some Eucalyptus species, although further studies are required to assess additional treatment schedules and to determine the effects of the process oil flexural properties of the poles.