Inferential, non-parametric statistics to assess quality of probabilistic forecast systems

Many statistical forecast systems are available to interested users. In order to be useful for decision-making, these systems must be based on evidence of underlying mechanisms. Once causal connections between the mechanism and their statistical manifestation have been firmly established, the forecasts must also provide some quantitative evidence of `quality’. However, the quality of statistical climate forecast systems (forecast quality) is an ill-defined and frequently misunderstood property. Often, providers and users of such forecast systems are unclear about what ‘quality’ entails and how to measure it, leading to confusion and misinformation. Here we present a generic framework to quantify aspects of forecast quality using an inferential approach to calculate nominal significance levels (p-values) that can be obtained either by directly applying non-parametric statistical tests such as Kruskal-Wallis (KW) or Kolmogorov-Smirnov (KS) or by using Monte-Carlo methods (in the case of forecast skill scores). Once converted to p-values, these forecast quality measures provide a means to objectively evaluate and compare temporal and spatial patterns of forecast quality across datasets and forecast systems. Our analysis demonstrates the importance of providing p-values rather than adopting some arbitrarily chosen significance levels such as p < 0.05 or p < 0.01, which is still common practice. This is illustrated by applying non-parametric tests (such as KW and KS) and skill scoring methods (LEPS and RPSS) to the 5-phase Southern Oscillation Index classification system using historical rainfall data from Australia, The Republic of South Africa and India. The selection of quality measures is solely based on their common use and does not constitute endorsement. We found that non-parametric statistical tests can be adequate proxies for skill measures such as LEPS or RPSS. The framework can be implemented anywhere, regardless of dataset, forecast system or quality measure. Eventually such inferential evidence should be complimented by descriptive statistical methods in order to fully assist in operational risk management.

Seasonal changes in pasture quality and diet selection and their relationship with liveweight gain of steers grazing tropical grass and grass-legume pastures in northern Australia.

The variation in liveweight gain in grazing beef cattle as influenced by pasture type, season and year effects has important economic implications for mixed crop-livestock systems and the ability to better predict such variation would benefit beef producers by providing a guide for decision making. To identify key determinants of liveweight change of Brahman-cross steers grazing subtropical pastures, measurements of pasture quality and quantity, and diet quality in parallel with liveweight were made over two consecutive grazing seasons (48 and 46 weeks, respectively), on mixed Clitoria ternatea/grass, Stylosanthes seabrana/grass and grass swards (grass being a mixture of Bothriochloa insculpta cv. Bisset, Dichanthium sericeum and Panicum maximum var. trichoglume cv. Petrie). Steers grazing the legume-based pastures had the highest growth rate and gained between 64 and 142 kg more than those grazing the grass pastures in under 12 months. Using an exponential model, green leaf mass, green leaf %, adjusted green leaf % (adjusted for inedible woody legume stems), faecal near infrared reflectance spectroscopy predictions of diet crude protein and diet dry matter digestibility, accounted for 77, 74, 80, 63 and 60%, respectively, of the variation in daily weight gain when data were pooled across pasture types and grazing seasons. The standard error of the regressions indicated that 95% prediction intervals were large (+/- 0.42-0.64 kg/head.day) suggesting that derived regression relationships have limited practical application for accurately estimating growth rate. In this study, animal factors, especially compensatory growth effects, appeared to have a major influence on growth rate in relation to pasture and diet attributes. It was concluded that predictions of growth rate based only on pasture or diet attributes are unlikely to be accurate or reliable. Nevertheless, key pasture attributes such as green leaf mass and green leaf% provide a robust indication of what proportion of the potential growth rate of the grazing animals can be achieved.

A comparison of plotless density estimators using Monte Carlo simulation on totally enumerated field data sets

Background: Plotless density estimators are those that are based on distance measures rather than counts per unit area (quadrats or plots) to estimate the density of some usually stationary event, e.g. burrow openings, damage to plant stems, etc. These estimators typically use distance measures between events and from random points to events to derive an estimate of density. The error and bias of these estimators for the various spatial patterns found in nature have been examined using simulated populations only. In this study we investigated eight plotless density estimators to determine which were robust across a wide range of data sets from fully mapped field sites. They covered a wide range of situations including animal damage to rice and corn, nest locations, active rodent burrows and distribution of plants. Monte Carlo simulations were applied to sample the data sets, and in all cases the error of the estimate (measured as relative root mean square error) was reduced with increasing sample size. The method of calculation and ease of use in the field were also used to judge the usefulness of the estimator. Estimators were evaluated in their original published forms, although the variable area transect (VAT) and ordered distance methods have been the subjects of optimization studies. Results: An estimator that was a compound of three basic distance estimators was found to be robust across all spatial patterns for sample sizes of 25 or greater. The same field methodology can be used either with the basic distance formula or the formula used with the Kendall-Moran estimator in which case a reduction in error may be gained for sample sizes less than 25, however, there is no improvement for larger sample sizes. The variable area transect (VAT) method performed moderately well, is easy to use in the field, and its calculations easy to undertake. Conclusion: Plotless density estimators can provide an estimate of density in situations where it would not be practical to layout a plot or quadrat and can in many cases reduce the workload in the field.

Non-destructive prediction of 'Hass' avocado dry matter via FT-NIR spectroscopy.

BACKGROUND: The inability to consistently guarantee internal quality of horticulture produce is of major importance to the primary producer, marketers and ultimately the consumer. Currently, commercial avocado maturity estimation is based on the destructive assessment of percentage dry matter (%DM), and sometimes percentage oil, both of which are highly correlated with maturity. In this study the utility of Fourier transform (FT) near-infrared spectroscopy (NIRS) was investigated for the first time as a non-invasive technique for estimating %DM of whole intact 'Hass' avocado fruit. Partial least squares regression models were developed from the diffuse reflectance spectra to predict %DM, taking into account effects of intra-seasonal variation and orchard conditions. RESULTS: It was found that combining three harvests (early, mid and late) from a single farm in the major production district of central Queensland yielded a predictive model for %DM with a coefficient of determination for the validation set of 0.76 and a root mean square error of prediction of 1.53% for DM in the range 19.4-34.2%. CONCLUSION: The results of the study indicate the potential of FT-NIRS in diffuse reflectance mode to non-invasively predict %DM of whole 'Hass' avocado fruit. When the FT-NIRS system was assessed on whole avocados, the results compared favourably against data from other NIRS systems identified in the literature that have been used in research applications on avocados.