Robust evaluation of fixed-event forecast rationality

In this paper we introduce a new testing procedure for evaluating the rationality of fixed-event forecasts based on a pseudo-maximum likelihood estimator. The procedure is designed to be robust to departures in the normality assumption. A model is introduced to show that such departures are likely when forecasters experience a credibility loss when they make large changes to their forecasts. The test is illustrated using monthly fixed-event forecasts produced by four UK institutions. Use of the robust test leads to the conclusion that certain forecasts are rational while use of the Gaussian-based test implies that certain forecasts are irrational. The difference in the results is due to the nature of the underlying data. Copyright © 2001 John Wiley & Sons, Ltd.

Evaluation of reach and grasp robot-assisted therapy suggests similar functional recovery patterns on proximal and distal arm segments in sub-acute hemiplegia

This paper provides some additional evidence in support of the hypothesis that robot therapies are clinically beneficial in neurorehabilitation. Although only 4 subjects were included in the study, the design of the intervention and the measures were done so as to minimise bias. The results are presented as single case studies, and can only be interpreted as such due to the study size. The intensity of intervention was 16 hours and the therapy philosophy (based on Carr and Shepherd) was that coordinated movements are preferable to joint based therapies, and that coordinating distal movements (in this case grasps) helps not only to recover function in these areas, but has greater value since the results are immediately transferable to daily skills such as reach and grasp movements.

Evaluation of biospheric components in Earth system models using modern and palaeo-observations: the state-of-the-art

Earth system models (ESMs) are increasing in complexity by incorporating more processes than their predecessors, making them potentially important tools for studying the evolution of climate and associated biogeochemical cycles. However, their coupled behaviour has only recently been examined in any detail, and has yielded a very wide range of outcomes. For example, coupled climate–carbon cycle models that represent land-use change simulate total land carbon stores at 2100 that vary by as much as 600 Pg C, given the same emissions scenario. This large uncertainty is associated with differences in how key processes are simulated in different models, and illustrates the necessity of determining which models are most realistic using rigorous methods of model evaluation. Here we assess the state-of-the-art in evaluation of ESMs, with a particular emphasis on the simulation of the carbon cycle and associated biospheric processes. We examine some of the new advances and remaining uncertainties relating to (i) modern and palaeodata and (ii) metrics for evaluation. We note that the practice of averaging results from many models is unreliable and no substitute for proper evaluation of individual models. We discuss a range of strategies, such as the inclusion of pre-calibration, combined process- and system-level evaluation, and the use of emergent constraints, that can contribute to the development of more robust evaluation schemes. An increasingly data-rich environment offers more opportunities for model evaluation, but also presents a challenge. Improved knowledge of data uncertainties is still necessary to move the field of ESM evaluation away from a "beauty contest" towards the development of useful constraints on model outcomes.

A low-cost wireless temperature sensor: evaluation for use in environmental applications

With a wide range of applications benefiting from dense network air temperature observations but with limitations of costs, existing siting guidelines and risk of damage to sensors, new methods are required to gain a high resolution understanding of the spatio-temporal patterns of urban meteorological phenomena such as the urban heat island or precision farming needs. With the launch of a new generation of low cost sensors it is possible to deploy a network to monitor air temperature at finer spatial resolutions. Here we investigate the Aginova Sentinel Micro (ASM) sensor with a bespoke radiation shield (together < US$150) which can provide secure near-real-time air temperature data to a server utilising existing (or user deployed) Wireless Fidelity (Wi-Fi) networks. This makes it ideally suited for deployment where wireless communications readily exist, notably urban areas. Assessment of the performance of the ASM relative to traceable standards in a water bath and atmospheric chamber show it to have good measurement accuracy with mean errors < ± 0.22 °C between -25 and 30 °C, with a time constant in ambient air of 110 ± 15 s. Subsequent field tests of it within the bespoke shield also had excellent performance (root-mean-square error = 0.13 °C) over a range of meteorological conditions relative to a traceable operational UK Met Office platinum resistance thermometer. These results indicate that the ASM and bespoke shield are more than fit-for-purpose for dense network deployment in urban areas at relatively low cost compared to existing observation techniques.

Performance evaluation of crowd image analysis using the PETS2009 dataset

This paper presents the PETS2009 outdoor crowd image analysis surveillance dataset and the performance evaluation of people counting, detection and tracking results using the dataset submitted to five IEEE Performance Evaluation of Tracking and Surveillance (PETS) workshops. The evaluation was carried out using well established metrics developed in the Video Analysis and Content Extraction (VACE) programme and the CLassification of Events, Activities, and Relationships (CLEAR) consortium. The comparative evaluation highlights the detection and tracking performance of the authors’ systems in areas such as precision, accuracy and robustness and provides a brief analysis of the metrics themselves to provide further insights into the performance of the authors’ systems.

Evaluation of ERA-Interim reanalysis precipitation products using England and Wales observations

Precipitation forecast data from the ERA-Interim reanalysis (33 years) are evaluated using the daily England and Wales Precipitation (EWP) observations obtained from a rain gauge network. Observed and reanalysis daily precipitation data are both described well by Weibull distributions with indistinguishable shapes but different scale parameters, such that the reanalysis underestimates the observations by an average factor of 22%. The correlation between the observed and ERA-Interim time series of regional, daily precipitation is 0.91. ERA-Interim also captures the statistics of extreme precipitation including a slightly lower likelihood of the heaviest precipitation events (>15 mm day− 1 for the regional average) than indicated by the Weibull fit. ERA-Interim is also closer to EWP for the high precipitation events. Since these carry weight in longer accumulations, a smaller underestimation of 19% is found for monthly mean precipitation. The partition between convective and stratiform precipitation in the ERA-Interim forecast is also examined. In summer both components contribute equally to the total precipitation amount, while in winter the stratiform precipitation is approximately double convective. These results are expected to be relevant to other regions with low orography on the coast of a continent at the downstream end of mid-latitude stormtracks.

Evaluation of seven European aerosol optical depth retrieval algorithms for climate analysis

Satellite data are increasingly used to provide observation-based estimates of the effects of aerosols on climate. The Aerosol-cci project, part of the European Space Agency's Climate Change Initiative (CCI), was designed to provide essential climate variables for aerosols from satellite data. Eight algorithms, developed for the retrieval of aerosol properties using data from AATSR (4), MERIS (3) and POLDER, were evaluated to determine their suitability for climate studies. The primary result from each of these algorithms is the aerosol optical depth (AOD) at several wavelengths, together with the Ångström exponent (AE) which describes the spectral variation of the AOD for a given wavelength pair. Other aerosol parameters which are possibly retrieved from satellite observations are not considered in this paper. The AOD and AE (AE only for Level 2) were evaluated against independent collocated observations from the ground-based AERONET sun photometer network and against “reference” satellite data provided by MODIS and MISR. Tools used for the evaluation were developed for daily products as produced by the retrieval with a spatial resolution of 10 × 10 km2 (Level 2) and daily or monthly aggregates (Level 3). These tools include statistics for L2 and L3 products compared with AERONET, as well as scoring based on spatial and temporal correlations. In this paper we describe their use in a round robin (RR) evaluation of four months of data, one month for each season in 2008. The amount of data was restricted to only four months because of the large effort made to improve the algorithms, and to evaluate the improvement and current status, before larger data sets will be processed. Evaluation criteria are discussed. Results presented show the current status of the European aerosol algorithms in comparison to both AERONET and MODIS and MISR data. The comparison leads to a preliminary conclusion that the scores are similar, including those for the references, but the coverage of AATSR needs to be enhanced and further improvements are possible for most algorithms. None of the algorithms, including the references, outperforms all others everywhere. AATSR data can be used for the retrieval of AOD and AE over land and ocean. PARASOL and one of the MERIS algorithms have been evaluated over ocean only and both algorithms provide good results.

Statistical-dynamical downscaling for wind energy potentials: evaluation and applications to decadal hindcasts and climate change projections

A statistical–dynamical downscaling (SDD) approach for the regionalization of wind energy output (Eout) over Europe with special focus on Germany is proposed. SDD uses an extended circulation weather type (CWT) analysis on global daily mean sea level pressure fields with the central point being located over Germany. Seventy-seven weather classes based on the associated CWT and the intensity of the geostrophic flow are identified. Representatives of these classes are dynamically downscaled with the regional climate model COSMO-CLM. By using weather class frequencies of different data sets, the simulated representatives are recombined to probability density functions (PDFs) of near-surface wind speed and finally to Eout of a sample wind turbine for present and future climate. This is performed for reanalysis, decadal hindcasts and long-term future projections. For evaluation purposes, results of SDD are compared to wind observations and to simulated Eout of purely dynamical downscaling (DD) methods. For the present climate, SDD is able to simulate realistic PDFs of 10-m wind speed for most stations in Germany. The resulting spatial Eout patterns are similar to DD-simulated Eout. In terms of decadal hindcasts, results of SDD are similar to DD-simulated Eout over Germany, Poland, Czech Republic, and Benelux, for which high correlations between annual Eout time series of SDD and DD are detected for selected hindcasts. Lower correlation is found for other European countries. It is demonstrated that SDD can be used to downscale the full ensemble of the Earth System Model of the Max Planck Institute (MPI-ESM) decadal prediction system. Long-term climate change projections in Special Report on Emission Scenarios of ECHAM5/MPI-OM as obtained by SDD agree well to the results of other studies using DD methods, with increasing Eout over northern Europe and a negative trend over southern Europe. Despite some biases, it is concluded that SDD is an adequate tool to assess regional wind energy changes in large model ensembles.

The AeroCom evaluation and intercomparison of organic aerosol in global models

This paper evaluates the current status of global modeling of the organic aerosol (OA) in the troposphere and analyzes the differences between models as well as between models and observations. Thirty-one global chemistry transport models (CTMs) and general circulation models (GCMs) have participated in this intercomparison, in the framework of AeroCom phase II. The simulation of OA varies greatly between models in terms of the magnitude of primary emissions, secondary OA (SOA) formation, the number of OA species used (2 to 62), the complexity of OA parameterizations (gas-particle partitioning, chemical aging, multiphase chemistry, aerosol microphysics), and the OA physical, chemical and optical properties. The diversity of the global OA simulation results has increased since earlier AeroCom experiments, mainly due to the increasing complexity of the SOA parameterization in models, and the implementation of new, highly uncertain, OA sources. Diversity of over one order of magnitude exists in the modeled vertical distribution of OA concentrations that deserves a dedicated future study. Furthermore, although the OA / OC ratio depends on OA sources and atmospheric processing, and is important for model evaluation against OA and OC observations, it is resolved only by a few global models. The median global primary OA (POA) source strength is 56 Tg a−1 (range 34–144 Tg a−1) and the median SOA source strength (natural and anthropogenic) is 19 Tg a−1 (range 13–121 Tg a−1). Among the models that take into account the semi-volatile SOA nature, the median source is calculated to be 51 Tg a−1 (range 16–121 Tg a−1), much larger than the median value of the models that calculate SOA in a more simplistic way (19 Tg a−1; range 13–20 Tg a−1, with one model at 37 Tg a−1). The median atmospheric burden of OA is 1.4 Tg (24 models in the range of 0.6–2.0 Tg and 4 between 2.0 and 3.8 Tg), with a median OA lifetime of 5.4 days (range 3.8–9.6 days). In models that reported both OA and sulfate burdens, the median value of the OA/sulfate burden ratio is calculated to be 0.77; 13 models calculate a ratio lower than 1, and 9 models higher than 1. For 26 models that reported OA deposition fluxes, the median wet removal is 70 Tg a−1 (range 28–209 Tg a−1), which is on average 85% of the total OA deposition. Fine aerosol organic carbon (OC) and OA observations from continuous monitoring networks and individual field campaigns have been used for model evaluation. At urban locations, the model–observation comparison indicates missing knowledge on anthropogenic OA sources, both strength and seasonality. The combined model–measurements analysis suggests the existence of increased OA levels during summer due to biogenic SOA formation over large areas of the USA that can be of the same order of magnitude as the POA, even at urban locations, and contribute to the measured urban seasonal pattern. Global models are able to simulate the high secondary character of OA observed in the atmosphere as a result of SOA formation and POA aging, although the amount of OA present in the atmosphere remains largely underestimated, with a mean normalized bias (MNB) equal to −0.62 (−0.51) based on the comparison against OC (OA) urban data of all models at the surface, −0.15 (+0.51) when compared with remote measurements, and −0.30 for marine locations with OC data. The mean temporal correlations across all stations are low when compared with OC (OA) measurements: 0.47 (0.52) for urban stations, 0.39 (0.37) for remote stations, and 0.25 for marine stations with OC data. The combination of high (negative) MNB and higher correlation at urban stations when compared with the low MNB and lower correlation at remote sites suggests that knowledge about the processes that govern aerosol processing, transport and removal, on top of their sources, is important at the remote stations. There is no clear change in model skill with increasing model complexity with regard to OC or OA mass concentration. However, the complexity is needed in models in order to distinguish between anthropogenic and natural OA as needed for climate mitigation, and to calculate the impact of OA on climate accurately.

Stated willingness-to-pay for organic fruit and pesticide ban: an evaluation using both web-based and face-to-face interviewing

Income growth in highly industrialised countries has resulted in consumer choice of foodstuffs no longer being primarily influenced by basic factors such as price and organoleptic features. From this perspective, the present study sets out to evaluate how and to what extent consumer choice is influenced by the possible negative effects on health and environment caused by the consumption of fruit containing deposits of pesticides and chemical products. The study describes the results of a survey which explores and estimates consumer willingness to pay in two forms: a yearly contribution for the abolition of the use of pesticides on fruit, and a premium price for organically grown apples guaranteed by a certified label. The same questionnaire was administered to two samples. The first was a conventional face-to-face survey of customers of large retail outlets located around Bologna (Italy); the second was an Internet sample. The discrete choice data were analysed by means of probit and tobit models to estimate the utility consumers attribute to organically grown fruit and to a pesticide ban. The research also addresses questions of validity and representativeness as a fundamental problem in web-based surveys.

Consumers perception and evaluation of fresh cut buying attributes: a survey on the Italian market

During the last twenty years, consumer choice in high income countries is no longer merely dictated by price and the organoleptic characteristics of a product, but also by other features some of which are not patently tangible. The growing importance of such attributes in the process of consumer choice is not only due to income increase, but also to changes in lifestyle such as migrations from the countryside, a generalized urbanization and consequential city life style, female emancipation and work outside the domestic walls for women, the drastic decrease in hard physical labor and the process of internationalization. The present survey study aims to explore the importance that Italian consumers give to fresh cut buying attributes and which of these attributes should be taken into consideration by industries in order to satisfy the needs of the most critical shoppers. Where possible, market and survey data for fresh cut products will be compared with those for cooked products and before presenting the results and conclusions of the study, the technical issues of processing will be highlighted owing to the fact that they affect the marketing of these products, the recent market situation with regard to consumption will be illustrated and the methodology used will be described.

Evaluation of water properties in HEA–HEMA hydrogels swollen in aqueous-PEG solutions using thermoanalytical techniques

Hydrogels are polymeric materials used in many pharmaceutical and biomedical applications due to their ability to form 3D hydrophilic polymeric networks, which can absorb large amounts of water. In the present work, polyethylene glycols (PEG) were introduced into the hydrogel liquid phase in order to improve the mechanical properties of hydrogels composed of 2-hydroxyethylacrylate and 2-hydroxyethylmethacrylate (HEA–HEMA) synthesized with different co-monomer compositions and equilibrated in water or in 20 % water–PEG 400 and 600 solutions. The thermoanalytical techniques [differential scanning calorimetry (DSC) and thermogravimetry (TG)] were used to evaluate the amount and properties of free and bound water in HEA–HEMA hydrogels. The internal structure and the mechanical properties of hydrogels were studied using scanning electron microscopy and friability assay. TG “loss-on-drying” experiments were applied to study the water-retention properties of hydrogels, whereas the combination of TG and DSC allowed estimating the total amount of freezable and non-freezing water in hydrogels. The results show that the addition of viscous co-solvent (PEG) to the liquid medium results in significant improvement of the mechanical properties of HEA–HEMA hydrogels and also slightly retards the water loss from the hydrogels. A redistribution of free and bound water in the hydrogels equilibrated in mixed solutions containing 20 vol% of PEGs takes place.

Evaluation of optical techniques for characterising soil organic matter quality in agricultural soils

Soil organic matter (SOM) is one of the main global carbon pools. It is a measure of soil quality as its presence increases carbon sequestration and improves physical and chemical soil properties. The determination and characterisation of humic substances gives essential information of the maturity and stresses of soils as well as of their health. However, the determination of the exact nature and molecular structure of these substances has been proven difficult. Several complex techniques exist to characterise SOM and mineralisation and humification processes. One of the more widely accepted for its accuracy is nuclear magnetic resonance (NMR) spectroscopy. Despite its efficacy, NMR needs significant economic resources, equipment, material and time. Proxy measures like the fluorescence index (FI), cold and hot-water extractable carbon (CWC and HWC) and SUVA-254 have the potential to characterise SOM and, in combination, provide qualitative and quantitative data of SOM and its processes. Spanish and British agricultural cambisols were used to measure SOM quality and determine whether similarities were found between optical techniques and 1H NMR results in these two regions with contrasting climatic conditions. High correlations (p < 0.001) were found between the specific aromatic fraction measured with 1H NMR and SUVA-254 (Rs = 0.95) and HWC (Rs = 0.90), which could be described using a linear model. A high correlation between FI and the aromatics fraction measured with 1H NMR (Rs = −0.976) was also observed. In view of our results, optical measures have a potential, in combination, to predict the aromatic fraction of SOM without the need of expensive and time consuming techniques.

Towards improving the framework for probabilistic forecast evaluation

The evaluation of forecast performance plays a central role both in the interpretation and use of forecast systems and in their development. Different evaluation measures (scores) are available, often quantifying different characteristics of forecast performance. The properties of several proper scores for probabilistic forecast evaluation are contrasted and then used to interpret decadal probability hindcasts of global mean temperature. The Continuous Ranked Probability Score (CRPS), Proper Linear (PL) score, and IJ Good’s logarithmic score (also referred to as Ignorance) are compared; although information from all three may be useful, the logarithmic score has an immediate interpretation and is not insensitive to forecast busts. Neither CRPS nor PL is local; this is shown to produce counter intuitive evaluations by CRPS. Benchmark forecasts from empirical models like Dynamic Climatology place the scores in context. Comparing scores for forecast systems based on physical models (in this case HadCM3, from the CMIP5 decadal archive) against such benchmarks is more informative than internal comparison systems based on similar physical simulation models with each other. It is shown that a forecast system based on HadCM3 out performs Dynamic Climatology in decadal global mean temperature hindcasts; Dynamic Climatology previously outperformed a forecast system based upon HadGEM2 and reasons for these results are suggested. Forecasts of aggregate data (5-year means of global mean temperature) are, of course, narrower than forecasts of annual averages due to the suppression of variance; while the average “distance” between the forecasts and a target may be expected to decrease, little if any discernible improvement in probabilistic skill is achieved.

An evaluation of the pedestrian classification in a multi-domain multi-modality setup

The objective of this article is to study the problem of pedestrian classification across different light spectrum domains (visible and far-infrared (FIR)) and modalities (intensity, depth and motion). In recent years, there has been a number of approaches for classifying and detecting pedestrians in both FIR and visible images, but the methods are difficult to compare, because either the datasets are not publicly available or they do not offer a comparison between the two domains. Our two primary contributions are the following: (1) we propose a public dataset, named RIFIR , containing both FIR and visible images collected in an urban environment from a moving vehicle during daytime; and (2) we compare the state-of-the-art features in a multi-modality setup: intensity, depth and flow, in far-infrared over visible domains. The experiments show that features families, intensity self-similarity (ISS), local binary patterns (LBP), local gradient patterns (LGP) and histogram of oriented gradients (HOG), computed from FIR and visible domains are highly complementary, but their relative performance varies across different modalities. In our experiments, the FIR domain has proven superior to the visible one for the task of pedestrian classification, but the overall best results are obtained by a multi-domain multi-modality multi-feature fusion.