Evaluating forecasts of extreme events for hydrological applications: an approach for screening unfamiliar performance measures

Many different performance measures have been developed to evaluate field predictions in meteorology. However, a researcher or practitioner encountering a new or unfamiliar measure may have difficulty in interpreting its results, which may lead to them avoiding new measures and relying on those that are familiar. In the context of evaluating forecasts of extreme events for hydrological applications, this article aims to promote the use of a range of performance measures. Some of the types of performance measures that are introduced in order to demonstrate a six-step approach to tackle a new measure. Using the example of the European Centre for Medium-Range Weather Forecasts (ECMWF) ensemble precipitation predictions for the Danube floods of July and August 2002, to show how to use new performance measures with this approach and the way to choose between different performance measures based on their suitability for the task at hand is shown. Copyright © 2008 Royal Meteorological Society

Determinants and measures of export performance – comprehensive literature review

As one of the key indicators of the firm’s ability to leverage successfully its resources and capabilities in the international context, export performance has been one of the most extensively studied phenomena. A plethora of studies have been conducted pertaining to provide better understanding of the factors (firm- or environment-specific) and behaviours (e.g., export strategy) that make exporting a successful venture. Following a comprehensive literature review undertaking in this study the current state of the export performance literature could be summarisedas (i) methodologically fragmented in that there is a variety of analytical and methodological approaches, (ii) conceptually diverse, a large number of determinants have been identified as having direct or indirect influence on the firm’s export performance, and a large number of indicators have been used to conceptualise and operationalise the export performance measures, and (iii) inconclusive, the studies have produced inconsistent results of the impact of different determinants on export performance.

Generic strategies and performance: evidence from manufacturing firms

Purpose – The purpose of this study is to examine the relationship between business-level strategy and organisational performance and to test the applicability of Porter's generic strategies in explaining differences in the performance of organisations. Design/methodology/approach – The study was focussed on manufacturing firms in the UK belonging to the electrical and mechanical engineering sectors. Data were collected through a postal survey using the survey instrument from 124 organisations and the respondents were all at CEO level. Both objective and subjective measures were used to assess performance. Non-response bias was assessed statistically and it was not found to be a major problem affecting this study. Appropriate measures were taken to ensure that common method variance (CMV) does not affect the results of this study. Statistical tests indicated that CMV problem does not affect the results of this study. Findings – The results of this study indicate that firms adopting one of the strategies, namely cost-leadership or differentiation, perform better than “stuck-in-the-middle” firms which do not have a dominant strategic orientation. The integrated strategy group has lower performance compared with cost-leaders and differentiators in terms of financial performance measures. This provides support for Porter's view that combination strategies are unlikely to be effective in organisations. However, the cost-leadership and differentiation strategies were not strongly correlated with the financial performance measures indicating the limitations of Porter's generic strategies in explaining performance heterogeneity in organisations. Originality/value – This study makes an important contribution to the literature by identifying some of the gaps in the literature through a systematic literature review and addressing those gaps.

Improving the evaluation of hydrological multi-model forecast performance in the Upper Danube Catchment

Medium range flood forecasting activities, driven by various meteorological forecasts ranging from high resolution deterministic forecasts to low spatial resolution ensemble prediction systems, share a major challenge in the appropriateness and design of performance measures. In this paper possible limitations of some traditional hydrological and meteorological prediction quality and verification measures are identified. Some simple modifications are applied in order to circumvent the problem of the autocorrelation dominating river discharge time-series and in order to create a benchmark model enabling the decision makers to evaluate the forecast quality and the model quality. Although the performance period is quite short the advantage of a simple cost-loss function as a measure of forecast quality can be demonstrated.

Problems with binary pattern measures for flood model evaluation

As the calibration and evaluation of flood inundation models are a prerequisite for their successful application, there is a clear need to ensure that the performance measures that quantify how well models match the available observations are fit for purpose. This paper evaluates the binary pattern performance measures that are frequently used to compare flood inundation models with observations of flood extent. This evaluation considers whether these measures are able to calibrate and evaluate model predictions in a credible and consistent way, i.e. identifying the underlying model behaviour for a number of different purposes such as comparing models of floods of different magnitudes or on different catchments. Through theoretical examples, it is shown that the binary pattern measures are not consistent for floods of different sizes, such that for the same vertical error in water level, a model of a flood of large magnitude appears to perform better than a model of a smaller magnitude flood. Further, the commonly used Critical Success Index (usually referred to as F<2 >) is biased in favour of overprediction of the flood extent, and is also biased towards correctly predicting areas of the domain with smaller topographic gradients. Consequently, it is recommended that future studies consider carefully the implications of reporting conclusions using these performance measures. Additionally, future research should consider whether a more robust and consistent analysis could be achieved by using elevation comparison methods instead.

Supply management capabilities, routine bundles and their impact on firm performance

Despite the generally positive contribution of supply management capabilities to firm performance their respective routines require more depth of assessment. Using the resource-based view we examine four routines bundles comprising ostensive and performative aspects of supply management capability – supply management integration, coordinated sourcing, collaboration management and performance assessment. Using structural equation modelling we measure supply management capability empirically as a second-order latent variable and estimate its effect on a series of financial and operational performance measures. The routines-based approach allows us to demonstrate a different, more fine-grained approach for assessing consistent bundles of homogeneous patterns of activity across firms. The results suggest supply management capability is formed of internally consistent routine bundles, which are significantly related to financial performance, mediated by operational performance. Our results confirm an indirect effect of firm performance for ‘core’ routines forming the architecture of a supply management capability. Supply management capability primarily improves the operational performance of the business, which is subsequently translated into improved financial performance. The study is significant for practice as it offers a different view about the face-valid rationale of supply management directly influencing firm financial performance. We confound this assumption, prompting caution when placing too much importance on directly assessing supply management capability using financial performance of the business.

Is it significant? Guidelines for reporting BCI performance

Recent growth in brain-computer interface (BCI) research has increased pressure to report improved performance. However, different research groups report performance in different ways. Hence, it is essential that evaluation procedures are valid and reported in sufficient detail. In this chapter we give an overview of available performance measures such as classification accuracy, cohen’s kappa, information transfer rate (ITR), and written symbol rate. We show how to distinguish results from chance level using confidence intervals for accuracy or kappa. Furthermore, we point out common pitfalls when moving from offline to online analysis and provide a guide on how to conduct statistical tests on (BCI) results.

Tracking performance evaluation on PETS 2015 Challenge datasets

This paper presents a quantitative evaluation of a tracking system on PETS 2015 Challenge datasets using well-established performance measures. Using the existing tools, the tracking system implements an end-to-end pipeline that include object detection, tracking and post- processing stages. The evaluation results are presented on the provided sequences of both ARENA and P5 datasets of PETS 2015 Challenge. The results show an encouraging performance of the tracker in terms of accuracy but a greater tendency of being prone to cardinality error and ID changes on both datasets. Moreover, the analysis show a better performance of the tracker on visible imagery than on thermal imagery.

Calibration of uncertain flood inundation models using remotely sensed water levels

A traditional method of validating the performance of a flood model when remotely sensed data of the flood extent are available is to compare the predicted flood extent to that observed. The performance measure employed often uses areal pattern-matching to assess the degree to which the two extents overlap. Recently, remote sensing of flood extents using synthetic aperture radar (SAR) and airborne scanning laser altimetry (LIDAR) has made more straightforward the synoptic measurement of water surface elevations along flood waterlines, and this has emphasised the possibility of using alternative performance measures based on height. This paper considers the advantages that can accrue from using a performance measure based on waterline elevations rather than one based on areal patterns of wet and dry pixels. The two measures were compared for their ability to estimate flood inundation uncertainty maps from a set of model runs carried out to span the acceptable model parameter range in a GLUE-based analysis. A 1 in 5-year flood on the Thames in 1992 was used as a test event. As is typical for UK floods, only a single SAR image of observed flood extent was available for model calibration and validation. A simple implementation of a two-dimensional flood model (LISFLOOD-FP) was used to generate model flood extents for comparison with that observed. The performance measure based on height differences of corresponding points along the observed and modelled waterlines was found to be significantly more sensitive to the channel friction parameter than the measure based on areal patterns of flood extent. The former was able to restrict the parameter range of acceptable model runs and hence reduce the number of runs necessary to generate an inundation uncertainty map. A result of this was that there was less uncertainty in the final flood risk map. The uncertainty analysis included the effects of uncertainties in the observed flood extent as well as in model parameters. The height-based measure was found to be more sensitive when increased heighting accuracy was achieved by requiring that observed waterline heights varied slowly along the reach. The technique allows for the decomposition of the reach into sections, with different effective channel friction parameters used in different sections, which in this case resulted in lower r.m.s. height differences between observed and modelled waterlines than those achieved by runs using a single friction parameter for the whole reach. However, a validation of the modelled inundation uncertainty using the calibration event showed a significant difference between the uncertainty map and the observed flood extent. While this was true for both measures, the difference was especially significant for the height-based one. This is likely to be due to the conceptually simple flood inundation model and the coarse application resolution employed in this case. The increased sensitivity of the height-based measure may lead to an increased onus being placed on the model developer in the production of a valid model

Changing patterns of cognitive-motor interference (CMI) over time during recovery from stroke

Objective: To explore the extent and nature of change in cognitive-motor interference (CMI) among rehabilitating stroke patients who showed dual-task gait decrement at initial assessment. Design: Experimental, with in-subjects, repeated measures design. Setting: Rehabilitation centre for adults with acquired, nonprogressive brain injury. Subjects: Ten patients with unilateral stroke, available for reassessment 1-9 months following their participation in a study of CMI after brain injury. Measures: Median stride duration; mean word generation. Methods: Two x one-minute walking trials, two x one-minute word generation trials, two x one-minute trials of simultaneous walking and word generation; 10-metre walking time; Barthel ADL Scale score. Results: Seven out of ten patients showed reduction over time in dual-task gait decrement. Three out of ten showed reduction in cognitive decrement. Only one showed concomitant reduction in gait and word generation decrement. Conclusion: Extent of CMI during relearning to walk after a stroke reduced over time in the majority of patients. Effects were more evident in improved stride duration than improved cognitive performance. Measures of multiple task performance should be included in assessment for functional recovery.

Knowledge transfer processes in PFI/PPP: critical success factors

Successful knowledge transfer is an important process which requires continuous improvement in today’s knowledge-intensive economy. However, improving knowledge transfer processes represents a challenge for construction practitioners due to the complexity of knowledge acquisition, codification and sharing. Although knowledge transfer is context based, understanding the critical success factors can lead to improvements in the transfer process. This paper seeks to identify and evaluate the most significant critical factors for improving knowledge transfer processes in Public Private Partnerships/Private Finance Initiatives (PPP/PFI) projects. Drawing upon a questionnaire survey of 52 construction firms located in the UK, data is analysed using Severity Index (SI) and Coefficient of Variation (COV), to examine and identify these factors in PPP/PFI schemes. The findings suggest that a supportive leadership, participation/commitment from the relevant parties, and good communication between the relevant parties are crucial to improving knowledge transfer processes in PFI schemes. Practitioners, managers and researchers can use the findings to efficiently design performance measures for analysing and improving knowledge transfer processes.

Disorder-specific functional abnormalities during sustained attention in youth with Attention Deficit Hyperactivity Disorder (ADHD) and with Autism

Attention Deficit Hyperactivity Disorder (ADHD) and Autism Spectrum Disorder (ASD) are often comorbid and share behavioural-cognitive abnormalities in sustained attention. A key question is whether this shared cognitive phenotype is based on common or different underlying pathophysiologies. To elucidate this question, we compared 20 boys with ADHD to 20 age and IQ matched ASD and 20 healthy boys using functional magnetic resonance imaging (fMRI) during a parametrically modulated vigilance task with a progressively increasing load of sustained attention. ADHD and ASD boys had significantly reduced activation relative to controls in bilateral striato–thalamic regions, left dorsolateral prefrontal cortex (DLPFC) and superior parietal cortex. Both groups also displayed significantly increased precuneus activation relative to controls. Precuneus was negatively correlated with the DLPFC activation, and progressively more deactivated with increasing attention load in controls, but not patients, suggesting problems with deactivation of a task-related default mode network in both disorders. However, left DLPFC underactivation was significantly more pronounced in ADHD relative to ASD boys, which furthermore was associated with sustained performance measures that were only impaired in ADHD patients. ASD boys, on the other hand, had disorder-specific enhanced cerebellar activation relative to both ADHD and control boys, presumably reflecting compensation. The findings show that ADHD and ASD boys have both shared and disorder-specific abnormalities in brain function during sustained attention. Shared deficits were in fronto–striato–parietal activation and default mode suppression. Differences were a more severe DLPFC dysfunction in ADHD and a disorder-specific fronto–striato–cerebellar dysregulation in ASD.

Assessing the reliability of probabilistic flood inundation model predictions

An ability to quantify the reliability of probabilistic flood inundation predictions is a requirement not only for guiding model development but also for their successful application. Probabilistic flood inundation predictions are usually produced by choosing a method of weighting the model parameter space, but previous study suggests that this choice leads to clear differences in inundation probabilities. This study aims to address the evaluation of the reliability of these probabilistic predictions. However, a lack of an adequate number of observations of flood inundation for a catchment limits the application of conventional methods of evaluating predictive reliability. Consequently, attempts have been made to assess the reliability of probabilistic predictions using multiple observations from a single flood event. Here, a LISFLOOD-FP hydraulic model of an extreme (>1 in 1000 years) flood event in Cockermouth, UK, is constructed and calibrated using multiple performance measures from both peak flood wrack mark data and aerial photography captured post-peak. These measures are used in weighting the parameter space to produce multiple probabilistic predictions for the event. Two methods of assessing the reliability of these probabilistic predictions using limited observations are utilized; an existing method assessing the binary pattern of flooding, and a method developed in this paper to assess predictions of water surface elevation. This study finds that the water surface elevation method has both a better diagnostic and discriminatory ability, but this result is likely to be sensitive to the unknown uncertainties in the upstream boundary condition

Rejoinder to 'Performance of different synchronisation measures in real data: A case study on electroencephalograhic signals

We agree with Duckrow and Albano [Phys. Rev. E 67, 063901 (2003)] and Quian Quiroga et al. [Phys. Rev. E 67, 063902 (2003)] that mutual information (MI) is a useful measure of dependence for electroencephalogram (EEG) data, but we show that the improvement seen in the performance of MI on extracting dependence trends from EEG is more dependent on the type of MI estimator rather than any embedding technique used. In an independent study we conducted in search for an optimal MI estimator, and in particular for EEG applications, we examined the performance of a number of MI estimators on the data set used by Quian Quiroga et al. in their original study, where the performance of different dependence measures on real data was investigated [Phys. Rev. E 65, 041903 (2002)]. We show that for EEG applications the best performance among the investigated estimators is achieved by k-nearest neighbors, which supports the conjecture by Quian Quiroga et al. in Phys. Rev. E 67, 063902 (2003) that the nearest neighbor estimator is the most precise method for estimating MI.

Comment on "Performance of different synchronization measures in real data: A case study on electroencephalographic signals"

We agree with Duckrow and Albano [Phys. Rev. E 67, 063901 (2003)] and Quian Quiroga [Phys. Rev. E 67, 063902 (2003)] that mutual information (MI) is a useful measure of dependence for electroencephalogram (EEG) data, but we show that the improvement seen in the performance of MI on extracting dependence trends from EEG is more dependent on the type of MI estimator rather than any embedding technique used. In an independent study we conducted in search for an optimal MI estimator, and in particular for EEG applications, we examined the performance of a number of MI estimators on the data set used by Quian Quiroga in their original study, where the performance of different dependence measures on real data was investigated [Phys. Rev. E 65, 041903 (2002)]. We show that for EEG applications the best performance among the investigated estimators is achieved by k-nearest neighbors, which supports the conjecture by Quian Quiroga in Phys. Rev. E 67, 063902 (2003) that the nearest neighbor estimator is the most precise method for estimating MI.