Statistical Tests for Joint Analysis of Performance Measures

Recently there has been an increasing interest in the development of new methods using Pareto optimality to deal with multi-objective criteria (for example, accuracy and architectural complexity). Once one has learned a model based on their devised method, the problem is then how to compare it with the state of art. In machine learning, algorithms are typically evaluated by comparing their performance on different data sets by means of statistical tests. Unfortunately, the standard tests used for this purpose are not able to jointly consider performance measures. The aim of this paper is to resolve this issue by developing statistical procedures that are able to account for multiple competing measures at the same time. In particular, we develop two tests: a frequentist procedure based on the generalized likelihood-ratio test and a Bayesian procedure based on a multinomial-Dirichlet conjugate model. We further extend them by discovering conditional independences among measures to reduce the number of parameter of such models, as usually the number of studied cases is very reduced in such comparisons. Real data from a comparison among general purpose classifiers is used to show a practical application of our tests.

Joint Analysis of Multiple Algorithms and Performance Measures

There has been an increasing interest in the development of new methods using Pareto optimality to deal with multi-objective criteria (for example, accuracy and time complexity). Once one has developed an approach to a problem of interest, the problem is then how to compare it with the state of art. In machine learning, algorithms are typically evaluated by comparing their performance on different data sets by means of statistical tests. Standard tests used for this purpose are able to consider jointly neither performance measures nor multiple competitors at once. The aim of this paper is to resolve these issues by developing statistical procedures that are able to account for multiple competing measures at the same time and to compare multiple algorithms altogether. In particular, we develop two tests: a frequentist procedure based on the generalized likelihood-ratio test and a Bayesian procedure based on a multinomial-Dirichlet conjugate model. We further extend them by discovering conditional independences among measures to reduce the number of parameters of such models, as usually the number of studied cases is very reduced in such comparisons. Data from a comparison among general purpose classifiers is used to show a practical application of our tests.

Developing management capability within a horizontal supply chain in performance measurement deployment and evolution: A Dynamic Capabilities and Goal Theory perspective

Complex collaboration in rapidly changing business environments create challenges for management capability in Utility Horizontal Supply Chains (UHSCs) involving the deploying and evolving of performance measures. The aim of the study is twofold. First, there is a need to explore how management capability can be developed and used to deploy and evolve Performance Measurement (PM), both across a UHSC and within its constituent organisations, drawing upon a theoretical nexus of Dynamic Capability (DC) theory and complementary Goal Theory. Second, to make a contribution to knowledge by empirically building theory using these constructs to show the management motivations and behaviours within PM-based DCs. The methodology uses an interpretive theory building, multiple case based approach (n=3) as part of a USHC. The data collection methods include, interviews (n=54), focus groups (n=10), document analysis and participant observation (reflective learning logs) over a five-year period giving longitudinal data. The empirical findings lead to the development of a conceptual framework showing that management capabilities in driving PM deployment and evolution can be represented as multilevel renewal and incremental Dynamic Capabilities, which can be further understood in terms of motivation and behaviour by Goal-Theoretic constructs. In addition three interrelated cross cutting themes of management capabilities in consensus building, goal setting and resource change were identified. These management capabilities require carefully planned development and nurturing within the UHSC. 

Reaching common ground:A patient-family-based conceptual framework of quality EOL care

Improvement in the quality of end-of-life (EOL) care is a priority health care issue since serious deficiencies in quality of care have been reported across care settings. Increasing pressure is now focused on Canadian health care organizations to be accountable for the quality of palliative and EOL care delivered. Numerous domains of quality EOL care upon which to create accountability frameworks are now published, with some derived from the patient/family perspective. There is a need to reach common ground on the domains of quality EOL care valued by patients and families in order to develop consistent performance measures and set priorities for health care improvement. This paper describes a meta-synthesis study to develop a common conceptual framework of quality EOL care integrating attributes of quality valued by patients and their families. © 2005 Centre for Bioethics, IRCM.

Rapid detection of health-care-associated bloodstream infection in critical care using multipathogen real-time polymerase chain reaction technology: a diagnostic accuracy study and systematic review

Background: There is growing interest in the potential utility of real-time polymerase chain reaction (PCR) in diagnosing bloodstream infection by detecting pathogen deoxyribonucleic acid (DNA) in blood samples within a few hours. SeptiFast (Roche Diagnostics GmBH, Mannheim, Germany) is a multipathogen probe-based system targeting ribosomal DNA sequences of bacteria and fungi. It detects and identifies the commonest pathogens causing bloodstream infection. As background to this study, we report a systematic review of Phase III diagnostic accuracy studies of SeptiFast, which reveals uncertainty about its likely clinical utility based on widespread evidence of deficiencies in study design and reporting with a high risk of bias. 

Objective: Determine the accuracy of SeptiFast real-time PCR for the detection of health-care-associated bloodstream infection, against standard microbiological culture. 

Design: Prospective multicentre Phase III clinical diagnostic accuracy study using the standards for the reporting of diagnostic accuracy studies criteria. 

Setting: Critical care departments within NHS hospitals in the north-west of England. 

Participants: Adult patients requiring blood culture (BC) when developing new signs of systemic inflammation. 

Main outcome measures: SeptiFast real-time PCR results at species/genus level compared with microbiological culture in association with independent adjudication of infection. Metrics of diagnostic accuracy were derived including sensitivity, specificity, likelihood ratios and predictive values, with their 95% confidence intervals (CIs). Latent class analysis was used to explore the diagnostic performance of culture as a reference standard. 

Results: Of 1006 new patient episodes of systemic inflammation in 853 patients, 922 (92%) met the inclusion criteria and provided sufficient information for analysis. Index test assay failure occurred on 69 (7%) occasions. Adult patients had been exposed to a median of 8 days (interquartile range 4–16 days) of hospital care, had high levels of organ support activities and recent antibiotic exposure. SeptiFast real-time PCR, when compared with culture-proven bloodstream infection at species/genus level, had better specificity (85.8%, 95% CI 83.3% to 88.1%) than sensitivity (50%, 95% CI 39.1% to 60.8%). When compared with pooled diagnostic metrics derived from our systematic review, our clinical study revealed lower test accuracy of SeptiFast real-time PCR, mainly as a result of low diagnostic sensitivity. There was a low prevalence of BC-proven pathogens in these patients (9.2%, 95% CI 7.4% to 11.2%) such that the post-test probabilities of both a positive (26.3%, 95% CI 19.8% to 33.7%) and a negative SeptiFast test (5.6%, 95% CI 4.1% to 7.4%) indicate the potential limitations of this technology in the diagnosis of bloodstream infection. However, latent class analysis indicates that BC has a low sensitivity, questioning its relevance as a reference test in this setting. Using this analysis approach, the sensitivity of the SeptiFast test was low but also appeared significantly better than BC. Blood samples identified as positive by either culture or SeptiFast real-time PCR were associated with a high probability (> 95%) of infection, indicating higher diagnostic rule-in utility than was apparent using conventional analyses of diagnostic accuracy. 

Conclusion: SeptiFast real-time PCR on blood samples may have rapid rule-in utility for the diagnosis of health-care-associated bloodstream infection but the lack of sensitivity is a significant limiting factor. Innovations aimed at improved diagnostic sensitivity of real-time PCR in this setting are urgently required. Future work recommendations include technology developments to improve the efficiency of pathogen DNA extraction and the capacity to detect a much broader range of pathogens and drug resistance genes and the application of new statistical approaches able to more reliably assess test performance in situation where the reference standard (e.g. blood culture in the setting of high antimicrobial use) is prone to error.

Performance Management in the UK Public Sector: addressing multiple stakeholder complexity

Purpose The aim of this paper is to explore the issues involved in developing and applying performance management approaches within a large UK public sector department using a multiple stakeholder perspective and an accompanying theoretical framework. Design/methodology/approach An initial short questionnaire was used to determine perceptions about the implementation and effectiveness of the new performance management system across the organisation. In total, 700 questionnaires were distributed. Running concurrently with an ethnographic approach, and informed by the questionnaire responses, was a series of semi-structured interviews and focus groups. Findings Staff at all levels had an understanding of the new system and perceived it as being beneficial. However, there were concerns that the approach was not continuously managed throughout the year and was in danger of becoming an annual event, rather than an ongoing process. Furthermore, the change process seemed to have advanced without corresponding changes to appraisal and reward and recognition systems. Thus, the business objectives were not aligned with motivating factors within the organisation. Research limitations/implications Additional research to test the validity and usefulness of the theoretical model, as discussed in this paper, would be beneficial. Practical implications The strategic integration of the stakeholder performance measures and scorecards was found to be essential to producing an overall stakeholder-driven strategy within the case study organisation. Originality/value This paper discusses in detail the approach adopted and the progress made by one large UK public sector organisation, as it attempts to develop better relationships with all of its stakeholders and hence improve its performance. This paper provides a concerted attempt to link theory with practice.

Probabilistic classification of acute myocardial infarction from multiple cardiac markers

Logistic regression and Gaussian mixture model (GMM) classifiers have been trained to estimate the probability of acute myocardial infarction (AMI) in patients based upon the concentrations of a panel of cardiac markers. The panel consists of two new markers, fatty acid binding protein (FABP) and glycogen phosphorylase BB (GPBB), in addition to the traditional cardiac troponin I (cTnI), creatine kinase MB (CKMB) and myoglobin. The effect of using principal component analysis (PCA) and Fisher discriminant analysis (FDA) to preprocess the marker concentrations was also investigated. The need for classifiers to give an accurate estimate of the probability of AMI is argued and three categories of performance measure are described, namely discriminatory ability, sharpness, and reliability. Numerical performance measures for each category are given and applied. The optimum classifier, based solely upon the samples take on admission, was the logistic regression classifier using FDA preprocessing. This gave an accuracy of 0.85 (95% confidence interval: 0.78-0.91) and a normalised Brier score of 0.89. When samples at both admission and a further time, 1-6 h later, were included, the performance increased significantly, showing that logistic regression classifiers can indeed use the information from the five cardiac markers to accurately and reliably estimate the probability AMI. © Springer-Verlag London Limited 2008.

Understanding New Product Project Performance

Purpose – The purpose of this paper is to summarize the accumulated body of knowledge on the performance of new product projects and provide directions for further research. Design/methodology/approach – Using a refined classification of antecedents of new product project performance the research results are meta-analyzed in the literature in order to identify the strength and stability of predictor-performance relationships. Findings – The results reveal that 22 variables have a significant relationship with new product project performance, of which only 12 variables have a sizable relationship. In order of importance these factors are the degree of organizational interaction, R&D and marketing interface, general product development proficiency, product advantage, financial/business analysis, technical proficiency, management skill, marketing proficiency, market orientation, technology synergy, project manager competency and launch activities. Of the 34 variables 16 predictors show potential for moderator effects. Research limitations/implications – The validity of the results is constrained by publication bias and heterogeneity of performance measures, and directions for the presentation of data in future empirical publications are provided. Practical implications – This study helps new product project managers in understanding and managing the performance of new product development projects. Originality/value – This paper provides unique insights into the importance of predictors of new product performance at the project level. Furthermore, it identifies which predictor-performance relations are contingent on other factors.

The development of University Technology Transfer stakeholder relationships at a regional level: Lessons for the future

The increasing importance placed upon regional development and the knowledge-based economy as economic growth stimuli has led to a changing role for Universities and their interaction with the business community through (though not limited to) the transfer of technology from academia to industry. With the emergence of Local Enterprise Partnerships (LEPs) replacing the Regional Development Agencies (RDAs), there is a need for policy and practice going forward to be clearly informed by a critique of TTO (Technology Transfer Office)–RDA stakeholder relationship in a lessons learned approach so that LEPs can benefit from a faster learning curve. Thus, the aim of this paper is to examine the stakeholder relationship between three regional universities in the context of its TTO and the RDA with a view to determining lessons learned for the emerging LEP approach. Although the issues raised are contextual, the abstracted stakeholder conceptualisation of the TTO–RDA relationship should enable wider generalisation of the issues raised beyond the UK. Stakeholder theory relationship and stage development models are used to guide a repeat interview study of the TTO and RDA stakeholder groupings. The findings, interpreted using combined category and stage based stakeholder models, show how the longitudinal development of the TTO–RDA stakeholder relationship for each case has progressed through different stakeholder pathways, and stages where specific targeting of funding was dependant on the stakeholder stage. Greater targeted policy and funding, based on the stakeholder relationship approach, led to the development of joint mechanisms and a closer alignment of performance measures between the TTO and the RDA. However, over-reliance on the unitary nature of the TTO–RDA relationship may lead to a lack of cultivation and dependency for funding from other stakeholders.

Modelling the Performance of Irish Credit Unions, 2002 to 2010

This study undertakes a modeling based performance assessment of all Irish credit unions between 2002 and 2010, a particularly turbulent period in their history. The analysis explicitly addresses the current challenges faced by credit unions in that the modeling approach used rewards credit unions for reducing undesirable outputs (impaired loans and investments) as well as for increasing desirable outputs (loans, earning assets and members’ funds) and decreasing inputs (labour expenditure, capital expenditure and fund expenses). The main findings are: credit unions are subject to increasing returns to scale; technical regression occurred in the years after 2007; there is significant scope for an improvement in efficiency through expansion of desirable outputs and contraction of undesirable outputs and inputs; and that larger credit unions, that are better capitalised and pay a higher dividend to members are more efficient than their smaller, less capitalised, and lower dividend paying counterparts.

Enabling Complexity-Performance Trade-Offs for Successive Cancellation Decoding of Polar Codes

Polar codes are one of the most recent advancements in coding theory and they have attracted significant interest. While they are provably capacity achieving over various channels, they have seen limited practical applications. Unfortunately, the successive nature of successive cancellation based decoders hinders fine-grained adaptation of the decoding complexity to design constraints and operating conditions. In this paper, we propose a systematic method for enabling complexity-performance trade-offs by constructing polar codes based on an optimization problem which minimizes the complexity under a suitably defined mutual information based performance constraint. Moreover, a low-complexity greedy algorithm is proposed in order to solve the optimization problem efficiently for very large code lengths.

Aggressiveness as a component of fighting ability in pigs using a game-theoretical framework

Understanding animal contests has benefited greatly from employing the concept of fighting ability, termed resource-holding potential (RHP), with body size/weight typically used as a proxy. However, victory does not always go to the larger/heavier contestant and the existing RHP approach thereby fails to accurately predict contest outcome. Aggressiveness, typically studied as a personality trait, might explain part of this discrepancy. We investigated whether aggressiveness forms a component of RHP, examining effects on contest outcome, duration and phases, plus physiological measures of costs (lactate and glucose). Furthermore, using the correct theoretical framework, we provide the first study to investigate whether individuals gather and use information on aggressiveness as part of an assessment strategy. Pigs, Sus scrofa, were assessed for aggressiveness in resident-intruder tests whereby attack latency reflects aggressiveness. Contests were then staged between size-matched animals diverging in aggressiveness. Individuals with a short attack latency in the resident-intruder test almost always initiated the first bite and fight in the subsequent contest. However, aggressiveness had no direct effect on contest outcome, whereas bite initiation did lead to winning in contests without an escalated fight. This indirect effect suggests that aggressiveness is not a component of RHP, but rather reflects a signal of intent. Winner and loser aggressiveness did not affect contest duration or its separate phases, suggesting aggressiveness is not part of an assessment strategy. A greater asymmetry in aggressiveness prolonged contest duration and the duration of displaying, which is in a direction contrary to assessment models based on morphological traits. Blood lactate and glucose increased with contest duration and peaked during escalated fights, highlighting the utility of physiological measures as proxies for fight cost. Integrating personality traits into the study of contest behaviour, as illustrated here, will enhance our understanding of the subtleties of agonistic interactions.