On the Problems of Mixing RCTs with Qualitative Research: The Case of the MRC Framework for the Evaluation of Complex Healthcare Interventions

In the perceived hierarchy of research designs, the results from randomized controlled trials are considered to provide the highest level of evidence. Indeed these trials have been upheld as the gold standard in research. The benefits and limitations of the randomized controlled trial as a method of evaluating the effectiveness of healthcare interventions are presented. The article then examines the different levels of complexity within healthcare interventions and the problems this poses in determining effectiveness. In an effort to provide a solution to this problem, the Medical Research Council produced a framework to assist investigators to develop and evaluate complex healthcare interventions. The framework is described with reference to an example of implementing and evaluating protocols for weaning patients in the intensive care unit. The framework is critiqued on the basis that it involves an ambiguous or contradictory ontology, which fails to articulate the relationship between the positivism of randomized controlled trials with the relativism of qualitative approaches. It is concluded that the use of realist strategies in combination with randomized controlled trials provides the most coherent solution to this quandary

A Framework for the Evaluation of Digital Musical Instruments

At the outset of a discussion of evaluating digital musical instruments, that is to say instruments whose sound generators are digital and separable though not necessarily separate from their control interfaces (Malloch, 2006), it is reasonable to ask what the term evaluation in this context really means. After all, there may be many perspectives from which to view the effectiveness or otherwise of the instruments we build. For most performers, performance on an instrument becomes a means of evaluating how well it functions in the context of live music making, and their measure of success is the response of the audience to their performance. Audiences evaluate performances on the basis of how engaged they feel they have been by what they have seen and heard. When questioned, they are likely to describe good performances as “exciting,” “skillful,” “musical.” Bad performances are “boring,” and those which are marred by technical malfunction are often dismissed out of hand. If performance is considered to be a valid means of evaluating a musical instrument, then it follows that, for the field of DMI design, a much broader definition of the term “evaluation” than that typically used in human-computer interaction (HCI) is required to reflect the fact that there are a number of stakeholders involved in the design and evaluation of DMIs. In addition to players and audiences, there are also composers, instrument builders, component manufacturers, and perhaps even customers, each of whom will have a different concept of what is meant by “evaluation.”

Implementation and Evaluation of Measurement-Based Admission Control Schemes Within a Converged Networks QoS Management Framework

Policy-based network management (PBNM) paradigms provide an effective tool for end-to-end resource
management in converged next generation networks by enabling unified, adaptive and scalable solutions
that integrate and co-ordinate diverse resource management mechanisms associated with heterogeneous
access technologies. In our project, a PBNM framework for end-to-end QoS management in converged
networks is being developed. The framework consists of distributed functional entities managed within a
policy-based infrastructure to provide QoS and resource management in converged networks. Within any
QoS control framework, an effective admission control scheme is essential for maintaining the QoS of
flows present in the network. Measurement based admission control (MBAC) and parameter basedadmission control (PBAC) are two commonly used approaches. This paper presents the implementationand analysis of various measurement-based admission control schemes developed within a Java-based
prototype of our policy-based framework. The evaluation is made with real traffic flows on a Linux-based experimental testbed where the current prototype is deployed. Our results show that unlike with classic MBAC or PBAC only schemes, a hybrid approach that combines both methods can simultaneously result in improved admission control and network utilization efficiency

THE SAAF STUDY: evaluation of the Safeguarding Children Assessment and Analysis Framework (SAAF), compared with management as usual, for improving outcomes for children and young people who have experienced, or are at risk of, maltreatment: study protocol for a randomised controlled trial

Background: Serious case reviews and research studies have indicated weaknesses in risk assessments conducted by child protection social workers. Social workers are adept at gathering information but struggle with analysis and assessment of risk. The Department for Education wants to know if the use of a structured decision-making tool can improve child protection assessments of risk.

Methods/design: This multi-site, cluster-randomised trial will assess the effectiveness of the Safeguarding Children Assessment and Analysis Framework (SAAF). This structured decision-making tool aims to improve social workers' assessments of harm, of future risk and parents' capacity to change. The comparison is management as usual.

Inclusion criteria: Children's Services Departments (CSDs) in England willing to make relevant teams available to be randomised, and willing to meet the trial's training and data collection requirements.

Exclusion criteria: CSDs where there were concerns about performance; where a major organisational restructuring was planned or under way; or where other risk assessment tools were in use.

Six CSDs are participating in this study. Social workers in the experimental arm will receive 2 days training in SAAF together with a range of support materials, and access to limited telephone consultation post-training. The primary outcome is child maltreatment. This will be assessed using data collected nationally on two key performance indicators: the first is the number of children in a year who have been subject to a second Child Protection Plan (CPP); the second is the number of re-referrals of children because of related concerns about maltreatment. Secondary outcomes are: i) the quality of assessments judged against a schedule of quality criteria and ii) the relationship between the three assessments required by the structured decision-making tool (level of harm, risk of (re) abuse and prospects for successful intervention).

Discussion: This is the first study to examine the effectiveness of SAAF. It will contribute to a very limited literature on the contribution that structured decision-making tools can make to improving risk assessment and case planning in child protection and on what is involved in their effective implementation.

Measuring healthcare integration: Operationalization of a framework for a systems evaluation of palliative care structures, processes, and outcomes

BACKGROUND: Healthcare integration is a priority in many countries, yet there remains little direction on how to systematically evaluate this construct to inform further development. The examination of community-based palliative care networks provides an ideal opportunity for the advancement of integration measures, in consideration of how fundamental provider cohesion is to effective care at end of life.

AIM: This article presents a variable-oriented analysis from a theory-based case study of a palliative care network to help bridge the knowledge gap in integration measurement.

DESIGN: Data from a mixed-methods case study were mapped to a conceptual framework for evaluating integrated palliative care and a visual array depicting the extent of key factors in the represented palliative care network was formulated.

SETTING/PARTICIPANTS: The study included data from 21 palliative care network administrators, 86 healthcare professionals, and 111 family caregivers, all from an established palliative care network in Ontario, Canada.

RESULTS: The framework used to guide this research proved useful in assessing qualities of integration and functioning in the palliative care network. The resulting visual array of elements illustrates that while this network performed relatively well at the multiple levels considered, room for improvement exists, particularly in terms of interventions that could facilitate the sharing of information.

CONCLUSION: This study, along with the other evaluative examples mentioned, represents important initial attempts at empirically and comprehensively examining network-integrated palliative care and healthcare integration in general.

Repply to: Brito, A.C., Brotas, V., Caetano, M., Coutinho, T.P., Bordalo, A.A., Icely, J., Neto, J.M., Serôdio, J., Moita, T. (2012) "Defining phytoplankton class boundaries in Portuguese transitional waters: an evaluation of the ecological quality status according to the Water Framework Directive" [Ecological Indicators 19, 5-14]

We write to comment on the recently published paper “Defining phytoplankton class boundaries in Portuguese transitional waters: an evaluation of the ecological quality status according to the Water Framework Directive” (Brito et al., 2012). This paper presents an integrated methodology to analyse the ecological quality status of several Portuguese transitional waters, using phytoplanktonrelated metrics. One of the systems analysed, the Guadiana estuary in southern Portugal, is considered the most problematic estuary, with its upstream water bodies classified as Poor in terms of ecological status. We strongly disagree with this conclusion and we would like to raise awareness to some methodological constraints that, in our opinion, are the basis of such deceptive conclusions and should therefore not be neglected when using phytoplankton to assess the ecological status of natural waters.

Specifying a programming exercises evaluation service on the e-Framework

The e-Framework is arguably the most prominent e-learning framework currently in use. For this reason it was selected as basis for modelling a programming exercises evaluation service. The purpose of this type of evaluator is to mark and grade exercises in computer programming courses and in programming contests. By exposing its functions as services a programming exercise evaluator is able to participate in business processes integrating different system types, such as Programming Contest Management Systems, Learning Management Systems, Integrated Development Environments and Learning Object Repositories. This paper formalizes the approaches to be used in the implementation of a programming exercise evaluator as a service on the e-Framework.

Impact evaluation of a Kenyan Eco Business: a pilot framework.

Promoting social and environmental entrepreneurship is suggested by many, including international institutions, national development agencies and non-governmental organizations, as critical in tackling longstanding complex global sustainable development problems. However while interest in this kind of alternative entrepreneurship grows, with a proliferation of claims made about its potential to catalyze societal transformation, research in this field remains nascent and fragmented, particularly in relation to Africa. There are few examples of work systematically examining the impacts of social and environmental enterprises on sustainable development and poverty alleviation, especially research based on rigorous empirical fieldwork. This paper begins addressing these limitations by proposing a framework for mapping the sustainable development and poverty alleviation impacts of social and environmental enterprises in Africa. This framework is then piloted with reference to a Kenyan ecobusiness.

Towards improving the framework for probabilistic forecast evaluation

The evaluation of forecast performance plays a central role both in the interpretation and use of forecast systems and in their development. Different evaluation measures (scores) are available, often quantifying different characteristics of forecast performance. The properties of several proper scores for probabilistic forecast evaluation are contrasted and then used to interpret decadal probability hindcasts of global mean temperature. The Continuous Ranked Probability Score (CRPS), Proper Linear (PL) score, and IJ Good’s logarithmic score (also referred to as Ignorance) are compared; although information from all three may be useful, the logarithmic score has an immediate interpretation and is not insensitive to forecast busts. Neither CRPS nor PL is local; this is shown to produce counter intuitive evaluations by CRPS. Benchmark forecasts from empirical models like Dynamic Climatology place the scores in context. Comparing scores for forecast systems based on physical models (in this case HadCM3, from the CMIP5 decadal archive) against such benchmarks is more informative than internal comparison systems based on similar physical simulation models with each other. It is shown that a forecast system based on HadCM3 out performs Dynamic Climatology in decadal global mean temperature hindcasts; Dynamic Climatology previously outperformed a forecast system based upon HadGEM2 and reasons for these results are suggested. Forecasts of aggregate data (5-year means of global mean temperature) are, of course, narrower than forecasts of annual averages due to the suppression of variance; while the average “distance” between the forecasts and a target may be expected to decrease, little if any discernible improvement in probabilistic skill is achieved.

A Computer Vision Framework For Finger-Tapping Evaluation In Parkinson's Disease

Objective: To define and evaluate a Computer-Vision (CV) method for scoring Paced Finger-Tapping (PFT) in Parkinson's disease (PD) using quantitative motion analysis of index-fingers and to compare the obtained scores to the UPDRS (Unified Parkinson's Disease Rating Scale) finger-taps (FT). Background: The naked-eye evaluation of PFT in clinical practice results in coarse resolution to determine PD status. Besides, sensor mechanisms for PFT evaluation may cause patients discomfort. In order to avoid cost and effort of applying wearable sensors, a CV system for non-invasive PFT evaluation is introduced. Methods: A database of 221 PFT videos from 6 PD patients was processed. The subjects were instructed to position their hands above their shoulders besides the face and tap the index-finger against the thumb consistently with speed. They were facing towards a pivoted camera during recording. The videos were rated by two clinicians between symptom levels 0-to-3 using UPDRS-FT. The CV method incorporates a motion analyzer and a face detector. The method detects the face of testee in each video-frame. The frame is split into two images from face-rectangle center. Two regions of interest are located in each image to detect index-finger motion of left and right hands respectively. The tracking of opening and closing phases of dominant hand index-finger produces a tapping time-series. This time-series is normalized by the face height. The normalization calibrates the amplitude in tapping signal which is affected by the varying distance between camera and subject (farther the camera, lesser the amplitude). A total of 15 features were classified using K-nearest neighbor (KNN) classifier to characterize the symptoms levels in UPDRS-FT. The target ratings provided by the raters were averaged. Results: A 10-fold cross validation in KNN classified 221 videos between 3 symptom levels with 75% accuracy. An area under the receiver operating characteristic curves of 82.6% supports feasibility of the obtained features to replicate clinical assessments. Conclusions: The system is able to track index-finger motion to estimate tapping symptoms in PD. It has certain advantages compared to other technologies (e.g. magnetic sensors, accelerometers etc.) for PFT evaluation to improve and automate the ratings

Evaluation of Resin Bond Strength to Yttria-stabilized Tetragonal Zirconia and Framework Marginal Fit: Comparison of Different Surface Conditionings

Fundação de Amparo à Pesquisa do Estado de São Paulo (FAPESP)