On the Problems of Mixing RCTs with Qualitative Research: The Case of the MRC Framework for the Evaluation of Complex Healthcare Interventions

In the perceived hierarchy of research designs, the results from randomized controlled trials are considered to provide the highest level of evidence. Indeed these trials have been upheld as the gold standard in research. The benefits and limitations of the randomized controlled trial as a method of evaluating the effectiveness of healthcare interventions are presented. The article then examines the different levels of complexity within healthcare interventions and the problems this poses in determining effectiveness. In an effort to provide a solution to this problem, the Medical Research Council produced a framework to assist investigators to develop and evaluate complex healthcare interventions. The framework is described with reference to an example of implementing and evaluating protocols for weaning patients in the intensive care unit. The framework is critiqued on the basis that it involves an ambiguous or contradictory ontology, which fails to articulate the relationship between the positivism of randomized controlled trials with the relativism of qualitative approaches. It is concluded that the use of realist strategies in combination with randomized controlled trials provides the most coherent solution to this quandary

An evaluation of an adapted United States model of pharmaceutical care to improve psychoactive prescribing for nursing home residents in Northern Ireland (Fleetwood NI Study).

OBJECTIVES: To test the effect of an adapted U.S. model of pharmaceutical care on prescribing of inappropriate psychoactive (anxiolytic, hypnotic, and antipsychotic) medications and falls in nursing homes for older people in Northern Ireland (NI).

DESIGN: Cluster randomized controlled trial.

SETTING: Nursing homes randomized to intervention (receipt of the adapted model of care; n=11) or control (usual care continued; n=11).

PARTICIPANTS: Residents aged 65 and older who provided informed consent (N=334; 173 intervention, 161 control).

INTERVENTION: Specially trained pharmacists visited intervention homes monthly for 12 months and reviewed residents' clinical and prescribing information, applied an algorithm that guided them in assessing the appropriateness of psychoactive medication, and worked with prescribers (general practitioners) to improve the prescribing of these drugs. The control homes received usual care.

MEASUREMENTS: The primary end point was the proportion of residents prescribed one or more inappropriate psychoactive medicine according to standardized protocols; falls were evaluated using routinely collected falls data mandated by the regulatory body for nursing homes in NI.

RESULTS: The proportion of residents taking inappropriate psychoactive medications at 12 months in the intervention homes (25/128, 19.5%) was much lower than in the control homes (62/124, 50.0%) (odds ratio=0.26, 95% confidence interval=0.14–0.49) after adjustment for clustering within homes. No differences were observed at 12 months in the falls rate between the intervention and control groups.

CONCLUSION: Marked reductions in inappropriate psychoactive medication prescribing in residents resulted from pharmacist review of targeted medications, but there was no effect on falls.

Novel macrocyclic and acyclic cationic lipids for gene transfer: Synthesis and in vitro evaluation

The synthesis and in vitro evaluation of four cationic lipid gene delivery vectors, characterized by acyclic or macrocyclic, and saturated or unsaturated hydrophobic regions, is described. The synthesis employed standard protocols, including ring-closing metathesis for macrocyclic lipid construction. All lipoplexes studied, formulated from plasmid DNA and a liposome composed of a synthesized lipid, 1,2-dimyristoyl-sn-glycero-3-ethylphosphocholine (EPC), and either 1,2-dioleoyl-sn-glycero-3-phosphoethanolamine (DOPE) or cholesterol as co-lipid, exhibited plasmid DNA binding and protection from DNase I degradation, and concentration dependent cytotoxicity using Chinese hamster ovary-K1 cells. The transfection efficiency of formulations with cholesterol outperformed those with DOPE, and in many cases the EPC/cholesterol control, and formulations with a macrocyclic lipid (+/- 10:1) outperformed their acyclic counterparts (+/- 3:1).

A Qualitative Evaluation of the Provision of Bereavement Care Accessed by Service Users Living in a Health and Social Care Trust Area in Northern Ireland

Within the health and social care sector in the United Kingdom, the
management of death and bereavement has become increasingly
challenging. This service evaluation sought to explore the bereavement
care offered to individuals living in one Health and Social Care
Trust catchment area of Northern Ireland. Qualitative interviews
were conducted with key government and voluntary agency staff.
The findings indicated that much of the bereavement provision is
based on the interest and initiative of individual staff members, with
few processes to assess the level of bereavement care needed and those
best skilled to provide it. Recommendations are made for a bereavement
care strategy that outlines a bereavement needs assessment process,
identifying the scope of interventions and protocols for practice.

Managing patient deterioration: evaluation of what works.

Background: Rapid Response Systems (RRS) have been implemented nationally and internationally to improve patient safety in hospital. However, to date the majority of the RRS research evidence has focused on measuring the effectiveness of the intervention on patient outcomes. To evaluate RRS it has been recommended that a multimodal approach is required to address the broad range of process and outcome measures required to determine the effectiveness of the RRS concept. Aim: The aim of this paper is to evaluate the official RRS programme theoretical assumptions regarding how the programme is meant to work against actual practice in order to determine what works. Methods: The research design was a multiple case study approach of four wards in two hospitals in Northern Ireland. It followed the principles of realist evaluation research which allowed empirical data to be gathered to test and refine RRS programme theory [1]. This approach used a variety of mixed methods to test the programme theories including individual and focus group interviews with a purposive sample of 75 nurses and doctors, observation of ward practices and documentary analysis. The findings from the case studies were analysed and compared within and across cases to identify what works for whom and in what circumstances. Results: The RRS programme theories were critically evaluated and compared with study findings to develop a mid-range theory to explain what works, for whom in what circumstances. The findings of what works suggests that clinical experience, established working relationships, flexible implementation of protocols, ongoing experiential learning, empowerment and pre-emptive management are key to the success of RRS implementation.  Conclusion:These findings highlight the combination of factors that can improve the implementation of RRS and in light of this evidence several recommendations are made to provide policymakers with guidance and direction for their success and sustainability.References: 1.Pawson R and Tilley N. (1997) Realistic Evaluation. Sage Publications; LondonType of submission: Concurrent session Source of funding: Sandra Ryan Fellowship funded by the School of Nursing & Midwifery, Queen’s University of Belfast

Realistic evaluation of EWS and ALERT: factors enabling and constraining implementation.

Realistic Evaluation of EWS and ALERT: factors enabling and constraining implementation Background The implementation of EWS and ALERT in practice is essential to the success of Rapid Response Systems but is dependent upon nurses utilising EWS protocols and applying ALERT best practice guidelines. To date there is limited evidence on the effectiveness of EWS or ALERT as research has primarily focused on measuring patient outcomes (cardiac arrests, ICU admissions) following the implementation of a Rapid Response Team. Complex interventions in healthcare aimed at changing service delivery and related behaviour of health professionals require a different research approach to evaluate the evidence. To understand how and why EWS and ALERT work, or might not work, research needs to consider the social, cultural and organisational influences that will impact on successful implementation in practice. This requires a research approach that considers both the processes and outcomes of complex interventions, such as EWS and ALERT, implemented in practice. Realistic Evaluation is such an approach and was used to explain the factors that enable and constrain the implementation of EWS and ALERT in practice [1]. Aim The aim of this study was to evaluate factors that enabled and constrained the implementation and service delivery of early warnings systems (EWS) and ALERT in practice in order to provide direction for enabling their success and sustainability. Methods The research design was a multiple case study approach of four wards in two hospitals in Northern Ireland. It followed the principles of realist evaluation research which allowed empirical data to be gathered to test and refine RRS programme theory. This approach used a variety of mixed methods to test the programme theories including individual and focus group interviews, observation and documentary analysis in a two stage process. A purposive sample of 75 key informants participated in individual and focus group interviews. Observation and documentary analysis of EWS compliance data and ALERT training records provided further evidence to support or refute the interview findings. Data was analysed using NVIVO8 to categorise interview findings and SPSS for ALERT documentary data. These findings were further synthesised by undertaking a within and cross case comparison to explain the factors enabling and constraining EWS and ALERT. Results A cross case analysis highlighted similarities, differences and factors enabling or constraining successful implementation across the case study sites. Findings showed that personal (confidence; clinical judgement; personality), social (ward leadership; communication), organisational (workload and staffing issues; pressure from managers to complete EWS audit and targets), educational (constraints on training; no clinical educator on ward) and cultural (routine task delegated) influences impact on EWS and acute care training outcomes. There were also differences noted between medical and surgical wards across both case sites. Conclusions Realist Evaluation allows refinement and development of the RRS programme theory to explain the realities of practice. These refined RRS programme theories are capable of informing the planning of future service provision and provide direction for enabling their success and sustainability. References: 1. McGaughey J, Blackwood B, O’Halloran P, Trinder T. J. & Porter S. (2010) A realistic evaluation of Track and Trigger systems and acute care training for early recognition and management of deteriorating ward–based patients. Journal of Advanced Nursing 66 (4), 923-932. Type of submission: Concurrent session Source of funding: Sandra Ryan Fellowship funded by the School of Nursing & Midwifery, Queen’s University of Belfast

A realistic evaluation of Early Warning Systems and acute care training in Northern Ireland: findings and recommendations

Statement of purpose The purpose of this concurrent session is to present the main findings and recommendations from a five year study evaluating the implementation of Early Warning Systems (EWS) and the Acute Life-threatening Events: Recognition and Treatment (ALERT) course in Northern Ireland. The presentation will provide delegates with an understanding of those factors that enable and constrain successful implementation of EWS and ALERT in practice in order to provide an impetus for change. Methods The research design was a multiple case study approach of four wards in two hospitals in Northern Ireland. It followed the principles of realist evaluation research which allowed empirical data to be gathered to test and refine RRS programme theory [1]. The stages included identifying the programme theories underpinning EWS and ALERT, generating hypotheses, gathering empirical evidence and refining the programme theories. This approach used a variety of mixed methods including individual and focus group interviews, observation and documentary analysis of EWS compliance data and ALERT training records. A within and across case comparison facilitated the development of mid-range theories from the research evidence. Results The official RRS theories developed from the realist synthesis were critically evaluated and compared with the study findings to develop a mid-range theory to explain what works, for whom in what circumstances. The findings of what works suggests that clinical experience, established working relationships, flexible implementation of protocols, ongoing experiential learning, empowerment and pre-emptive management are key to the success of EWS and ALERT implementation. Each concept is presented as ‘context, mechanism and outcome configurations’ to provide an understanding of how the context impacts on individual reasoning or behaviour to produce certain outcomes. Conclusion These findings highlight the combination of factors that can improve the implementation and sustainability of EWS and ALERT and in light of this evidence several recommendations are made to provide policymakers with guidance and direction for future policy development. References: 1. Pawson R and Tilley N. (1997) Realistic Evaluation. Sage Publications; London Type of submission: Concurrent session Source of funding: Sandra Ryan Fellowship funded by the School of Nursing & Midwifery, Queen’s University of Belfast

Factors that impact on the use of mechanical ventilation weaning protocols in critically ill adults and children: a qualitative-evidence synthesis

Background:
Prolonged mechanical ventilation is associated with a longer intensive care unit (ICU) length of stay and higher mortality. Consequently, methods to improve ventilator weaning processes have been sought. Two recent Cochrane systematic reviews in ICU adult and paediatric populations concluded that protocols can be effective in reducing the duration of mechanical ventilation, but there was significant heterogeneity in study findings. Growing awareness of the benefits of understanding the contextual factors impacting on effectiveness has encouraged the integration of qualitative evidence syntheses with effectiveness reviews, which has delivered important insights into the reasons underpinning (differential) effectiveness of healthcare interventions.

Objectives:
1. To locate, appraise and synthesize qualitative evidence concerning the barriers and facilitators of the use of protocols for weaning critically-ill adults and children from mechanical ventilation;

2. To integrate this synthesis with two Cochrane effectiveness reviews of protocolized weaning to help explain observed heterogeneity by identifying contextual factors that impact on the use of protocols for weaning critically-ill adults and children from mechanical ventilation;

3. To use the integrated body of evidence to suggest the circumstances in which weaning protocols are most likely to be used.

Search methods:
We used a range of search terms identified with the help of the SPICE (Setting, Perspective, Intervention, Comparison, Evaluation) mnemonic. Where available, we used appropriate methodological filters for specific databases. We searched the following databases: Ovid MEDLINE, Embase, OVID, PsycINFO, CINAHL Plus, EBSCOHost, Web of Science Core Collection, ASSIA, IBSS, Sociological Abstracts, ProQuest and LILACS on the 26th February 2015. In addition, we searched: the grey literature; the websites of professional associations for relevant publications; and the reference lists of all publications reviewed. We also contacted authors of the trials included in the effectiveness reviews as well as of studies (potentially) included in the qualitative synthesis, conducted citation searches of the publications reporting these studies, and contacted content experts.

We reran the search on 3rd July 2016 and found three studies, which are awaiting classification.

Selection criteria:
We included qualitative studies that described: the circumstances in which protocols are designed, implemented or used, or both, and the views and experiences of healthcare professionals either involved in the design, implementation or use of weaning protocols or involved in the weaning of critically-ill adults and children from mechanical ventilation not using protocols. We included studies that: reflected on any aspect of the use of protocols, explored contextual factors relevant to the development, implementation or use of weaning protocols, and reported contextual phenomena and outcomes identified as relevant to the effectiveness of protocolized weaning from mechanical ventilation.

Data collection and analysis:
At each stage, two review authors undertook designated tasks, with the results shared amongst the wider team for discussion and final development. We independently reviewed all retrieved titles, abstracts and full papers for inclusion, and independently extracted selected data from included studies. We used the findings of the included studies to develop a new set of analytic themes focused on the barriers and facilitators to the use of protocols, and further refined them to produce a set of summary statements. We used the Confidence in the Evidence from Reviews of Qualitative Research (CERQual) framework to arrive at a final assessment of the overall confidence of the evidence used in the synthesis. We included all studies but undertook two sensitivity analyses to determine how the removal of certain bodies of evidence impacted on the content and confidence of the synthesis. We deployed a logic model to integrate the findings of the qualitative evidence synthesis with those of the Cochrane effectiveness reviews.

Main results:
We included 11 studies in our synthesis, involving 267 participants (one study did not report the number of participants). Five more studies are awaiting classification and will be dealt with when we update the review.

The quality of the evidence was mixed; of the 35 summary statements, we assessed 17 as ‘low’, 13 as ‘moderate’ and five as ‘high’ confidence. Our synthesis produced nine analytical themes, which report potential barriers and facilitators to the use of protocols. The themes are: the need for continual staff training and development; clinical experience as this promotes felt and perceived competence and confidence to wean; the vulnerability of weaning to disparate interprofessional working; an understanding of protocols as militating against a necessary proactivity in clinical practice; perceived nursing scope of practice and professional risk; ICU structure and processes of care; the ability of protocols to act as a prompt for shared care and consistency in weaning practice; maximizing the use of protocols through visibility and ease of implementation; and the ability of protocols to act as a framework for communication with parents.

Authors' conclusions:
There is a clear need for weaning protocols to take account of the social and cultural environment in which they are to be implemented. Irrespective of its inherent strengths, a protocol will not be used if it does not accommodate these complexities. In terms of protocol development, comprehensive interprofessional input will help to ensure broad-based understanding and a sense of ‘ownership’. In terms of implementation, all relevant ICU staff will benefit from general weaning as well as protocol-specific training; not only will this help secure a relevant clinical knowledge base and operational understanding, but will also demonstrate to others that this knowledge and understanding is in place. In order to maximize relevance and acceptability, protocols should be designed with the patient profile and requirements of the target ICU in mind. Predictably, an under-resourced ICU will impact adversely on protocol implementation, as staff will prioritize management of acutely deteriorating and critically-ill patients.

Impact of Referral Protocols on the Dental Management of Patients Undergoing Treatment for Head and Neck Oncology in Northern Ireland

Objective: To evaluate the handling, by a group of practice-based researchers, of a recently introduced bulk fill resin-based composite restorative material, Filtek Bulk Fill Restorative (3M ESPE).

Methods: The twelve selected evaluators were sent explanatory letters, a pack of the material under investigation to use for 8 weeks, and a questionnaire.

Results: The evaluators rated the ease of use of the bulk fill restorative the same as the previously used posterior composite material. The provision of one shade only for evaluation may have compromised the score for aesthetic quality. No post-operative sensitivity was reported.

Conclusions: The bulk fill material was well received as indicated by the high number of evaluators who would both purchase the material and recommend it to colleagues.

Clinical relevance: A recently introduced bulk fill restorative material achieved a rating for handling which was similar to the evaluators’ previously used resin composite, although there were some concerns regarding the translucency of the material.

Systematic evaluation of patient-reported outcome (PRO) protocol content and reporting in UK cancer clinical trials: the EPiC study protocol

Introduction Emerging evidence suggests that patient-reported outcome (PRO)-specific information may be omitted in trial protocols and that PRO results are poorly reported, limiting the use of PRO data to inform cancer care. This study aims to evaluate the standards of PRO-specific content in UK cancer trial protocols and their arising publications and to highlight examples of best-practice PRO protocol content and reporting where they occur. The objective of this study is to determine if these early findings are generalisable to UK cancer trials, and if so, how best we can bring about future improvements in clinical trials methodology to enhance the way PROs are assessed, managed and reported. Hypothesis: Trials in which the primary end point is based on a PRO will have more complete PRO protocol and publication components than trials in which PROs are secondary end points.

Methods and analysis Completed National Institute for Health Research (NIHR) Portfolio Cancer clinical trials (all cancer specialities/age-groups) will be included if they contain a primary/secondary PRO end point. The NIHR portfolio includes cancer trials, supported by a range of funders, adjudged as high-quality clinical research studies. The sample will be drawn from studies completed between 31 December 2000 and 1 March 2014 (n=1141) to allow sufficient time for completion of the final trial report and publication. Two reviewers will then review the protocols and arising publications of included trials to: (1) determine the completeness of their PRO-specific protocol content; (2) determine the proportion and completeness of PRO reporting in UK Cancer trials and (3) model factors associated with PRO protocol and reporting completeness and with PRO reporting proportion.

Ethics and dissemination The study was approved by the ethics committee at University of Birmingham (ERN_15-0311). Trial findings will be disseminated via presentations at local, national and international conferences, peer-reviewed journals and social media including the CPROR twitter account and UOB departmental website (http://www.birmingham.ac.uk/cpro0r).