Realist evaluation - promise, problems and practicalities

Realist evaluation is an innovative, multi-method approach to evaluating the effectiveness of complex health care interventions that is having an increasing impact on the research community. Drawing on their experience doing four realist evaluations in diverse areas of healthcare, the authors offer a comprehensive overview and critique of essential theory and practice. The first paper (Realist review and realist evaluation: an introduction) introduces the main components of the approach and shows how realist review can support realist evaluation. The second paper (Concepts and methodology for realist evaluation: help or hindrance?) provides further detail on the key concepts, shows how they can be operationalised, and discusses the advantages and difficulties of using these ideas. Following these two papers introducing and illustrating the major concepts, the third paper (Realist Evaluation: a critical realist critique) takes a step back to re-consider realist evaluation in relation to its critical realist roots, asking whether it leads to evaluators abandoning the attempt to understand (and if necessary challenge) the underlying values of health care interventions and contenting themselves merely with explicating the factors that help or hinder implementation. The fourth and final paper (Data analysis and theory development in realist evaluation) plunges back into the tangled undergrowth of multiple-method data collection and shows how disparate forms of data can be synthesised for theory development, and the results presented in a form that is useful to practitioners and policy-makers.

Factors affecting the successful implementation and sustainability of the Liverpool Care Pathway for dying patients – a realist evaluation

Objectives: The Liverpool Care Pathway for the dying patient (LCP) was designed to improve end-of-life care in generalist health care settings. Controversy has led to its withdrawal in some jurisdictions. The main objective of this research was to identify the influences that facilitated or hindered successful LCP implementation.

Method: An organisational case study using realist evaluation in one health and social care trust in Northern Ireland. Two rounds of semi-structured interviews were conducted with two policy makers and twenty two participants with experience and/or involvement in management of the LCP during 2011 and 2012.

Results: Key resource inputs included facilitation with a view to maintaining LCP ‘visibility’, reducing anxiety among nurses and increasing their confidence regarding the delivery of end-of-life care; and nurse and medical education designed to increase professional self-efficacy and reduce misuse and misunderstanding of the LCP. Key enabling contexts were consistent senior management support; ongoing education and training tailored to the needs of each professional group; and an organisational cultural change in the hospital setting that encompassed end-of-life care.

Conclusion: There is a need to appreciate the organizationally complex nature of intervening to improve end-of-life care. Successful implementation of evidence-based interventions for end-of-life care requires commitment to planning, training and ongoing review that takes account of different perspectives, institutional hierarchies and relationships and the educational needs of professional disciplines. There is a need also to recognise that medical consultants require particular support in their role as gatekeepers and as a lead communication channel with patients and their relatives.

The Management of Long-Term Sickness Absence in Large Public Sector Healthcare Organisations: A Realist Evaluation Using Mixed Methods

Purpose The success of measures to reduce long-term sickness absence (LTSA) in public sector organisations is contingent on organisational context. This realist evaluation investigates how interventions interact with context to influence successful management of LTSA. Methods Multi-method case study in three Health and Social Care Trusts in Northern Ireland comprising realist literature review, semi-structured interviews (61 participants), Process-Mapping and feedback meetings (59 participants), observation of training, analysis of documents. Results Important activities included early intervention; workplace-based occupational rehabilitation; robust sickness absence policies with clear trigger points for action. Used appropriately, in a context of good interpersonal and interdepartmental communication and shared goals, these are able to increase the motivation of staff to return to work. Line managers are encouraged to take a proactive approach when senior managers provide support and accountability. Hindering factors: delayed intervention; inconsistent implementation of policy and procedure; lack of resources; organisational complexity; stakeholders misunderstanding each other’s goals and motives. Conclusions Different mechanisms have the potential to encourage common motivations for earlier return from LTSA, such as employees feeling that they have the support of their line manager to return to work and having the confidence to do so. Line managers’ proactively engage when they have confidence in the support of seniors and in their own ability to address LTSA. Fostering these motivations calls for a thoughtful, diagnostic process, taking into account the contextual factors (and whether they can be modified) and considering how a given intervention can be used to trigger the appropriate mechanisms.

The uncritical realism of realist evaluation

This article is a response to Ray Pawson’s critique of critical realism, the philosophy of science elaborated by Roy Bhaskar. I argue with Pawson’s interpretation of critical realism’s positions on both natural and social science and his charges concerning its totalizing ontology, its arrogant epistemology and its naive methodology. The differences between critical realism and realist evaluation are not as significant as Pawson contends. The main differences between the two realisms lie in their approaches to the relationship between social structures and human agency, and between facts and values. I argue that evaluation scientists need to clearly distinguish structure and agency. They should also make their values explicit. The uncritical approach of realist evaluation, combined with its underplaying of the importance of agency, leaves it open to implication in the abuses of bureaucratic instrumentalism.

Realist evaluation: an immanent critique

This paper critically analyses realist evaluation, focussing on its primary analytical concepts: mechanisms, contexts, and outcomes. Noting that nursing investigators have had difficulty in operationalizing the concepts of mechanism and context, it is argued that their confusion is at least partially the result of ambiguities, inconsistencies, and contradictions in the realist evaluation model. Problematic issues include the adoption of empiricist and idealist positions, oscillation between determinism and voluntarism, subsumption of agency under structure, and categorical confusion between context and mechanism. In relation to outcomes, it is argued that realist evaluation's adoption of the fact/value distinction prevents it from taking into account the concerns of those affected by interventions. The aim of the paper is to use these immanent critiques of realist evaluation to construct an internally consistent realist approach to evaluation that is more amenable to being operationalized by nursing researchers.

Realist Evaluation: a theory driven approach.

Realistic Evaluation assumes that all programmes implemented in practice have an underlying theory to explain how a particular intervention is meant to work. The purpose of realist evaluation is to test the theoretical propositions underpinning the implementation of a programme in order to understand how and why it works, or might not work, in certain circumstances. The first stage of the realist evaluation is to track and articulate the programme theories to determine the evidence on the ‘official conjecture’ (Pawson et al 2004 pg 16) of how the programme is suppose to work in practice. These official conjectures are then tested and refined by gathering empirical evidence to establish causal relationships between a programme and its outcome. Evaluation of the factors and interactions between factors, supporting or hindering the implementation of a programme in practice facilitate theory refinement. Theory refinement is viewed as an iterative and cyclical process undertaken to synthesise the empirical evidence and develop mid-range theories which can be generalised and applied to other programmes to improve implementation and sustainability. In this symposium an example of realist evaluation used to test and refine the theory underpinning the implementation of Early Warning Systems (EWS) is provided to clarify how this theory driven approach can be applied in practice.

How do primary health care teams learn to integrate intimate partner violence (IPV) management? A realist evaluation protocol

Background: Despite the existence of ample literature dealing, on the one hand, with the integration of innovations within health systems and team learning, and, on the other hand, with different aspects of the detection and management of intimate partner violence (IPV) within healthcare facilities, research that explores how health innovations that go beyond biomedical issues—such as IPV management—get integrated into health systems, and that focuses on healthcare teams’ learning processes is, to the best of our knowledge, very scarce if not absent. This realist evaluation protocol aims to ascertain: why, how, and under what circumstances primary healthcare teams engage (if at all) in a learning process to integrate IPV management in their practices; and why, how, and under what circumstances team learning processes lead to the development of organizational culture and values regarding IPV management, and the delivery of IPV management services. Methods: This study will be conducted in Spain using a multiple-case study design. Data will be collected from selected cases (primary healthcare teams) through different methods: individual and group interviews, routinely collected statistical data, documentary review, and observation. Cases will be purposively selected in order to enable testing the initial middle-range theory (MRT). After in-depth exploration of a limited number of cases, additional cases will be chosen for their ability to contribute to refining the emerging MRT to explain how primary healthcare learn to integrate intimate partner violence management. Discussion: Evaluations of health sector responses to IPV are scarce, and even fewer focus on why, how, and when the healthcare services integrate IPV management. There is a consensus that healthcare professionals and healthcare teams play a key role in this integration, and that training is important in order to realize changes. However, little is known about team learning of IPV management, both in terms of how to trigger such learning and how team learning is connected with changes in organizational culture and values, and in service delivery. This realist evaluation protocol aims to contribute to this knowledge by conducting this project in a country, Spain, where great endeavours have been made towards the integration of IPV management within the health system.

Mechanisms that Trigger a Good Health-Care Response to Intimate Partner Violence in Spain. Combining Realist Evaluation and Qualitative Comparative Analysis Approaches

Background. Health care professionals, especially those working in primary health-care services, can play a key role in preventing and responding to intimate partner violence. However, there are huge variations in the way health care professionals and primary health care teams respond to intimate partner violence. In this study we tested a previously developed programme theory on 15 primary health care center teams located in four different Spanish regions: Murcia, C Valenciana, Castilla-León and Cantabria. The aim was to identify the key combinations of contextual factors and mechanisms that trigger a good primary health care center team response to intimate partner violence. Methods. A multiple case-study design was used. Qualitative and quantitative information was collected from each of the 15 centers (cases). In order to handle the large amount of information without losing familiarity with each case, qualitative comparative analysis was undertaken. Conditions (context and mechanisms) and outcomes, were identified and assessed for each of the 15 cases, and solution formulae were calculated using qualitative comparative analysis software. Results. The emerging programme theory highlighted the importance of the combination of each team’s self-efficacy, perceived preparation and women-centredness in generating a good team response to intimate partner violence. The use of the protocol and accumulated experience in primary health care were the most relevant contextual/intervention conditions to trigger a good response. However in order to achieve this, they must be combined with other conditions, such as an enabling team climate, having a champion social worker and having staff with training in intimate partner violence. Conclusions. Interventions to improve primary health care teams’ response to intimate partner violence should focus on strengthening team’s self-efficacy, perceived preparation and the implementation of a woman-centred approach. The use of the protocol combined with a large working experience in primary health care, and other factors such as training, a good team climate, and having a champion social worker on the team, also played a key role. Measures to sustain such interventions and promote these contextual factors should be encouraged.

Using realist evaluation to assess primary healthcare teams’ responses to intimate partner violence in Spain

Objective: Few evaluations have assessed the factors triggering an adequate health care response to intimate partner violence. This article aimed to: 1) describe a realist evaluation carried out in Spain to ascertain why, how and under what circumstances primary health care teams respond to intimate partner violence, and 2) discuss the strengths and challenges of its application. Methods: We carried out a series of case studies in four steps. First, we developed an initial programme theory (PT1), based on interviews with managers. Second, we refined PT1 into PT2 by testing it in a primary healthcare team that was actively responding to violence. Third, we tested the refined PT2 by incorporating three other cases located in the same region. Qualitative and quantitative data were collected and thick descriptions were produced and analysed using a retroduction approach. Fourth, we analysed a total of 15 cases, and identified combinations of contextual factors and mechanisms that triggered an adequate response to violence by using qualitative comparative analysis. Results: There were several key mechanisms —the teams’ self-efficacy, perceived preparation, women-centred care—, and contextual factors —an enabling team environment and managerial style, the presence of motivated professionals, the use of the protocol and accumulated experience in primary health care—that should be considered to develop adequate primary health-care responses to violence. Conclusion: The full application of this realist evaluation was demanding, but also well suited to explore a complex intervention reflecting the situation in natural settings.

Managing patient deterioration: evaluation of what works.

Background: Rapid Response Systems (RRS) have been implemented nationally and internationally to improve patient safety in hospital. However, to date the majority of the RRS research evidence has focused on measuring the effectiveness of the intervention on patient outcomes. To evaluate RRS it has been recommended that a multimodal approach is required to address the broad range of process and outcome measures required to determine the effectiveness of the RRS concept. Aim: The aim of this paper is to evaluate the official RRS programme theoretical assumptions regarding how the programme is meant to work against actual practice in order to determine what works. Methods: The research design was a multiple case study approach of four wards in two hospitals in Northern Ireland. It followed the principles of realist evaluation research which allowed empirical data to be gathered to test and refine RRS programme theory [1]. This approach used a variety of mixed methods to test the programme theories including individual and focus group interviews with a purposive sample of 75 nurses and doctors, observation of ward practices and documentary analysis. The findings from the case studies were analysed and compared within and across cases to identify what works for whom and in what circumstances. Results: The RRS programme theories were critically evaluated and compared with study findings to develop a mid-range theory to explain what works, for whom in what circumstances. The findings of what works suggests that clinical experience, established working relationships, flexible implementation of protocols, ongoing experiential learning, empowerment and pre-emptive management are key to the success of RRS implementation.  Conclusion:These findings highlight the combination of factors that can improve the implementation of RRS and in light of this evidence several recommendations are made to provide policymakers with guidance and direction for their success and sustainability.References: 1.Pawson R and Tilley N. (1997) Realistic Evaluation. Sage Publications; LondonType of submission: Concurrent session Source of funding: Sandra Ryan Fellowship funded by the School of Nursing & Midwifery, Queen’s University of Belfast

Realistic evaluation of EWS and ALERT: factors enabling and constraining implementation.

Realistic Evaluation of EWS and ALERT: factors enabling and constraining implementation Background The implementation of EWS and ALERT in practice is essential to the success of Rapid Response Systems but is dependent upon nurses utilising EWS protocols and applying ALERT best practice guidelines. To date there is limited evidence on the effectiveness of EWS or ALERT as research has primarily focused on measuring patient outcomes (cardiac arrests, ICU admissions) following the implementation of a Rapid Response Team. Complex interventions in healthcare aimed at changing service delivery and related behaviour of health professionals require a different research approach to evaluate the evidence. To understand how and why EWS and ALERT work, or might not work, research needs to consider the social, cultural and organisational influences that will impact on successful implementation in practice. This requires a research approach that considers both the processes and outcomes of complex interventions, such as EWS and ALERT, implemented in practice. Realistic Evaluation is such an approach and was used to explain the factors that enable and constrain the implementation of EWS and ALERT in practice [1]. Aim The aim of this study was to evaluate factors that enabled and constrained the implementation and service delivery of early warnings systems (EWS) and ALERT in practice in order to provide direction for enabling their success and sustainability. Methods The research design was a multiple case study approach of four wards in two hospitals in Northern Ireland. It followed the principles of realist evaluation research which allowed empirical data to be gathered to test and refine RRS programme theory. This approach used a variety of mixed methods to test the programme theories including individual and focus group interviews, observation and documentary analysis in a two stage process. A purposive sample of 75 key informants participated in individual and focus group interviews. Observation and documentary analysis of EWS compliance data and ALERT training records provided further evidence to support or refute the interview findings. Data was analysed using NVIVO8 to categorise interview findings and SPSS for ALERT documentary data. These findings were further synthesised by undertaking a within and cross case comparison to explain the factors enabling and constraining EWS and ALERT. Results A cross case analysis highlighted similarities, differences and factors enabling or constraining successful implementation across the case study sites. Findings showed that personal (confidence; clinical judgement; personality), social (ward leadership; communication), organisational (workload and staffing issues; pressure from managers to complete EWS audit and targets), educational (constraints on training; no clinical educator on ward) and cultural (routine task delegated) influences impact on EWS and acute care training outcomes. There were also differences noted between medical and surgical wards across both case sites. Conclusions Realist Evaluation allows refinement and development of the RRS programme theory to explain the realities of practice. These refined RRS programme theories are capable of informing the planning of future service provision and provide direction for enabling their success and sustainability. References: 1. McGaughey J, Blackwood B, O’Halloran P, Trinder T. J. & Porter S. (2010) A realistic evaluation of Track and Trigger systems and acute care training for early recognition and management of deteriorating ward–based patients. Journal of Advanced Nursing 66 (4), 923-932. Type of submission: Concurrent session Source of funding: Sandra Ryan Fellowship funded by the School of Nursing & Midwifery, Queen’s University of Belfast

Realist synthesis: Critical care rapid response systems.

Background Rapid Response Systems (RRS) consist of four interrelated and interdependent components; an event detection and trigger mechanism, a response strategy, a governance structure and process improvement system. These multiple components of the RRS pose problems in evaluation as the intervention is complex and cannot be evaluated using a traditional systematic review. Complex interventions in healthcare aimed at changing service delivery and related behaviour of health professionals require a different approach to summarising the evidence. Realist synthesis is such an approach to reviewing research evidence on complex interventions to provide an explanatory analysis of how and why an intervention works or doesn’t work in practice. The core principle is to make explicit the underlying assumptions about how an intervention is suppose to work (ie programme theory) and then use this theory to guide evaluation. Methods A realist synthesis process was used to explain those factors that enable or constrain the success of RRS programmes. Results The findings from the review include the articulation of the RRS programme theories, evaluation of whether these theories are supported or refuted by the research evidence and an evaluation of evidence to explain the underlying reasons why RRS works or doesn’t work in practice. Rival conjectured RRS programme theories were identified to explain the constraining factors regarding implementation of RRS in practice. These programme theories are presented using a logic model to highlight all the components which impact or influence the delivery of RRS programmes in the practice setting. The evidence from the realist synthesis provided the foundation for the development of hypothesis to test and refine the theories in the subsequent stages of the Realist Evaluation PhD study [1]. This information will be useful in providing evidence and direction for strategic and service planning of acute care to improve patient safety in hospital. References: McGaughey J, Blackwood B, O’Halloran P, Trinder T. J. & Porter S. (2010) Realistic Evaluation of Early Warning Systems and the Acute Life-threatening Events – Recognition and Treatment training course for early recognition and management of deteriorating ward-based patients: research protocol. Journal of Advanced Nursing 66 (4), 923-932.

A realistic evaluation of Early Warning Systems and acute care training in Northern Ireland: findings and recommendations

Statement of purpose The purpose of this concurrent session is to present the main findings and recommendations from a five year study evaluating the implementation of Early Warning Systems (EWS) and the Acute Life-threatening Events: Recognition and Treatment (ALERT) course in Northern Ireland. The presentation will provide delegates with an understanding of those factors that enable and constrain successful implementation of EWS and ALERT in practice in order to provide an impetus for change. Methods The research design was a multiple case study approach of four wards in two hospitals in Northern Ireland. It followed the principles of realist evaluation research which allowed empirical data to be gathered to test and refine RRS programme theory [1]. The stages included identifying the programme theories underpinning EWS and ALERT, generating hypotheses, gathering empirical evidence and refining the programme theories. This approach used a variety of mixed methods including individual and focus group interviews, observation and documentary analysis of EWS compliance data and ALERT training records. A within and across case comparison facilitated the development of mid-range theories from the research evidence. Results The official RRS theories developed from the realist synthesis were critically evaluated and compared with the study findings to develop a mid-range theory to explain what works, for whom in what circumstances. The findings of what works suggests that clinical experience, established working relationships, flexible implementation of protocols, ongoing experiential learning, empowerment and pre-emptive management are key to the success of EWS and ALERT implementation. Each concept is presented as ‘context, mechanism and outcome configurations’ to provide an understanding of how the context impacts on individual reasoning or behaviour to produce certain outcomes. Conclusion These findings highlight the combination of factors that can improve the implementation and sustainability of EWS and ALERT and in light of this evidence several recommendations are made to provide policymakers with guidance and direction for future policy development. References: 1. Pawson R and Tilley N. (1997) Realistic Evaluation. Sage Publications; London Type of submission: Concurrent session Source of funding: Sandra Ryan Fellowship funded by the School of Nursing & Midwifery, Queen’s University of Belfast