Realist evaluation - promise, problems and practicalities

Realist evaluation is an innovative, multi-method approach to evaluating the effectiveness of complex health care interventions that is having an increasing impact on the research community. Drawing on their experience doing four realist evaluations in diverse areas of healthcare, the authors offer a comprehensive overview and critique of essential theory and practice. The first paper (Realist review and realist evaluation: an introduction) introduces the main components of the approach and shows how realist review can support realist evaluation. The second paper (Concepts and methodology for realist evaluation: help or hindrance?) provides further detail on the key concepts, shows how they can be operationalised, and discusses the advantages and difficulties of using these ideas. Following these two papers introducing and illustrating the major concepts, the third paper (Realist Evaluation: a critical realist critique) takes a step back to re-consider realist evaluation in relation to its critical realist roots, asking whether it leads to evaluators abandoning the attempt to understand (and if necessary challenge) the underlying values of health care interventions and contenting themselves merely with explicating the factors that help or hinder implementation. The fourth and final paper (Data analysis and theory development in realist evaluation) plunges back into the tangled undergrowth of multiple-method data collection and shows how disparate forms of data can be synthesised for theory development, and the results presented in a form that is useful to practitioners and policy-makers.

Factors affecting the successful implementation and sustainability of the Liverpool Care Pathway for dying patients – a realist evaluation

Objectives: The Liverpool Care Pathway for the dying patient (LCP) was designed to improve end-of-life care in generalist health care settings. Controversy has led to its withdrawal in some jurisdictions. The main objective of this research was to identify the influences that facilitated or hindered successful LCP implementation.

Method: An organisational case study using realist evaluation in one health and social care trust in Northern Ireland. Two rounds of semi-structured interviews were conducted with two policy makers and twenty two participants with experience and/or involvement in management of the LCP during 2011 and 2012.

Results: Key resource inputs included facilitation with a view to maintaining LCP ‘visibility’, reducing anxiety among nurses and increasing their confidence regarding the delivery of end-of-life care; and nurse and medical education designed to increase professional self-efficacy and reduce misuse and misunderstanding of the LCP. Key enabling contexts were consistent senior management support; ongoing education and training tailored to the needs of each professional group; and an organisational cultural change in the hospital setting that encompassed end-of-life care.

Conclusion: There is a need to appreciate the organizationally complex nature of intervening to improve end-of-life care. Successful implementation of evidence-based interventions for end-of-life care requires commitment to planning, training and ongoing review that takes account of different perspectives, institutional hierarchies and relationships and the educational needs of professional disciplines. There is a need also to recognise that medical consultants require particular support in their role as gatekeepers and as a lead communication channel with patients and their relatives.

The Management of Long-Term Sickness Absence in Large Public Sector Healthcare Organisations: A Realist Evaluation Using Mixed Methods

Purpose The success of measures to reduce long-term sickness absence (LTSA) in public sector organisations is contingent on organisational context. This realist evaluation investigates how interventions interact with context to influence successful management of LTSA. Methods Multi-method case study in three Health and Social Care Trusts in Northern Ireland comprising realist literature review, semi-structured interviews (61 participants), Process-Mapping and feedback meetings (59 participants), observation of training, analysis of documents. Results Important activities included early intervention; workplace-based occupational rehabilitation; robust sickness absence policies with clear trigger points for action. Used appropriately, in a context of good interpersonal and interdepartmental communication and shared goals, these are able to increase the motivation of staff to return to work. Line managers are encouraged to take a proactive approach when senior managers provide support and accountability. Hindering factors: delayed intervention; inconsistent implementation of policy and procedure; lack of resources; organisational complexity; stakeholders misunderstanding each other’s goals and motives. Conclusions Different mechanisms have the potential to encourage common motivations for earlier return from LTSA, such as employees feeling that they have the support of their line manager to return to work and having the confidence to do so. Line managers’ proactively engage when they have confidence in the support of seniors and in their own ability to address LTSA. Fostering these motivations calls for a thoughtful, diagnostic process, taking into account the contextual factors (and whether they can be modified) and considering how a given intervention can be used to trigger the appropriate mechanisms.

The uncritical realism of realist evaluation

This article is a response to Ray Pawson’s critique of critical realism, the philosophy of science elaborated by Roy Bhaskar. I argue with Pawson’s interpretation of critical realism’s positions on both natural and social science and his charges concerning its totalizing ontology, its arrogant epistemology and its naive methodology. The differences between critical realism and realist evaluation are not as significant as Pawson contends. The main differences between the two realisms lie in their approaches to the relationship between social structures and human agency, and between facts and values. I argue that evaluation scientists need to clearly distinguish structure and agency. They should also make their values explicit. The uncritical approach of realist evaluation, combined with its underplaying of the importance of agency, leaves it open to implication in the abuses of bureaucratic instrumentalism.

Realist evaluation: an immanent critique

This paper critically analyses realist evaluation, focussing on its primary analytical concepts: mechanisms, contexts, and outcomes. Noting that nursing investigators have had difficulty in operationalizing the concepts of mechanism and context, it is argued that their confusion is at least partially the result of ambiguities, inconsistencies, and contradictions in the realist evaluation model. Problematic issues include the adoption of empiricist and idealist positions, oscillation between determinism and voluntarism, subsumption of agency under structure, and categorical confusion between context and mechanism. In relation to outcomes, it is argued that realist evaluation's adoption of the fact/value distinction prevents it from taking into account the concerns of those affected by interventions. The aim of the paper is to use these immanent critiques of realist evaluation to construct an internally consistent realist approach to evaluation that is more amenable to being operationalized by nursing researchers.

Realist Evaluation: a theory driven approach.

Realistic Evaluation assumes that all programmes implemented in practice have an underlying theory to explain how a particular intervention is meant to work. The purpose of realist evaluation is to test the theoretical propositions underpinning the implementation of a programme in order to understand how and why it works, or might not work, in certain circumstances. The first stage of the realist evaluation is to track and articulate the programme theories to determine the evidence on the ‘official conjecture’ (Pawson et al 2004 pg 16) of how the programme is suppose to work in practice. These official conjectures are then tested and refined by gathering empirical evidence to establish causal relationships between a programme and its outcome. Evaluation of the factors and interactions between factors, supporting or hindering the implementation of a programme in practice facilitate theory refinement. Theory refinement is viewed as an iterative and cyclical process undertaken to synthesise the empirical evidence and develop mid-range theories which can be generalised and applied to other programmes to improve implementation and sustainability. In this symposium an example of realist evaluation used to test and refine the theory underpinning the implementation of Early Warning Systems (EWS) is provided to clarify how this theory driven approach can be applied in practice.

Managing patient deterioration: evaluation of what works.

Background: Rapid Response Systems (RRS) have been implemented nationally and internationally to improve patient safety in hospital. However, to date the majority of the RRS research evidence has focused on measuring the effectiveness of the intervention on patient outcomes. To evaluate RRS it has been recommended that a multimodal approach is required to address the broad range of process and outcome measures required to determine the effectiveness of the RRS concept. Aim: The aim of this paper is to evaluate the official RRS programme theoretical assumptions regarding how the programme is meant to work against actual practice in order to determine what works. Methods: The research design was a multiple case study approach of four wards in two hospitals in Northern Ireland. It followed the principles of realist evaluation research which allowed empirical data to be gathered to test and refine RRS programme theory [1]. This approach used a variety of mixed methods to test the programme theories including individual and focus group interviews with a purposive sample of 75 nurses and doctors, observation of ward practices and documentary analysis. The findings from the case studies were analysed and compared within and across cases to identify what works for whom and in what circumstances. Results: The RRS programme theories were critically evaluated and compared with study findings to develop a mid-range theory to explain what works, for whom in what circumstances. The findings of what works suggests that clinical experience, established working relationships, flexible implementation of protocols, ongoing experiential learning, empowerment and pre-emptive management are key to the success of RRS implementation.  Conclusion:These findings highlight the combination of factors that can improve the implementation of RRS and in light of this evidence several recommendations are made to provide policymakers with guidance and direction for their success and sustainability.References: 1.Pawson R and Tilley N. (1997) Realistic Evaluation. Sage Publications; LondonType of submission: Concurrent session Source of funding: Sandra Ryan Fellowship funded by the School of Nursing & Midwifery, Queen’s University of Belfast

Realistic evaluation of EWS and ALERT: factors enabling and constraining implementation.

Realistic Evaluation of EWS and ALERT: factors enabling and constraining implementation Background The implementation of EWS and ALERT in practice is essential to the success of Rapid Response Systems but is dependent upon nurses utilising EWS protocols and applying ALERT best practice guidelines. To date there is limited evidence on the effectiveness of EWS or ALERT as research has primarily focused on measuring patient outcomes (cardiac arrests, ICU admissions) following the implementation of a Rapid Response Team. Complex interventions in healthcare aimed at changing service delivery and related behaviour of health professionals require a different research approach to evaluate the evidence. To understand how and why EWS and ALERT work, or might not work, research needs to consider the social, cultural and organisational influences that will impact on successful implementation in practice. This requires a research approach that considers both the processes and outcomes of complex interventions, such as EWS and ALERT, implemented in practice. Realistic Evaluation is such an approach and was used to explain the factors that enable and constrain the implementation of EWS and ALERT in practice [1]. Aim The aim of this study was to evaluate factors that enabled and constrained the implementation and service delivery of early warnings systems (EWS) and ALERT in practice in order to provide direction for enabling their success and sustainability. Methods The research design was a multiple case study approach of four wards in two hospitals in Northern Ireland. It followed the principles of realist evaluation research which allowed empirical data to be gathered to test and refine RRS programme theory. This approach used a variety of mixed methods to test the programme theories including individual and focus group interviews, observation and documentary analysis in a two stage process. A purposive sample of 75 key informants participated in individual and focus group interviews. Observation and documentary analysis of EWS compliance data and ALERT training records provided further evidence to support or refute the interview findings. Data was analysed using NVIVO8 to categorise interview findings and SPSS for ALERT documentary data. These findings were further synthesised by undertaking a within and cross case comparison to explain the factors enabling and constraining EWS and ALERT. Results A cross case analysis highlighted similarities, differences and factors enabling or constraining successful implementation across the case study sites. Findings showed that personal (confidence; clinical judgement; personality), social (ward leadership; communication), organisational (workload and staffing issues; pressure from managers to complete EWS audit and targets), educational (constraints on training; no clinical educator on ward) and cultural (routine task delegated) influences impact on EWS and acute care training outcomes. There were also differences noted between medical and surgical wards across both case sites. Conclusions Realist Evaluation allows refinement and development of the RRS programme theory to explain the realities of practice. These refined RRS programme theories are capable of informing the planning of future service provision and provide direction for enabling their success and sustainability. References: 1. McGaughey J, Blackwood B, O’Halloran P, Trinder T. J. & Porter S. (2010) A realistic evaluation of Track and Trigger systems and acute care training for early recognition and management of deteriorating ward–based patients. Journal of Advanced Nursing 66 (4), 923-932. Type of submission: Concurrent session Source of funding: Sandra Ryan Fellowship funded by the School of Nursing & Midwifery, Queen’s University of Belfast

Realist synthesis: Critical care rapid response systems.

Background Rapid Response Systems (RRS) consist of four interrelated and interdependent components; an event detection and trigger mechanism, a response strategy, a governance structure and process improvement system. These multiple components of the RRS pose problems in evaluation as the intervention is complex and cannot be evaluated using a traditional systematic review. Complex interventions in healthcare aimed at changing service delivery and related behaviour of health professionals require a different approach to summarising the evidence. Realist synthesis is such an approach to reviewing research evidence on complex interventions to provide an explanatory analysis of how and why an intervention works or doesn’t work in practice. The core principle is to make explicit the underlying assumptions about how an intervention is suppose to work (ie programme theory) and then use this theory to guide evaluation. Methods A realist synthesis process was used to explain those factors that enable or constrain the success of RRS programmes. Results The findings from the review include the articulation of the RRS programme theories, evaluation of whether these theories are supported or refuted by the research evidence and an evaluation of evidence to explain the underlying reasons why RRS works or doesn’t work in practice. Rival conjectured RRS programme theories were identified to explain the constraining factors regarding implementation of RRS in practice. These programme theories are presented using a logic model to highlight all the components which impact or influence the delivery of RRS programmes in the practice setting. The evidence from the realist synthesis provided the foundation for the development of hypothesis to test and refine the theories in the subsequent stages of the Realist Evaluation PhD study [1]. This information will be useful in providing evidence and direction for strategic and service planning of acute care to improve patient safety in hospital. References: McGaughey J, Blackwood B, O’Halloran P, Trinder T. J. & Porter S. (2010) Realistic Evaluation of Early Warning Systems and the Acute Life-threatening Events – Recognition and Treatment training course for early recognition and management of deteriorating ward-based patients: research protocol. Journal of Advanced Nursing 66 (4), 923-932.

A realistic evaluation of Early Warning Systems and acute care training in Northern Ireland: findings and recommendations

Statement of purpose The purpose of this concurrent session is to present the main findings and recommendations from a five year study evaluating the implementation of Early Warning Systems (EWS) and the Acute Life-threatening Events: Recognition and Treatment (ALERT) course in Northern Ireland. The presentation will provide delegates with an understanding of those factors that enable and constrain successful implementation of EWS and ALERT in practice in order to provide an impetus for change. Methods The research design was a multiple case study approach of four wards in two hospitals in Northern Ireland. It followed the principles of realist evaluation research which allowed empirical data to be gathered to test and refine RRS programme theory [1]. The stages included identifying the programme theories underpinning EWS and ALERT, generating hypotheses, gathering empirical evidence and refining the programme theories. This approach used a variety of mixed methods including individual and focus group interviews, observation and documentary analysis of EWS compliance data and ALERT training records. A within and across case comparison facilitated the development of mid-range theories from the research evidence. Results The official RRS theories developed from the realist synthesis were critically evaluated and compared with the study findings to develop a mid-range theory to explain what works, for whom in what circumstances. The findings of what works suggests that clinical experience, established working relationships, flexible implementation of protocols, ongoing experiential learning, empowerment and pre-emptive management are key to the success of EWS and ALERT implementation. Each concept is presented as ‘context, mechanism and outcome configurations’ to provide an understanding of how the context impacts on individual reasoning or behaviour to produce certain outcomes. Conclusion These findings highlight the combination of factors that can improve the implementation and sustainability of EWS and ALERT and in light of this evidence several recommendations are made to provide policymakers with guidance and direction for future policy development. References: 1. Pawson R and Tilley N. (1997) Realistic Evaluation. Sage Publications; London Type of submission: Concurrent session Source of funding: Sandra Ryan Fellowship funded by the School of Nursing & Midwifery, Queen’s University of Belfast

General practitioners’ management of the long-term sick role

In this paper, we use qualitative research techniques to examine the role of general practitioners in the management of the long-term sickness absence. In order to uncover the perspectives of all the main agents affected by the actions of general practitioners, a case study approach focussing on one particular employment sector, the public health service, is adopted. The role of family physicians is viewed from the perspectives of health service managers, occupational health physicians, employees / patients, and general practitioners. Our argument is theoretically framed by Talcott Parsons’s model of the medical contribution to the sick role, along with subsequent conceptualisations of the social role and position of physicians. Sixty one semi-structured interviews and three focus group interviews were conducted in three Health and Social Care Trusts in Northern Ireland between 2010 and 2012. There was a consensus among respondents that general practitioners put far more weight on the preferences and needs of their patients than they did on the requirements of employing organisations. This was explained by respondents in terms of the propinquity and longevity of relationships between doctors and their patients, and by the ideology of holistic care and patient advocacy that general practitioners viewed as providing the foundations of their approach to patients. The approach of general practitioners was viewed negatively by managers and occupational health physicians, and more positively by general practitioners and patients. However, there is some evidence that general practitioners would be prepared to forfeit their role as validators of sick leave. Given the imperatives of both state and capital to reduce the financial burden of long-term sickness, this preparedness puts into doubt the continued role of general practitioners as gatekeepers to legitimate long-term sickness absence.

Stakeholders and perspectivism in qualitative policy evaluation: A realist reflection

This paper examines the epistemological bases for the inclusion of stakeholders in policy research. While it concedes that the input of stakeholders provides essential expert and experiential knowledge for the understanding of complex policies and programmes, it contends that the approach which assumes that all interpretations of policy including those of stakeholders should be afforded equal validity, which we term relativist perspectivism, undermines the possibility of robust research by allowing power to replace methodological rigour as the primary research dynamic. It is noted that this problem tends to be more acute when the research is qualitative. A study into the gendered effects of Common Agricultural Policy reforms is used as an illustrative example of how research can be compromised by relativist perspectivism. It is argued that realist research methodologies uniquely provide the capacity to maintain epistemological robustness, while also being able to take due account of the perspectives of stakeholders.

The use and limitation of realistic evaluation as a tool for evidence-based practice: a critical realist perspective

In this paper, we assess realistic evaluation’s articulation with evidence-based practice (EBP) from the perspective of critical realism. We argue that the adoption by realistic evaluation of a realist causal ontology means that it is better placed to explain complex healthcare interventions than the traditional method used by EBP, the randomized controlled trial (RCT). However, we do not conclude from this that the use of RCTs is without merit, arguing that it is possible to use both methods in combination under the rubric of realist theory. More negatively, we contend that the rejection of critical theory and utopianism by realistic evaluation in favour of the pragmatism of piecemeal social engineering means that it is vulnerable to accusations that it promotes technocratic interpretations of human problems. We conclude that, insofar as realistic evaluation adheres to the ontology of critical realism, it provides a sound contribution to EBP, but insofar as it rejects the critical turn of Bhaskar’s realism, it replicates the technocratic tendencies inherent in EBP.