Realistic evaluation of EWS and ALERT: factors enabling and constraining implementation.

Realistic Evaluation of EWS and ALERT: factors enabling and constraining implementation Background The implementation of EWS and ALERT in practice is essential to the success of Rapid Response Systems but is dependent upon nurses utilising EWS protocols and applying ALERT best practice guidelines. To date there is limited evidence on the effectiveness of EWS or ALERT as research has primarily focused on measuring patient outcomes (cardiac arrests, ICU admissions) following the implementation of a Rapid Response Team. Complex interventions in healthcare aimed at changing service delivery and related behaviour of health professionals require a different research approach to evaluate the evidence. To understand how and why EWS and ALERT work, or might not work, research needs to consider the social, cultural and organisational influences that will impact on successful implementation in practice. This requires a research approach that considers both the processes and outcomes of complex interventions, such as EWS and ALERT, implemented in practice. Realistic Evaluation is such an approach and was used to explain the factors that enable and constrain the implementation of EWS and ALERT in practice [1]. Aim The aim of this study was to evaluate factors that enabled and constrained the implementation and service delivery of early warnings systems (EWS) and ALERT in practice in order to provide direction for enabling their success and sustainability. Methods The research design was a multiple case study approach of four wards in two hospitals in Northern Ireland. It followed the principles of realist evaluation research which allowed empirical data to be gathered to test and refine RRS programme theory. This approach used a variety of mixed methods to test the programme theories including individual and focus group interviews, observation and documentary analysis in a two stage process. A purposive sample of 75 key informants participated in individual and focus group interviews. Observation and documentary analysis of EWS compliance data and ALERT training records provided further evidence to support or refute the interview findings. Data was analysed using NVIVO8 to categorise interview findings and SPSS for ALERT documentary data. These findings were further synthesised by undertaking a within and cross case comparison to explain the factors enabling and constraining EWS and ALERT. Results A cross case analysis highlighted similarities, differences and factors enabling or constraining successful implementation across the case study sites. Findings showed that personal (confidence; clinical judgement; personality), social (ward leadership; communication), organisational (workload and staffing issues; pressure from managers to complete EWS audit and targets), educational (constraints on training; no clinical educator on ward) and cultural (routine task delegated) influences impact on EWS and acute care training outcomes. There were also differences noted between medical and surgical wards across both case sites. Conclusions Realist Evaluation allows refinement and development of the RRS programme theory to explain the realities of practice. These refined RRS programme theories are capable of informing the planning of future service provision and provide direction for enabling their success and sustainability. References: 1. McGaughey J, Blackwood B, O’Halloran P, Trinder T. J. & Porter S. (2010) A realistic evaluation of Track and Trigger systems and acute care training for early recognition and management of deteriorating ward–based patients. Journal of Advanced Nursing 66 (4), 923-932. Type of submission: Concurrent session Source of funding: Sandra Ryan Fellowship funded by the School of Nursing & Midwifery, Queen’s University of Belfast

Realist synthesis: Critical care rapid response systems.

Background Rapid Response Systems (RRS) consist of four interrelated and interdependent components; an event detection and trigger mechanism, a response strategy, a governance structure and process improvement system. These multiple components of the RRS pose problems in evaluation as the intervention is complex and cannot be evaluated using a traditional systematic review. Complex interventions in healthcare aimed at changing service delivery and related behaviour of health professionals require a different approach to summarising the evidence. Realist synthesis is such an approach to reviewing research evidence on complex interventions to provide an explanatory analysis of how and why an intervention works or doesn’t work in practice. The core principle is to make explicit the underlying assumptions about how an intervention is suppose to work (ie programme theory) and then use this theory to guide evaluation. Methods A realist synthesis process was used to explain those factors that enable or constrain the success of RRS programmes. Results The findings from the review include the articulation of the RRS programme theories, evaluation of whether these theories are supported or refuted by the research evidence and an evaluation of evidence to explain the underlying reasons why RRS works or doesn’t work in practice. Rival conjectured RRS programme theories were identified to explain the constraining factors regarding implementation of RRS in practice. These programme theories are presented using a logic model to highlight all the components which impact or influence the delivery of RRS programmes in the practice setting. The evidence from the realist synthesis provided the foundation for the development of hypothesis to test and refine the theories in the subsequent stages of the Realist Evaluation PhD study [1]. This information will be useful in providing evidence and direction for strategic and service planning of acute care to improve patient safety in hospital. References: McGaughey J, Blackwood B, O’Halloran P, Trinder T. J. & Porter S. (2010) Realistic Evaluation of Early Warning Systems and the Acute Life-threatening Events – Recognition and Treatment training course for early recognition and management of deteriorating ward-based patients: research protocol. Journal of Advanced Nursing 66 (4), 923-932.

A realistic evaluation of Early Warning Systems and acute care training in Northern Ireland: findings and recommendations

Statement of purpose The purpose of this concurrent session is to present the main findings and recommendations from a five year study evaluating the implementation of Early Warning Systems (EWS) and the Acute Life-threatening Events: Recognition and Treatment (ALERT) course in Northern Ireland. The presentation will provide delegates with an understanding of those factors that enable and constrain successful implementation of EWS and ALERT in practice in order to provide an impetus for change. Methods The research design was a multiple case study approach of four wards in two hospitals in Northern Ireland. It followed the principles of realist evaluation research which allowed empirical data to be gathered to test and refine RRS programme theory [1]. The stages included identifying the programme theories underpinning EWS and ALERT, generating hypotheses, gathering empirical evidence and refining the programme theories. This approach used a variety of mixed methods including individual and focus group interviews, observation and documentary analysis of EWS compliance data and ALERT training records. A within and across case comparison facilitated the development of mid-range theories from the research evidence. Results The official RRS theories developed from the realist synthesis were critically evaluated and compared with the study findings to develop a mid-range theory to explain what works, for whom in what circumstances. The findings of what works suggests that clinical experience, established working relationships, flexible implementation of protocols, ongoing experiential learning, empowerment and pre-emptive management are key to the success of EWS and ALERT implementation. Each concept is presented as ‘context, mechanism and outcome configurations’ to provide an understanding of how the context impacts on individual reasoning or behaviour to produce certain outcomes. Conclusion These findings highlight the combination of factors that can improve the implementation and sustainability of EWS and ALERT and in light of this evidence several recommendations are made to provide policymakers with guidance and direction for future policy development. References: 1. Pawson R and Tilley N. (1997) Realistic Evaluation. Sage Publications; London Type of submission: Concurrent session Source of funding: Sandra Ryan Fellowship funded by the School of Nursing & Midwifery, Queen’s University of Belfast

'Motor magic' : evaluation of a community capacity-building approach to supporting the development of preschool children (part 2)

Background and Aims: This paper is the second in a series of two that presents and discusses an exploratory evaluation study of the 'Motor Magic' program conducted in Adelaide, South Australia.

Methods: A realist evaluation approach was used, and two focus groups (one with parents and one with kindergarten staff) were held to identify program outcomes and to develop and refine key hypotheses about how these outcomes occurred.

Results and Conclusions: Results for kindergarten staff involved in the program are presented and discussed in this paper, including improved early identification and early intervention for children with, or at risk of, fine motor developmental difficulties; increased confidence in the effectiveness of their practice with these children; improved practice of kindergarten staff with both targeted and all children; improved empathy with, and support for, parents; and increased interagency links and ongoing advocacy for further resources. These results suggest that the program provides an effective model for building the capacity of kindergarten staff to support children with, or at risk of, developmental difficulties.

Addressing Multilevel Program Complexity by Evaluation Design

In this paper, we present the evaluation design for a complex multilevel program recently introduced in Switzerland. The evaluation embraces the federal level, the cantonal program level, and the project level where target groups are directly addressed. We employ Pawson and Tilley’s realist evaluation approach, in order to do justice to the varying context factors that impact the cantonal programs leading to varying effectiveness of the implemented activities. The application of the model to the canton of Uri shows that the numerous vertical and horizontal relations play a crucial role for the program’s effectiveness. As a general learning for the evaluation of complex programs, we state that there is a need to consider all affected levels of a program and that no monocausal effects can be singled out in programs where multiple interventions address the same problem. Moreover, considering all affected levels of a program can mean going beyond the borders of the actual program organization and including factors that do not directly interfere with the policy delivery as such. In particular, we found that the relationship between the cantonal and the federal level was a crucial organizational factor influencing the effectiveness of the cantonal program.

Developing a programme theory to explain how primary health care teams learn to respond to intimate partner violence: a realist case-study

Background: Despite the progress made on policies and programmes to strengthen primary health care teams’ response to Intimate Partner Violence, the literature shows that encounters between women exposed to IPV and health-care providers are not always satisfactory, and a number of barriers that prevent individual health-care providers from responding to IPV have been identified. We carried out a realist case study, for which we developed and tested a programme theory that seeks to explain how, why and under which circumstances a primary health care team in Spain learned to respond to IPV. Methods: A realist case study design was chosen to allow for an in-depth exploration of the linkages between context, intervention, mechanisms and outcomes as they happen in their natural setting. The first author collected data at the primary health care center La Virgen (pseudonym) through the review of documents, observation and interviews with health systems’ managers, team members, women patients, and members of external services. The quality of the IPV case management was assessed with the PREMIS tool. Results: This study found that the health care team at La Virgen has managed 1) to engage a number of staff members in actively responding to IPV, 2) to establish good coordination, mutual support and continuous learning processes related to IPV, 3) to establish adequate internal referrals within La Virgen, and 4) to establish good coordination and referral systems with other services. Team and individual level factors have triggered the capacity and interest in creating spaces for team leaning, team work and therapeutic responses to IPV in La Virgen, although individual motivation strongly affected this mechanism. Regional interventions did not trigger individual and/ or team responses but legitimated the workings of motivated professionals. Conclusions: The primary health care team of La Virgen is involved in a continuous learning process, even as participation in the process varies between professionals. This process has been supported, but not caused, by a favourable policy for integration of a health care response to IPV. Specific contextual factors of La Virgen facilitated the uptake of the policy. To some extent, the performance of La Virgen has the potential to shape the IPV learning processes of other primary health care teams in Murcia.

A realist case study of a regional hospital's response to improve emergency department access in the context of Australian healthcare reforms

Introduction: Major health-care reforms have extended across all Australian public hospitals in recent years. Improving emergency department (ED) access has been a focus of these reforms.

Objective: This study evaluates how the national reforms have led to improvement in ED access in a regional hospital in remote Australia.Methods: Assessing a complex scenario such as national reforms and the challenges faced by the regional hospital to implement these reforms requires in-depth analysis. A realist evaluation theory-based approach was employed, allowing investigation of what, how, why, and for whom change occurred. A case study mixed methods design was adopted within the realist framework to answer these questions about change.

Results and Conclusion: The study identified moderate improvement in ED access as a result of the reforms (investment in infrastructure and workforce and the introduction of ED targets). Clinical leadership and support from management were essential for the improvement. Without ongoing investment and clinical redesign activities, however, sustainability of the improvement may prove difficult.

General practitioners’ management of the long-term sick role

In this paper, we use qualitative research techniques to examine the role of general practitioners in the management of the long-term sickness absence. In order to uncover the perspectives of all the main agents affected by the actions of general practitioners, a case study approach focussing on one particular employment sector, the public health service, is adopted. The role of family physicians is viewed from the perspectives of health service managers, occupational health physicians, employees / patients, and general practitioners. Our argument is theoretically framed by Talcott Parsons’s model of the medical contribution to the sick role, along with subsequent conceptualisations of the social role and position of physicians. Sixty one semi-structured interviews and three focus group interviews were conducted in three Health and Social Care Trusts in Northern Ireland between 2010 and 2012. There was a consensus among respondents that general practitioners put far more weight on the preferences and needs of their patients than they did on the requirements of employing organisations. This was explained by respondents in terms of the propinquity and longevity of relationships between doctors and their patients, and by the ideology of holistic care and patient advocacy that general practitioners viewed as providing the foundations of their approach to patients. The approach of general practitioners was viewed negatively by managers and occupational health physicians, and more positively by general practitioners and patients. However, there is some evidence that general practitioners would be prepared to forfeit their role as validators of sick leave. Given the imperatives of both state and capital to reduce the financial burden of long-term sickness, this preparedness puts into doubt the continued role of general practitioners as gatekeepers to legitimate long-term sickness absence.

Do healthy cities work? A logic of method for assessing impact and outcome of healthy cities

In this article, we discuss an appropriate methodology for assessing complex urban programs such as the WHO European Healthy Cities Network. The basic tenets and parameters for this project are reviewed, and situated in the broader urban health tradition. This leads to a delineation of the types of questions researchers can address when looking at a complex urban health program. Such questions reach appropriately beyond traditional public health concepts involving proximal and distal determinants of health (and associated upstream, midstream, and downstream rhetoric). Espousing a multi-level, reciprocal pathways perspective on Healthy Cities research, we also adopt a distinction between impacts and outcomes of Healthy Cities. The former are value driven, the latter intervention-driven. These approaches lead to the acknowledgment of a logic of method that includes situational and contextual appreciation of unique Healthy City experiences in a Realist Evaluation paradigm. The article concludes with a reflection of evaluation and assessment procedures applied to Phase IV (2003-2008) of the WHO European Healthy Cities Network and an interpretation of response rates to the range of methods that have been adopted.

La governance del sociosanitario in Emilia Romagna: Esperienze e prospettive

In un contesto dominato da invecchiamento della popolazione, prevalenza della cronicità e presenza crescente di pazienti multiproblematici e non autosufficienti è indispensabile spostare il baricentro delle cure dall'acuzie alla cronicità, e quindi assicurare la continuità e la coerenza fra i diversi setting di cura, sia sanitari che socio-sanitari (ospedale, servizi sanitari territoriali, domicilio, strutture residenziali di Long term care). Dall'analisi della letteratura emerge che il maggiore ostacolo a realizzare questa continuità è rappresentato dalla presenza, caratteristica del sistema di welfare italiano, di molteplici attori e strutture con competenze, obiettivi e funzioni diverse e separate, e la raccomandazione di lavorare per l'integrazione contemporaneamente su più livelli: - normativo-istituzionale - programmatorio - professionale e gestionale Il sistema della "governance" realizzato in Emilia-Romagna per l'integrazione socio-sanitaria è stato valutato alla luce di queste raccomandazioni, seguendo il modello della Realist evaluation per i Social complex interventions: enucleando le "teorie" alla base dell'intervento ed analizzando i diversi step della sua implementazione. Alla luce di questa valutazione, il modello della "governance" è risultato coerente con le indicazioni delle linee guida, ed effettivamente capace di produrre risultati al fine della continuità e della coerenza fra cure sanitarie e assistenza sociale e sanitaria complessa. Resta da realizzare una valutazione complessiva dell'impatto su efficacia, costi e soddisfazione dei pazienti.

Evaluating EU regional policy

This article presents a challenge to the ways in which EU regional policy has been evaluated in the past. Given the complexity of the 'policy framework' and its objectives, it is argued that existing evaluation methodologies are not only inappropriate but create a real risk of misleading policy-makers in their search for identifying which programmes and initiatives are the most effective in tackling the scale of regional disparity that exists across the European Union. For example, the search for an 'average effect' of intervention, whether in terms of jobs created or GVA generated, does not adequately recognise the context within which policy operates. The article argues that only by attempting to adopt a realist evaluation framework can the discourse on effective regional policy be advanced. Examples are provided from a body of work on the evaluation of business support interventions in the UK as well as a broader study of the way in which regulations impacts upon firm performance and growth. This methodological approach provides an opportunity for the evaluator to identify the causal mechanisms which connect the range of policy interventions and their outcomes. In brief, it has greater potential to inform the policy-maker as to what works and, more importantly, in what contexts.

Policy transfer in practice:implementing supplier diversity in the UK

This article draws on the policy transfer literature to examine a UK-based initiative to promote supplier diversity and provides insights into three areas neglected in current research, namely: the dynamics of non-governmental policy transfer; the factors that mediate policy transfer in different jurisdictions; and the integration of research and practice in small business related policy transfer. To this end, an innovative action research approach is deployed with the dual purpose of effecting practical change and advancing knowledge. This comprises two phases: first, a realist analysis of the US National Minority Supplier Development Council (NMSDC), an exemplar intermediary; and second, the implementation and concurrent realist evaluation of a supplier diversity initiative modelled on NMSDC, referred to as ‘Supplier Development East Midlands’ (SDEM). The findings provide lessons for academics and practitioners dealing with small and medium-sized enterprise (SME) policy transfer in general and supplier diversity intermediaries in particular.

Assessing the impacts of community participation policy and practice in Scotland and England

Background: Community participation has become an integral part of many areas of public policy over the last two decades. For a variety of reasons, ranging from concerns about social cohesion and unrest to perceived failings in public services, governments in the UK and elsewhere have turned to communities as both a site of intervention and a potential solution. In contemporary policy, the shift to community is exemplified by the UK Government’s Big Society/Localism agenda and the Scottish Government’s emphasis on Community Empowerment. Through such policies, communities have been increasingly encouraged to help themselves in various ways, to work with public agencies in reshaping services, and to become more engaged in the democratic process. These developments have led some theorists to argue that responsibilities are being shifted from the state onto communities, representing a new form of 'government through community' (Rose, 1996; Imrie and Raco, 2003). Despite this policy development, there is surprisingly little evidence which demonstrates the outcomes of the different forms of community participation. This study attempts to address this gap in two ways. Firstly, it explores the ways in which community participation policy in Scotland and England are playing out in practice. And secondly, it assesses the outcomes of different forms of community participation taking place within these broad policy contexts. Methodology: The study employs an innovative combination of the two main theory-based evaluation methodologies, Theories of Change (ToC) and Realist Evaluation (RE), building on ideas generated by earlier applications of each approach (Blamey and Mackenzie, 2007). ToC methodology is used to analyse the national policy frameworks and the general approach of community organisations in six case studies, three in Scotland and three in England. The local evidence from the community organisations’ theories of change is then used to analyse and critique the assumptions which underlie the Localism and Community Empowerment policies. Alongside this, across the six case studies, a RE approach is utilised to examine the specific mechanisms which operate to deliver outcomes from community participation processes, and to explore the contextual factors which influence their operation. Given the innovative methodological approach, the study also engages in some focused reflection on the practicality and usefulness of combining ToC and RE approaches. Findings: The case studies provide significant evidence of the outcomes that community organisations can deliver through directly providing services or facilities, and through influencing public services. Important contextual factors in both countries include particular strengths within communities and positive relationships with at least part of the local state, although this often exists in parallel with elements of conflict. Notably this evidence suggests that the idea of responsibilisation needs to be examined in a more nuanced fashion, incorporating issues of risk and power, as well the active agency of communities and the local state. Thus communities may sometimes willingly take on responsibility in return for power, although this may also engender significant risk, with the balance between these three elements being significantly mediated by local government. The evidence also highlights the impacts of austerity on community participation, with cuts to local government budgets in particular increasing the degree of risk and responsibility for communities and reducing opportunities for power. Furthermore, the case studies demonstrate the importance of inequalities within and between communities, operating through a socio-economic gradient in community capacity. This has the potential to make community participation policy regressive as more affluent communities are more able to take advantage of additional powers and local authorities have less resource to support the capacity of more disadvantaged communities. For Localism in particular, the findings suggest that some of the ‘new community rights’ may provide opportunities for communities to gain power and generate positive social outcomes. However, the English case studies also highlight the substantial risks involved and the extent to which such opportunities are being undermined by austerity. The case studies suggest that cuts to local government budgets have the potential to undermine some aspects of Localism almost entirely, and that the very limited interest in inequalities means that Localism may be both ‘empowering the powerful’ (Hastings and Matthews, 2014) and further disempowering the powerless. For Community Empowerment, the study demonstrates the ways in which community organisations can gain power and deliver positive social outcomes within the broad policy framework. However, whilst Community Empowerment is ostensibly less regressive, there are still significant challenges to be addressed. In particular, the case studies highlight significant constraints on the notion that communities can ‘choose their own level of empowerment’, and the assumption of partnership working between communities and the local state needs to take into account the evidence of very mixed relationships in practice. Most importantly, whilst austerity has had more limited impacts on local government in Scotland so far, the projected cuts in this area may leave Community Empowerment vulnerable to the dangers of regressive impact highlighted for Localism. Methodologically, the study shows that ToC and RE can be practically applied together and that there may be significant benefits of the combination. ToC offers a productive framework for policy analysis and combining this with data derived from local ToCs provides a powerful lens through which to examine and critique the aims and assumptions of national policy. ToC models also provide a useful framework within which to identify specific causal mechanisms, using RE methodology and, again, the data from local ToC work can enable significant learning about ‘what works for whom in what circumstances’ (Pawson and Tilley, 1997).

Stakeholders and perspectivism in qualitative policy evaluation: A realist reflection

This paper examines the epistemological bases for the inclusion of stakeholders in policy research. While it concedes that the input of stakeholders provides essential expert and experiential knowledge for the understanding of complex policies and programmes, it contends that the approach which assumes that all interpretations of policy including those of stakeholders should be afforded equal validity, which we term relativist perspectivism, undermines the possibility of robust research by allowing power to replace methodological rigour as the primary research dynamic. It is noted that this problem tends to be more acute when the research is qualitative. A study into the gendered effects of Common Agricultural Policy reforms is used as an illustrative example of how research can be compromised by relativist perspectivism. It is argued that realist research methodologies uniquely provide the capacity to maintain epistemological robustness, while also being able to take due account of the perspectives of stakeholders.

The use and limitation of realistic evaluation as a tool for evidence-based practice: a critical realist perspective

In this paper, we assess realistic evaluation’s articulation with evidence-based practice (EBP) from the perspective of critical realism. We argue that the adoption by realistic evaluation of a realist causal ontology means that it is better placed to explain complex healthcare interventions than the traditional method used by EBP, the randomized controlled trial (RCT). However, we do not conclude from this that the use of RCTs is without merit, arguing that it is possible to use both methods in combination under the rubric of realist theory. More negatively, we contend that the rejection of critical theory and utopianism by realistic evaluation in favour of the pragmatism of piecemeal social engineering means that it is vulnerable to accusations that it promotes technocratic interpretations of human problems. We conclude that, insofar as realistic evaluation adheres to the ontology of critical realism, it provides a sound contribution to EBP, but insofar as it rejects the critical turn of Bhaskar’s realism, it replicates the technocratic tendencies inherent in EBP.