Establishing the validity of English GP Patient Survey items evaluating out-of-hours care

Background A 2014 national audit used the English General Practice Patient Survey (GPPS) to compare service users’ experience of out-of-hours general practitioner (GP) services, yet there is no published evidence on the validity of these GPPS items. Objectives Establish the construct and concurrent validity of GPPS items evaluating service users’ experience of GP out-of-hours care. Methods Cross-sectional postal survey of service users (n=1396) of six English out-of-hours providers. Participants reported on four GPPS items evaluating out-of-hours care (three items modified following cognitive interviews with service users), and 14 evaluative items from the Out-of-hours Patient Questionnaire (OPQ). Construct validity was assessed through correlations between any reliable (Cochran's α>0.7) scales, as suggested by a principal component analysis of the modified GPPS items, with the ‘entry access’ (four items) and ‘consultation satisfaction’ (10 items) OPQ subscales. Concurrent validity was determined by investigating whether each modified GPPS item was associated with thematically related items from the OPQ using linear regressions. Results The modified GPPS item-set formed a single scale (α=0.77), which summarised the two-component structure of the OPQ moderately well; explaining 39.7% of variation in the ‘entry access’ scores (r=0.63) and 44.0% of variation in the ‘consultation satisfaction’ scores (r=0.66), demonstrating acceptable construct validity. Concurrent validity was verified as each modified GPPS item was highly associated with a distinct set of related items from the OPQ. Conclusions Minor modifications are required for the English GPPS items evaluating out-of-hours care to improve comprehension by service users. A modified question set was demonstrated to comprise a valid measure of service users’ overall satisfaction with out-of-hours care received. This demonstrates the potential for the use of as few as four items in benchmarking providers and assisting services in identifying, implementing and assessing quality improvement initiatives.

The use of patient experience survey data by out-of-hours primary care services: a qualitative interview study

Background English National Quality Requirements mandate out-of-hours primary care services to routinely audit patient experience, but do not state how it should be done.

Objectives We explored how providers collect patient feedback data and use it to inform service provision. We also explored staff views on the utility of out-of-hours questions from the English General Practice Patient Survey (GPPS).

Methods A qualitative study was conducted with 31 staff (comprising service managers, general practitioners and administrators) from 11 out-of-hours primary care providers in England, UK. Staff responsible for patient experience audits within their service were sampled and data collected via face-to-face semistructured interviews.

Results Although most providers regularly audited their patients’ experiences by using patient surveys, many participants expressed a strong preference for additional qualitative feedback. Staff provided examples of small changes to service delivery resulting from patient feedback, but service-wide changes were not instigated. Perceptions that patients lacked sufficient understanding of the urgent care system in which out-of-hours primary care services operate were common and a barrier to using feedback to enable change. Participants recognised the value of using patient experience feedback to benchmark services, but perceived weaknesses in the out-of-hours items from the GPPS led them to question the validity of using these data for benchmarking in its current form.

Conclusions The lack of clarity around how out-of-hours providers should audit patient experience hinders the utility of the National Quality Requirements. Although surveys were common, patient feedback data had only a limited role in service change. Data derived from the GPPS may be used to benchmark service providers, but refinement of the out-of-hours items is needed.

Managing patient deterioration: evaluation of what works.

Background: Rapid Response Systems (RRS) have been implemented nationally and internationally to improve patient safety in hospital. However, to date the majority of the RRS research evidence has focused on measuring the effectiveness of the intervention on patient outcomes. To evaluate RRS it has been recommended that a multimodal approach is required to address the broad range of process and outcome measures required to determine the effectiveness of the RRS concept. Aim: The aim of this paper is to evaluate the official RRS programme theoretical assumptions regarding how the programme is meant to work against actual practice in order to determine what works. Methods: The research design was a multiple case study approach of four wards in two hospitals in Northern Ireland. It followed the principles of realist evaluation research which allowed empirical data to be gathered to test and refine RRS programme theory [1]. This approach used a variety of mixed methods to test the programme theories including individual and focus group interviews with a purposive sample of 75 nurses and doctors, observation of ward practices and documentary analysis. The findings from the case studies were analysed and compared within and across cases to identify what works for whom and in what circumstances. Results: The RRS programme theories were critically evaluated and compared with study findings to develop a mid-range theory to explain what works, for whom in what circumstances. The findings of what works suggests that clinical experience, established working relationships, flexible implementation of protocols, ongoing experiential learning, empowerment and pre-emptive management are key to the success of RRS implementation.  Conclusion:These findings highlight the combination of factors that can improve the implementation of RRS and in light of this evidence several recommendations are made to provide policymakers with guidance and direction for their success and sustainability.References: 1.Pawson R and Tilley N. (1997) Realistic Evaluation. Sage Publications; LondonType of submission: Concurrent session Source of funding: Sandra Ryan Fellowship funded by the School of Nursing & Midwifery, Queen’s University of Belfast

Realistic evaluation of EWS and ALERT: factors enabling and constraining implementation.

Realistic Evaluation of EWS and ALERT: factors enabling and constraining implementation Background The implementation of EWS and ALERT in practice is essential to the success of Rapid Response Systems but is dependent upon nurses utilising EWS protocols and applying ALERT best practice guidelines. To date there is limited evidence on the effectiveness of EWS or ALERT as research has primarily focused on measuring patient outcomes (cardiac arrests, ICU admissions) following the implementation of a Rapid Response Team. Complex interventions in healthcare aimed at changing service delivery and related behaviour of health professionals require a different research approach to evaluate the evidence. To understand how and why EWS and ALERT work, or might not work, research needs to consider the social, cultural and organisational influences that will impact on successful implementation in practice. This requires a research approach that considers both the processes and outcomes of complex interventions, such as EWS and ALERT, implemented in practice. Realistic Evaluation is such an approach and was used to explain the factors that enable and constrain the implementation of EWS and ALERT in practice [1]. Aim The aim of this study was to evaluate factors that enabled and constrained the implementation and service delivery of early warnings systems (EWS) and ALERT in practice in order to provide direction for enabling their success and sustainability. Methods The research design was a multiple case study approach of four wards in two hospitals in Northern Ireland. It followed the principles of realist evaluation research which allowed empirical data to be gathered to test and refine RRS programme theory. This approach used a variety of mixed methods to test the programme theories including individual and focus group interviews, observation and documentary analysis in a two stage process. A purposive sample of 75 key informants participated in individual and focus group interviews. Observation and documentary analysis of EWS compliance data and ALERT training records provided further evidence to support or refute the interview findings. Data was analysed using NVIVO8 to categorise interview findings and SPSS for ALERT documentary data. These findings were further synthesised by undertaking a within and cross case comparison to explain the factors enabling and constraining EWS and ALERT. Results A cross case analysis highlighted similarities, differences and factors enabling or constraining successful implementation across the case study sites. Findings showed that personal (confidence; clinical judgement; personality), social (ward leadership; communication), organisational (workload and staffing issues; pressure from managers to complete EWS audit and targets), educational (constraints on training; no clinical educator on ward) and cultural (routine task delegated) influences impact on EWS and acute care training outcomes. There were also differences noted between medical and surgical wards across both case sites. Conclusions Realist Evaluation allows refinement and development of the RRS programme theory to explain the realities of practice. These refined RRS programme theories are capable of informing the planning of future service provision and provide direction for enabling their success and sustainability. References: 1. McGaughey J, Blackwood B, O’Halloran P, Trinder T. J. & Porter S. (2010) A realistic evaluation of Track and Trigger systems and acute care training for early recognition and management of deteriorating ward–based patients. Journal of Advanced Nursing 66 (4), 923-932. Type of submission: Concurrent session Source of funding: Sandra Ryan Fellowship funded by the School of Nursing & Midwifery, Queen’s University of Belfast

Realist synthesis: Critical care rapid response systems.

Background Rapid Response Systems (RRS) consist of four interrelated and interdependent components; an event detection and trigger mechanism, a response strategy, a governance structure and process improvement system. These multiple components of the RRS pose problems in evaluation as the intervention is complex and cannot be evaluated using a traditional systematic review. Complex interventions in healthcare aimed at changing service delivery and related behaviour of health professionals require a different approach to summarising the evidence. Realist synthesis is such an approach to reviewing research evidence on complex interventions to provide an explanatory analysis of how and why an intervention works or doesn’t work in practice. The core principle is to make explicit the underlying assumptions about how an intervention is suppose to work (ie programme theory) and then use this theory to guide evaluation. Methods A realist synthesis process was used to explain those factors that enable or constrain the success of RRS programmes. Results The findings from the review include the articulation of the RRS programme theories, evaluation of whether these theories are supported or refuted by the research evidence and an evaluation of evidence to explain the underlying reasons why RRS works or doesn’t work in practice. Rival conjectured RRS programme theories were identified to explain the constraining factors regarding implementation of RRS in practice. These programme theories are presented using a logic model to highlight all the components which impact or influence the delivery of RRS programmes in the practice setting. The evidence from the realist synthesis provided the foundation for the development of hypothesis to test and refine the theories in the subsequent stages of the Realist Evaluation PhD study [1]. This information will be useful in providing evidence and direction for strategic and service planning of acute care to improve patient safety in hospital. References: McGaughey J, Blackwood B, O’Halloran P, Trinder T. J. & Porter S. (2010) Realistic Evaluation of Early Warning Systems and the Acute Life-threatening Events – Recognition and Treatment training course for early recognition and management of deteriorating ward-based patients: research protocol. Journal of Advanced Nursing 66 (4), 923-932.

Early warning systems: problems, pragmatics and potential

Symposium Chair: Dr Jennifer McGaughey

Title: Early Warning Systems: problems, pragmatics and potential

Early Warning Systems (EWS) provide a mechanism for staff to recognise, refer and manage deteriorating patients on general hospital wards. Implementation of EWS in practice has required considerable change in the delivery of critical care across hospitals. Drawing their experience of these changes the authors will demonstrate the problems and potential of using EWS to improve patient outcomes.

The first paper (Dr Jennifer McGaughey: Early Warning Systems: what works?) reviews the research evidence regarding the factors that support or constrain the implementation of Early Warning System (EWS) in practice. These findings explain those processes which impact on the successful achievement of patient outcomes. In order to improve detection and standardise practice National EWS have been implemented in the United Kingdom. The second paper (Catherine Plowright: The implementation of the National EWS in a District General Hospital) focuses on the process of implementing and auditing a National EWS. This process improvement is essential to contribute to future collaborative research and collection of robust datasets to improve patient safety as recommended by the Royal College of Physicians (RCP 2012). To successfully implement NEWS in practice requires strategic planning and staff education. The practical issues of training staff is discussed in the third paper. This paper (Collette Laws-Chapman: Simulation as a modality to embed the use of Early Warning Systems) focuses on using simulation and structured debrief to enhance learning in the early recognition and management of deteriorating patients. This session emphasises the importance of cognitive and social skills developed alongside practical skills in the simulated setting.

Factors that impact on the use of mechanical ventilation weaning protocols in critically ill adults and children: a qualitative-evidence synthesis

Background:
Prolonged mechanical ventilation is associated with a longer intensive care unit (ICU) length of stay and higher mortality. Consequently, methods to improve ventilator weaning processes have been sought. Two recent Cochrane systematic reviews in ICU adult and paediatric populations concluded that protocols can be effective in reducing the duration of mechanical ventilation, but there was significant heterogeneity in study findings. Growing awareness of the benefits of understanding the contextual factors impacting on effectiveness has encouraged the integration of qualitative evidence syntheses with effectiveness reviews, which has delivered important insights into the reasons underpinning (differential) effectiveness of healthcare interventions.

Objectives:
1. To locate, appraise and synthesize qualitative evidence concerning the barriers and facilitators of the use of protocols for weaning critically-ill adults and children from mechanical ventilation;

2. To integrate this synthesis with two Cochrane effectiveness reviews of protocolized weaning to help explain observed heterogeneity by identifying contextual factors that impact on the use of protocols for weaning critically-ill adults and children from mechanical ventilation;

3. To use the integrated body of evidence to suggest the circumstances in which weaning protocols are most likely to be used.

Search methods:
We used a range of search terms identified with the help of the SPICE (Setting, Perspective, Intervention, Comparison, Evaluation) mnemonic. Where available, we used appropriate methodological filters for specific databases. We searched the following databases: Ovid MEDLINE, Embase, OVID, PsycINFO, CINAHL Plus, EBSCOHost, Web of Science Core Collection, ASSIA, IBSS, Sociological Abstracts, ProQuest and LILACS on the 26th February 2015. In addition, we searched: the grey literature; the websites of professional associations for relevant publications; and the reference lists of all publications reviewed. We also contacted authors of the trials included in the effectiveness reviews as well as of studies (potentially) included in the qualitative synthesis, conducted citation searches of the publications reporting these studies, and contacted content experts.

We reran the search on 3rd July 2016 and found three studies, which are awaiting classification.

Selection criteria:
We included qualitative studies that described: the circumstances in which protocols are designed, implemented or used, or both, and the views and experiences of healthcare professionals either involved in the design, implementation or use of weaning protocols or involved in the weaning of critically-ill adults and children from mechanical ventilation not using protocols. We included studies that: reflected on any aspect of the use of protocols, explored contextual factors relevant to the development, implementation or use of weaning protocols, and reported contextual phenomena and outcomes identified as relevant to the effectiveness of protocolized weaning from mechanical ventilation.

Data collection and analysis:
At each stage, two review authors undertook designated tasks, with the results shared amongst the wider team for discussion and final development. We independently reviewed all retrieved titles, abstracts and full papers for inclusion, and independently extracted selected data from included studies. We used the findings of the included studies to develop a new set of analytic themes focused on the barriers and facilitators to the use of protocols, and further refined them to produce a set of summary statements. We used the Confidence in the Evidence from Reviews of Qualitative Research (CERQual) framework to arrive at a final assessment of the overall confidence of the evidence used in the synthesis. We included all studies but undertook two sensitivity analyses to determine how the removal of certain bodies of evidence impacted on the content and confidence of the synthesis. We deployed a logic model to integrate the findings of the qualitative evidence synthesis with those of the Cochrane effectiveness reviews.

Main results:
We included 11 studies in our synthesis, involving 267 participants (one study did not report the number of participants). Five more studies are awaiting classification and will be dealt with when we update the review.

The quality of the evidence was mixed; of the 35 summary statements, we assessed 17 as ‘low’, 13 as ‘moderate’ and five as ‘high’ confidence. Our synthesis produced nine analytical themes, which report potential barriers and facilitators to the use of protocols. The themes are: the need for continual staff training and development; clinical experience as this promotes felt and perceived competence and confidence to wean; the vulnerability of weaning to disparate interprofessional working; an understanding of protocols as militating against a necessary proactivity in clinical practice; perceived nursing scope of practice and professional risk; ICU structure and processes of care; the ability of protocols to act as a prompt for shared care and consistency in weaning practice; maximizing the use of protocols through visibility and ease of implementation; and the ability of protocols to act as a framework for communication with parents.

Authors' conclusions:
There is a clear need for weaning protocols to take account of the social and cultural environment in which they are to be implemented. Irrespective of its inherent strengths, a protocol will not be used if it does not accommodate these complexities. In terms of protocol development, comprehensive interprofessional input will help to ensure broad-based understanding and a sense of ‘ownership’. In terms of implementation, all relevant ICU staff will benefit from general weaning as well as protocol-specific training; not only will this help secure a relevant clinical knowledge base and operational understanding, but will also demonstrate to others that this knowledge and understanding is in place. In order to maximize relevance and acceptability, protocols should be designed with the patient profile and requirements of the target ICU in mind. Predictably, an under-resourced ICU will impact adversely on protocol implementation, as staff will prioritize management of acutely deteriorating and critically-ill patients.

Identification of novel Factor XI mutations in a patient requiring dental extractions

Factor XI is a serine protease that participates in the intrinsic pathway of blood coagulation. Patients deficient in factor XI exhibit varying degrees of post operative bleeding following invasive surgical procedures such as dental extractions. Objectives: The aim of the study was to identify the specific mutations in a patient from a family with known factor XI deficiency. Methods: Samples were obtained from the patient, his mother and his father and subjected to DNA sequencing. Each protein coding exon 2-15 of the factor XI gene was amplified by polymerase chain reaction (PCR) followed by bidirectional sequencing utilizing di-deoxy chain termination chemistry. Results: The patient had a factor XI level of 20% of normal. Initial sequencing of factor XI from the patient identified a point mutation (646G>A) and a putative splice site mutation (1567+4A>T) in intron 13. These are novel previously unreported mutations. DNA sequence analysis of the mother revealed the 1567+4A>T mutation and the father exhibited the 646G>A mutation. As a consequence the treatment proceeded without serious bleeding complication and required administration only of transexamic acid though factor XI was available as haemostatic cover. Conclusion: The two mutations identified in this family are novel; further laboratory investigation of the functional consequences of those mutations is currently underway. Although factor XI deficiency is rare in the Northern Irish population this study highlights the techniques available to sequence and analyse this and similar haematological disorders.