The Evaluation of health and safety auditing systems

The specific objective of the research was to evaluate proprietary audit systems. Proprietary audit systems comprise question sets containing approximately 500 questions dealing with selected aspects of health and safety management. Each question is allotted a number of points and an organisation seeks to judge its health and safety performance by the overall score achieved in the audit. Initially it was considered that the evaluation method might involve comparing the proprietary audit scores with other methods of measuring safety performance. However, what appeared to be missing in the first instance was information that organisations could use to compare the contrast question set content against their own needs. A technique was developed using the computer database FileMaker Pro. This enables questions in an audit to be sorted into categories using a process of searching for key words. Questions that are not categorised by word searching can be identified and sorted manually. The process can be completed in 2-3 hours which is considerably faster than manual categorisation of questions which typically takes about 10 days. The technique was used to compare and contrast three proprietary audits: ISRS, CHASE and QSA. Differences and similarities between these audits were successfully identified. It was concluded that in general proprietary audits need to focus to a greater extent on identifying strengths and weaknesses in occupational health and safety management systems. To do this requires the inclusion of more probing questions which consider whether risk control measures are likely to be successful.

+Psychometric evaluation of the MacDQoL individualised measure of the impact of macular degeneration on quality of life

Background: The MacDQoL is an individualised measure of the impact of macular degeneration (MD) on quality of life (QoL). There is preliminary evidence of its psychometric properties and sensitivity to severity of MD. The aim of this study was to carry out further psychometric evaluation with a larger sample and investigate the measure's sensitivity to MD severity. Methods: Patients with MD (n = 156: 99 women, 57 men, mean age 79 ± 13 years), recruited from eye clinics (one NHS, one private) completed the MacDQoL by telephone interview and later underwent a clinic vision assessment including near and distance visual acuity (VA), comfortable near VA, contrast sensitivity, colour recognition, recovery from glare and presence or absence of distortion or scotoma in the central 10° of the visual field. Results: The completion rate for the MacDQoL items was 99.8%. Of the 26 items, three were dropped from the measure due to redundancy. A fourth was retained in the questionnaire but excluded when computing the scale score. Principal components analysis and Cronbach's alpha (0.944) supported combining the remaining 22 items in a single scale. Lower MacDQoL scores, indicating more negative impact of MD on QoL, were associated with poorer distance VA (better eye r = -0.431 p < 0.001; worse eye r = -0.350 p < 0.001; binocular vision r = -0.419 p < 0.001) and near VA (better eye r -0.326 p < 0.001; worse eye r = -0.226 p < 0.001; binocular vision r = -0.326 p < 0.001). Poorer MacDQoL scores were associated with poorer contrast sensitivity (better eye r = 0.392 p < 0.001; binocular vision r = 0.423 p < 0.001), poorer colour recognition (r = 0.417 p < 0.001) and poorer comfortable near VA (r = -0.283, p < 0.001). The MacDQoL differentiated between those with and without binocular scotoma (U = 1244 p < 0.001). Conclusion: The MacDQoL 22-item scale has excellent internal consistency reliability and a single-factor structure. The measure is acceptable to respondents and the generic QoL item, MD-specific QoL item and average weighted impact score are related to several measures of vision. The MacDQoL demonstrates that MD has considerable negative impact on many aspects of QoL, particularly independence, leisure activities, dealing with personal affairs and mobility. The measure may be valuable for use in clinical trials and routine clinical care. © 2005 Mitchell et al; licensee BioMed Central Ltd.

The evaluation of information systems in the organisational context of the National Health Service. Available in 2 volumes.

This thesis describes a project which has investigated the evaluation of information systems. The work took place in, and is related to, a specific organisational context, that of the National Health Service (NHS). It aims to increase understanding of the evaluation which takes place in the service and the way in which this is affected by the NHS environment. It also investigates the issues which surround some important types of evaluation and their use in this context. The first stage of the project was a postal survey in which respondents were asked to describe the evaluation which took place in their authorities and to give their opinions about it. This was used to give an overview of the practice of IS evaluation in the NHS and to identify its uses and the problems experienced. Three important types of evaluation were then examined in more detail by means of action research studies. One of these dealt with the selection and purchase of a large hospital information system. The study took the form of an evaluation of the procurement process, and examined the methods used and the influence of organisational factors. The other studies are concerned with post-implementation evaluation, and examine the choice of an evaluation approach as well as its application. One was an evaluation of a community health system which had been operational for some time but was of doubtful value, and suffered from a number of problems. The situation was explored by means of a study of the costs and benefits of the system. The remaining study was the initial review of a system which was used in the administration of a Breast Screening Service. The service itself was also newly operational and the relationship between the service and the system was of interest.

An evaluation of the effectiveness of intervention strategies for the promotion of health and safety performance in small firms

Recognition of the contribution of small firms to the UK economy has grown considerably since 1995 when this research first began. The poor record of small firms in managing health and safety effectively has caused concern, and efforts made to improve knowledge and awareness of the target group through various initiatives have had some success. This research thesis attempts to identify the range of intervention routes and methods available to reach the target group, and to consider ways of evaluating the outcome of such efforts. Various interventions were tested with small firms, including a Workshop; use of Questionnaires; short postal Reply Slip survey; leading to a closer evaluation of a specific industry- the Licensed Trade. Attitudes and beliefs of the sample were identified, and observations carried out to consider actions taken by workers and others in the workplace. These empirical research findings were used to develop the theme of Primary and Secondary interventions intended to change behaviours, and to confirm assumptions about what small firms currently do to manage health and safety risks. Guidance for small firms was developed as a Secondary intervention tool to support Primary interventions, such as inspection or insurance provision.

An evaluation of certain aspects of an occupational health service

This thesis has been concerned with obtaining evidence to explore the proposition that the provision of occupational health services as arranged at the present time represents a misallocation of resources. The research has been undertaken within the occupational health service of a large Midlands food factory. As the research progressed it became evident that questions were being raised about the nature and scope of occupational health as well as the contribution, in combating danger at work, that occupational health services can make to the health and safety team. These questions have been scrutinized in depth, as they are clearly important, and a resolution of the problem of the definition of occupational health has been proposed. I have taken the approach of attempting to identify specific objectives or benefits of occupational health activities so that it is possible to assess how far these objectives are being achieved. I have looked at three aspects of occupational health; audiometry, physiotherapy and pre-employment medical examinations as these activities embody crucial concepts which are common to all activities in an occupational health programme. A three category classification of occupational health activities is proposed such that the three activities provide examples within each category. These are called personnel therapy, personnel input screening and personnel throughput screening. I conclude that I have not shown audiometry to be cost-effective. My observations of the physiotherapy service lead me to support the suggestion that there is a decline in sickness absence rates due to physiotherapy in industry. With pre-employment medical examinations I have shown that the service is product safety oriented and that benefits are extremely difficult to identify. In regard to the three services studied, in the one factory investigated, and because of the immeasurability of certain activities, I find support for the proposition that the mix of occupational health services as provided at the present time represents a misallocation of resources.

A pre-post test evaluation of the impact of the PELICAN MDT-TME development programme on the working lives of colorectal cancer team members

Background - The PELICAN Multidisciplinary Team Total Mesorectal Excision (MDT-TME) Development Programme aimed to improve clinical outcomes for rectal cancer by educating colorectal cancer teams in precision surgery and related aspects of multidisciplinary care. The Programme reached almost all colorectal cancer teams across England. We took the opportunity to assess the impact of participating in this novel team-based Development Programme on the working lives of colorectal cancer team members. Methods - The impact of participating in the programme on team members' self-reported job stress, job satisfaction and team performance was assessed in a pre-post course study. 333/568 (59%) team members, from the 75 multidisciplinary teams who attended the final year of the Programme, completed questionnaires pre-course, and 6-8 weeks post-course. Results - Across all team members, the main sources of job satisfaction related to working in multidisciplinary teams; whilst feeling overloaded was the main source of job stress. Surgeons and clinical nurse specialists reported higher levels of job satisfaction than team members who do not provide direct patient care, whilst MDT coordinators reported the lowest levels of job satisfaction and job stress. Both job stress and satisfaction decreased after participating in the Programme for all team members. There was a small improvement in team performance. Conclusions - Participation in the Development Programme had a mixed impact on the working lives of team members in the immediate aftermath of attending. The decrease in team members' job stress may reflect the improved knowledge and skills conferred by the Programme. The decrease in job satisfaction may be the consequence of being unable to apply these skills immediately in clinical practice because of a lack of required infrastructure and/or equipment. In addition, whilst the Programme raised awareness of the challenges of teamworking, a greater focus on tackling these issues may have improved working lives further.

Multiple component patient safety intervention in English hospitals:controlled evaluation of second phase

Objective: To independently evaluate the impact of the second phase of the Health Foundation's Safer Patients Initiative (SPI2) on a range of patient safety measures. Design: A controlled before and after design. Five substudies: survey of staff attitudes; review of case notes from high risk (respiratory) patients in medical wards; review of case notes from surgical patients; indirect evaluation of hand hygiene by measuring hospital use of handwashing materials; measurement of outcomes (adverse events, mortality among high risk patients admitted to medical wards, patients' satisfaction, mortality in intensive care, rates of hospital acquired infection). Setting: NHS hospitals in England. Participants: Nine hospitals participating in SPI2 and nine matched control hospitals. Intervention The SPI2 intervention was similar to the SPI1, with somewhat modified goals, a slightly longer intervention period, and a smaller budget per hospital. Results: One of the scores (organisational climate) showed a significant (P=0.009) difference in rate of change over time, which favoured the control hospitals, though the difference was only 0.07 points on a five point scale. Results of the explicit case note reviews of high risk medical patients showed that certain practices improved over time in both control and SPI2 hospitals (and none deteriorated), but there were no significant differences between control and SPI2 hospitals. Monitoring of vital signs improved across control and SPI2 sites. This temporal effect was significant for monitoring the respiratory rate at both the six hour (adjusted odds ratio 2.1, 99% confidence interval 1.0 to 4.3; P=0.010) and 12 hour (2.4, 1.1 to 5.0; P=0.002) periods after admission. There was no significant effect of SPI for any of the measures of vital signs. Use of a recommended system for scoring the severity of pneumonia improved from 1.9% (1/52) to 21.4% (12/56) of control and from 2.0% (1/50) to 41.7% (25/60) of SPI2 patients. This temporal change was significant (7.3, 1.4 to 37.7; P=0.002), but the difference in difference was not significant (2.1, 0.4 to 11.1; P=0.236). There were no notable or significant changes in the pattern of prescribing errors, either over time or between control and SPI2 hospitals. Two items of medical history taking (exercise tolerance and occupation) showed significant improvement over time, across both control and SPI2 hospitals, but no additional SPI2 effect. The holistic review showed no significant changes in error rates either over time or between control and SPI2 hospitals. The explicit case note review of perioperative care showed that adherence rates for two of the four perioperative standards targeted by SPI2 were already good at baseline, exceeding 94% for antibiotic prophylaxis and 98% for deep vein thrombosis prophylaxis. Intraoperative monitoring of temperature improved over time in both groups, but this was not significant (1.8, 0.4 to 7.6; P=0.279), and there were no additional effects of SPI2. A dramatic rise in consumption of soap and alcohol hand rub was similar in control and SPI2 hospitals (P=0.760 and P=0.889, respectively), as was the corresponding decrease in rates of Clostridium difficile and meticillin resistant Staphylococcus aureus infection (P=0.652 and P=0.693, respectively). Mortality rates of medical patients included in the case note reviews in control hospitals increased from 17.3% (42/243) to 21.4% (24/112), while in SPI2 hospitals they fell from 10.3% (24/233) to 6.1% (7/114) (P=0.043). Fewer than 8% of deaths were classed as avoidable; changes in proportions could not explain the divergence of overall death rates between control and SPI2 hospitals. There was no significant difference in the rate of change in mortality in intensive care. Patients' satisfaction improved in both control and SPI2 hospitals on all dimensions, but again there were no significant changes between the two groups of hospitals. Conclusions: Many aspects of care are already good or improving across the NHS in England, suggesting considerable improvements in quality across the board. These improvements are probably due to contemporaneous policy activities relating to patient safety, including those with features similar to the SPI, and the emergence of professional consensus on some clinical processes. This phenomenon might have attenuated the incremental effect of the SPI, making it difficult to detect. Alternatively, the full impact of the SPI might be observable only in the longer term. The conclusion of this study could have been different if concurrent controls had not been used.

Development and psychometric evaluation of a new team effectiveness scale for all types of community adult mental health teams:a mixed-methods approach

Defining 'effectiveness' in the context of community mental health teams (CMHTs) has become increasingly difficult under the current pattern of provision required in National Health Service mental health services in England. The aim of this study was to establish the characteristics of multi-professional team working effectiveness in adult CMHTs to develop a new measure of CMHT effectiveness. The study was conducted between May and November 2010 and comprised two stages. Stage 1 used a formative evaluative approach based on the Productivity Measurement and Enhancement System to develop the scale with multiple stakeholder groups over a series of qualitative workshops held in various locations across England. Stage 2 analysed responses from a cross-sectional survey of 1500 members in 135 CMHTs from 11 Mental Health Trusts in England to determine the scale's psychometric properties. Based on an analysis of its structural validity and reliability, the resultant 20-item scale demonstrated good psychometric properties and captured one overall latent factor of CMHT effectiveness comprising seven dimensions: improved service user well-being, creative problem-solving, continuous care, inter-team working, respect between professionals, engagement with carers and therapeutic relationships with service users. The scale will be of significant value to CMHTs and healthcare commissioners both nationally and internationally for monitoring, evaluating and improving team functioning in practice.

GPs' perspectives on managing the NHS Health Check in primary care:a qualitative evaluation of implementation in one area of England

OBJECTIVES: To evaluate the implementation of the National Health Service (NHS) Health Check programme in one area of England from the perspective of general practitioners (GPs). DESIGN: A qualitative exploratory study was conducted with GPs and other healthcare professionals involved in delivering the NHS Health Check and with patients. This paper reports the experience of GPs and focuses on the management of the Heath Check programme in primary care. SETTING: Primary care surgeries in the Heart of Birmingham region (now under the auspices of the Birmingham Cross City Clinical Commissioning Group) were invited to take part in the larger scale evaluation. This study focuses on a subset of those surgeries whose GPs were willing to participate. PARTICIPANTS: 9 GPs from different practices volunteered. GPs served an ethnically diverse region with areas of socioeconomic deprivation. Ethnicities of participant GPs included South Asian, South Asian British, white, black British and Chinese. METHODS: Individual semistructured interviews were conducted with GPs face to face or via telephone. Thematic analysis was used to analyse verbatim transcripts. RESULTS: Themes were generated which represent GPs' experiences of managing the NHS Health Check: primary care as a commercial enterprise; 'buy in' to concordance in preventive healthcare; following protocol and support provision. These themes represent the key issues raised by GPs. They reveal variability in the implementation of NHS Health Checks. GPs also need support in allocating resources to the Health Check including training on how to conduct checks in a concordant (or collaborative) way. CONCLUSIONS: The variability observed in this small-scale evaluation corroborates existing findings suggesting a need for more standardisation. Further large-scale research is needed to determine how that could be achieved. Work needs to be done to further develop a concordant approach to lifestyle advice which involves tailored individual goal setting rather than a paternalistic advice-giving model.

Evaluation of low vision services on the quality of life of individuals with age-related macular degeneration

Background: Age-related macular degeneration (ARMD) is a major cause of irreversible visual loss in the elderly and a significant threat to their quality of life. Although low vision services often improve the functional outcomes of individuals with macular disease, it remains unclear whether or not they have any impact on quality of life. The principal aim of this study was to determine the effect of a hospital-based low vision clinic on the quality of life of individuals with ARMD. Methods: Forty patients with ARMD attended the low vision clinic at Milton Keynes University Hospital. Quality of life was measured with the vision-specific Low Vision Quality of Life (LVQOL) questionnaire and the general health EuroQol (EQ-5D-5L) questionnaire. Measures were completed at baseline (time zero, T0), and at three- (T3) and six-month (T6) follow-up visits. Results: The near visual acuity of individuals attending the low vision clinic for the first time improved significantly between visits T0 and T3 (p=0.005), reflecting the practiced use of their newly-dispensed low vision aids. As expected, there was no significant change in near acuity over this time period for existing patients. For both new and existing patients, a significant increase in LVQOL score was evident between visits T0 and T3, with a further significant improvement between T3 and T6. Similarly, there was a significant decrease in EQ-5D-5L questionnaire scores between visits T0 and T6. Conclusions: The higher LVQOL scores obtained at the end of the study period (T6) provide evidence that low vision services at Milton Keynes University Hospital served to improve patient quality of life. The reduction in EQ-5D-5L scores over the same time period suggests that low vision services also provide for an improvement in general health-related quality of life. Impact: The findings support the cause of low vision services to improve not only the vision and functional outcomes of individuals with macular disease but also their quality of life. Moreover, the findings suggest that a more efficient allocation of resources at low vision clinics may be possible through the standardisation of patient follow-up frequency.