Student evaluation of teaching : keeping in touch with reality

Student evaluation of teaching is commonplace in many universities and may be the predominant input into the performance evaluation of staff and organisational units. This article used publicly available student evaluation of teaching data to present examples of where institutional responses to evaluation processes appeared to be educationally ineffective and where the pursuit of the ‘right’ student evaluation results appears to have been mistakenly equated with the aim of improved teaching and learning. If the vast resources devoted to student evaluation of teaching are to be effective, then the data produced by student evaluation systems must lead to real and sustainable improvements in teaching quality and student learning, rather than becoming an end in itself.

The performance of a student evaluation of teaching system

Student evaluation of teaching (SET) is now commonplace in many universities internationally. While much effort has been devoted to examining the statistical validity of SET instruments, there has been limited examination of the methodological and consequential validity (together referred to as ‘utility’) of the ways in which SET data are used. This paper examines the SET system at Deakin University from the perspective of utility. It draws on publicly available SET results for an entire annual cycle of unit offerings. Consideration is given to the representativeness of the data produced, and to the utility of the data reported, by the system. While this investigation focuses on the SET system currently employed at Deakin University, it offers both an analysis methodology and conclusions that can be applied more generally.

Scaling up diabetes prevention in Victoria, Australia: policy development, implementation and evaluation

Objective
The Australian lifestyle intervention program Life! is only the second reported, large-scale diabetes prevention program. This paper describes the genesis and the successful establishment of Life! and its key outcomes for participants and implementation.

Research
Design and Methods Life!, a behavior change intervention, comprises six group sessions over eight months. The Victorian Department of Health funded Diabetes Australia-Victoria to implement the program. Experience of the Greater Green Triangle diabetes prevention implementation trial was used for intervention design, workforce development, training and infrastructure. Clinical and anthropometric data from participants, used for program evaluation, was recorded on a central database.

Results
Life! has a state-wide workforce of 302 trained facilitators within 137 organizations. 29,000 Victorians showed interest in Life! and 15,000 individuals have been referred to the program. In total, 8,412 participants commenced a Life! program between October 2007 and June 2011. 37% of the original participants completed the eight month program. Participants completing sessions one to five lost an average of 1·4 kg weight (p<0·001) and waist circumference of 2·5cm (p<0.001). Those completing six sessions lost an average of 2·4 kg weight (p<0·001) and waist circumference of 3·8 cm (p<0·001). The weight loss of 2.4 kg represents 2.7% of participants’ starting body weight.

Conclusion
The impact of Life! is attributable to applying available evidence for the systems design of the intervention, and collaboration between policy makers, implementers and evaluators using the principles of continuous quality improvement to support successful, large scale recruitment and implementation.

Clinical sensemaking: a systematic approach to reduce the impact of normalised deviance in the medical profession

organisational culture, quality assurance, health care, state medicine, Great Britain, Health care reform, quality improvement

Elements underpinning successful implementation of a national best-practice child investigative interviewing framework

The complexity and effort required to achieve the widespread implementation of best-practice child interview guidelines justifies the establishment of structures to enhance cross-jurisdictional sharing of expertise, resources and training delivery support. Australia has made great strides toward such a system via work currently being undertaken by police jurisdictions to facilitate greater consistency in education and training for practitioners in the area of investigative interviewing, strengthening collaboration between police and tertiary education institutions, and growing commitment to evidence-based policy and practice among police executives. To maximise progress, however, organisations need to consider the development of a coordinated continual quality improvement approach. This will be impeded by three structural elements: access to field interviews for practitioner feedback and organisational evaluation, interviewer tenure and case tracking. This article discusses each element, their roles within a national best-practice interview framework, and attempts by some jurisdictions to address them. It also provides recommendations to guide further reform.

Convergence and divergence dynamics in British and French business schools: how will the pressure for accreditation influence these dynamics?

This paper focuses on convergence and divergence dynamics among leading British and French business schools and explores how the pressure for accreditation influences these dynamics. We illustrate that despite historical differences in approaches to management education in Britain and France, these approaches have converged partly based on the influence of the American model of management education but more recently through the pursuit of accreditation, in particular from the Association to Advance Collegiate Schools of Business and the European Quality Improvement Standard. We explore these dynamics through the application of the resource-based view of the firm and institutional theory and suggest that, whilst achieving accreditation is a necessary precursor for international competition, it is no longer a form of competitive advantage. The pursuit of accreditation has fostered a form of competitive mimicry reducing national distinctiveness. The resource-based view of the firm suggests that the top schools need a more heterogeneous approach that is not easily replicable if they are to outperform the competitors. Consequently, the convergence of management education in Britain and France will become a new impetus for divergence. We assert that future growth and competitive advantage might be better achieved through the reassertion of national, regional and local cultural characteristics. © 2013 British Academy of Management.

Understanding context in knowledge translation: a concept analysis study protocol.

AIM: To conduct a concept analysis of clinical practice contexts (work environments) that facilitate or militate against the uptake of research evidence by healthcare professionals in clinical practice. This will involve developing a clear definition of context by describing its features, domains and defining characteristics. BACKGROUND: The context where clinical care is delivered influences that care. While research shows that context is important to knowledge translation (implementation), we lack conceptual clarity on what is context, which contextual factors probably modify the effect of knowledge translation interventions (and hence should be considered when designing interventions) and which contextual factors themselves could be targeted as part of a knowledge translation intervention (context modification). DESIGN: Concept analysis. METHODS: The Walker and Avant concept analysis method, comprised of eight systematic steps, will be used: (1) concept selection; (2) determination of aims; (3) identification of uses of context; (4) determination of defining attributes of context; (5) identification/construction of a model case of context; (6) identification/construction of additional cases of context; (7) identification/construction of antecedents and consequences of context; and (8) definition of empirical referents of context. This study is funded by the Canadian Institutes of Health Research (January 2014). DISCUSSION: This study will result in a much needed framework of context for knowledge translation, which identifies specific elements that, if assessed and used to tailor knowledge translation activities, will result in increased research use by nurses and other healthcare professionals in clinical practice, ultimately leading to better patient care.

Cross-cultural examination of the structure of the Revised American Pain Society Patient Outcome Questionnaire (APS-POQ-R)

This study investigated the cross-cultural factor stability and internal consistency of the Revised American Pain Society Patient Outcome Questionnaire (APS-POQ-R); a measure of the quality of postoperative pain management employed internationally. We conducted exploratory factor analysis (EFA) of APS-POQ-R data from two point-prevalence studies comprising 268 and 311 surveys of Danish and Australian medical-surgical patients. Parallel analysis indicated four and three factor solutions for Danish and Australian patients respectively, which accounted for 58.1% and 52.9% of variance. Internal consistency was unsatisfactory among both Danish (Cronbach α=.54) and Australian (Cronbach α=.63) cohorts. There was a high degree of between-group similarity in item-factor loadings of variables coded as "pain experience", but not "pain management". This reflected cross-cultural differences in ratings of treatment satisfaction. For Danish patients, satisfaction was associated with the degree of pain severity and activity interference whereas for Australian patients, satisfaction was associated with their perceived ability to participate in treatment. To facilitate further cross-cultural comparison, we compared our findings to past research conducted in the U.S. and Iceland. EFA supported the construct validity of the APS-POQ-R as a measure of "pain experience", but indicated that items measuring "pain management" may vary cross-culturally. Findings highlighted the need for further validation of the APS-POQ-R internationally. PERSPECTIVE: This study revealed the APS-POQ-R as a valid measure of postoperative pain experience for Danish and Australian patients. Measures of patients' perception of pain management were not robust to group differences in treatment expectations and demonstrated cross-cultural instability. Results highlighted difficulties in establishing stable cross-cultural, cross-population subscales for the APS-POQ-R.

Understanding feedback report uptake: process evaluation findings from a 13-month feedback intervention in long-term care settings

BACKGROUND: Long-term care settings provide care to a large proportion of predominantly older, highly disabled adults across the United States and Canada. Managing and improving quality of care is challenging, in part because staffing is highly dependent on relatively non-professional health care aides and resources are limited. Feedback interventions in these settings are relatively rare, and there has been little published information about the process of feedback intervention. Our objectives were to describe the key components of uptake of the feedback reports, as well as other indicators of participant response to the intervention. METHODS: We conducted this project in nine long-term care units in four facilities in Edmonton, Canada. We used mixed methods, including observations during a 13-month feedback report intervention with nine post-feedback survey cycles, to conduct a process evaluation of a feedback report intervention in these units. We included all facility-based direct care providers (staff) in the feedback report distribution and survey administration. We conducted descriptive analyses of the data from observations and surveys, presenting this in tabular and graphic form. We constructed a short scale to measure uptake of the feedback reports. Our analysis evaluated feedback report uptake by provider type over the 13 months of the intervention. RESULTS: We received a total of 1,080 survey responses over the period of the intervention, which varied by type of provider, facility, and survey month. Total number of reports distributed ranged from 103 in cycle 12 to 229 in cycle 3, although the method of delivery varied widely across the period, from 12% to 65% delivered directly to individuals and 15% to 84% left for later distribution. The key elements of feedback uptake, including receiving, reading, understanding, discussing, and reporting a perception that the reports were useful, varied by survey cycle and provider type, as well as by facility. Uptake, as we measured it, was consistently high overall, but varied widely by provider type and time period. CONCLUSIONS: We report detailed process data describing the aspects of uptake of a feedback report during an intensive, longitudinal feedback intervention in long-term care facilities. Uptake is a complex process for which we used multiple measures. We demonstrate the feasibility of conducting a complex longitudinal feedback intervention in relatively resource-poor long-term care facilities to a wider range of provider types than have been included in prior feedback interventions.

A systematic review of reliable and valid tools for the measurement of patient participation in healthcare

INTRODUCTION: Patient participation in healthcare is recognised internationally as essential for consumer-centric, high-quality healthcare delivery. Its measurement as part of continuous quality improvement requires development of agreed standards and measurable indicators. AIM: This systematic review sought to identify strategies to measure patient participation in healthcare and to report their reliability and validity. In the context of this review, patient participation was constructed as shared decision-making, acknowledging the patient as having critical knowledge regarding their own health and care needs and promoting self-care/autonomy. METHODS: Following a comprehensive search, studies reporting reliability or validity of an instrument used in a healthcare setting to measure patient participation, published in English between January 2004 and March 2014 were eligible for inclusion. RESULTS: From an initial search, which identified 1582 studies, 156 studies were retrieved and screened against inclusion criteria. Thirty-three studies reporting 24 patient participation measurement tools met inclusion criteria, and were critically appraised. The majority of studies were descriptive psychometric studies using prospective, cross-sectional designs. Almost all the tools completed by patients, family caregivers, observers or more than one stakeholder focused on aspects of patient-professional communication. Few tools designed for completion by patients or family caregivers provided valid and reliable measures of patient participation. There was low correlation between many of the tools and other measures of patient satisfaction. CONCLUSION: Few reliable and valid tools for measurement of patient participation in healthcare have been recently developed. Of those reported in this review, the dyadic Observing Patient Involvement in Decision Making (dyadic-OPTION) tool presents the most promise for measuring core components of patient participation. There remains a need for further study into valid, reliable and feasible strategies for measuring patient participation as part of continuous quality improvement.

Health services should collect feedback from inpatients at the point of service: opinions from patients and staff in acute and subacute facilities

OBJECTIVE: Point of service feedback (POSF) enables patients to give health services feedback about their experiences during or immediately after care. Despite the increasing use of POSF, little is known regarding patients' and staffs' opinions of this practice and whether they consider it acceptable or useful. The study aimed to determine patient and staff opinions regarding POSF. DESIGN: A cross-sectional survey. SETTING: Acute and subacute healthcare facilities. PARTICIPANTS: Two hundred and forty-seven patients and 221 staff. RESULTS: Participants indicated that patients should be invited to evaluate health services when they are in hospital or subacute care and improving services was the most important reason for doing so. Staff indicated that:• collecting patients' feedback during their stay was an important part of providing care and not an interruption to it (n = 187 of 221, 85%).• collecting patients' feedback was best done with a variety of methods; talking directly with patients during their stay was the preferred option (n = 161 of 219, 74%).More patients preferred to:• give feedback during their stay (51%) than after discharge from care (15%).• give feedback by talking with someone (45%) than completing a questionnaire (31%).Some patients (14%) were concerned about reprisal from staff if they gave negative feedback. CONCLUSIONS: POSF can be acceptable and useful for evaluating health services and should be incorporated into a person-centred approach that allows patients to choose from a variety of feedback options both during and after their stay. To be most useful, feedback should be incorporated into a quality improvement system.

Responding to student evaluation of teaching : closing the loop

Our undergraduate curriculum focuses on preparing students to be critical thinkers, problem solvers, and knowledgeable, responsible information technology (IT) professionals. Students often question the relevance of core subjects which are designed to introduce and develop these attributes; such units regularly perform poorly in student evaluation of teaching. Although these subjects are strongly grounded in meeting industry expectations regarding employability skills, students do not value the learning as the curriculum is deemed non-technical and not worthy of inclusion in an IT qualification. As these are core subjects, improving student perceptions and learning outcomes is of critical importance. Simply responding to student feedback received via student evaluation of teaching had not resulted in any improvement; so a holistic approach was adopted. A workshop was organised to explore the issues and develop a strategy to improve outcomes and perceptions of these subjects. Rather than addressing student feedback specifically, the underlying issues that led to students’ negative perceptions were identified and addressed. Recommendations were implemented in 2014 resulting in improvements in student evaluation for relevant subjects. In this paper, we will discuss the process that was adopted to respond effectively to student evaluations, and explore the impact that this had on them.

An evaluation of an action research nursing documentation project

AIMS AND OBJECTIVES: The aim of this study is to describe healthcare professionals' experiences and perceptions of an intervention implemented in an action research project conducted to improve nursing documentation practices in four municipalities in Norway.

BACKGROUND: Documentation of individualized patient care is a continuing concern in healthcare services and could impacts the quality and safety of healthcare. Use of electronic systems has made some aspects of documentation more comprehensive, but creation of an individualized care plan remains a pressing issue.

DESIGN: A qualitative descriptive design was used.

METHODS: An action research project was conducted between 2010 and 2012 to improve the content and quality of nursing documentation in community healthcare services in four municipalities. One year after the project was completed four focus group interviews were conducted with healthcare professionals, one for each involved municipality. Two unit managers were interviewed individually. Qualitative content analysis was used.

RESULTS: Three themes emerged: healthcare professionals perceived competing interest; they experienced that they had to manage complexity and changes; and they highlighted a clear and visible leader as important for success.

CONCLUSIONS: Quality improvement activities are essential. Healthcare professionals experience a complicated situation when electronic health record systems do not support workflow. Further research is recommended to focus on the functionality and user interface of EHR systems, and on the role of leadership when implementing changes in clinical practice. This article is protected by copyright. All rights reserved.