Quality assurance of simulated patient feedback in communication training for fourth-year medical students

Introduction In our program, simulated patients (SPs) give feedback to medical students in the course of communication skills training. To ensure effective training, quality control of the SPs’ feedback should be implemented. At other institutions, medical students evaluate the SPs’ feedback for quality control (Bouter et al., 2012). Thinking about implementing quality control for SPs’ feedback in our program, we wondered whether the evaluation by students would result in the same scores as evaluation by experts. Methods Consultations simulated by 4th-year medical students with SPs were video taped including the SP’s feedback to the students (n=85). At the end of the training sessions students rated the SPs’ performance using a rating instrument called Bernese Assessment for Role-play and Feedback (BARF) containing 11 items concerning feedback quality. Additionally the videos were evaluated by 3 trained experts using the BARF. Results The experts showed a high interrater agreement when rating identical feedbacks (ICCunjust=0.953). Comparing the rating of students and experts, high agreement was found with regard to the following items: 1. The SP invited the student to reflect on the consultation first, Amin (= minimal agreement) 97% 2. The SP asked the student what he/she liked about the consultation, Amin = 88%. 3. The SP started with positive feedback, Amin = 91%. 4. The SP was comparing the student with other students, Amin = 92%. In contrast the following items showed differences between the rating of experts and students: 1. The SP used precise situations for feedback, Amax (=maximal agreement) 55%, Students rated 67 of SPs’ feedbacks to be perfect with regard to this item (highest rating on a 5 point Likert scale), while only 29 feedbacks were rated this way by the experts. 2. The SP gave precise suggestions for improvement, Amax 75%, 62 of SPs’ feedbacks obtained the highest rating from students, while only 44 of SPs’ feedbacks achieved the highest rating in the view of the experts. 3. The SP speaks about his/her role in the third person, Amax 60%. Students rated 77 feedbacks with the highest score, while experts judged only 43 feedbacks this way. Conclusion Although evaluation by the students was in agreement with that of experts concerning some items, students rated the SPs’ feedback more often with the optimal score than experts did. Moreover it seems difficult for students to notice when SPs talk about the role in the first instead of the third person. Since precision and talking about the role in the third person are important quality criteria of feedback, this result should be taken into account when thinking about students’ evaluation of SPs’ feedback for quality control. Bouter, S., E. van Weel-Baumgarten, and S. Bolhuis. 2012. Construction and Validation of the Nijmegen Evaluation of the Simulated Patient (NESP): Assessing Simulated Patients’ Ability to Role-Play and Provide Feedback to Students. Academic Medicine: Journal of the Association of American Medical Colleges

Cost-effectiveness of peer role play and standardized patients in undergraduate communication training.

BACKGROUND The few studies directly comparing the methodological approach of peer role play (RP) and standardized patients (SP) for the delivery of communication skills all suggest that both methods are effective. In this study we calculated the costs of both methods (given comparable outcomes) and are the first to generate a differential cost-effectiveness analysis of both methods. METHODS Medical students in their prefinal year were randomly assigned to one of two groups receiving communication training in Pediatrics either with RP (N = 34) or 19 individually trained SP (N = 35). In an OSCE with standardized patients using the Calgary-Cambridge Referenced Observation Guide both groups achieved comparable high scores (results published). In this study, corresponding costs were assessed as man-hours resulting from hours of work of SP and tutors. A cost-effectiveness analysis was performed. RESULTS Cost-effectiveness analysis revealed a major advantage for RP as compared to SP (112 vs. 172 man hours; cost effectiveness ratio .74 vs. .45) at comparable performance levels after training with both methods. CONCLUSIONS While both peer role play and training with standardized patients have their value in medical curricula, RP has a major advantage in terms of cost-effectiveness. This could be taken into account in future decisions.

Individual Video Training iVT and Annotating Academic Videos AAV: two complementing technologies

Individual Video Training iVT and Annotating Academic Videos AAV: two complementing technologies 1. Recording communication skills training sessions and reviewing them by oneself, with peers, and with tutors has become standard in medical education. Increasing numbers of students paired with restrictions of financial and human resources create a big obstacle to this important teaching method. 2. Everybody who wants to increase efficiency and effectiveness of communication training can get new ideas from our technical solution. 3. Our goal was to increase the effectiveness of communication skills training by supporting self, peer and tutor assessment over the Internet. Two technologies of SWITCH, the national foundation to support IT solutions for Swiss universities, came handy for our project. The first is the authentication and authorization infrastructure providing all Swiss students with a nationwide single login. The second is SWITCHcast which allows automated recording, upload and publication of videos in the Internet. Students start the recording system by entering their single login. This automatically links the video with their password. Within a few hours, they find their video password protected on the Internet. They now can give access to peers and tutors. Additionally, an annotation interface was developed. This software has free text as well as checklist annotations capabilities. Tutors as well as students can create checklists. Tutor’s checklists are not editable by students. Annotations are linked to tracks. Tracks can be private or public. Public means visible to all who have access to the video. Annotation data can be exported for statistical evaluation. 4. The system was well received by students and tutors. Big numbers of videos were processed simultaneously without any problems. 5. iVT http://www.switch.ch/aaa/projects/detail/UNIBE.7 AAV http://www.switch.ch/aaa/projects/detail/ETHZ.9

Validity evidence and reliability of a Simulated Patient Feedback Instrument

In the training of healthcare professionals, one of the advantages of communication training with simulated patients (SPs) is the SP's ability to provide direct feedback to students after a simulated clinical encounter. The quality of SP feedback must be monitored, especially because it is well known that feedback can have a profound effect on student performance. Due to the current lack of valid and reliable instruments to assess the quality of SP feedback, our study examined the validity and reliability of one potential instrument, the 'modified Quality of Simulated Patient Feedback Form' (mQSF). Methods Content validity of the mQSF was assessed by inviting experts in the area of simulated clinical encounters to rate the importance of the mQSF items. Moreover, generalizability theory was used to examine the reliability of the mQSF. Our data came from videotapes of clinical encounters between six simulated patients and six students and the ensuing feedback from the SPs to the students. Ten faculty members judged the SP feedback according to the items on the mQSF. Three weeks later, this procedure was repeated with the same faculty members and recordings. Results All but two items of the mQSF received importance ratings of > 2.5 on a four-point rating scale. A generalizability coefficient of 0.77 was established with two judges observing one encounter. Conclusions The findings for content validity and reliability with two judges suggest that the mQSF is a valid and reliable instrument to assess the quality of feedback provided by simulated patients.

Communication skills training in oncology: a position paper based on a consensus meeting among European experts in 2009

Background: Communication in cancer care has become a major topic of interest. Since there is evidence that ineffective communication affects both patients and oncology clinicians (physicians and nurses), so-called communication skills trainings (CSTs) have been developed over the last decade. While these trainings have been demonstrated to be effective, there is an important heterogeneity with regard to implementation and with regard to evidence of different aspects of CST. Methods: In order to review and discuss the scientific literature on CST in oncology and to formulate recommendations, the Swiss Cancer League has organised a consensus meeting with European opinion leaders and experts in the field of CST, as well as oncology clinicians, representatives of oncology societies and patient organisations. On the basis of a systematic review and a meta-analysis, recommendations have been developed and agreed upon. Results: Recommendations address (i) the setting, objectives and participants of CST, (ii) its content and pedagogic tools, (iii) organisational aspects, (iv) outcome and (v) future directions and research. Conclusion: This consensus meeting, on the basis of European expert opinions and a systematic review and meta-analysis, defines key elements for the current provision and future development and evaluation of CST in oncology.

Benefits of communication skills training after real patient exposure

The established communication skills training (CST) curriculum consists of continuous education, frequent practice with real patients, supervision and feedback from medical teachers. The limitation of this curriculum is that students have to directly apply the theoretical basis they are provided with to real patients. To improve the existing CST curriculum a CST module involving simulated patients was introduced in order to develop more complex communication skills.

Efficacy of communication skills training courses in oncology: a systematic review and meta-analysis

Objective: Group training in communication skills [communication skills training (CST)] has become partly mandatory for oncology staff. However, so far, a comprehensive meta-analysis on the efficacy is lacking. Design: Included studies either compare the efficacy of a specific training with a control group or look at the additional effect of booster sessions on communication behaviour, attitudes or patient outcomes. Methods: Four electronic databases were searched up to July 2008 without language restriction, and reference lists of earlier reviews were screened. Effect sizes (ESs) were extracted and pooled in random effects meta-analyses. Results: We included 13 trials (three non-randomised), 10 with no specific intervention in the control group. Meta-analysis showed a moderate effect of CST on communication behaviour ES = 0.54. Three trials compared basic training courses with more extensive training courses and showed a small additional effect on communication skills ES = 0.37. Trials investigating participants' attitudes ES = 0.35 and patient outcomes ES = 0.13 (trend) confirmed this effect. Conclusions: Training health professionals by CST is a promising approach to change communication behaviour and attitudes. Patients might also benefit from specifically trained health professionals but strong studies are lacking. However, feasibility and economic aspects have to be kept in mind when considering providing a training of optimal length.