FAIR_OSCE – Feedback structure for assessment of interactive roleplay in Objective Structured Clinical Exams

Introduction Since the quality of patient portrayal of standardized patients (SPs) during an Objective Structured Clinical Exam (OSCE) has a major impact on the reliability and validity of the exam, quality control should be initiated. Literature about quality control of SPs’ performance focuses on feedback [1, 2] or completion of checklists [3, 4]. Since we did not find a published instrument meeting our needs for the assessment of patient portrayal, we developed such an instrument after being inspired by others [5] and used it in our high-stakes exam. Project description SP trainers from five medical faculties collected and prioritized quality criteria for patient portrayal. Items were revised twice, based on experiences during OSCEs. The final instrument contains 14 criteria for acting (i.e. adequate verbal and non-verbal expression) and standardization (i.e. verbatim delivery of the first sentence). All partners used the instrument during a high-stakes OSCE. SPs and trainers were introduced to the instrument. The tool was used in training (more than 100 observations) and during the exam (more than 250 observations). Outcome High quality of SPs’ patient portrayal during the exam was documented. More than 90% of SP performances were rated to be completely correct or sufficient. An increase in quality of performance between training and exam was noted. For example, the rate of completely correct reaction in medical tests increased from 88% to 95%. Together with 4% of sufficient performances these 95% add up to 99% of the reactions in medical tests meeting the standards of the exam. SP educators using the instrument reported an augmentation of SPs’ performance induced by the use of the instrument. Disadvantages mentioned were the high concentration needed to observe all criteria and the cumbersome handling of the paper-based forms. Discussion We were able to document a very high quality of SP performance in our exam. The data also indicates that our training is effective. We believe that the high concentration needed using the instrument is well invested, considering the observed enhancement of performance. The development of an iPad-based application for the form is planned to address the cumbersome handling of the paper.

QUADAS-2: a revised tool for the quality assessment of diagnostic accuracy studies

In 2003, the QUADAS tool for systematic reviews of diagnostic accuracy studies was developed. Experience, anecdotal reports, and feedback suggested areas for improvement; therefore, QUADAS-2 was developed. This tool comprises 4 domains: patient selection, index test, reference standard, and flow and timing. Each domain is assessed in terms of risk of bias, and the first 3 domains are also assessed in terms of concerns regarding applicability. Signalling questions are included to help judge risk of bias. The QUADAS-2 tool is applied in 4 phases: summarize the review question, tailor the tool and produce review-specific guidance, construct a flow diagram for the primary study, and judge bias and applicability. This tool will allow for more transparent rating of bias and applicability of primary diagnostic accuracy studies.

Self-assessment insufficiently predicts performance in emergency skills

Background: Residents demonstrate a broad range of performance levels for clinical skills, with some at an inadequate level. Adequate self-assessment is important for life long learning. However, its accuracy is questioned extensively. The aim of this study was to evaluate how far the residents’ self-assessment predicts their performance in an expert assessment of emergency skills. Summary of work: Twelve skills were identified as being relevant for the emergency duties of residents in smaller hospitals. Fifteen first-year residents from the departments of internal medicine and general surgery at a district hospital rated their performance on a questionnaire (self-assessment). This was followed by a structured, practical in vivo assessment by an anaesthesiologist (expert assessment). For both, a visual analogue scale from 0 to 10 was used, on which 0 stands for novice and 10 for expert. Predictive validity was described by Spearman’s correlation, which was significant in 3 out of 12 skills only. Median correlation (r) was 0.50 (range 0.16 to 0.93). Conclusion: At the beginning of postgraduate training, self-assessment alone is not sufficient to guide self-directed learning. Take-home message: At the beginning of their residency, physicians need structured feedback in emergency skills which can be offered by anaesthesiologists.

Endocardial versus epicardial ventricular radiofrequency ablation: utility of in vivo contact force assessment.

BACKGROUND Contact force (CF) is an important determinant of lesion formation for atrial endocardial radiofrequency ablation. There are minimal published data on CF and ventricular lesion formation. We studied the impact of CF on lesion formation using an ovine model both endocardially and epicardially. METHODS AND RESULTS Twenty sheep received 160 epicardial and 160 endocardial ventricular radiofrequency applications using either a 3.5-mm irrigated-tip catheter (Thermocool, Biosense-Webster, n=160) or a 3.5 irrigated-tip catheter with CF assessment (Tacticath, Endosense, n=160), via percutaneous access. Power was delivered at 30 watts for 60 seconds, when either catheter/tissue contact was felt to be good or when CF>10 g with Tacticath. After completion of all lesions, acute dimensions were taken at pathology. Identifiable lesion formation from radiofrequency application was improved with the aid of CF information, from 78% to 98% on the endocardium (P<0.001) and from 90% to 100% on the epicardium (P=0.02). The mean total force was greater on the endocardium (39±18 g versus 21±14 g for the epicardium; P<0.001) mainly because of axial force. Despite the force-time integral being greater endocardially, epicardial lesions were larger (231±182 mm(3) versus 209±131 mm(3); P=0.02) probably because of the absence of the heat sink effect of the circulating blood and covered a greater area (41±27 mm(2) versus 29±17 mm(2); P=0.03) because of catheter orientation. CONCLUSIONS In the absence of CF feedback, 22% of endocardial radiofrequency applications that are thought to have good contact did not result in lesion formation. Epicardial ablation is associated with larger lesions.

A virtual environment for the assessment and the rehabilitation of the visuo-constructional ability in dementia patients

This study proposes a VE that offers a reliable diagnosis of the stage of cognitive decline in dementia patients and assists the delay of this decline in terms of the visuo-constructional ability. The proposed VE, in the case of the assessment, presents a visuo-constructional completion task, which requires spatial perception, motor memory and the perception of the target object. In the case of the rehabilitation the VE uses sound as audio-feedback that, with the aid of the music perception, tends to develop an enhancement in the visuo-construction ability of the dementia patients that can be generalized even outside of the VE. The study examined 30 subjects that were normal controls (N), 30 patients suffering from memory disorders (Age-Associated Memory Impairment--AAMI) and 30 suffering from Alzheimer's Disease (AD). The results showed that there is a significant correlation between the performance in the visuo-constructional task and the dementia diagnosis. It also seems that the visuo-constructional ability of the (AD) patients can be statistically improved by the audio experience in the VE. The empirical results of this study offer an alternative diagnosis and treatment of dementia patients and could share some light in the brain sub-systems that are responsible for the visuo-constructional ability. Further studies are required in order to investigate the nature of this phenomenon more.

Feedback von Schülerinnen und Schülern als Quelle des Lernprozesses im Praktikum von Lehramtsstudierenden

Rückmeldungen von Schülerinnen und Schülern zum Unterricht nehmen eine wichtige Rolle zur Verbesserung der Unterrichtsqualität ein. Während der Ausbildung von Lehrpersonen wird dieses Potenzial bisher aber selten genutzt. Der Beitrag stellt Ergebnisse aus dem Forschungsprojekt "Lernen im Praktikum" vor. Befragt wurden 150 Studierende des Lehramts für 7.-9. Klassen (Sekundarstufe 1) zur Bedeutung von Schülerrückmeldungen im Praktikum sowie 1331 Schülerinnen und Schüler aus Praktikumsklassen zum Praktikum und zum Lernprozess der Praktikantinnen und Praktikanten. Die Ergebnisse unterstützen einerseits die Relevanz systematischer Schülerrückmeldungen für das Lernen im Praktikum. Andererseits machen sie auf die Notwendigkeit aufmerksam, die Kriterien, die Schülerbeurteilungen zugrunde liegen, weiter zu erforschen.

Cardiopulmonary performance testing using a robotics-assisted tilt table: Feasibility assessment in able-bodied subjects

BACKGROUND: Robotics-assisted tilt table technology was introduced for early rehabilitation of neurological patients. It provides cyclical stepping movement and physiological loading of the legs. The aim of the present study was to assess the feasibility of this type of device for peak cardiopulmonary performance testing using able-bodied subjects. METHODS: A robotics-assisted tilt table was augmented with force sensors in the thigh cuffs and a work rate estimation algorithm. A custom visual feedback system was employed to guide the subjects' work rate and to provide real time feedback of actual work rate. Feasibility assessment focused on: (i) implementation (technical feasibility), and (ii) responsiveness (was there a measurable, high-level cardiopulmonary reaction?). For responsiveness testing, each subject carried out an incremental exercise test to the limit of functional capacity with a work rate increment of 5 W/min in female subjects and 8 W/min in males. RESULTS: 11 able-bodied subjects were included (9 male, 2 female; age 29.6 ± 7.1 years: mean ± SD). Resting oxygen uptake (O_{2}) was 4.6 ± 0.7 mL/min/kg and O_{2}peak was 32.4 ± 5.1 mL/min/kg; this mean O_{2}peak was 81.1% of the predicted peak value for cycle ergometry. Peak heart rate (HRpeak) was 177.5 ± 9.7 beats/min; all subjects reached at least 85% of their predicted HRpeak value. Respiratory exchange ratio (RER) at O_{2}peak was 1.02 ± 0.07. Peak work rate) was 61.3 ± 15.1 W. All subjects reported a Borg CR10 value for exertion and leg fatigue of 7 or more. CONCLUSIONS: The robotics-assisted tilt table is deemed feasible for peak cardiopulmonary performance testing: the approach was found to be technically implementable and substantial cardiopulmonary responses were observed. Further testing in neurologically-impaired subjects is warranted.

Formatives Selbst- und Peer-Assessment im Unterricht peripher venöser Venenpunktion

Das Lernen einer Fertigkeit durch Demonstration und anschließendes Üben wird „Modeling“ genannt. Es basiert darauf, die Diskrepanz zwischen dem Soll-Zustand (Demonstration) und dem Ist-Zustand (Üben) zu erkennen und zu beheben. Dafür ist die exakte Analyse der eigenen Fertigkeiten beim Üben unentbehrlich. Entsprechend ist auch bekannt, dass formative Evaluationen wesentlich zum erfolgreichen Lernen beitragen. Wir haben deshalb im Kurs für periphere Venenpunktion im 3. Studienjahr formatives Selbst- und Peer-Assessment eingeführt. Die Struktur des Assessment entspricht einem DOPS (dircect observation of procedural skills). DOPS stammt aus dem Arbeitsplatz-basieren Assessment und beinhaltet die Beurteilung folgender Kriterien: Vorbereitung/Nachsorge, technische Fertigkeit, Asepsis/Sicherheit, klinische Urteilsfähigkeit, Organisation/Effizienz, professionelles Verhalten, Gesamteindruck. Diese Kriterien wurden für den Unterricht konkretisiert (z.B. Vorbereitung mit Beschriftung der Röhrchen, etc.) und den Studierenden als Merkblätter ausgeteilt. Die Studierenden beurteilten ihre eigene Performance bzw. die eines Kommilitonen, gaben sich Feedback und legten individuelle Lernziele zur Verbesserung fest. Dieses Vorgehen hat den Vorteil, dass sowohl der Übende, als auch der beobachtende Kommilitone, die optimale Ausführung der jeweiligen Tätigkeit reflektieren, welches für beide eine Möglichkeit zum Lernen bietet . Bei der Evaluation des Kurses wurden die Handouts mit den Kriterien der DOPS von Teilnehmern von 9 der 10 Gruppen positiv erwähnt. Im Rahmen eines Debriefing mit den studentischen Tutoren wurde jedoch kritisch angemerkt, dass der Prozess der formativen Selbst- und Fremdevaluation den Studierenden im 3. Studienjahr nicht vertraut war. Es war für die Teilnehmer schwierig konkretes Feedback zu geben und individuelle Lernziele festzulegen. Für das kommende Jahr planen wir in Bezug auf den Kurs folgendes: Die Kriterien der korrekten Durchführung einer Fertigkeit zu formulieren wird von den Teilnehmern als hilfreich empfunden und soll deshalb beibehalten werden. Die Studierenden, die dieses Jahr an dem Kurs teilnehmen, haben bereits ein Feedbacktraining absolviert. Der Kurs kann deshalb neu an Vorkenntnisse anknüpfen. Darüber hinaus soll der Prozess der Festlegung der individuellen Lernziele in der Schulung der studentischen Tutoren des Kurses mehr Gewicht erhalten, damit die Tutoren die Teilnehmer hier gezielt unterstützen können.

Quality assurance of simulated patient feedback in communication training for fourth-year medical students

Introduction In our program, simulated patients (SPs) give feedback to medical students in the course of communication skills training. To ensure effective training, quality control of the SPs’ feedback should be implemented. At other institutions, medical students evaluate the SPs’ feedback for quality control (Bouter et al., 2012). Thinking about implementing quality control for SPs’ feedback in our program, we wondered whether the evaluation by students would result in the same scores as evaluation by experts. Methods Consultations simulated by 4th-year medical students with SPs were video taped including the SP’s feedback to the students (n=85). At the end of the training sessions students rated the SPs’ performance using a rating instrument called Bernese Assessment for Role-play and Feedback (BARF) containing 11 items concerning feedback quality. Additionally the videos were evaluated by 3 trained experts using the BARF. Results The experts showed a high interrater agreement when rating identical feedbacks (ICCunjust=0.953). Comparing the rating of students and experts, high agreement was found with regard to the following items: 1. The SP invited the student to reflect on the consultation first, Amin (= minimal agreement) 97% 2. The SP asked the student what he/she liked about the consultation, Amin = 88%. 3. The SP started with positive feedback, Amin = 91%. 4. The SP was comparing the student with other students, Amin = 92%. In contrast the following items showed differences between the rating of experts and students: 1. The SP used precise situations for feedback, Amax (=maximal agreement) 55%, Students rated 67 of SPs’ feedbacks to be perfect with regard to this item (highest rating on a 5 point Likert scale), while only 29 feedbacks were rated this way by the experts. 2. The SP gave precise suggestions for improvement, Amax 75%, 62 of SPs’ feedbacks obtained the highest rating from students, while only 44 of SPs’ feedbacks achieved the highest rating in the view of the experts. 3. The SP speaks about his/her role in the third person, Amax 60%. Students rated 77 feedbacks with the highest score, while experts judged only 43 feedbacks this way. Conclusion Although evaluation by the students was in agreement with that of experts concerning some items, students rated the SPs’ feedback more often with the optimal score than experts did. Moreover it seems difficult for students to notice when SPs talk about the role in the first instead of the third person. Since precision and talking about the role in the third person are important quality criteria of feedback, this result should be taken into account when thinking about students’ evaluation of SPs’ feedback for quality control. Bouter, S., E. van Weel-Baumgarten, and S. Bolhuis. 2012. Construction and Validation of the Nijmegen Evaluation of the Simulated Patient (NESP): Assessing Simulated Patients’ Ability to Role-Play and Provide Feedback to Students. Academic Medicine: Journal of the Association of American Medical Colleges

«Toolbox Assessment» - ein hochschuldidaktischer Service im virtuellen Raum

Zeitgemäße Hochschullehre verlangt nach zeitgemäßen Prüfungsformen. Mit «Assessment», wie es hier verstanden wird, ist jedoch nicht nur die abschließende Lernerfolgskontrolle gemeint. Im Ideal der kompetenzorientierten Hochschullehre erhalten die Studierenden auch auf dem Weg zur angestrebten Handlungskompetenz immer wieder Gelegenheit zur Standortbestimmung. Eine Benotung ist dabei nicht zwingend, informierendes Feedback jedoch schon. Mit der «Toolbox Assessment» erhalten Lehrende dazu nun ein attraktives Hilfsmittel buchstäblich in die Hand.

Effect of Health Risk Assessment and Counselling on Health Behaviour and Survival in Older People: A Pragmatic Randomised Trial.

BACKGROUND Potentially avoidable risk factors continue to cause unnecessary disability and premature death in older people. Health risk assessment (HRA), a method successfully used in working-age populations, is a promising method for cost-effective health promotion and preventive care in older individuals, but the long-term effects of this approach are unknown. The objective of this study was to evaluate the effects of an innovative approach to HRA and counselling in older individuals for health behaviours, preventive care, and long-term survival. METHODS AND FINDINGS This study was a pragmatic, single-centre randomised controlled clinical trial in community-dwelling individuals aged 65 y or older registered with one of 19 primary care physician (PCP) practices in a mixed rural and urban area in Switzerland. From November 2000 to January 2002, 874 participants were randomly allocated to the intervention and 1,410 to usual care. The intervention consisted of HRA based on self-administered questionnaires and individualised computer-generated feedback reports, combined with nurse and PCP counselling over a 2-y period. Primary outcomes were health behaviours and preventive care use at 2 y and all-cause mortality at 8 y. At baseline, participants in the intervention group had a mean ± standard deviation of 6.9 ± 3.7 risk factors (including unfavourable health behaviours, health and functional impairments, and social risk factors) and 4.3 ± 1.8 deficits in recommended preventive care. At 2 y, favourable health behaviours and use of preventive care were more frequent in the intervention than in the control group (based on z-statistics from generalised estimating equation models). For example, 70% compared to 62% were physically active (odds ratio 1.43, 95% CI 1.16-1.77, p = 0.001), and 66% compared to 59% had influenza vaccinations in the past year (odds ratio 1.35, 95% CI 1.09-1.66, p = 0.005). At 8 y, based on an intention-to-treat analysis, the estimated proportion alive was 77.9% in the intervention and 72.8% in the control group, for an absolute mortality difference of 4.9% (95% CI 1.3%-8.5%, p = 0.009; based on z-test for risk difference). The hazard ratio of death comparing intervention with control was 0.79 (95% CI 0.66-0.94, p = 0.009; based on Wald test from Cox regression model), and the number needed to receive the intervention to prevent one death was 21 (95% CI 12-79). The main limitations of the study include the single-site study design, the use of a brief self-administered questionnaire for 2-y outcome data collection, the unavailability of other long-term outcome data (e.g., functional status, nursing home admissions), and the availability of long-term follow-up data on mortality for analysis only in 2014. CONCLUSIONS This is the first trial to our knowledge demonstrating that a collaborative care model of HRA in community-dwelling older people not only results in better health behaviours and increased use of recommended preventive care interventions, but also improves survival. The intervention tested in our study may serve as a model of how to implement a relatively low-cost but effective programme of disease prevention and health promotion in older individuals. TRIAL REGISTRATION International Standard Randomized Controlled Trial Number: ISRCTN 28458424.

Sustained impact of a short small group course with systematic feedback in addition to regular clinical clerkship activities on musculoskeletal examination skills-a controlled study.

BACKGROUND The discrepancy between the extensive impact of musculoskeletal complaints and the common deficiencies in musculoskeletal examination skills lead to increased emphasis on structured teaching and assessment. However, studies of single interventions are scarce and little is known about the time-dependent effect of assisted learning in addition to a standard curriculum. We therefore evaluated the immediate and long-term impact of a small group course on musculoskeletal examination skills. METHODS All 48 Year 4 medical students of a 6 year curriculum, attending their 8 week clerkship of internal medicine at one University department in Berne, participated in this controlled study. Twenty-seven students were assigned to the intervention of a 6×1 h practical course (4-7 students, interactive hands-on examination of real patients; systematic, detailed feedback to each student by teacher, peers and patients). Twenty-one students took part in the regular clerkship activities only and served as controls. In all students clinical skills (CS, 9 items) were assessed in an Objective Structured Clinical Examination (OSCE) station, including specific musculoskeletal examination skills (MSES, 7 items) and interpersonal skills (IPS, 2 items). Two raters assessed the skills on a 4-point Likert scale at the beginning (T0), the end (T1) and 4-12 months after (T2) the clerkship. Statistical analyses included Friedman test, Wilcoxon rank sum test and Mann-Whitney U test. RESULTS At T0 there were no significant differences between the intervention and control group. At T1 and T2 the control group showed no significant changes of CS, MSES and IPS compared to T0. In contrast, the intervention group significantly improved CS, MSES and IPS at T1 (p < 0.001). This enhancement was sustained for CS and MSES (p < 0.05), but not for IPS at T2. CONCLUSIONS Year 4 medical students were incapable of improving their musculoskeletal examination skills during regular clinical clerkship activities. However, an additional small group, interactive clinical skills course with feedback from various sources, improved these essential examination skills immediately after the teaching and several months later. We conclude that supplementary specific teaching activities are needed. Even a single, short-lasting targeted module can have a long lasting effect and is worth the additional effort.

Usability and Acceptability of ASSESS MS: Assessment of Motor Dysfunction in Multiple Sclerosis Using Depth-Sensing Computer Vision

Background: Sensor-based recordings of human movements are becoming increasingly important for the assessment of motor symptoms in neurological disorders beyond rehabilitative purposes. ASSESS MS is a movement recording and analysis system being developed to automate the classification of motor dysfunction in patients with multiple sclerosis (MS) using depth-sensing computer vision. It aims to provide a more consistent and finer-grained measurement of motor dysfunction than currently possible. Objective: To test the usability and acceptability of ASSESS MS with health professionals and patients with MS. Methods: A prospective, mixed-methods study was carried out at 3 centers. After a 1-hour training session, a convenience sample of 12 health professionals (6 neurologists and 6 nurses) used ASSESS MS to capture recordings of standardized movements performed by 51 volunteer patients. Metrics for effectiveness, efficiency, and acceptability were defined and used to analyze data captured by ASSESS MS, video recordings of each examination, feedback questionnaires, and follow-up interviews. Results: All health professionals were able to complete recordings using ASSESS MS, achieving high levels of standardization on 3 of 4 metrics (movement performance, lateral positioning, and clear camera view but not distance positioning). Results were unaffected by patients’ level of physical or cognitive disability. ASSESS MS was perceived as easy to use by both patients and health professionals with high scores on the Likert-scale questions and positive interview commentary. ASSESS MS was highly acceptable to patients on all dimensions considered, including attitudes to future use, interaction (with health professionals), and overall perceptions of ASSESS MS. Health professionals also accepted ASSESS MS, but with greater ambivalence arising from the need to alter patient interaction styles. There was little variation in results across participating centers, and no differences between neurologists and nurses. Conclusions: In typical clinical settings, ASSESS MS is usable and acceptable to both patients and health professionals, generating data of a quality suitable for clinical analysis. An iterative design process appears to have been successful in accounting for factors that permit ASSESS MS to be used by a range of health professionals in new settings with minimal training. The study shows the potential of shifting ubiquitous sensing technologies from research into the clinic through a design approach that gives appropriate attention to the clinic environment.

Automatic assessment of time-resolved OCT images for selective retina therapy

Purpose In recent years, selective retina laser treatment (SRT), a sub-threshold therapy method, avoids widespread damage to all retinal layers by targeting only a few. While these methods facilitate faster healing, their lack of visual feedback during treatment represents a considerable shortcoming as induced lesions remain invisible with conventional imaging and make clinical use challenging. To overcome this, we present a new strategy to provide location-specific and contact-free automatic feedback of SRT laser applications. Methods We leverage time-resolved optical coherence tomography (OCT) to provide informative feedback to clinicians on outcomes of location-specific treatment. By coupling an OCT system to SRT treatment laser, we visualize structural changes in the retinal layers as they occur via time-resolved depth images. We then propose a novel strategy for automatic assessment of such time-resolved OCT images. To achieve this, we introduce novel image features for this task that when combined with standard machine learning classifiers yield excellent treatment outcome classification capabilities. Results Our approach was evaluated on both ex vivo porcine eyes and human patients in a clinical setting, yielding performances above 95 % accuracy for predicting patient treatment outcomes. In addition, we show that accurate outcomes for human patients can be estimated even when our method is trained using only ex vivo porcine data. Conclusion The proposed technique presents a much needed strategy toward noninvasive, safe, reliable, and repeatable SRT applications. These results are encouraging for the broader use of new treatment options for neovascularization-based retinal pathologies.