Self-assessment insufficiently predicts performance in emergency skills

Background: Residents demonstrate a broad range of performance levels for clinical skills, with some at an inadequate level. Adequate self-assessment is important for life long learning. However, its accuracy is questioned extensively. The aim of this study was to evaluate how far the residents’ self-assessment predicts their performance in an expert assessment of emergency skills. Summary of work: Twelve skills were identified as being relevant for the emergency duties of residents in smaller hospitals. Fifteen first-year residents from the departments of internal medicine and general surgery at a district hospital rated their performance on a questionnaire (self-assessment). This was followed by a structured, practical in vivo assessment by an anaesthesiologist (expert assessment). For both, a visual analogue scale from 0 to 10 was used, on which 0 stands for novice and 10 for expert. Predictive validity was described by Spearman’s correlation, which was significant in 3 out of 12 skills only. Median correlation (r) was 0.50 (range 0.16 to 0.93). Conclusion: At the beginning of postgraduate training, self-assessment alone is not sufficient to guide self-directed learning. Take-home message: At the beginning of their residency, physicians need structured feedback in emergency skills which can be offered by anaesthesiologists.

Exploring the validity and reliability of a questionnaire for evaluating virtual patient design with a special emphasis on fostering clinical reasoning

Background: The design of Virtual Patients (VPs) is essential. So far there are no validated evaluation instruments for VP design published. Summary of work: We examined three sources of validity evidence of an instrument to be filled out by students aimed at measuring the quality of VPs with a special emphasis on fostering clinical reasoning: (1) Content was examined based on theory of clinical reasoning and an international VP expert team. (2) Response process was explored in think aloud pilot studies with students and content analysis of free text questions accompanying each item of the instrument. (3) Internal structure was assessed by confirmatory factor analysis (CFA) using 2547 student evaluations and reliability was examined utilizing generalizability analysis. Summary of results: Content analysis was supported by theory underlying Gruppen and Frohna’s clinical reasoning model on which the instrument is based and an international VP expert team. The pilot study and analysis of free text comments supported the validity of the instrument. The CFA indicated that a three factor model comprising 6 items showed a good fit with the data. Alpha coefficients per factor were 0,74 - 0,82. The findings of the generalizability studies indicated that 40-200 student responses are needed in order to obtain reliable data on one VP. Conclusions: The described instrument has the potential to provide faculty with reliable and valid information about VP design. Take-home messages: We present a short instrument which can be of help in evaluating the design of VPs.

Alignment between learning needs and learning goals of Mini-CEX in clerkships

Background: Defining learning goals (LG) in alignment with learning needs (LN) is one of the key purposes of formative workplace-based assessment, but studies about this topic are scarce. Summary of Work: We analysed quantitatively and qualitatively how often trainer-student pairs identified the same LN during Mini Clinical Evaluation Exercises (Mini-CEX) in clerkships and to what degree those LNs were in line with the recorded LGs. Multilevel logistic regression models were used to predict LGs by identified LNs, controlling for context variables. Summary of Results: 512 trainers and 165 students conducted 1783 Mini-CEX (98% completion rate). Concordantly, trainer-student pairs most often identified LNs in the domains ‘clinical reasoning’ (23% of 1167 complete forms), ‘organisation / efficiency’ (20%) and ‘physical examination’ (20%). At least one ‘defined’ LG was noted on 313 student forms (18% of 1710), with a total of 446 LGs. Of these, the most frequent LGs were ‘physical examination’ (49% of 446 LGs) and ‘history taking’ (21%); corresponding LNs as well as context variables (e.g. clinic size) were found to be predictors of these LGs. Discussion and Conclusions: Although trainer-student pairs often agreed in their identified LNs, many assessments did not result in an aligned LG or a LG at all. Interventions are needed to enhance the proportion of (aligned) LGs in Mini-CEX in order to tap into its full potential not only as a ‘diagnostic’ but also as an ‘educational tool’. Take-home messages: The sparseness of LGs, their dependency on context variables and their partial non-alignment with students’ LNs raise the question of how the effectiveness of Mini-CEX can be further enhanced.

A Specific Mapping Study Using Fluorescence Sentinel Lymph Node Detection in Patients with Intermediate- and High-risk Prostate Cancer Undergoing Extended Pelvic Lymph Node Dissection.

Sentinel lymph node (SLN) detection techniques have the potential to change the standard of surgical care for patients with prostate cancer. We performed a lymphatic mapping study and determined the value of fluorescence SLN detection with indocyanine green (ICG) for the detection of lymph node metastases in intermediate- and high-risk patients undergoing radical prostatectomy and extended pelvic lymph node dissection. A total of 42 patients received systematic or specific ICG injections into the prostate base, the midportion, the apex, the left lobe, or the right lobe. We found (1) that external and internal iliac regions encompass the majority of SLNs, (2) that common iliac regions contain up to 22% of all SLNs, (3) that a prostatic lobe can drain into the contralateral group of pelvic lymph nodes, and (4) that the fossa of Marcille also receives significant drainage. Among the 12 patients who received systematic ICG injections, 5 (42%) had a total of 29 lymph node metastases. Of these, 16 nodes were ICG positive, yielding 55% sensitivity. The complex drainage pattern of the prostate and the low sensitivity of ICG for the detection of lymph node metastases reported in our study highlight the difficulties related to the implementation of SNL techniques in prostate cancer. PATIENT SUMMARY There is controversy about how extensive lymph node dissection (LND) should be during prostatectomy. We investigated the lymphatic drainage of the prostate and whether sentinel node fluorescence techniques would be useful to detect node metastases. We found that the drainage pattern is complex and that the sentinel node technique is not able to replace extended pelvic LND.

Enhancing Authenticity in Simulation Using Appropriate Techniques of Moulage

Workshop Overview The use of special effects (moulage) is a way to augment the authenticity of a scenario in simulation. This workshop will introduce different techniques of moulage (oil based cream colors, watercolors, transfer tattoos and 3D Prosthetics). The participants will have the opportunity to explore these techniques by applying various moulages. They will compare the techniques and discuss their advantages and disadvantages. Moreover, strategies for standardization and quality assurance will be discussed. Workshop Rationale Moulage supports the sensory perception in an scenario (1). It can provide evaluation clues (2) and help learners (and SPs) to engage in the simulation. However, it is of crucial importance that the simulated physical pathologies are represented accurate and reliable. Accuracy is achieved by using the appropriate technique, which requires knowledge and practice . With information about different moulage techniques, we hope to increases the knowledge of moulage during the workshop. By applying moulages in various techniques we will practice together. As standardization is critical for simulation scenarios in assessment (3, 4) strategies for standardization of moulage will be introduced and discussed. Workshop Objectives During the workshop participants will: - gain knowledge about different techniques of moulages - practice moulages in various techniques - discuss the advantages and disadvantages of moulage techniques - describe strategies for standardization and quality assurance of moulage Planned Format 5 min Introduction 15 min Overview – Background & Theory (presentation) 15 min Application of moulage for ankle sprain in 4 different techniques (oil based cream color, water color, temporary tatoo, 3D prosthetic) in small groups 5 min Comparing the results by interactive viewing of prepared moulages 15 min Application of moulages for burn in different techniques in small groups 5 min Comparing results the results by interactive viewing of prepared moulages 5 min Sharing experiences with different techniques in small groups 20 min Discussion of the techniques including standardization and quality assurance strategies (plenary discussion) 5 min Summary / Take home points

Do multiple true-false items beat the commonly used one-best-answers questions regarding to the Ottawa Criteria for Good Assessment? Results of a literature review

Background: Multiple True-False-Items (MTF-Items) might offer some advantages compared to one-best-answer-questions (TypeA) as they allow more than one correct answer and may better represent clinical decisions. However, in medical education assessment MTF-Items are seldom used. Summary of Work: With this literature review existing findings on MTF-items and on TypeA were compared along the Ottawa Criteria for Good Assessment, i.e. (1) reproducibility, (2) feasibility, (3) validity, (4) acceptance, (5) educational effect, (6) catalytic effects, and (7) equivalence. We conducted a literature research on ERIC and Google Scholar including papers from the years 1935 to 2014. We used the search terms “multiple true-false”, “true-false”, “true/false”, and “Kprim” combined with “exam”, “test”, and “assessment”. Summary of Results: We included 29 out of 33 studies. Four of them were carried out in the medical field Compared to TypeA, MTF-Items are associated with (1) higher reproducibility (2) lower feasibility (3) similar validity (4) higher acceptance (5) higher educational effect (6) no studies on catalytic effects or (7) equivalence. Discussion and Conclusions: While studies show overall good characteristics of MTF items according to the Ottawa criteria, this type of question seems to be rather seldom used. One reason might be the reported lower feasibility. Overall the literature base is still weak. Furthermore, only 14 % of literature is from the medical domain. Further studies to better understand the characteristics of MTF-Items in the medical domain are warranted. Take-home messages: Overall the literature base is weak and therefore further studies are needed. Existing studies show that: MTF-Items show higher reliability, acceptance and educational effect; MTF-Items are more difficult to produce

Comparability of Outcome Frameworks in Medical Education: Implications for framework development

Background: The demand for international harmonization in medical education increases with the growing mobility of students and health professionals. Many medical societies and governmental offices have issued outcome frameworks (OF), which describe aims and contents of medical education based on competencies. These national standards affect the development of curricula as well as assessment and licensing procedures. Comparing OF and identifying factors that limit their comparability may thus foster international harmonization of medical education. Summary of Work: We conducted a systematic search for national OF in MedLine, EmBase and the internet. We included all OF in German or English that resulted from a national consensus process and were published or endorsed by a national society or governmental body. We extracted information in five predetermined categories: history of origin, audience, formal structure, medical schooling system and key terms. Summary of Results: Out of 1816 results, 13 OF were included into further analyses. OF reference each other, often without addressing existing differences (e.g. in target audiences). The two most cited OF are “CanMEDs” and “Scottish Doctor”. OF differ especially in their level of detail as well as in the underlying educational system. Discussion and Conclusions: Based on our results we propose a two-step blueprint for OF, that may help to establish comparability for internationally aligned key features – so-called “core competencies” – while at the same time allowing for necessary regional adaptations in terms of “secondary competencies”. Take-home messages: Considerable differences in at least five categories of OF currently hinder the comparability of outcome frameworks.

Computer-based summative assessment of clinical reasoning in clerkships: A mixed-method comparison of key feature problems with case-based multiple choice questions

Background: It is yet unclear if there are differences between using electronic key feature problems (KFPs) or electronic case-based multiple choice questions (cbMCQ) for the assessment of clinical decision making. Summary of Work: Fifth year medical students were exposed to clerkships which ended with a summative exam. Assessment of knowledge per exam was done by 6-9 KFPs, 9-20 cbMCQ and 9-28 MC questions. Each KFP consisted of a case vignette and three key features (KF) using “long menu” as question format. We sought students’ perceptions of the KFPs and cbMCQs in focus groups (n of students=39). Furthermore statistical data of 11 exams (n of students=377) concerning the KFPs and (cb)MCQs were compared. Summary of Results: The analysis of the focus groups resulted in four themes reflecting students’ perceptions of KFPs and their comparison with (cb)MCQ: KFPs were perceived as (i) more realistic, (ii) more difficult, (iii) more motivating for the intense study of clinical reasoning than (cb)MCQ and (iv) showed an overall good acceptance when some preconditions are taken into account. The statistical analysis revealed that there was no difference in difficulty; however KFP showed a higher discrimination and reliability (G-coefficient) even when corrected for testing times. Correlation of the different exam parts was intermediate. Conclusions: Students perceived the KFPs as more motivating for the study of clinical reasoning. Statistically KFPs showed a higher discrimination and higher reliability than cbMCQs. Take-home messages: Including KFPs with long menu questions into summative clerkship exams seems to offer positive educational effects.

Improving residency training with the help of multisource feedback: Generation of a model of influencing factors

Background: Feedback is considered to be one of the most important drivers of learning. One form of structured feedback used in medical settings is multisource feedback (MSF). This feedback technique provides the opportunity to gain a differentiated view on a doctor’s performance from several perspectives using a questionnaire and a facilitating conversation, in which learning goals are formulated. While many studies have been conducted on the validity, reliability and feasibility of the instrument, little is known about the impact of factors that might influence the effects of MSF on clinical performance. Summary of Work: To study under which circumstances MSF is most effective, we performed a literature review on Google Scholar with focus on MSF and feedback in general. Main key-words were: MSF, multi-source-feedback, multi source feedback, and feedback each combined with influencing/ hindering/ facilitating factors, effective, effectiveness, doctors-intraining, and surgery. Summary of Results: Based on the literature, we developed a preliminary model of facilitating factors. This model includes five main factors influencing MSF: questionnaire, doctor-in-training, group of raters, facilitating supervisor, and facilitating conversation. Discussion and Conclusions: Especially the following points that might influence MSF have not yet been sufficiently studied: facilitating conversation with the supervisor, individual aspects of doctors-in-training, and the causal relations between influencing factors. Overall there are only very few studies focusing on the impact of MSF on actual and long-term performance. We developed a preliminary model of hindering and facilitating factors on MSF. Further studies are needed to better understand under which circumstances MSF is most effective. Take-home messages: The preliminary model might help to guide further studies on how to implement MSF to use it at its full potential.

Improving the Assessment of Communication Skills at the Swiss Federal Licensing Exam: Results of a Student's Needs Assessment

Introduction With a three year project the assessment of communication skills within the Swiss Federal Licensing Examinations (FLE) shall be improved. As a first step a needs assessment among communication experts and medical students of the Swiss Medical Faculties will be performed. In this presentation the results of the students’ needs assessment will be presented. Methods A bilingual student’s online questionnaire will be developed by an expert panel taking relevant literature, the Swiss Catalogue of Learning Objectives and other consensus statements for communication (e.g., the European and Basler consensus statements) into account. With a think aloud study response process validity evidence will be sought. The questionnaire will focus on the following topics related to communication skills: (1) What has been taught?, (2) What has been assessed in the faculty exams?, (3) What has been assessed in the FLE?, (4) What should have been assessed in the FLE and how should the assessment be improved? Results Results of the students’ needs assessment will be available by the end of 2015 and be presented. Conclusions/ Take-home message We hope for valuable input for improving the assessment of communications skills within the FLE also from the students’ side. Results of the needs assessment from the students and experts will be combined and taken as input for an international expert symposium on how to improve the communication skills assessment within the FLE.

Fiber penetration and copper access regulation

Research question, methodology and messages to take home - Impact of copper (legacy networks)regulation(LLU and bitstream)on NGN adoption - Methodology: fixed effects regression and forecasting with s- curves(logistic) - Key message:(excessive) copper regulation discourages NGN adoption (deployment, investments), in particular an increase ofunbundling access charges and/or a decrease of wholesale access would increase penetration of fibre and cable modem.

Performance audit of the Department of Central Management Services' operation of the state vehicle fleet /

"House Resolution 658 directed the Office of the Auditor General (OAG) to audit the Department of Central Management Services' (CMS) operation of the fleet of passenger cars used by State executive agencies. The Resolution called for the audit to address the total number of cars; number of take-home cars; necessity of take-home vehicles; costs of vehicles in Fiscal Year 2009; and the adequacy of CMS' system to record their use and maintenance and to check for official use, including whether it is possible to implement a system to track vehicles for business only."

Pitch jump probability measures for the analysis of snoring sounds in apnea

Obstructive sleep apnea (OSA) is a highly prevalent disease in which upper airways are collapsed during sleep, leading to serious consequences. The gold standard of diagnosis, called polysomnography (PSG), requires a full-night hospital stay connected to over ten channels of measurements requiring physical contact with sensors. PSG is inconvenient, expensive and unsuited for community screening. Snoring is the earliest symptom of OSA, but its potential in clinical diagnosis is not fully recognized yet. Diagnostic systems intent on using snore-related sounds (SRS) face the tough problem of how to define a snore. In this paper, we present a working definition of a snore, and propose algorithms to segment SRS into classes of pure breathing, silence and voiced/unvoiced snores. We propose a novel feature termed the 'intra-snore-pitch-jump' (ISPJ) to diagnose OSA. Working on clinical data, we show that ISPJ delivers OSA detection sensitivities of 86-100% while holding specificity at 50-80%. These numbers indicate that snore sounds and the ISPJ have the potential to be good candidates for a take-home device for OSA screening. Snore sounds have the significant advantage in that they can be conveniently acquired with low-cost non-contact equipment. The segmentation results presented in this paper have been derived using data from eight patients as the training set and another eight patients as the testing set. ISPJ-based OSA detection results have been derived using training data from 16 subjects and testing data from 29 subjects.

Project-based assessment for graduate coursework in physics

Project-based assessment, in the form of take-home exams, was trialed in an honours/masters level electromagnetic theory course. This assessment formed an integral part of the learning experience of the students, and students felt that this was effective method of learning.

Patients’ and physicians’ experiences of atrial fibrillation and anticoagulation therapy:a qualitative journey

Background: Oral anticoagulation (OAC) reduces stroke risk in patients with atrial fibrillation (AF) however it is often underutilized and sometimes refused by patients. This programme of work included a meta-synthesis and two inter-linking studies aiming to explore patients’ and physicians’ experiences of AF and OAC. Methods: A meta-synthesis of qualitative evidence was conducted which informed the empirical work. Semi-structured individual interviews were utilised. Study 1: Three AF patient sub-groups were interviewed; accepted (n=4), refused (n=4), or discontinued (n=3) warfarin. Study 2: Four physician sub-groups (n=4 each group) prescribing OAC to AF patients were interviewed: consultant cardiologists, consultant general physicians, general practitioners and cardiology registrars. Data was analysed using interpretative phenomenological analysis. Results: Study 1: Three over-arching themes comprised patients’ experiences: (1) the initial consultation, (2) life after the consultation, and (3) patients’ reflections. Patients commented on the relief and reassurance experienced during the consultation but they perceived the decision making process mostly led by the physician. Lack of education and take-home materials distributed during the initial consultation was highlighted. Patients who had experienced stroke themselves or were caregivers, were more receptive to education aimed towards stroke risk reduction rather than bleeding risk. Warfarin monitoring was challenging for patients, however some patients perceived it as beneficial as it served to enhance patient-physician relationship. Study 2: Two over-arching themes emerged from physicians’ experiences: (1) communicating information and (2) challenges with OAC prescription for AF. Physicians’ approach to the consultation style shifted through a continuum of compliance-adherence-concordance during the consultation. They aimed for concordance, however challenges such as time and the perceived patient trust in them as the expert, led to physicians adopting a paternalistic approach. Physicians also pointed out challenges associated with guideline adherence and the need to adopt a multi-disciplinary approach, where other health professionals could provide on-going education. Conclusion: This programme of work has illustrated the benefit of taking an in depth phenomenological approach to understanding the lived experience of the physician-patient consultation. Together with the meta-synthesis, this work has strengthened the evidence base and demonstrated that there is a need to target patients' and physicians' ability to communicate with each other in a comprehensible way.