Validity evidence and reliability of a Simulated Patient Feedback Instrument

In the training of healthcare professionals, one of the advantages of communication training with simulated patients (SPs) is the SP's ability to provide direct feedback to students after a simulated clinical encounter. The quality of SP feedback must be monitored, especially because it is well known that feedback can have a profound effect on student performance. Due to the current lack of valid and reliable instruments to assess the quality of SP feedback, our study examined the validity and reliability of one potential instrument, the 'modified Quality of Simulated Patient Feedback Form' (mQSF). Methods Content validity of the mQSF was assessed by inviting experts in the area of simulated clinical encounters to rate the importance of the mQSF items. Moreover, generalizability theory was used to examine the reliability of the mQSF. Our data came from videotapes of clinical encounters between six simulated patients and six students and the ensuing feedback from the SPs to the students. Ten faculty members judged the SP feedback according to the items on the mQSF. Three weeks later, this procedure was repeated with the same faculty members and recordings. Results All but two items of the mQSF received importance ratings of > 2.5 on a four-point rating scale. A generalizability coefficient of 0.77 was established with two judges observing one encounter. Conclusions The findings for content validity and reliability with two judges suggest that the mQSF is a valid and reliable instrument to assess the quality of feedback provided by simulated patients.

Comparing the validity and output of the GT1M and GT3X accelerometer in 5- to 9-year-old children

The purpose of this study was to compare the validity and output of the biaxial ActiGraph GT1M and the triaxial GT3X (ActiGraph, LLC, Pensacola, FL, USA)accelerometer in 5- to 9-year-old children. Thirty-two children wore the two monitors while their energy expenditure was measured with indirect calorimetry. They performed four locomotor and four play activities in an exercise laboratory and were further measured during 12 minutes of a sports lesson. Validity evidence in relation to indirect calorimetry was examined with linear regression equations applied to the laboratory data. During the sports lessons predicted energy expenditure according to the regression equations was compared to measured energy expenditure with the Wilcoxon-signed rank test and the Spearman correlation. To compare the output, agreement between counts of the two monitors during the laboratory activities was assessed with Bland-Altman plots. The evidence of validity was similar for both monitors. Agreement between the output of the two monitors was good for vertical counts (mean bias = −14 ± 22 counts) but not for horizontal counts (−17 ± 32 counts). The current results indicate that the two accelerometer models are able to estimate energy expenditure of a range of physical activities equally well in young children. However, they show output differences for movement in the horizontal direction.

Exploring the validity and reliability of a questionnaire for evaluating virtual patient design with a special emphasis on fostering clinical reasoning

Background: The design of Virtual Patients (VPs) is essential. So far there are no validated evaluation instruments for VP design published. Summary of work: We examined three sources of validity evidence of an instrument to be filled out by students aimed at measuring the quality of VPs with a special emphasis on fostering clinical reasoning: (1) Content was examined based on theory of clinical reasoning and an international VP expert team. (2) Response process was explored in think aloud pilot studies with students and content analysis of free text questions accompanying each item of the instrument. (3) Internal structure was assessed by confirmatory factor analysis (CFA) using 2547 student evaluations and reliability was examined utilizing generalizability analysis. Summary of results: Content analysis was supported by theory underlying Gruppen and Frohna’s clinical reasoning model on which the instrument is based and an international VP expert team. The pilot study and analysis of free text comments supported the validity of the instrument. The CFA indicated that a three factor model comprising 6 items showed a good fit with the data. Alpha coefficients per factor were 0,74 - 0,82. The findings of the generalizability studies indicated that 40-200 student responses are needed in order to obtain reliable data on one VP. Conclusions: The described instrument has the potential to provide faculty with reliable and valid information about VP design. Take-home messages: We present a short instrument which can be of help in evaluating the design of VPs.

Exploring the validity and reliability of a questionnaire for evaluating virtual patient design with a special emphasis on fostering clinical reasoning

Background: Virtual patients (VPs) are increasingly used to train clinical reasoning. So far, no validated evaluation instruments for VP design are available. Aims: We examined the validity of an instrument for assessing the perception of VP design by learners. Methods: Three sources of validity evidence were examined: (i) Content was examined based on theory of clinical reasoning and an international VP expert team. (ii) The response process was explored in think-aloud pilot studies with medical students and in content analyses of free text questions accompanying each item of the instrument. (iii) Internal structure was assessed by exploratory factor analysis (EFA) and inter-rater reliability by generalizability analysis. Results: Content analysis was reasonably supported by the theoretical foundation and the VP expert team. The think-aloud studies and analysis of free text comments supported the validity of the instrument. In the EFA, using 2547 student evaluations of a total of 78 VPs, a three-factor model showed a reasonable fit with the data. At least 200 student responses are needed to obtain a reliable evaluation of a VP on all three factors. Conclusion: The instrument has the potential to provide valid information about VP design, provided that many responses per VP are available.

Entwicklung eines deutschsprachigen Fragebogens zur Durchführung von Multisource-Feedback in der ärztlichen Weiterbildung

Fragestellung/Einleitung: Multisource-Feedback (MSF) ist ein anerkanntes Instrument zur Überprüfung und Verbesserung der ärztlichen Tätigkeit [1]. Es beinhaltet Feedback, das von MitarbeiterInnen verschiedener Tätigkeitsbereiche und verschiedener Hierarchiestufen gegeben wird. Das Feedback wird anonym mithilfe eines Fragebogens gegeben, der verschiedene Kriterien der ärztlichen Kompetenz beschreibt. Das Feedback wird anschlieβend für die zu beurteilenden ÄrztInnen in einem Gespräch von einer/m SupervisorIn zusammengefasst. Bislang existiert kein deutschsprachiger Fragebogen für Multisource-Feedback für die ärztliche Tätigkeit. Unsere Zielsetzung war es daher, einen deutschsprachigen Fragebogen zu erstellen und diesen bzgl. relevanter Validitätskriterien zu untersuchen. Methoden: Zur Erstellung des Fragebogens sammelten wir die beste verfügbare Evidenz der entsprechenden Literatur. Wir wählten einen validierten englischen Fragebogen, der bereits in der Weiterbildung in Groβbritannien angewendet wird [2] und den wichtigsten Kriterien entspricht. Dieser wurde übersetzt und in einigen Bereichen erweitert, um ihn sprachlichen Gegebenheiten und lokalen Bedürfnissen anzupassen. Bezüglich der Validität wurden zwei Kriterien untersucht: Inhaltsvalidität (content validity evidence) und Antwortprozesse (response process validity evidence). Um die Inhaltsvalidität zu untersuchen, wurde in einer Expertenrunde diskutiert, ob der übersetzte Fragebogen die erwarteten Kompetenzen widerspiegelt. Im Anschluss wurden die Antwortprozesse mithilfe eines sog. „think-alouds“ mit ÄrztInnen in Weiterbildung und ihren AusbilderInnen untersucht. Ergebnisse: Der resultierende Fragebogen umfasst 20 Fragen. Davon sind 15 Items den Bereichen „Klinische Fähigkeiten“, „Umgang mit Patienten“, „Umgang mit Kollegen“ und „Arbeitsweise“ zuzuordnen. Diese Fragen werden auf einer fünfstufigen Likert-Skala beantwortet. Zusätzlich bietet jede Frage die Möglichkeit, einen Freitext zu besonderen Stärken und Schwächen der KandidatInnen aufzuführen. Weiterhin gibt es fünf globale Fragen zu Stärken und Verbesserungsmöglichkeiten, äuβeren Einflüssen, den Arbeitsbedingungen und nach Zweifeln an der Gesundheit oder Integrität des Arztes/ der Ärztin. In der Expertenrunde wurde der Fragebogen als für den deutschsprachigen Raum ohne Einschränkungen anwendbar eingeschätzt. Die Analyse der Antwortprozesse führte zu kleineren sprachlichen Anpassungen und bestätigt, dass der Fragebogen verständlich und eindeutig zu beantworten ist und das gewählte Konstrukt der ärztlichen Tätigkeit vollständig umschreibt. Diskussion/Schlussfolgerung: Wir entwickelten einen deutschsprachigen Fragebogen zur Durchführung von Multisource-Feedback in der ärztlichen Weiterbildung. Wir fanden Hinweise für die Validität dieses Fragebogens bzgl. des Inhalts und der Antwortprozesse. Zusätzliche Untersuchungen zur Validität wie z.B. die durch den Fragebogen entstehenden Auswirkungen (consequences) sind vorgesehen. Dieser Fragebogen könnte zum breiteren Einsatz von MSF in der ärztlichen Weiterbildung auch im deutschsprachigen Raum beitragen. This is an Open Access article distributed under the terms of the Creative Commons Attribution License. You are free: to Share - to copy, distribute and transmit the work, provided the original author and source are credited. See license information at http://creativecommons.org/licenses/by-nc-nd/3.0/.

Improving the Assessment of Communication Skills at the Swiss Federal Licensing Exam: Results of a Student's Needs Assessment

Introduction With a three year project the assessment of communication skills within the Swiss Federal Licensing Examinations (FLE) shall be improved. As a first step a needs assessment among communication experts and medical students of the Swiss Medical Faculties will be performed. In this presentation the results of the students’ needs assessment will be presented. Methods A bilingual student’s online questionnaire will be developed by an expert panel taking relevant literature, the Swiss Catalogue of Learning Objectives and other consensus statements for communication (e.g., the European and Basler consensus statements) into account. With a think aloud study response process validity evidence will be sought. The questionnaire will focus on the following topics related to communication skills: (1) What has been taught?, (2) What has been assessed in the faculty exams?, (3) What has been assessed in the FLE?, (4) What should have been assessed in the FLE and how should the assessment be improved? Results Results of the students’ needs assessment will be available by the end of 2015 and be presented. Conclusions/ Take-home message We hope for valuable input for improving the assessment of communications skills within the FLE also from the students’ side. Results of the needs assessment from the students and experts will be combined and taken as input for an international expert symposium on how to improve the communication skills assessment within the FLE.

Measuring depression with a well-being index: Further evidence for the validity of the WHO Well-Being Index (WHO-5) as a measure of the severity of depression

BACKGROUND: In recent years, the WHO Wellbeing Index (WHO-5) has been used as a screening measure for depression. Nevertheless, research on the validity of this measure in the context of clinical depression is sparse. QUESTIONS: The aim of the present study was to investigate the measurement invariance of the WHO-5 across depressed and non-depressed individuals, as well as the shape and specificity of its relationship to measures of depression severity. METHOD: Of the 414 subjects who completed the WHO-5 and the Beck Depression Inventory-II (BDI-II), 207 had a diagnosis of a major depressive episode (MDE). A subsample also completed the Beck Anxiety Inventory (BAI) and was assessed by clinicians using the Hamilton Depression Rating Scale (HAM-D) and the Hamilton Anxiety Rating Scale (HAM-A). RESULTS: The WHO-5 demonstrated strong measurement invariance regarding the presence or absence of a current MDE. The WHO-5 showed a very high negative association with self- and observer-rated measures of depressive symptoms, especially in the range of mild to moderate symptoms. These associations were still substantial after controlling for measures of anxiety symptoms. LIMITATIONS: In addition to a diagnostic interview, only one measure for self- and observer-rated symptoms of depression was used. Furthermore, the observer-rated measure was only assessed in one subsample that exhibited a somewhat restricted range of depression severity. CONCLUSION: Although this index was originally designed as a measure of well-being, the results support the use of the WHO-5 in the context of depression research.

Testing the structural and cross-cultural validity of the KIDSCREEN-27 quality of life questionnaire

OBJECTIVES: The aim of this study is to assess the structural and cross-cultural validity of the KIDSCREEN-27 questionnaire. METHODS: The 27-item version of the KIDSCREEN instrument was derived from a longer 52-item version and was administered to young people aged 8-18 years in 13 European countries in a cross-sectional survey. Structural and cross-cultural validity were tested using multitrait multi-item analysis, exploratory and confirmatory factor analysis, and Rasch analyses. Zumbo's logistic regression method was applied to assess differential item functioning (DIF) across countries. Reliability was assessed using Cronbach's alpha. RESULTS: Responses were obtained from n = 22,827 respondents (response rate 68.9%). For the combined sample from all countries, exploratory factor analysis with procrustean rotations revealed a five-factor structure which explained 56.9% of the variance. Confirmatory factor analysis indicated an acceptable model fit (RMSEA = 0.068, CFI = 0.960). The unidimensionality of all dimensions was confirmed (INFIT: 0.81-1.15). Differential item functioning (DIF) results across the 13 countries showed that 5 items presented uniform DIF whereas 10 displayed non-uniform DIF. Reliability was acceptable (Cronbach's alpha = 0.78-0.84 for individual dimensions). CONCLUSIONS: There was substantial evidence for the cross-cultural equivalence of the KIDSCREEN-27 across the countries studied and the factor structure was highly replicable in individual countries. Further research is needed to correct scores based on DIF results. The KIDSCREEN-27 is a new short and promising tool for use in clinical and epidemiological studies.

Systematic review of the empirical evidence of study publication bias and outcome reporting bias

BACKGROUND: The increased use of meta-analysis in systematic reviews of healthcare interventions has highlighted several types of bias that can arise during the completion of a randomised controlled trial. Study publication bias has been recognised as a potential threat to the validity of meta-analysis and can make the readily available evidence unreliable for decision making. Until recently, outcome reporting bias has received less attention. METHODOLOGY/PRINCIPAL FINDINGS: We review and summarise the evidence from a series of cohort studies that have assessed study publication bias and outcome reporting bias in randomised controlled trials. Sixteen studies were eligible of which only two followed the cohort all the way through from protocol approval to information regarding publication of outcomes. Eleven of the studies investigated study publication bias and five investigated outcome reporting bias. Three studies have found that statistically significant outcomes had a higher odds of being fully reported compared to non-significant outcomes (range of odds ratios: 2.2 to 4.7). In comparing trial publications to protocols, we found that 40-62% of studies had at least one primary outcome that was changed, introduced, or omitted. We decided not to undertake meta-analysis due to the differences between studies. CONCLUSIONS: Recent work provides direct empirical evidence for the existence of study publication bias and outcome reporting bias. There is strong evidence of an association between significant results and publication; studies that report positive or significant results are more likely to be published and outcomes that are statistically significant have higher odds of being fully reported. Publications have been found to be inconsistent with their protocols. Researchers need to be aware of the problems of both types of bias and efforts should be concentrated on improving the reporting of trials.

Completeness of Follow-Up Determines Validity of Study Findings: Results of a Prospective Repeated Measures Cohort Study.

BACKGROUND Current reporting guidelines do not call for standardised declaration of follow-up completeness, although study validity depends on the representativeness of measured outcomes. The Follow-Up Index (FUI) describes follow-up completeness at a given study end date as ratio between the investigated and the potential follow-up period. The association between FUI and the accuracy of survival-estimates was investigated. METHODS FUI and Kaplan-Meier estimates were calculated twice for 1207 consecutive patients undergoing aortic repair during an 11-year period: in a scenario A the population's clinical routine follow-up data (available from a prospective registry) was analysed conventionally. For the control scenario B, an independent survey was completed at the predefined study end. To determine the relation between FUI and the accuracy of study findings, discrepancies between scenarios regarding FUI, follow-up duration and cumulative survival-estimates were evaluated using multivariate analyses. RESULTS Scenario A noted 89 deaths (7.4%) during a mean considered follow-up of 30±28months. Scenario B, although analysing the same study period, detected 304 deaths (25.2%, P<0.001) as it scrutinized the complete follow-up period (49±32months). FUI (0.57±0.35 versus 1.00±0, P<0.001) and cumulative survival estimates (78.7% versus 50.7%, P<0.001) differed significantly between scenarios, suggesting that incomplete follow-up information led to underestimation of mortality. Degree of follow-up completeness (i.e. FUI-quartiles and FUI-intervals) correlated directly with accuracy of study findings: underestimation of long-term mortality increased almost linearly by 30% with every 0.1 drop in FUI (adjusted HR 1.30; 95%-CI 1.24;1.36, P<0.001). CONCLUSION Follow-up completeness is a pre-requisite for reliable outcome assessment and should be declared systematically. FUI represents a simple measure suited as reporting standard. Evidence lacking such information must be challenged as potentially flawed by selection bias.

Total disc arthroplasty versus anterior cervical interbody fusion: use of the Spine Tango registry to supplement the evidence from randomized control trials.

BACKGROUND CONTEXT Several randomized controlled trials (RCTs) have compared patient outcomes of anterior (cervical) interbody fusion (AIF) with those of total disc arthroplasty (TDA). Because RCTs have known limitations with regard to their external validity, the comparative effectiveness of the two therapies in daily practice remains unknown. PURPOSE This study aimed to compare patient-reported outcomes after TDA versus AIF based on data from an international spine registry. STUDY DESIGN AND SETTING A retrospective analysis of registry data was carried out. PATIENT SAMPLE Inclusion criteria were degenerative disc or disc herniation of the cervical spine treated by single-level TDA or AIF, no previous surgery, and a Core Outcome Measures Index (COMI) completed at baseline and at least 3 months' follow-up. Overall, 987 patients were identified. OUTCOME MEASURES Neck and arm pain relief and COMI score improvement were the outcome measures. METHODS Three separate analyses were performed to compare TDA and AIF surgical outcomes: (1) mimicking an RCT setting, with admission criteria typical of those in published RCTs, a 1:1 matched analysis was carried out in 739 patients; (2) an analysis was performed on 248 patients outside the classic RCT spectrum, that is, with one or more typical RCT exclusion criteria; (3) a subgroup analysis of all patients with additional follow-up longer than 2 years (n=149). RESULTS Matching resulted in 190 pairs with an average follow-up of 17 months that had no residual significant differences for any patient characteristics. Small but statistically significant differences in outcome were observed in favor of TDA, which are potentially clinically relevant. Subgroup analyses of atypical patients and of patients with longer-term follow-up showed no significant differences in outcome between the treatments. CONCLUSIONS The results of this observational study were in accordance with those of the published RCTs, suggesting substantial pain reduction both after AIF and TDA, with slightly greater benefit after arthroplasty. The analysis of atypical patients suggested that, in patients outside the spectrum of clinical trials, both surgical interventions appeared to work to a similar extent to that shown for the cohort in the matched study. Also, in the longer-term perspective, both therapies resulted in similar benefits to the patients.