Effect of standardized training on the reliability of the Cochrane risk of bias assessment tool: a study protocol

BACKGROUND The Cochrane risk of bias (RoB) tool has been widely embraced by the systematic review community, but several studies have reported that its reliability is low. We aim to investigate whether training of raters, including objective and standardized instructions on how to assess risk of bias, can improve the reliability of this tool. We describe the methods that will be used in this investigation and present an intensive standardized training package for risk of bias assessment that could be used by contributors to the Cochrane Collaboration and other reviewers. METHODS/DESIGN This is a pilot study. We will first perform a systematic literature review to identify randomized clinical trials (RCTs) that will be used for risk of bias assessment. Using the identified RCTs, we will then do a randomized experiment, where raters will be allocated to two different training schemes: minimal training and intensive standardized training. We will calculate the chance-corrected weighted Kappa with 95% confidence intervals to quantify within- and between-group Kappa agreement for each of the domains of the risk of bias tool. To calculate between-group Kappa agreement, we will use risk of bias assessments from pairs of raters after resolution of disagreements. Between-group Kappa agreement will quantify the agreement between the risk of bias assessment of raters in the training groups and the risk of bias assessment of experienced raters. To compare agreement of raters under different training conditions, we will calculate differences between Kappa values with 95% confidence intervals. DISCUSSION This study will investigate whether the reliability of the risk of bias tool can be improved by training raters using standardized instructions for risk of bias assessment. One group of inexperienced raters will receive intensive training on risk of bias assessment and the other will receive minimal training. By including a control group with minimal training, we will attempt to mimic what many review authors commonly have to do, that is-conduct risk of bias assessment in RCTs without much formal training or standardized instructions. If our results indicate that an intense standardized training does improve the reliability of the RoB tool, our study is likely to help improve the quality of risk of bias assessments, which is a central component of evidence synthesis.

Comprehensive assessment of gesture production: a new test of upper limb apraxia (TULIA)

BACKGROUND: Only few standardized apraxia scales are available and they do not cover all domains and semantic features of gesture production. Therefore, the objective of the present study was to evaluate the reliability and validity of a newly developed test of upper limb apraxia (TULIA), which is comprehensive and still short to administer. METHODS: The TULIA consists of 48 items including imitation and pantomime domain of non-symbolic (meaningless), intransitive (communicative) and transitive (tool related) gestures corresponding to 6 subtests. A 6-point scoring method (0-5) was used (score range 0-240). Performance was assessed by blinded raters based on videos in 133 stroke patients, 84 with left hemisphere damage (LHD) and 49 with right hemisphere damage (RHD), as well as 50 healthy subjects (HS). RESULTS: The clinimetric findings demonstrated mostly good to excellent internal consistency, inter- and intra-rater (test-retest) reliability, both at the level of the six subtests and at individual item level. Criterion validity was evaluated by confirming hypotheses based on the literature. Construct validity was demonstrated by a high correlation (r = 0.82) with the De Renzi-test. CONCLUSION: These results show that the TULIA is both a reliable and valid test to systematically assess gesture production. The test can be easily applied and is therefore useful for both research purposes and clinical practice.

Relevance of in vitro tests of adhesive and composite dental materials, a review in 3 parts. Part 1: Approval requirements and standardized testing of composite materials according to ISO specifications

The first part of this three-part review on the relevance of laboratory testing of composites and adhesives deals with approval requirements for composite materials. We compare the in vivo and in vitro literature data and discuss the relevance of in vitro analyses. The standardized ISO protocols are presented, with a focus on the evaluation of physical parameters. These tests all have a standardized protocol that describes the entire test set-up. The tests analyse flexural strength, depth of cure, susceptibility to ambient light, color stability, water sorption and solubility, and radiopacity. Some tests have a clinical correlation. A high flexural strength, for instance, decreases the risk of fractures of the marginal ridge in posterior restorations and incisal edge build-ups of restored anterior teeth. Other tests do not have a clinical correlation or the threshold values are too low, which results in an approval of materials that show inferior clinical properties (e.g., radiopacity). It is advantageous to know the test set-ups and the ideal threshold values to correctly interpret the material data. Overall, however, laboratory assessment alone cannot ensure the clinical success of a product.

[Assessment of musculoskeletal pain]

The rationale for a successful treatment of musculoskeletal pain is an adequate initial assessment. Standardized questionnaires, modern imaging modalities such as computed tomography, magnetic resonance imaging and musculoskeletal ultrasound or electrophysiology have enriched our armamentarium in the last decades. Pain inducing pathologies can often be identified and treated in a targeted way due to these procedures. But none of these techniques allows an adequate judgment of the acquired findings. Supplementary tests have to be indicated and interpreted in the context of the patient's entire history and the clinical findings. These two remain to be the cornerstones of the assessment of painful musculoskeletal disorders.

Assessment of smoking behaviour in the dental setting. A study comparing self-reported questionnaire data and exhaled carbon monoxide levels

The present study validated the accuracy of data from a self-reported questionnaire on smoking behaviour with the use of exhaled carbon monoxide (CO) level measurements in two groups of patients. Group 1 included patients referred to an oral medicine unit, whereas group 2 was recruited from the daily outpatient service. All patients filled in a standardized questionnaire regarding their current and former smoking habits. Additionally, exhaled CO levels were measured using a monitor. A total of 121 patients were included in group 1, and 116 patients were included in group 2. The mean value of exhaled CO was 7.6 ppm in the first group and 9.2 ppm in the second group. The mean CO values did not statistically significantly differ between the two groups. The two exhaled CO level measurements taken for each patient exhibited very good correlation (Spearman's coefficient of 0.9857). Smokers had a mean difference of exhaled CO values of 13.95 ppm (p < 0.001) compared to non-smokers adjusted for the first or second group. The consumption of one additional pack year resulted in an increase in CO values of 0.16 ppm (p = 0.003). The consumption of one additional cigarette per day elevated the CO measurements by 0.88 ppm (p < 0.001). Based on these results, the correlations between the self-reported smoking habits and exhaled CO values are robust and highly reproducible. CO monitors may offer a non-invasive method to objectively assess current smoking behaviour and to monitor tobacco use cessation attempts in the dental setting.

Proposal for a 10-high-power-fields scoring method for the assessment of tumor budding in colorectal cancer

Although tumor budding is linked to adverse prognosis in colorectal cancer, it remains largely unreported in daily diagnostic work due to the absence of a standardized scoring method. Our aim was to assess the inter-observer agreement of a novel 10-high-power-fields method for assessment of tumor budding at the invasive front and to confirm the prognostic value of tumor budding in our setting of colorectal cancers. Whole tissue sections of 215 colorectal cancers with full clinico-pathological and follow-up information were stained with cytokeratin AE1/AE3 antibody. Presence of buds was scored across 10-high-power fields at the invasive front by two pathologists and two additional observers were asked to score 50 cases of tumor budding randomly selected from the larger cohort. The measurements were correlated to the patient and tumor characteristics. Inter-observer agreement and correlation between observers' scores were excellent (P<0.0001; intraclass correlation coefficient=0.96). A test subgroup of 65 patients (30%) was used to define a valid cutoff score for high-grade tumor budding and the remaining 70% of the patients were entered into the analysis. High-grade budding was defined as an average of ≥10 buds across 10-high-power fields. High-grade budding was associated with a higher tumor grade (P<0.0001), higher TNM stage (P=0.0003), vascular invasion (P<0.0001), infiltrating tumor border configuration (P<0.0001) and reduced survival (P<0.0001). Multivariate analysis confirmed its independent prognostic effect (P=0.007) when adjusting for TNM stage and adjuvant therapy. Using 10-high-power fields for evaluating tumor budding has independent prognostic value and shows excellent inter-observer agreement. Like the BRE and Gleason scores in breast and prostate cancers, respectively, tumor budding could be a basis for a prognostic score in colorectal cancer.

Clinical end points in coronary stent trials: a case for standardized definitions

BACKGROUND: Although most clinical trials of coronary stents have measured nominally identical safety and effectiveness end points, differences in definitions and timing of assessment have created confusion in interpretation. METHODS AND RESULTS: The Academic Research Consortium is an informal collaboration between academic research organizations in the United States and Europe. Two meetings, in Washington, DC, in January 2006 and in Dublin, Ireland, in June 2006, sponsored by the Academic Research Consortium and including representatives of the US Food and Drug Administration and all device manufacturers who were working with the Food and Drug Administration on drug-eluting stent clinical trial programs, were focused on consensus end point definitions for drug-eluting stent evaluations. The effort was pursued with the objective to establish consistency among end point definitions and provide consensus recommendations. On the basis of considerations from historical legacy to key pathophysiological mechanisms and relevance to clinical interpretability, criteria for assessment of death, myocardial infarction, repeat revascularization, and stent thrombosis were developed. The broadly based consensus end point definitions in this document may be usefully applied or recognized for regulatory and clinical trial purposes. CONCLUSION: Although consensus criteria will inevitably include certain arbitrary features, consensus criteria for clinical end points provide consistency across studies that can facilitate the evaluation of safety and effectiveness of these devices.

From text to codings: intercoder reliability assessment in qualitative content analysis

BACKGROUND: High intercoder reliability (ICR) is required in qualitative content analysis for assuring quality when more than one coder is involved in data analysis. The literature is short of standardized procedures for ICR procedures in qualitative content analysis. OBJECTIVE: To illustrate how ICR assessment can be used to improve codings in qualitative content analysis. METHODS: Key steps of the procedure are presented, drawing on data from a qualitative study on patients' perspectives on low back pain. RESULTS: First, a coding scheme was developed using a comprehensive inductive and deductive approach. Second, 10 transcripts were coded independently by two researchers, and ICR was calculated. A resulting kappa value of .67 can be regarded as satisfactory to solid. Moreover, varying agreement rates helped to identify problems in the coding scheme. Low agreement rates, for instance, indicated that respective codes were defined too broadly and would need clarification. In a third step, the results of the analysis were used to improve the coding scheme, leading to consistent and high-quality results. DISCUSSION: The quantitative approach of ICR assessment is a viable instrument for quality assurance in qualitative content analysis. Kappa values and close inspection of agreement rates help to estimate and increase quality of codings. This approach facilitates good practice in coding and enhances credibility of analysis, especially when large samples are interviewed, different coders are involved, and quantitative results are presented.

Updated standardized endpoint definitions for transcatheter aortic valve implantation: the Valve Academic Research Consortium-2 consensus document

OBJECTIVES The aim of the current Valve Academic Research Consortium (VARC)-2 initiative was to revisit the selection and definitions of transcatheter aortic valve implantation (TAVI) clinical endpoints to make them more suitable to the present and future needs of clinical trials. In addition, this document is intended to expand the understanding of patient risk stratification and case selection. BACKGROUND A recent study confirmed that VARC definitions have already been incorporated into clinical and research practice and represent a new standard for consistency in reporting clinical outcomes of patients with symptomatic severe aortic stenosis (AS) undergoing TAVI. However, as the clinical experience with this technology has matured and expanded, certain definitions have become unsuitable or ambiguous. METHODS AND RESULTS Two in-person meetings (held in September 2011 in Washington, DC, and in February 2012 in Rotterdam, The Netherlands) involving VARC study group members, independent experts (including surgeons, interventional and noninterventional cardiologists, imaging specialists, neurologists, geriatric specialists, and clinical trialists), the US Food and Drug Administration (FDA), and industry representatives, provided much of the substantive discussion from which this VARC-2 consensus manuscript was derived. This document provides an overview of risk assessment and patient stratification that need to be considered for accurate patient inclusion in studies. Working groups were assigned to define the following clinical endpoints: mortality, stroke, myocardial infarction, bleeding complications, acute kidney injury, vascular complications, conduction disturbances and arrhythmias, and a miscellaneous category including relevant complications not previously categorized. Furthermore, comprehensive echocardiographic recommendations are provided for the evaluation of prosthetic valve (dys)function. Definitions for the quality of life assessments are also reported. These endpoints formed the basis for several recommended composite endpoints. CONCLUSIONS This VARC-2 document has provided further standardization of endpoint definitions for studies evaluating the use of TAVI, which will lead to improved comparability and interpretability of the study results, supplying an increasingly growing body of evidence with respect to TAVI and/or surgical aortic valve replacement. This initiative and document can furthermore be used as a model during current endeavors of applying definitions to other transcatheter valve therapies (for example, mitral valve repair).

Psychoendocrine validation of a short measure for assessment of perceived stress management skills in different non-clinical populations

BACKGROUND: We investigated the psychometric properties of a short questionnaire for combined assessment of different perceived stress management skills in the general population and tested whether scores relate to physiological stress reactivity. METHODS: For psychometric evaluation, we determined the factor structure of the questionnaire and investigated its measurement invariance in the participant groups and over time in three different independent samples representing the general population (total N=332). Reliability was tested by estimating test-retest reliability, internal consistency, and item reliabilities. We examined convergent and criterion validity using selected criterion variables. For endocrine validation, 35 healthy non-smoking and medication-free men in a laboratory study and 35 male and female employees in a workplace study underwent an acute standardized psychosocial stress task. We assessed stress management skills and measured salivary cortisol before and several times up to 60 min (workplace study) and 120 min (laboratory study) after stress. Potential confounders were controlled. RESULTS: The factor structure of the questionnaire consists of five scales reflecting acceptably distinct stress management skills such as cognitive strategies, use of social support, relaxation strategies, anger regulation, and perception of bodily tension. This factor structure was stable across participant groups and over time. Internal consistencies, item reliabilities, and test-retest reliabilities met established statistical requirements. Convergent and criterion validity were also established. In both endocrine validation studies, higher stress management skills were independently associated with lower cortisol stress reactivity (p's<.029). CONCLUSIONS: Our findings suggest that the questionnaire has good psychometric properties and that it relates to subjective psychological and objective physiological stress indicators. Therefore, the instrument seems a suitable measure for differential assessment of stress management skills in the general population.

High resolution quantitative computed tomography-based assessment of trabecular microstructure and strength estimates by finite-element analysis of the spine, but not DXA, reflects vertebral fracture status in men with glucocorticoid-induced osteoporosis

High-resolution quantitative computed tomography (HRQCT)-based analysis of spinal bone density and microstructure, finite element analysis (FEA), and DXA were used to investigate the vertebral bone status of men with glucocorticoid-induced osteoporosis (GIO). DXA of L1–L3 and total hip, QCT of L1–L3, and HRQCT of T12 were available for 73 men (54.6±14.0years) with GIO. Prevalent vertebral fracture status was evaluated on radiographs using a semi-quantitative (SQ) score (normal=0 to severe fracture=3), and the spinal deformity index (SDI) score (sum of SQ scores of T4 to L4 vertebrae). Thirty-one (42.4%) subjects had prevalent vertebral fractures. Cortical BMD (Ct.BMD) and thickness (Ct.Th), trabecular BMD (Tb.BMD), apparent trabecular bone volume fraction (app.BV/TV), and apparent trabecular separation (app.Tb.Sp) were analyzed by HRQCT. Stiffness and strength of T12 were computed by HRQCT-based nonlinear FEA for axial compression, anterior bending and axial torsion. In logistic regressions adjusted for age, glucocorticoid dose and osteoporosis treatment, Tb.BMD was most closely associated with vertebral fracture status (standardized odds ratio [sOR]: Tb.BMD T12: 4.05 [95% CI: 1.8–9.0], Tb.BMD L1–L3: 3.95 [1.8–8.9]). Strength divided by cross-sectional area for axial compression showed the most significant association with spine fracture status among FEA variables (2.56 [1.29–5.07]). SDI was best predicted by a microstructural model using Ct.Th and app.Tb.Sp (r2=0.57, p<0.001). Spinal or hip DXA measurements did not show significant associations with fracture status or severity. In this cross-sectional study of males with GIO, QCT, HRQCT-based measurements and FEA variables were superior to DXA in discriminating between patients of differing prevalent vertebral fracture status. A microstructural model combining aspects of cortical and trabecular bone reflected fracture severity most accurately.

Dental erosive wear assessment among adolescents and adults utilizing the basic erosive wear examination (BEWE) scoring system.

OBJECTIVES To investigate erosive tooth wear and related variables among adolescents and adults in Israel, utilizing the new basic erosive wear examination (BEWE) scoring system, in an attempt to contribute to the ongoing review, evaluation, and further development of an international standardized index. MATERIAL AND METHODS A cross-sectional, descriptive, and analytic survey was conducted among 500 subjects of five age groups. Dental erosion was measured according to the new BEWE scoring system. Independent variables included gender, age, origin, education, employment status, and diet. A backward stepwise linear regression model was applied to identify significantly associated variables. RESULTS Fifty percent of the survey subjects demonstrated erosive tooth wear; among them, 10 % had distinct erosion of over 50 % of the dental surface. Total BEWE score differences by age groups were statistically significant; as the age increased, the mean total BEWE scores increased (p < 0.001). The association between acidic foods and erosion was evident among the younger population (p = 0.038). In a multiple regression model, age (p < 0.001) and diet (p = 0.044) achieved statistical significance as variables associated with dental erosive wear. CONCLUSIONS Our study is one of the first to use the BEWE scoring system in an epidemiological survey among adolescents and adults. It was found that the BEWE index is straightforward, easy to conduct, and comfortably accepted by the examinees. CLINICAL RELEVANCE The present findings, together with further international research, should contribute toward continued evaluation of the BEWE system as an international standard and thereby, toward more optimal understanding, evidence-based treatment, and prevention of dental erosive wear.

Approaches to Ensure Quality Standards for Standardized/ Simulated Patient Performance in High Stakes Objective Structured Clinical Exams (OSCEs)

Introduction To meet the quality standards for high-stakes OSCEs, it is necessary to ensure high quality standardized performance of the SPs involved.[1] One of the ways this can be assured is through the assessment of the quality of SPs` performance in training and during the assessment. There is some literature concerning validated instruments that have been used to assess SP performance in formative contexts but very little related to high stakes contexts.[2], [3], [4]. Content and structure During this workshop different approaches to quality control for SPs` performance, developed in medicine, pharmacy and nursing OSCEs, will be introduced. Participants will have the opportunity to use these approaches in simulated interactions. Advantages and disadvantages of these approaches will be discussed. Anticipated outcomes By the end of this session, participants will be able to discuss the rationale for quality control of SPs` performance in high stakes OSCEs, outline key factors in creating strategies for quality control, identify various strategies for assuring quality control, and reflect on applications to their own practice. Who should attend The workshop is designed for those interested in quality assurance of SP performance in high stakes OSCEs. Level All levels are welcome. References Adamo G. 2003. Simulated and standardized patients in OSCEs: achievements and challenges:1992-2003. Med Teach. 25(3), 262- 270. Wind LA, Van Dalen J, Muijtjens AM, Rethans JJ. Assessing simulated patients in an educational setting: the MaSP (Maastricht Assessment of Simulated Patients). Med Educ 2004, 38(1):39-44. Bouter S, van Weel-Baumgarten E, Bolhuis S. Construction and validation of the Nijmegen Evaluation of the Simulated Patient (NESP): Assessing Simulated Patients' ability to role-play and provide feedback to students. Acad Med: Journal of the Association of American Medical Colleges 2012. May W, Fisher D, Souder D: Development of an instrument to measure the quality of standardized/simulated patient verbal feedback. Med Educ 2012, 2(1).