Can decision making in general surgery be based on evidence? An empirical study of Cochrane Reviews

BACKGROUND: This empirical study analyzes the current status of Cochrane Reviews (CRs) and their strength of recommendation for evidence-based decision making in the field of general surgery. METHODS: Systematic literature search of the Cochrane Database of Systematic Reviews and the Cochrane Collaboration's homepage to identify available CRs on surgical topics. Quantitative and qualitative characteristics, utilization, and formulated treatment recommendations were evaluated by 2 independent reviewers. Association of review characteristics with treatment recommendation was analyzed using univariate and multivariate logistic regression models. RESULTS: Ninety-three CRs, including 1,403 primary studies and 246,473 patients, were identified. Mean number of included primary studies per CR was 15.1 (standard deviation [SD] 14.5) including 2,650 (SD 3,340) study patients. Two and a half (SD 8.3) nonrandomized trials were included per analyzed CR. Seventy-two (77%) CRs were published or updated in 2005 or later. Explicit treatment recommendations were given in 45 (48%). Presence of a treatment recommendation was associated with the number of included primary studies and the proportion of randomized studies. Utilization of surgical CRs remained low and showed large inter-country differences. The most surgical CRs were accessed in UK, USA, and Australia, followed by several Western and Eastern European countries. CONCLUSION: Only a minority of available CRs address surgical questions and their current usage is low. Instead of unsystematically increasing the number of surgical CRs it would be far more efficient to focus the review process on relevant surgical questions. Prioritization of CRs needs valid methods which should be developed by the scientific surgical community.

Cochrane and non-Cochrane systematic reviews in leading orthodontic journals: a quality paradigm?

The aims of this study were to assess and compare the methodological quality of Cochrane and non-Cochrane systematic reviews (SRs) published in leading orthodontic journals and the Cochrane Database of Systematic Reviews (CDSR) using AMSTAR and to compare the prevalence of meta-analysis in both review types. A literature search was undertaken to identify SRs that consisted of hand-searching five major orthodontic journals [American Journal of Orthodontics and Dentofacial Orthopedics, Angle Orthodontist, European Journal of Orthodontics, Journal of Orthodontics and Orthodontics and Craniofacial Research (February 2002 to July 2011)] and the Cochrane Database of Systematic Reviews from January 2000 to July 2011. Methodological quality of the included reviews was gauged using the AMSTAR tool involving 11 key methodological criteria with a score of 0 or 1 given for each criterion. A cumulative grade was given for the paper overall (0-11); an overall score of 4 or less represented poor methodological quality, 5-8 was considered fair and 9 or greater was deemed to be good. In total, 109 SRs were identified in the five major journals and on the CDSR. Of these, 26 (23.9%) were in the CDSR. The mean overall AMSTAR score was 6.2 with 21.1% of reviews satisfying 9 or more of the 11 criteria; a similar prevalence of poor reviews (22%) was also noted. Multiple linear regression indicated that reviews published in the CDSR (P < 0.01); and involving meta-analysis (β = 0.50, 95% confidence interval 0.72, 2.07, P < 0.001) showed greater concordance with AMSTAR.

Updated method guidelines for cochrane musculoskeletal group systematic reviews and metaanalyses.

The Cochrane Musculoskeletal Group (CMSG), one of 53 groups of the not-for-profit, international Cochrane Collaboration, prepares, maintains, and disseminates systematic reviews of treatments for musculoskeletal diseases. It is important that authors conducting CMSG reviews and the readers of our reviews be aware of and use updated, state-of-the-art systematic review methodology. One hundred sixty reviews have been published. Previous method guidelines for systematic reviews of interventions in the musculoskeletal field published in 2006 have been substantially updated to incorporate methodological advances that are mandatory or highly desirable in Cochrane reviews and knowledge translation advances. The methodological advances include new guidance on searching, new risk-of-bias assessment, grading the quality of the evidence, the new Summary of Findings table, and comparative effectiveness using network metaanalysis. Method guidelines specific to musculoskeletal disorders are provided by CMSG editors for various aspects of undertaking a systematic review. These method guidelines will help improve the quality of reporting and ensure high standards of conduct as well as consistency across CMSG reviews.

Poor Reliability between Cochrane Reviewers and Blinded External Reviewers When Applying the Cochrane Risk of Bias Tool in Physical Therapy Trials

OBJECTIVES To test the inter-rater reliability of the RoB tool applied to Physical Therapy (PT) trials by comparing ratings from Cochrane review authors with those of blinded external reviewers. METHODS Randomized controlled trials (RCTs) in PT were identified by searching the Cochrane Database of Systematic Reviews for meta-analysis of PT interventions. RoB assessments were conducted independently by 2 reviewers blinded to the RoB ratings reported in the Cochrane reviews. Data on RoB assessments from Cochrane reviews and other characteristics of reviews and trials were extracted. Consensus assessments between the two reviewers were then compared with the RoB ratings from the Cochrane reviews. Agreement between Cochrane and blinded external reviewers was assessed using weighted kappa (κ). RESULTS In total, 109 trials included in 17 Cochrane reviews were assessed. Inter-rater reliability on the overall RoB assessment between Cochrane review authors and blinded external reviewers was poor (κ  =  0.02, 95%CI: -0.06, 0.06]). Inter-rater reliability on individual domains of the RoB tool was poor (median κ  = 0.19), ranging from κ  =  -0.04 ("Other bias") to κ  =  0.62 ("Sequence generation"). There was also no agreement (κ  =  -0.29, 95%CI: -0.81, 0.35]) in the overall RoB assessment at the meta-analysis level. CONCLUSIONS Risk of bias assessments of RCTs using the RoB tool are not consistent across different research groups. Poor agreement was not only demonstrated at the trial level but also at the meta-analysis level. Results have implications for decision making since different recommendations can be reached depending on the group analyzing the evidence. Improved guidelines to consistently apply the RoB tool and revisions to the tool for different health areas are needed.

Effect of standardized training on the reliability of the Cochrane risk of bias assessment tool: a study protocol

BACKGROUND The Cochrane risk of bias (RoB) tool has been widely embraced by the systematic review community, but several studies have reported that its reliability is low. We aim to investigate whether training of raters, including objective and standardized instructions on how to assess risk of bias, can improve the reliability of this tool. We describe the methods that will be used in this investigation and present an intensive standardized training package for risk of bias assessment that could be used by contributors to the Cochrane Collaboration and other reviewers. METHODS/DESIGN This is a pilot study. We will first perform a systematic literature review to identify randomized clinical trials (RCTs) that will be used for risk of bias assessment. Using the identified RCTs, we will then do a randomized experiment, where raters will be allocated to two different training schemes: minimal training and intensive standardized training. We will calculate the chance-corrected weighted Kappa with 95% confidence intervals to quantify within- and between-group Kappa agreement for each of the domains of the risk of bias tool. To calculate between-group Kappa agreement, we will use risk of bias assessments from pairs of raters after resolution of disagreements. Between-group Kappa agreement will quantify the agreement between the risk of bias assessment of raters in the training groups and the risk of bias assessment of experienced raters. To compare agreement of raters under different training conditions, we will calculate differences between Kappa values with 95% confidence intervals. DISCUSSION This study will investigate whether the reliability of the risk of bias tool can be improved by training raters using standardized instructions for risk of bias assessment. One group of inexperienced raters will receive intensive training on risk of bias assessment and the other will receive minimal training. By including a control group with minimal training, we will attempt to mimic what many review authors commonly have to do, that is-conduct risk of bias assessment in RCTs without much formal training or standardized instructions. If our results indicate that an intense standardized training does improve the reliability of the RoB tool, our study is likely to help improve the quality of risk of bias assessments, which is a central component of evidence synthesis.

PEDro or Cochrane to Assess the Quality of Clinical Trials? A Meta-Epidemiological Study.

OBJECTIVE There is debate on how the methodological quality of clinical trials should be assessed. We compared trials of physical therapy (PT) judged to be of adequate quality based on summary scores from the Physiotherapy Evidence Database (PEDro) scale with trials judged to be of adequate quality by Cochrane Risk of Bias criteria. DESIGN Meta-epidemiological study within Cochrane Database of Systematic Reviews. METHODS Meta-analyses of PT trials were identified in the Cochrane Database of Systematic Reviews. For each trial PeDro and Cochrane assessments were extracted from the PeDro and Cochrane databases. Adequate quality was defined as adequate generation of random sequence, concealment of allocation, and blinding of outcome assessors (Cochrane criteria) or as trials with a PEDro summary score ≥5 or ≥6 points. We combined trials of adequate quality using random-effects meta-analysis. RESULTS Forty-one Cochrane reviews and 353 PT trials were included. All meta-analyses included trials with PEDro scores ≥5, 37 (90.2%) included trials with PEDro scores ≥6 and only 22 (53.7%) meta-analyses included trials of adequate quality according to the Cochrane criteria. Agreement between PeDro and Cochrane was poor for PeDro scores of ≥5 points (kappa = 0.12; 95% CI 0.07 to 0.16) and slight for ≥6 points (kappa 0.24; 95% CI 0.16-0.32). When combining effect sizes of trials deemed to be of adequate quality according to PEDro or Cochrane criteria, we found that a substantial difference in the combined effect size (≥0.15) was evident in 9 (22%) out of the 41 meta-analyses for PEDro cutoff ≥5 and 10 (24%) for cutoff ≥6. CONCLUSIONS The PeDro and Cochrane approaches lead to different sets of trials of adequate quality, and different combined treatment estimates from meta-analyses of these trials. A consistent approach to assessing RoB in trials of physical therapy should be adopted.

Current State of Reporting Pain Outcomes in Cochrane Reviews of Chronic Musculoskeletal Pain Conditions and Considerations for an OMERACT Research Agenda.

OBJECTIVE To assess the current state of reporting of pain outcomes in Cochrane reviews on chronic musculoskeletal painful conditions and to elicit opinions of patients, healthcare practitioners, and methodologists on presenting pain outcomes to patients, clinicians, and policymakers. METHODS We identified all reviews in the Cochrane Library of chronic musculoskeletal pain conditions from Cochrane review groups (Back, Musculoskeletal, and Pain, Palliative, and Supportive Care) that contained a summary of findings (SoF) table. We extracted data on reported pain domains and instruments and conducted a survey and interviews on considerations for SoF tables (e.g., pain domains, presentation of results). RESULTS Fifty-seven SoF tables in 133 Cochrane reviews were eligible. SoF tables reported pain in 56/57, with all presenting results for pain intensity (20 different outcome instruments), pain interference in 8 SoF tables (5 different outcome instruments), and pain frequency in 1 multiple domain instrument. Other domains like pain quality or pain affect were not reported. From the survey and interviews [response rate 80% (36/45)], we derived 4 themes for a future research agenda: pain domains, considerations for assessing truth, discrimination, and feasibility; clinically important thresholds for responder analyses and presenting results; and establishing hierarchies of outcome instruments. CONCLUSION There is a lack of standardization in the domains of pain selected and the manner that pain outcomes are reported in SoF tables, hampering efforts to synthesize evidence. Future research should focus on the themes identified, building partnerships to achieve consensus and develop guidance on best practices for reporting pain outcomes.

Discrepancies in Outcome Reporting Exist Between Protocols and Published Oral Health Cochrane Systematic Reviews

OBJECTIVES To assess discrepancies in the analyzed outcomes between protocols and published reviews within Cochrane oral health systematic reviews (COHG) on the Cochrane Database of Systematic Reviews (CDSR). STUDY DESIGN AND SETTING All COHG systematic reviews on the CDSR and the corresponding protocols were retrieved in November 2014 and information on the reported outcomes was recorded. Data was collected at the systematic review level by two reviewers independently. RESULTS One hundred and fifty two reviews were included. In relation to primary outcomes, 11.2% were downgraded to secondary outcomes, 9.9% were omitted altogether in the final publication and new primary outcomes were identified in 18.4% of publications. For secondary outcomes, 2% were upgraded to primary, 12.5% were omitted and 30.9% were newly introduced in the publication. Overall, 45.4% of reviews had at least one discrepancy when compared to the protocol; these were reported in 14.5% reviews. The number of review updates appears to be associated with discrepancies between final review and protocol (OR: 3.18, 95% CI: 1.77, 5.74, p<0.001). The risk of reporting significant results was lower for both downgraded outcomes [RR: 0.52, 95% CI: 0.17, 1.58, p = 0.24] and upgraded or newly introduced outcomes [RR: 0.77, 95% CI: 0.36, 1.64, p = 0.50] compared to outcomes with no discrepancies. The risk of reporting significant results was higher for upgraded or newly introduced outcomes compared to downgraded outcomes (RR = 1.19, 95% CI: 0.65, 2.16, p = 0.57). None of the comparisons reached statistical significance. CONCLUSION While no evidence of selective outcome reporting was found in this study, based on the present analysis of SRs published within COHG systematic reviews, discrepancies between outcomes in pre-published protocols and final reviews continue to be common. Solutions such as the use of standardized outcomes to reduce the prevalence of this issue may need to be explored.

Reporting and handling missing outcome data in mental health: a systematic review of Cochrane systematic reviews and meta-analyses

OBJECTIVES The purpose of the study was to provide empirical evidence about the reporting of methodology to address missing outcome data and the acknowledgement of their impact in Cochrane systematic reviews in the mental health field. METHODS Systematic reviews published in the Cochrane Database of Systematic Reviews after January 1, 2009 by three Cochrane Review Groups relating to mental health were included. RESULTS One hundred ninety systematic reviews were considered. Missing outcome data were present in at least one included study in 175 systematic reviews. Of these 175 systematic reviews, 147 (84%) accounted for missing outcome data by considering a relevant primary or secondary outcome (e.g., dropout). Missing outcome data implications were reported only in 61 (35%) systematic reviews and primarily in the discussion section by commenting on the amount of the missing outcome data. One hundred forty eligible meta-analyses with missing data were scrutinized. Seventy-nine (56%) of them had studies with total dropout rate between 10 and 30%. One hundred nine (78%) meta-analyses reported to have performed intention-to-treat analysis by including trials with imputed outcome data. Sensitivity analysis for incomplete outcome data was implemented in less than 20% of the meta-analyses. CONCLUSIONS Reporting of the techniques for handling missing outcome data and their implications in the findings of the systematic reviews are suboptimal.

Combined modality treatment improves tumor control and overall survival in patients with early stage Hodgkin's lymphoma: a systematic review

Combined modality treatment (CMT) of chemotherapy followed by localized radiotherapy is standard treatment for patients with early stage Hodgkin's lymphoma. However, the role of radiotherapy has been questioned recently and some clinical study groups advocate chemotherapy only for this indication. We thus performed a systematic review with meta-analysis of randomized controlled trials comparing chemotherapy alone with CMT in patients with early stage Hodgkin's lymphoma with respect to response rate, tumor control and overall survival (OS). We searched Medline, EMBASE and the Cochrane Library as well as conference proceedings from January 1980 to February 2009 for randomized controlled trials comparing chemotherapy alone versus the same chemotherapy regimen plus radiotherapy. Progression free survival and similar outcomes were analyzed together as tumor control. Effect measures used were hazard ratios for OS and tumor control as well as relative risks for complete response (CR). Meta-analyses were performed using RevMan5. Five randomized controlled trials involving 1,245 patients were included. The hazard ratio (HR) was 0.41 (95% confidence interval (CI) 0.25 to 0.66) for tumor control and 0.40 (95% CI 0.27 to 0.59) for OS for patients receiving CMT compared to chemotherapy alone. CR rates were similar between treatment groups. In sensitivity analyses another 6 trials were included that did not fulfill the inclusion criteria of our protocol but were considered relevant to the topic. These trials underlined the results of the main analysis. In conclusion, adding radiotherapy to chemotherapy improves tumor control and OS in patients with early stage Hodgkin's lymphoma.

Hand and ultrasonic instrumentation for orthograde root canal treatment of permanent teeth

Root canal treatment is a frequently performed dental procedure and is carried out on teeth in which irreversible pulpitis has led to necrosis of the dental pulp. Removal of the necrotic tissue remnants and cleaning and shaping of the root canal are important phases of root canal treatment. Treatment options include the use of hand and rotary instruments and methods using ultrasonic or sonic equipment. OBJECTIVES: The objectives of this systematic review of randomized controlled trials were to determine the relative clinical effectiveness of hand instrumentation versus ultrasonic instrumentation alone or in conjunction with hand instrumentation for orthograde root canal treatment of permanent teeth. MATERIAL AND METHODS: The search strategy retrieved 226 references from the Cochrane Oral Health Group Trials Register (7), the Cochrane Central Register of Controlled Trials (CENTRAL) (12), MEDLINE (192), EMBASE (8) and LILACS (7). No language restriction was applied. The last electronic search was conducted on December 13th, 2007. Screening of eligible studies was conducted in duplicate and independently. RESULTS: Results were to be expressed as fixed-effect or random-effects models using mean differences for continuous outcomes and risk ratios for dichotomous outcomes with 95% confidence intervals. Heterogeneity was to be investigated including both clinical and methodological factors. No eligible randomized controlled trials were identified. CONCLUSIONS: This review illustrates the current lack of published or ongoing randomized controlled trials and the unavailability of high-level evidence based on clinically relevant outcomes referring to the effectiveness of ultrasonic instrumentation used alone or as an adjunct to hand instrumentation for orthograde root canal treatment. In the absence of reliable research-based evidence, clinicians should base their decisions on clinical experience, individual circumstances and in conjunction with patients' preferences where appropriate. Future randomized controlled trials might focus more closely on evaluating the effectiveness of combinations of these interventions with an emphasis on not only clinically relevant, but also patient-centered outcomes.

Orthodontic therapy and gingival recession: a systematic review

To perform a systematic review on the effect of changes in incisor inclination owing to orthodontic treatment and the occurrence of gingival recession. PubMed, EMBASE Excerpta Medica and CENTRAL of the Cochrane Library were searched and a hand search was performed. From 1925 articles identified, 17 articles were finally included: six experimental animal studies and 11 retrospective clinical studies in humans. More proclined teeth compared with less proclined teeth or untreated teeth had in most studies a higher occurrence or severity of gingival recession. Contradictory results were found regarding a possible statistically significant correlation between the extent of gingival recession and the amount of incisor proclination during treatment, width of attached gingiva, hygiene, periodontal condition or thickness of the symphysis. There are no high quality animal or clinical studies on this topic. Movement of the incisors out of the osseous envelope of the alveolar process may be associated with a higher tendency for developing gingival recessions. The amount of recession found in studies with statistically significant differences between proclined and non-proclined incisors is small and the clinical consequence questionable. Because of the low level of evidence of the included studies, the results should be considered with caution. Further randomized clinical studies including clinical examination of hygiene and gingival condition before, during and after treatment are needed to clarify the effect of orthodontic changes in incisor inclination and the occurrence of gingival recession.

Soft tissue profile changes after bilateral sagittal split osteotomy for mandibular setback: a systematic review

PURPOSE: To evaluate the ratio of soft tissue to hard tissue in bilateral sagittal split setback osteotomy with rigid internal fixation or wire fixation. MATERIALS AND METHODS: A literature search was performed using PubMed, Medline, CINAHL, Web of Science, the Cochrane Library, and Google Scholar Beta. From the original 766 articles identified, 8 articles were included. Two articles were prospective and 6 retrospective. The follow-up period ranged from 1 year to 12.7 years for rigid internal fixation. Two articles on wire fixation were found to be appropriate for inclusion. RESULTS: The differences between short- and long-term ratios of the lower lip to lower incisors for bilateral sagittal split setback osteotomy with rigid internal fixation or wire fixation were quite small. The ratio was 1:1 in the long term and by trend slightly lower in the short term. No distinction was seen between the short- and long-term ratios for mentolabial fold. The ratio was found to be 1:1 for the mentolabial fold to point B. In the short term, the ratio of the soft tissue pogonion to the pogonion showed a 1:1 ratio, with a trend to be lower in the long term. The upper lip showed mainly protrusion, but the amount was highly variable. CONCLUSIONS: This systematic review shows that evidence-based conclusions on soft tissue changes are difficult to draw. This is mostly because of inherent problems of retrospective studies, inferior study designs, and the lack of standardized outcome measurements. Well-designed prospective studies with sufficient samples and excluding additional surgery, ie, genioplasty or maxillary surgery, are needed.

Soft tissue profile changes after bilateral sagittal split osteotomy for mandibular advancement: a systematic review

PURPOSE: The purpose of the present systematic review was to evaluate the soft tissue/hard tissue ratio in bilateral sagittal split advancement osteotomy (BSSO) with rigid internal fixation (RIF) or wire fixation (WF). MATERIALS AND METHODS: The databases PubMed, Medline, CINAHL, Web of Science, Cochrane Library, and Google Scholar Beta were searched. From the original 711 articles identified, 12 were finally included. Only 3 studies were prospective and 9 were retrospective. The postoperative follow-up ranged from 3 months to 12.7 years for RIF and 6 months to 5 years for WF. RESULTS: The short- and long-term ratios for the lower lip to lower incisor for BSSO with RIF or WF were 50%. No difference between the short- and long-term ratios for the mentolabial-fold to point B and soft tissue pogonion to pogonion could be observed. It was a 1:1 ratio. One exception was seen for the long-term results of the soft tissue pogonion to pogonion in BSSO with RIF; they tended to be greater than a 1:1 ratio. The upper lip mainly showed retrusion but with high variability. CONCLUSIONS: Despite a large number of studies on the short- and long-term effects of mandibular advancement by BSSO, the results of the present systematic review have shown that evidence-based conclusions on soft tissue changes are still unknown. This is mostly because of the inherent problems of retrospective studies, inferior study designs, and the lack of standardized outcome measures. Well-designed prospective studies with sufficient sample sizes that have excluded patients undergoing additional surgery (ie, genioplasty or maxillary surgery) are needed.