Can decision making in general surgery be based on evidence? An empirical study of Cochrane Reviews

BACKGROUND: This empirical study analyzes the current status of Cochrane Reviews (CRs) and their strength of recommendation for evidence-based decision making in the field of general surgery. METHODS: Systematic literature search of the Cochrane Database of Systematic Reviews and the Cochrane Collaboration's homepage to identify available CRs on surgical topics. Quantitative and qualitative characteristics, utilization, and formulated treatment recommendations were evaluated by 2 independent reviewers. Association of review characteristics with treatment recommendation was analyzed using univariate and multivariate logistic regression models. RESULTS: Ninety-three CRs, including 1,403 primary studies and 246,473 patients, were identified. Mean number of included primary studies per CR was 15.1 (standard deviation [SD] 14.5) including 2,650 (SD 3,340) study patients. Two and a half (SD 8.3) nonrandomized trials were included per analyzed CR. Seventy-two (77%) CRs were published or updated in 2005 or later. Explicit treatment recommendations were given in 45 (48%). Presence of a treatment recommendation was associated with the number of included primary studies and the proportion of randomized studies. Utilization of surgical CRs remained low and showed large inter-country differences. The most surgical CRs were accessed in UK, USA, and Australia, followed by several Western and Eastern European countries. CONCLUSION: Only a minority of available CRs address surgical questions and their current usage is low. Instead of unsystematically increasing the number of surgical CRs it would be far more efficient to focus the review process on relevant surgical questions. Prioritization of CRs needs valid methods which should be developed by the scientific surgical community.

Current State of Reporting Pain Outcomes in Cochrane Reviews of Chronic Musculoskeletal Pain Conditions and Considerations for an OMERACT Research Agenda.

OBJECTIVE To assess the current state of reporting of pain outcomes in Cochrane reviews on chronic musculoskeletal painful conditions and to elicit opinions of patients, healthcare practitioners, and methodologists on presenting pain outcomes to patients, clinicians, and policymakers. METHODS We identified all reviews in the Cochrane Library of chronic musculoskeletal pain conditions from Cochrane review groups (Back, Musculoskeletal, and Pain, Palliative, and Supportive Care) that contained a summary of findings (SoF) table. We extracted data on reported pain domains and instruments and conducted a survey and interviews on considerations for SoF tables (e.g., pain domains, presentation of results). RESULTS Fifty-seven SoF tables in 133 Cochrane reviews were eligible. SoF tables reported pain in 56/57, with all presenting results for pain intensity (20 different outcome instruments), pain interference in 8 SoF tables (5 different outcome instruments), and pain frequency in 1 multiple domain instrument. Other domains like pain quality or pain affect were not reported. From the survey and interviews [response rate 80% (36/45)], we derived 4 themes for a future research agenda: pain domains, considerations for assessing truth, discrimination, and feasibility; clinically important thresholds for responder analyses and presenting results; and establishing hierarchies of outcome instruments. CONCLUSION There is a lack of standardization in the domains of pain selected and the manner that pain outcomes are reported in SoF tables, hampering efforts to synthesize evidence. Future research should focus on the themes identified, building partnerships to achieve consensus and develop guidance on best practices for reporting pain outcomes.

Outcome Measures in Glaucoma : a systematic review of Cochrane Reviews and Protocols

Peer reviewed

Outcome Measures in Glaucoma : a systematic review of Cochrane Reviews and Protocols

Peer reviewed

Outcome Measures in Glaucoma : a systematic review of Cochrane Reviews and Protocols

Peer reviewed

Updated method guidelines for cochrane musculoskeletal group systematic reviews and metaanalyses.

The Cochrane Musculoskeletal Group (CMSG), one of 53 groups of the not-for-profit, international Cochrane Collaboration, prepares, maintains, and disseminates systematic reviews of treatments for musculoskeletal diseases. It is important that authors conducting CMSG reviews and the readers of our reviews be aware of and use updated, state-of-the-art systematic review methodology. One hundred sixty reviews have been published. Previous method guidelines for systematic reviews of interventions in the musculoskeletal field published in 2006 have been substantially updated to incorporate methodological advances that are mandatory or highly desirable in Cochrane reviews and knowledge translation advances. The methodological advances include new guidance on searching, new risk-of-bias assessment, grading the quality of the evidence, the new Summary of Findings table, and comparative effectiveness using network metaanalysis. Method guidelines specific to musculoskeletal disorders are provided by CMSG editors for various aspects of undertaking a systematic review. These method guidelines will help improve the quality of reporting and ensure high standards of conduct as well as consistency across CMSG reviews.

Feasibility of a knowledge translation CME program: courriels Cochrane.

INTRODUCTION: Systematic literature reviews provide best evidence, but are underused by clinicians. Thus, integrating Cochrane reviews into continuing medical education (CME) is challenging. We designed a pilot CME program where summaries of Cochrane reviews (Courriels Cochrane) were disseminated by e-mail. Program participants automatically received CME credit for each Courriel Cochrane they rated. The feasibility of this program is reported (delivery, participation, and participant evaluation). METHOD: We recruited French-speaking physicians through the Canadian Medical Association. Program delivery and participation were documented. Participants rated the informational value of Courriels Cochrane using the Information Assessment Method (IAM), which documented their reflective learning (relevance, cognitive impact, use for a patient, expected health benefits). IAM responses were aggregated and analyzed. RESULTS: The program was delivered as planned. Thirty Courriels Cochrane were delivered to 985 physicians, and 127 (12.9%) completed at least one IAM questionnaire. Out of 1109 Courriels Cochrane ratings, 973 (87.7%) conta-ined 1 or more types of positive cognitive impact, while 835 (75.3%) were clinically relevant. Participants reported the use of information for a patient and expected health benefits in 595 (53.7%) and 569 (51.3%) ratings, respectively. DISCUSSION: Program delivery required partnering with 5 organizations. Participants valued Courriels Cochrane. IAM ratings documented their reflective learning. The aggregation of IAM ratings documented 3 levels of CME outcomes: participation, learning, and performance. This evaluation study demonstrates the feasibility of the Courriels Cochrane as an approach to further disseminate Cochrane systematic literature reviews to clinicians and document self-reported knowledge translation associated with Cochrane reviews.

Systematic reviews showed insufficient evidence for clinical practice in 2004: What about in 2011? the next appeal for the evidence-based medicine age

Rationale and aim The aims of the Cochrane systematic reviews are to make readily available and up-to-date information for clinical practice, offering consistent evidence and straightforward recommendations. In 2004, we evaluated the conclusions from Cochrane systematic reviews of randomized controlled trials in terms of their recommendations for clinical practice and found that 47.83% of them had insufficient evidence for use in clinical practice. We proposed to reanalyze the reviews to evaluate whether this percentage had significantly decreased. Methods A cross-sectional study of systematic reviews published in the Cochrane Library (Issue 7, 2011) was conducted. We randomly selected reviews across all 52 Cochrane Collaborative Review Groups. Results We analyzed 1128 completed systematic reviews. Of these, 45.30% concluded that the interventions studied were likely to be beneficial, of which only 2.04% recommended no further research. In total, 45.04% of the reviews reported that the evidence did not support either benefit or harm, of which 0.8% did not recommend further studies and 44.24% recommended additional studies; the latter has decreased from our previous study with a difference of 3.59%. Conclusion Only a small number of the Cochrane collaboration's systematic reviews support clinical interventions with no need for additional research. A larger number of high-quality randomized clinical trials are necessary to change the 'insufficient evidence' scenario for clinical practice illustrated by the Cochrane database. It is recommended that we should produce higher-quality primary studies in active collaboration and consultation with global scholars and societies so that this can represent a major component of methodological advance in this context. © 2012 John Wiley & Sons Ltd.

A PRISMA assessment of the reporting quality of systematic reviews in orthodontics

Abstract Objectives: To assess the reporting quality of Cochrane and non-Cochrane systematic reviews (SR) in orthodontics and to compare the reporting quality (PRISMA score) with methodological quality (AMSTAR criteria). Materials and Methods: Systematic reviews (n  =  109) published between January 2000 and July 2011 in five leading orthodontic journals were identified and included. The quality of reporting of the included reviews was assessed by two authors in accordance with the PRISMA guidelines. Each article was assigned a cumulative grade based on fulfillment of the applicable criteria, and an overall percentage score was assigned. Descriptive statistics and simple and multiple linear regression analyses were undertaken. Results: The mean overall PRISMA score was 64.1% (95% confidence interval [CI], 62%-65%). The quality of reporting was considerably better in reviews published in the Cochrane Database of Systematic Reviews (P < .001) than in non-Cochrane reviews. Both multivariable and univariable analysis indicated that journal of publication and number of authors was significantly associated with the PRISMA score. The association between AMSTAR score and modified PRISMA score was also found to be highly statistically significant. Conclusion: Compliance of orthodontic SRs published in orthodontic journals with PRISMA guidelines was deficient in several areas. The quality of reporting assessed using PRISMA guidelines was significantly better in orthodontic SRs published in the Cochrane Database of Systematic Reviews.

Reporting completeness of abstracts of systematic reviews published in leading dental specialty journals

The aim of this study was to investigate the reporting completeness of systematic review (SR) abstracts in leading dental specialty journals. Electronic and supplementary hand searching were undertaken to identify SRs published in seven dental specialty journals and in the Cochrane Database of Systematic Reviews. Abstract reporting completeness was evaluated using a checklist derived from the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (prisma) guidelines. Descriptive statistics followed by univariate and multivariate analyses were conducted. Two-hundred and eighteen SR abstracts were identified. Reporting of interventions (94%), objectives (96%), data sources (81%), eligibility criteria (77%), and conclusions (97%) was adequate in the majority of reviews. However, inadequate reporting of participants (18%), results (42%), effect size (14%), level of significance (60%), and trial registration (100%) was commonplace. The mean overall reporting score was 79.1% (95% CI, 77.6-80.6). Only journal of publication was a significant predictor of overall reporting, with inferior results for all journals relative to Cochrane reviews, with scores ranging from -4.3% (95% CI, -8.74 to 0.08) to -35.6% (95% CI, -42.0 to -24.3) for the International Journal of Prosthodontics and the British Journal of Oral and Maxillofacial Surgery, respectively. Improved reporting of dental SR abstracts is needed and should be encouraged, as these abstracts may underpin influential clinical decisions.

Poor Reliability between Cochrane Reviewers and Blinded External Reviewers When Applying the Cochrane Risk of Bias Tool in Physical Therapy Trials

OBJECTIVES To test the inter-rater reliability of the RoB tool applied to Physical Therapy (PT) trials by comparing ratings from Cochrane review authors with those of blinded external reviewers. METHODS Randomized controlled trials (RCTs) in PT were identified by searching the Cochrane Database of Systematic Reviews for meta-analysis of PT interventions. RoB assessments were conducted independently by 2 reviewers blinded to the RoB ratings reported in the Cochrane reviews. Data on RoB assessments from Cochrane reviews and other characteristics of reviews and trials were extracted. Consensus assessments between the two reviewers were then compared with the RoB ratings from the Cochrane reviews. Agreement between Cochrane and blinded external reviewers was assessed using weighted kappa (κ). RESULTS In total, 109 trials included in 17 Cochrane reviews were assessed. Inter-rater reliability on the overall RoB assessment between Cochrane review authors and blinded external reviewers was poor (κ  =  0.02, 95%CI: -0.06, 0.06]). Inter-rater reliability on individual domains of the RoB tool was poor (median κ  = 0.19), ranging from κ  =  -0.04 ("Other bias") to κ  =  0.62 ("Sequence generation"). There was also no agreement (κ  =  -0.29, 95%CI: -0.81, 0.35]) in the overall RoB assessment at the meta-analysis level. CONCLUSIONS Risk of bias assessments of RCTs using the RoB tool are not consistent across different research groups. Poor agreement was not only demonstrated at the trial level but also at the meta-analysis level. Results have implications for decision making since different recommendations can be reached depending on the group analyzing the evidence. Improved guidelines to consistently apply the RoB tool and revisions to the tool for different health areas are needed.

PEDro or Cochrane to Assess the Quality of Clinical Trials? A Meta-Epidemiological Study.

OBJECTIVE There is debate on how the methodological quality of clinical trials should be assessed. We compared trials of physical therapy (PT) judged to be of adequate quality based on summary scores from the Physiotherapy Evidence Database (PEDro) scale with trials judged to be of adequate quality by Cochrane Risk of Bias criteria. DESIGN Meta-epidemiological study within Cochrane Database of Systematic Reviews. METHODS Meta-analyses of PT trials were identified in the Cochrane Database of Systematic Reviews. For each trial PeDro and Cochrane assessments were extracted from the PeDro and Cochrane databases. Adequate quality was defined as adequate generation of random sequence, concealment of allocation, and blinding of outcome assessors (Cochrane criteria) or as trials with a PEDro summary score ≥5 or ≥6 points. We combined trials of adequate quality using random-effects meta-analysis. RESULTS Forty-one Cochrane reviews and 353 PT trials were included. All meta-analyses included trials with PEDro scores ≥5, 37 (90.2%) included trials with PEDro scores ≥6 and only 22 (53.7%) meta-analyses included trials of adequate quality according to the Cochrane criteria. Agreement between PeDro and Cochrane was poor for PeDro scores of ≥5 points (kappa = 0.12; 95% CI 0.07 to 0.16) and slight for ≥6 points (kappa 0.24; 95% CI 0.16-0.32). When combining effect sizes of trials deemed to be of adequate quality according to PEDro or Cochrane criteria, we found that a substantial difference in the combined effect size (≥0.15) was evident in 9 (22%) out of the 41 meta-analyses for PEDro cutoff ≥5 and 10 (24%) for cutoff ≥6. CONCLUSIONS The PeDro and Cochrane approaches lead to different sets of trials of adequate quality, and different combined treatment estimates from meta-analyses of these trials. A consistent approach to assessing RoB in trials of physical therapy should be adopted.

The Quality of the Evidence According to GRADE Is Predominantly Low or Very Low in Oral Health Systematic Reviews

OBJECTIVES The main objective was to assess the credibility of the evidence using Grades of Recommendation, Assessment, Development, and Evaluation (GRADE) in oral health systematic reviews on the Cochrane Database of Systematic Reviews (CDSR) and elsewhere. STUDY DESIGN AND SETTING Systematic Reviews or meta-analyses (January 2008-December 2013) from 14 high impact general dental and specialty dental journals and the Cochrane Database of Systematic Reviews were screened for meta-analyses. Data was collected at the systematic review, meta-analysis and trial level. Two reviewers applied and agreed on the GRADE rating for the selected meta-analyses. RESULTS From the 510 systematic reviews initially identified 91 reviews (41 Cochrane and 50 non-Cochrane) were eligible for inclusion. The quality of evidence was high in 2% and moderate in 18% of the included meta-analyses with no difference between Cochrane and non-Cochrane reviews, journal impact factor or year of publication. The most common domains prompting downgrading of the evidence were study limitations (risk of bias) and imprecision (risk of play of chance). CONCLUSION The quality of the evidence in oral health assessed using GRADE is predominantly low or very low suggesting a pressing need for more randomised clinical trials and other studies of higher quality in order to inform clinical decisions thereby reducing the risk of instituting potentially ineffective and/or harmful therapies.

Cautionary tales in the interpretation of systematic reviews of therapy trials

This is the second in a series of articles emphasizing the cautions in the interpretation of health-care studies. Systematic reviews are presented as comprehensive, unbiased summaries of evidence and are often referred to by clinicians, guideline developers and health policy-makers. Their strengths and limitations, and how their results can be subject to bias and misinterpretation, are discussed.