Discrepancies in Outcome Reporting Exist Between Protocols and Published Oral Health Cochrane Systematic Reviews

OBJECTIVES To assess discrepancies in the analyzed outcomes between protocols and published reviews within Cochrane oral health systematic reviews (COHG) on the Cochrane Database of Systematic Reviews (CDSR). STUDY DESIGN AND SETTING All COHG systematic reviews on the CDSR and the corresponding protocols were retrieved in November 2014 and information on the reported outcomes was recorded. Data was collected at the systematic review level by two reviewers independently. RESULTS One hundred and fifty two reviews were included. In relation to primary outcomes, 11.2% were downgraded to secondary outcomes, 9.9% were omitted altogether in the final publication and new primary outcomes were identified in 18.4% of publications. For secondary outcomes, 2% were upgraded to primary, 12.5% were omitted and 30.9% were newly introduced in the publication. Overall, 45.4% of reviews had at least one discrepancy when compared to the protocol; these were reported in 14.5% reviews. The number of review updates appears to be associated with discrepancies between final review and protocol (OR: 3.18, 95% CI: 1.77, 5.74, p<0.001). The risk of reporting significant results was lower for both downgraded outcomes [RR: 0.52, 95% CI: 0.17, 1.58, p = 0.24] and upgraded or newly introduced outcomes [RR: 0.77, 95% CI: 0.36, 1.64, p = 0.50] compared to outcomes with no discrepancies. The risk of reporting significant results was higher for upgraded or newly introduced outcomes compared to downgraded outcomes (RR = 1.19, 95% CI: 0.65, 2.16, p = 0.57). None of the comparisons reached statistical significance. CONCLUSION While no evidence of selective outcome reporting was found in this study, based on the present analysis of SRs published within COHG systematic reviews, discrepancies between outcomes in pre-published protocols and final reviews continue to be common. Solutions such as the use of standardized outcomes to reduce the prevalence of this issue may need to be explored.

The Quality of the Evidence According to GRADE Is Predominantly Low or Very Low in Oral Health Systematic Reviews

OBJECTIVES The main objective was to assess the credibility of the evidence using Grades of Recommendation, Assessment, Development, and Evaluation (GRADE) in oral health systematic reviews on the Cochrane Database of Systematic Reviews (CDSR) and elsewhere. STUDY DESIGN AND SETTING Systematic Reviews or meta-analyses (January 2008-December 2013) from 14 high impact general dental and specialty dental journals and the Cochrane Database of Systematic Reviews were screened for meta-analyses. Data was collected at the systematic review, meta-analysis and trial level. Two reviewers applied and agreed on the GRADE rating for the selected meta-analyses. RESULTS From the 510 systematic reviews initially identified 91 reviews (41 Cochrane and 50 non-Cochrane) were eligible for inclusion. The quality of evidence was high in 2% and moderate in 18% of the included meta-analyses with no difference between Cochrane and non-Cochrane reviews, journal impact factor or year of publication. The most common domains prompting downgrading of the evidence were study limitations (risk of bias) and imprecision (risk of play of chance). CONCLUSION The quality of the evidence in oral health assessed using GRADE is predominantly low or very low suggesting a pressing need for more randomised clinical trials and other studies of higher quality in order to inform clinical decisions thereby reducing the risk of instituting potentially ineffective and/or harmful therapies.

Use of quality assessment tools in systematic reviews was varied and inconsistent

OBJECTIVES To assess the use of quality assessment tools among a cross-section of systematic reviews (SRs) and to further evaluate whether quality was used as a parameter in the decision to include primary studies within subsequent meta-analysis. STUDY DESIGN AND SETTING We searched PubMed for SRs (interventional, observational, and diagnostic) published in Core Clinical Journals between January 1 and March 31, 2014. RESULTS Three hundred nine SRs were identified. Quality assessment was undertaken in 222 (71.8%) with isolated use of the Cochrane risk of bias tool (26.1%, n = 58) and the Newcastle-Ottawa Scale (15.3%, n = 34) most common. A threshold level of primary study quality for subsequent meta-analysis was used in 12.9% (40 of 309) of reviews. Overall, fifty-four combinations of quality assessment tools were identified with a similar preponderance of tools used among observational and interventional reviews. Multiple tools were used in 11.7% (n = 36) of SRs overall. CONCLUSION We found that quality assessment tools were used in a majority of SRs; however, a threshold level of quality for meta-analysis was stipulated in just 12.9% (n = 40). This cross-sectional analysis provides further evidence of the need for more active or intuitive editorial processes to enhance the reporting of SRs.

Reporting and handling missing outcome data in mental health: a systematic review of Cochrane systematic reviews and meta-analyses

OBJECTIVES The purpose of the study was to provide empirical evidence about the reporting of methodology to address missing outcome data and the acknowledgement of their impact in Cochrane systematic reviews in the mental health field. METHODS Systematic reviews published in the Cochrane Database of Systematic Reviews after January 1, 2009 by three Cochrane Review Groups relating to mental health were included. RESULTS One hundred ninety systematic reviews were considered. Missing outcome data were present in at least one included study in 175 systematic reviews. Of these 175 systematic reviews, 147 (84%) accounted for missing outcome data by considering a relevant primary or secondary outcome (e.g., dropout). Missing outcome data implications were reported only in 61 (35%) systematic reviews and primarily in the discussion section by commenting on the amount of the missing outcome data. One hundred forty eligible meta-analyses with missing data were scrutinized. Seventy-nine (56%) of them had studies with total dropout rate between 10 and 30%. One hundred nine (78%) meta-analyses reported to have performed intention-to-treat analysis by including trials with imputed outcome data. Sensitivity analysis for incomplete outcome data was implemented in less than 20% of the meta-analyses. CONCLUSIONS Reporting of the techniques for handling missing outcome data and their implications in the findings of the systematic reviews are suboptimal.

The evidence from systematic reviews and meta-analyses published in orthodontic literature. Where do we stand?

AIM To analyse meta-analyses included in systematic reviews (SRs) published in leading orthodontic journals and the Cochrane Database of Systematic Reviews (CDSR) focusing on orthodontic literature and to assess the quality of the existing evidence. MATERIALS AND METHODS Electronic searching was undertaken to identify SRs published in five major orthodontic journals and the CDSR between January 2000 and June 2014. Quality assessment of the overall body of evidence from meta-analyses was conducted using the Grading of Recommendations Assessment, Development and Evaluation working group (GRADE) tool. RESULTS One hundred and fifty-seven SRs were identified; meta-analysis was present in 43 of these (27.4 per cent). The highest proportion of SRs that included a meta-analysis was found in Orthodontics and Craniofacial Research (6/13; 46.1 per cent), followed by the CDSR (12/33; 36.4 per cent) and the American Journal of Orthodontics and Dentofacial Orthopaedics (15/44; 34.1 per cent). Class II treatment was the most commonly addressed topic within SRs in orthodontics (n = 18/157; 11.5 per cent). The number of trials combined to produce a summary estimate was small for most meta-analyses with a median of 4 (range: 2-52). Only 21 per cent (n = 9) of included meta-analyses were considered to have a high/moderate quality of evidence according to GRADE, while the majority were of low or very low quality (n = 34; 79.0 per cent). CONCLUSIONS Overall, approximately one quarter of orthodontic SRs included quantitative synthesis, with a median of four trials per meta-analysis. The overall quality of evidence from the selected orthodontic SRs was predominantly low to very low indicating the relative lack of high quality of evidence from SRs to inform clinical practice guidelines.