Cochrane and non-Cochrane systematic reviews in leading orthodontic journals: a quality paradigm?

The aims of this study were to assess and compare the methodological quality of Cochrane and non-Cochrane systematic reviews (SRs) published in leading orthodontic journals and the Cochrane Database of Systematic Reviews (CDSR) using AMSTAR and to compare the prevalence of meta-analysis in both review types. A literature search was undertaken to identify SRs that consisted of hand-searching five major orthodontic journals [American Journal of Orthodontics and Dentofacial Orthopedics, Angle Orthodontist, European Journal of Orthodontics, Journal of Orthodontics and Orthodontics and Craniofacial Research (February 2002 to July 2011)] and the Cochrane Database of Systematic Reviews from January 2000 to July 2011. Methodological quality of the included reviews was gauged using the AMSTAR tool involving 11 key methodological criteria with a score of 0 or 1 given for each criterion. A cumulative grade was given for the paper overall (0-11); an overall score of 4 or less represented poor methodological quality, 5-8 was considered fair and 9 or greater was deemed to be good. In total, 109 SRs were identified in the five major journals and on the CDSR. Of these, 26 (23.9%) were in the CDSR. The mean overall AMSTAR score was 6.2 with 21.1% of reviews satisfying 9 or more of the 11 criteria; a similar prevalence of poor reviews (22%) was also noted. Multiple linear regression indicated that reviews published in the CDSR (P < 0.01); and involving meta-analysis (β = 0.50, 95% confidence interval 0.72, 2.07, P < 0.001) showed greater concordance with AMSTAR.

Discrepancies in Outcome Reporting Exist Between Protocols and Published Oral Health Cochrane Systematic Reviews

OBJECTIVES To assess discrepancies in the analyzed outcomes between protocols and published reviews within Cochrane oral health systematic reviews (COHG) on the Cochrane Database of Systematic Reviews (CDSR). STUDY DESIGN AND SETTING All COHG systematic reviews on the CDSR and the corresponding protocols were retrieved in November 2014 and information on the reported outcomes was recorded. Data was collected at the systematic review level by two reviewers independently. RESULTS One hundred and fifty two reviews were included. In relation to primary outcomes, 11.2% were downgraded to secondary outcomes, 9.9% were omitted altogether in the final publication and new primary outcomes were identified in 18.4% of publications. For secondary outcomes, 2% were upgraded to primary, 12.5% were omitted and 30.9% were newly introduced in the publication. Overall, 45.4% of reviews had at least one discrepancy when compared to the protocol; these were reported in 14.5% reviews. The number of review updates appears to be associated with discrepancies between final review and protocol (OR: 3.18, 95% CI: 1.77, 5.74, p<0.001). The risk of reporting significant results was lower for both downgraded outcomes [RR: 0.52, 95% CI: 0.17, 1.58, p = 0.24] and upgraded or newly introduced outcomes [RR: 0.77, 95% CI: 0.36, 1.64, p = 0.50] compared to outcomes with no discrepancies. The risk of reporting significant results was higher for upgraded or newly introduced outcomes compared to downgraded outcomes (RR = 1.19, 95% CI: 0.65, 2.16, p = 0.57). None of the comparisons reached statistical significance. CONCLUSION While no evidence of selective outcome reporting was found in this study, based on the present analysis of SRs published within COHG systematic reviews, discrepancies between outcomes in pre-published protocols and final reviews continue to be common. Solutions such as the use of standardized outcomes to reduce the prevalence of this issue may need to be explored.

Reporting and handling missing outcome data in mental health: a systematic review of Cochrane systematic reviews and meta-analyses

OBJECTIVES The purpose of the study was to provide empirical evidence about the reporting of methodology to address missing outcome data and the acknowledgement of their impact in Cochrane systematic reviews in the mental health field. METHODS Systematic reviews published in the Cochrane Database of Systematic Reviews after January 1, 2009 by three Cochrane Review Groups relating to mental health were included. RESULTS One hundred ninety systematic reviews were considered. Missing outcome data were present in at least one included study in 175 systematic reviews. Of these 175 systematic reviews, 147 (84%) accounted for missing outcome data by considering a relevant primary or secondary outcome (e.g., dropout). Missing outcome data implications were reported only in 61 (35%) systematic reviews and primarily in the discussion section by commenting on the amount of the missing outcome data. One hundred forty eligible meta-analyses with missing data were scrutinized. Seventy-nine (56%) of them had studies with total dropout rate between 10 and 30%. One hundred nine (78%) meta-analyses reported to have performed intention-to-treat analysis by including trials with imputed outcome data. Sensitivity analysis for incomplete outcome data was implemented in less than 20% of the meta-analyses. CONCLUSIONS Reporting of the techniques for handling missing outcome data and their implications in the findings of the systematic reviews are suboptimal.

Which components of heart failure programs are effective? A meta-analysis of the outcomes of structured telephone support or telemonitoring as the primary component of heart failure management in 9,560 patients (Abridged Cochrane Review)

Aims--Telemonitoring (TM) and structured telephone support (STS) have the potential to deliver specialised management to more patients with chronic heart failure (CHF), but their efficacy is still to be proven. Objectives To review randomised controlled trials (RCTs) of TM or STS on all- cause mortality and all-cause and CHF-related hospitalisations in patients with CHF, as a non-invasive remote model of specialised disease-management intervention.--Methods and Results--Data sources:We searched 15 electronic databases and hand-searched bibliographies of relevant studies, systematic reviews, and meeting abstracts. Two reviewers independently extracted all data. Study eligibility and participants: We included any randomised controlled trials (RCT) comparing TM or STS to usual care of patients with CHF. Studies that included intensified management with additional home or clinic visits were excluded. Synthesis: Primary outcomes (mortality and hospitalisations) were analysed; secondary outcomes (cost, length of stay, quality of life) were tabulated.--Results: Thirty RCTs of STS and TM were identified (25 peer-reviewed publications (n=8,323) and five abstracts (n=1,482)). Of the 25 peer-reviewed studies, 11 evaluated TM (2,710 participants), 16 evaluated STS (5,613 participants) and two tested both interventions. TM reduced all-cause mortality (risk ratio (RR 0•66 [95% CI 0•54-0•81], p<0•0001) and STS showed similar trends (RR 0•88 [95% CI 0•76-1•01], p=0•08). Both TM (RR 0•79 [95% CI 0•67-0•94], p=0•008) and STS (RR 0•77 [95% CI 0•68-0•87], p<0•0001) reduced CHF-related hospitalisations. Both interventions improved quality of life, reduced costs, and were acceptable to patients. Improvements in prescribing, patient-knowledge and self-care, and functional class were observed.--Conclusion: TM and STS both appear effective interventions to improve outcomes in patients with CHF.

Re-examining the evidence in radiation dermatitis management literature : an overview and a critical appraisal of systematic reviews

Purpose: To provide an overview and a critical appraisal of systematic reviews (SRs) of published interventions for the prevention/management of radiation dermatitis. Methods and Materials: We searched Medline, CINAHL, Embase, and the Cochrane Library. We also manually searched through individual reference lists of potentially eligible articles and a number of key journals in the topic area. Two authors screened all potential articles and included eligible SRs. Two authors critically appraised and extracted key findings from the included reviews using AMSTAR (the measurement tool for “assessment of multiple systematic reviews”). Results: Of 1837 potential titles, 6 SRs were included. A number of interventions have been reported to be potentially beneficial for managing radiation dermatitis. Interventions evaluated in these reviews included skin care advice, steroidal/nonsteroidal topical agents, systemic therapies, modes of radiation delivery, and dressings. However, all the included SRs reported that there is insufficient evidence supporting any single effective intervention. The methodological quality of the included studies varied, and methodological shortfalls in these reviews might create biases to the overall results or recommendations for clinical practice. Conclusions: An up-to-date high-quality SR in the prevention/management of radiation dermatitis is needed to guide practice and direction for future research. We recommend that clinicians or guideline developers critically evaluate the information of SRs in their decision making.

Community interventions for physical activity : the role of a new Cochrane systematic review

Introduction: Systematic reviews are essential in summarising the results of a range of research studies on a specific topic into a single report. They serve as a key source of evidence-based information to support and develop policy and practice for healthy communities. This presentation will examine a new review of community-wide strategies to increase population levels of physical activity and compare it to an earlier Community Guide Review (CGR) of Community-wide campaigns to increase physical activity which recommended community wide interventions. Methods: We registered a Cochrane Systematic Review (CSR) title, published a protocol and recently completed the review of Community-wide interventions to increase physical activity. We compared the definitions, design and findings of the CSR to the CGR. Results: The two reviews differed remarkably in their conclusions with the CGR recommending “strong evidence exists that community-wide campaigns are effective in increasing levels of physical activity”, and the new CSR stating “The body of evidence in this review does not support the hypothesis that multi-component community wide interventions effectively increase population levels of physical activity”. We observed that both reviews examined multi-component interventions as a “combined package”. Possible explanations for the different conclusions may be due to the definition of community (CSR defined community as “comprising those persons residing in a geographically defined community, such as a village, town, or city”, excluding interventions which were whole of state or country, and CGR as “a group of individuals who share one or more characteristics. The CSR utilised a logic model at various stages of the review process and explicitly defined a combination of strategies encompassed within the intervention. The CSR included 25 and CGR 10 studies, respectively. Six of the 10 studies that were included in CGR were excluded from the CSR due to issues relating to study design, intervention definition or duration. The two reviews also differ in function as the CSR seeks to summarise global evidence and included 7 studies in low-income countries, where as the CGR contained only studies deemed relevant to the USA context. Discussion: Differences in the findings between older and newer reviews can be due to a variety of factors. For example, in updating a review the definition of an intervention can be changed. Further, differences may also be due to improvements in the standards and methodologies for systematic reviews as well as the inclusion of newer studies. These factors need to be understood whenever differences between newer and older reviews are considered.

Not all systematic reviews are systematic : a meta-review of the quality of current systematic reviews and meta-analyses for remote monitoring in heart failure

Background/aims: Remote monitoring for heart failure has not only been evaluated in a large number of randomised controlled trials, but also in many systematic reviews and meta-analyses. The aim of this meta-review was to identify, appraise and synthesise existing systematic reviews that have evaluated the effects of remote monitoring in heart failure. Methods: Using a Cochrane methodology, we electronically searched all relevant online databases and search engines, performed a forward citation search as well as hand-searched bibliographies. Only fully published systematic reviews of invasive and/or non-invasive remote monitoring interventions were included. Two reviewers independently extracted data. Results: Sixty-five publications from 3333 citations were identified. Seventeen fulfilled the inclusion and exclusion criteria. Quality varied with A Measurement Tool to Assess Systematic Reviews (AMSTAR scores) ranging from 2 to 11 (mean 5.88). Seven reviews (41%) pooled results from individual studies for meta-analysis. Eight (47%) considered all non-invasive remote monitoring strategies. Four (24%) focused specifically on telemonitoring. Four (24%) included studies investigating both non-invasive and invasive technologies. Population characteristics of the included studies were not reported consistently. Mortality and hospitalisations were the most frequently reported outcomes 12 (70%). Only five reviews (29%) reported healthcare costs and compliance. A high degree of heterogeneity was reported in many of the meta-analyses. Conclusions: These results should be considered in context of two negative RCTs of remote monitoring for heart failure that have been published since the meta-analyses (TIM-HF and Tele-HF). However, high quality reviews demonstrated improved mortality, quality of life, reduction in hospitalisations and healthcare costs.

The benefits and challenges of conducting an overview of systematic reviews in public health: a focus on physical activity

Introduction The ultimate aim of Cochrane systematic reviews is to inform policy and practice decisions for better health outcomes. However, due to the increasing numbers of scientific publications, wading through the available evidence of both individual studies and systematic reviews can be challenging and overwhelming even for avid authors and readers. This paper briefly describes the first overview (a systematic review of reviews) of the Cochrane Public Health Group (CPHG) in development and proposes a way forward for the methodologies under consideration.

Text mining for screening efficiency? Testing within a Cochrane public health review

Background The requirement for dual screening of titles and abstracts to select papers to examine in full text can create a huge workload, not least when the topic is complex and a broad search strategy is required, resulting in a large number of results. An automated system to reduce this burden, while still assuring high accuracy, has the potential to provide huge efficiency savings within the review process. Objectives To undertake a direct comparison of manual screening with a semi‐automated process (priority screening) using a machine classifier. The research is being carried out as part of the current update of a population‐level public health review. Methods Authors have hand selected studies for the review update, in duplicate, using the standard Cochrane Handbook methodology. A retrospective analysis, simulating a quasi‐‘active learning’ process (whereby a classifier is repeatedly trained based on ‘manually’ labelled data) will be completed, using different starting parameters. Tests will be carried out to see how far different training sets, and the size of the training set, affect the classification performance; i.e. what percentage of papers would need to be manually screened to locate 100% of those papers included as a result of the traditional manual method. Results From a search retrieval set of 9555 papers, authors excluded 9494 papers at title/abstract and 52 at full text, leaving 9 papers for inclusion in the review update. The ability of the machine classifier to reduce the percentage of papers that need to be manually screened to identify all the included studies, under different training conditions, will be reported. Conclusions The findings of this study will be presented along with an estimate of any efficiency gains for the author team if the screening process can be semi‐automated using text mining methodology, along with a discussion of the implications for text mining in screening papers within complex health reviews.