Failure of a numerical quality assessment scale to identify potential risk of bias in a systematic review: a comparison study

BACKGROUND: Assessing methodological quality of primary studies is an essential component of systematic reviews. Following a systematic review which used a domain based system [United States Preventative Services Task Force (USPSTF)] to assess methodological quality, a commonly used numerical rating scale (Downs and Black) was also used to evaluate the included studies and comparisons were made between quality ratings assigned using the two different methods. Both tools were used to assess the 20 randomized and quasi-randomized controlled trials examining an exercise intervention for chronic musculoskeletal pain which were included in the review. Inter-rater reliability and levels of agreement were determined using intraclass correlation coefficients (ICC). Influence of quality on pooled effect size was examined by calculating the between group standardized mean difference (SMD).

RESULTS: Inter-rater reliability indicated at least substantial levels of agreement for the USPSTF system (ICC 0.85; 95% CI 0.66, 0.94) and Downs and Black scale (ICC 0.94; 95% CI 0.84, 0.97). Overall level of agreement between tools (ICC 0.80; 95% CI 0.57, 0.92) was also good. However, the USPSTF system identified a number of studies (n = 3/20) as "poor" due to potential risks of bias. Analysis revealed substantially greater pooled effect sizes in these studies (SMD -2.51; 95% CI -4.21, -0.82) compared to those rated as "fair" (SMD -0.45; 95% CI -0.65, -0.25) or "good" (SMD -0.38; 95% CI -0.69, -0.08).

CONCLUSIONS: In this example, use of a numerical rating scale failed to identify studies at increased risk of bias, and could have potentially led to imprecise estimates of treatment effect. Although based on a small number of included studies within an existing systematic review, we found the domain based system provided a more structured framework by which qualitative decisions concerning overall quality could be made, and was useful for detecting potential sources of bias in the available evidence.

Infection control strategies for preventing the transmission of meticillin-resistant Staphylococcus aureus (MRSA) in nursing homes for older people (Review)

Background: Nursing homes for older people provide an environment likely to promote the acquisition and spread of meticillin-resistant Staphylococcus aureus (MRSA), putting residents at increased risk of colonisation and infection. It is recognised that infection prevention and control strategies are important in preventing and controlling MRSA transmission.

Objectives: To determine the effects of infection prevention and control strategies for preventing the transmission of MRSA in nursing homes for older people.

Search methods: In August 2013, for this third update, we searched the Cochrane Wounds Group Specialised Register, the Cochrane Central Register of Controlled Trials (CENTRAL, The Cochrane Library), Database of Abstracts of Reviews of Effects (DARE, The Cochrane Library), Ovid MEDLINE, OVID MEDLINE (In-process and Other Non-Indexed Citations), Ovid EMBASE, EBSCO CINAHL, Web of Science and the Health Technology Assessment (HTA) website. Research in progress was sought through Current Clinical Trials, Gateway to Reseach, and HSRProj (Health Services Research Projects in Progress).

Selection criteria: All randomised and controlled clinical trials, controlled before and after studies and interrupted time series studies of infection prevention and control interventions in nursing homes for older people were eligible for inclusion.

Data collection and analysis: Two review authors independently reviewed the results of the searches. Another review author appraised identified papers and undertook data extraction which was checked by a second review author.

Main results: For this third update only one study was identified, therefore it was not possible to undertake a meta-analysis. A cluster randomised controlled trial in 32 nursing homes evaluated the effect of an infection control education and training programme on MRSA prevalence. The primary outcome was MRSA prevalence in residents and staff, and a change in infection control audit scores which measured adherence to infection control standards. At the end of the 12 month study, there was no change in MRSA prevalence between intervention and control sites, while mean infection control audit scores were significantly higher in the intervention homes compared with control homes.

Authors' conclusions: There is a lack of research evaluating the effects on MRSA transmission of infection prevention and control strategies in nursing homes. Rigorous studies should be conducted in nursing homes, involving residents and staff to test interventions that have been specifically designed for this unique environment.

Medication Reconciliation at Discharge from Hospital: A Systematic Review of the Quantitative Literature

Medicines reconciliation is a way to identify and act on discrepancies in patients’ medical histories and it is found to play a key role in patient safety. This review focuses on discrepancies and medical errors that occurred at point of discharge from hospital. Studies were identified through the following electronic databases: PubMed, Sciences Direct, EMBASE, Google Scholar, Cochrane Reviews and CINAHL. Each of the six databases was screened from inception to end of January 2014. To determine eligibility of the studies; the title, abstract and full manuscript were screened to find 15 articles that meet the inclusion criteria. The median number of discrepancies across the articles was found to be 60%. In average patient had between 1.2–5.3 discrepancies when leaving the hospital. More studies also found a relation between the numbers of drugs a patient was on and the number of discrepancies. The variation in the number of discrepancies found in the 15 studies could be due to the fact that some studies excluded patient taking more than 5 drugs at admission. Medication reconciliation would be a way to avoid the high number of discrepancies that was found in this literature review and thereby increase patient safety.

Interventions to improve the appropriate use of polypharmacy in older people: a Cochrane systematic review

Objective: To summarise the findings of an updated Cochrane review of interventions aimed at improving the appropriate use of polypharmacy in older people. Design: Cochrane systematic review. Multiple electronic databases were searched including MEDLINE, EMBASE and the Cochrane Central Register of Controlled Trials (from inception to November 2013). Hand searching of references was also performed. Randomised controlled trials (RCTs), controlled clinical trials, controlled before-and-after studies and interrupted time series analyses reporting on interventions targeting appropriate polypharmacy in older people in any healthcare setting were included if they used a validated measure of prescribing appropriateness. Evidence quality was assessed using the Cochrane risk of bias tool and GRADE (Grades of Recommendation, Assessment, Development and Evaluation).
Setting: All healthcare settings. 
Participants: Older people (≥65 years) with ≥1 long-term condition who were receiving polypharmacy (≥4 regular medicines).
Primary and secondary outcome measures: Primary outcomes were the change in prevalence of appropriate polypharmacy and hospital admissions. Medication-related problems (eg, adverse drug reactions), medication adherence and quality of life were included as secondary outcomes.
Results: 12 studies were included: 8 RCTs, 2 cluster RCTs and 2 controlled before-and-after studies. 1 study involved computerised decision support and 11 comprised pharmaceutical care approaches across various settings. Appropriateness was measured using validated tools, including the Medication Appropriateness Index, Beers’ criteria and Screening Tool of Older Person’s Prescriptions (STOPP)/ Screening Tool to Alert doctors to Right Treatment (START). The interventions demonstrated a reduction in inappropriate prescribing. Evidence of effect on hospital admissions and medication-related problems was conflicting. No differences in health-related quality of life were reported.
Conclusions: The included interventions demonstrated improvements in appropriate polypharmacy based on reductions in inappropriate prescribing. However, it remains unclear if interventions resulted in clinically significant improvements (eg, in terms of hospital admissions). Future intervention studies would benefit from available guidance on intervention development, evaluation and reporting to facilitate replication in clinical practice.

Interventions to increase physical activity in patients with COPD: A comprehensive review

It is unknown how interventions aimed at increasing physical activity (PA), other than traditional pulmonary rehabilitation, are structured and whether they are effective in increasing PA in chronic obstructive pulmonary disease (COPD). The primary aim of this review was to outline the typical components of PA interventions in patients with COPD. This review followed the PRISMA guidelines. A structured literature search of relevant electronic databases from inception to April 2014 was undertaken to outline typical components and examine outcome variables of PA interventions in patients with COPD. Over 12000 articles were screened and 20 relevant studies involving 31 PA interventions were included. Data extracted included patient demographics, components of the PA intervention, PA outcome measures and effects of the intervention. Quality was assessed using the PEDro and CASP scales. There were 13 randomised controlled trials and three randomised trials (PEDro score 5-7/10) and four cohort studies (CASP score 5/10). Interventions varied in duration, number of participant/researcher contacts and mode of delivery. The most common behaviour change techniques included information on when and where (n = 26/31) and how (n = 22/31) to perform PA behaviour and self-monitoring (n = 18/31). Significant between-group differences post-intervention in favour of the PA intervention, compared to a control group or to other PA interventions, in one or more PA assessments were found in 7/16 studies. All seven studies used walking as the main type of PA/exercise. In conclusion, although the components of PA interventions were variable, there is some evidence that PA interventions have the potential to increase PA in patients with COPD

Abdominal Aortic Aneurysm Genetic Associations: Mostly False? A Systematic Review and Meta-analysis

OBJECTIVE/BACKGROUND: Many associations between abdominal aortic aneurysm (AAA) and genetic polymorphisms have been reported. It is unclear which are genuine and which may be caused by type 1 errors, biases, and flexible study design. The objectives of the study were to identify associations supported by current evidence and to investigate the effect of study design on reporting associations.

METHODS: Data sources were MEDLINE, Embase, and Web of Science. Reports were dual-reviewed for relevance and inclusion against predefined criteria (studies of genetic polymorphisms and AAA risk). Study characteristics and data were extracted using an agreed tool and reports assessed for quality. Heterogeneity was assessed using I(2) and fixed- and random-effects meta-analyses were conducted for variants that were reported at least twice, if any had reported an association. Strength of evidence was assessed using a standard guideline.

RESULTS: Searches identified 467 unique articles, of which 97 were included. Of 97 studies, 63 reported at least one association. Of 92 studies that conducted multiple tests, only 27% corrected their analyses. In total, 263 genes were investigated, and associations were reported in polymorphisms in 87 genes. Associations in CDKN2BAS, SORT1, LRP1, IL6R, MMP3, AGTR1, ACE, and APOA1 were supported by meta-analyses.

CONCLUSION: Uncorrected multiple testing and flexible study design (particularly testing many inheritance models and subgroups, and failure to check for Hardy-Weinberg equilibrium) contributed to apparently false associations being reported. Heterogeneity, possibly due to the case mix, geographical, temporal, and environmental variation between different studies, was evident. Polymorphisms in nine genes had strong or moderate support on the basis of the literature at this time. Suggestions are made for improving AAA genetics study design and conduct.

The effectiveness of disaster risk communication: a systematic review of intervention studies

INTRODUCTION: A disaster is a serious disruption to the functioning of a community that exceeds its capacity to cope within its own resources. Risk communication in disasters aims to prevent and mitigate harm from disasters, prepare the population before a disaster, disseminate information during disasters and aid subsequent recovery. The aim of this systematic review is to identify, appraise and synthesise the findings of studies of the effects of risk communication interventions during four stages of the disaster cycle.

METHODS: We searched the Cochrane Central Register of Controlled Trials, Embase, MEDLINE, PsycInfo, Sociological Abstracts, Web of Science and grey literature sources for randomised trials, cluster randomised trials, controlled and uncontrolled before and after studies, interrupted time series studies and qualitative studies of any method of disaster risk communication to at-risk populations. Outcome criteria were disaster-related knowledge and behaviour, and health outcomes.

RESULTS: Searches yielded 5,224 unique articles, of which 100 were judged to be potentially relevant. Twenty-five studies met the inclusion criteria, and two additional studies were identified from other searching. The studies evaluated interventions in all four stages of the disaster cycle, included a variety of man-made, natural and infectious disease disasters, and were conducted in many disparate settings. Only one randomised trial and one cluster randomised trial were identified, with less robust designs used in the other studies. Several studies reported improvements in disaster-related knowledge and behaviour.

DISCUSSION: We identified and appraised intervention studies of disaster risk communication and present an overview of the contemporary literature. Most studies used non-randomised designs that make interpretation challenging. We do not make specific recommendations for practice but highlight the need for high-quality randomised trials and appropriately-analysed cluster randomised trials in the field of disaster risk communication where these can be conducted within an appropriate research ethics framework.

A review of the costs and benefits of demand response for electricity in the UK

The recent policy discussion in the UK on the economic case for demand response (DR) calls for a reflection on available evidence regarding its costs and benefits. Existing studies tend to consider the size of investments and returns of certain forms of DR in isolation and do not consider economic welfare effects. From review of existing studies, policy documents, and some simple modelling of benefits of DR in providing reserve for unforeseen events, we demonstrate that the economic case for DR in UK electricity markets is positive. Consideration of economic welfare gains is provided.

Measurement issues in trials of pediatric acute diarrheal diseases: a systematic review

BACKGROUND: Worldwide, diarrheal diseases rank second among conditions that afflict children. Despite the disease burden, there is limited consensus on how to define and measure pediatric acute diarrhea in trials. OBJECTIVES: In RCTs of children involving acute diarrhea as the primary outcome, we documented (1) how acute diarrhea and its resolution were defined, (2) all primary outcomes, (3) the psychometric properties of instruments used to measure acute diarrhea and (4) the methodologic quality of included trials, as reported. METHODS: We searched CENTRAL, Embase, Global Health, and Medline from inception to February 2009. English-language RCTs of children younger than 19 years that measured acute diarrhea as a primary outcome were chosen. RESULTS: We identified 138 RCTs reporting on 1 or more primary outcomes related to pediatric acute diarrhea/diseases. Included trials used 64 unique definitions of diarrhea, 69 unique definitions of diarrhea resolution, and 46 unique primary outcomes. The majority of included trials evaluated short-term clinical disease activity (incidence and duration of diarrhea), laboratory outcomes, or a composite of these end points. Thirty-two trials used instruments (eg, single and multidomain scoring systems) to support assessment of disease activity. Of these, 3 trials stated that their instrument was valid; however, none of the trials (or their citations) reported evidence of this validity. The overall methodologic quality of included trials was good. CONCLUSIONS: Even in what would be considered methodologically sound clinical trials, definitions of diarrhea, primary outcomes, and instruments employed in RCTs of pediatric acute diarrhea are heterogeneous, lack evidence of validity, and focus on indices that may not be important to participants.

Supporting Families through Short-Term Foster Care: An Essay Review

Aldgate, J. and Bradley, M. (1999). Supporting families through short-term fostering. London: The Stationery Office. This essay reviews a British qualitative study of short-term foster care from the perspectives of birth parents, children, foster parents, and social workers. Respondents highlighted the value of short-term foster care as a family support service and also offered many recommendations for improving service delivery. The study provides useful implications for restructuring child welfare services in the United States and for promoting cross-national collaboration in future research activities in the area of child and family services.