Using interviewer random effects to remove selection bias from HIV prevalence estimates

Background: Selection bias in HIV prevalence estimates occurs if non-participation in testing is correlated with HIV status. Longitudinal data suggests that individuals who know or suspect they are HIV positive are less likely to participate in testing in HIV surveys, in which case methods to correct for missing data which are based on imputation and observed characteristics will produce biased results. Methods: The identity of the HIV survey interviewer is typically associated with HIV testing participation, but is unlikely to be correlated with HIV status. Interviewer identity can thus be used as a selection variable allowing estimation of Heckman-type selection models. These models produce asymptotically unbiased HIV prevalence estimates, even when non-participation is correlated with unobserved characteristics, such as knowledge of HIV status. We introduce a new random effects method to these selection models which overcomes non-convergence caused by collinearity, small sample bias, and incorrect inference in existing approaches. Our method is easy to implement in standard statistical software, and allows the construction of bootstrapped standard errors which adjust for the fact that the relationship between testing and HIV status is uncertain and needs to be estimated. Results: Using nationally representative data from the Demographic and Health Surveys, we illustrate our approach with new point estimates and confidence intervals (CI) for HIV prevalence among men in Ghana (2003) and Zambia (2007). In Ghana, we find little evidence of selection bias as our selection model gives an HIV prevalence estimate of 1.4% (95% CI 1.2% – 1.6%), compared to 1.6% among those with a valid HIV test. In Zambia, our selection model gives an HIV prevalence estimate of 16.3% (95% CI 11.0% - 18.4%), compared to 12.1% among those with a valid HIV test. Therefore, those who decline to test in Zambia are found to be more likely to be HIV positive. Conclusions: Our approach corrects for selection bias in HIV prevalence estimates, is possible to implement even when HIV prevalence or non-participation is very high or very low, and provides a practical solution to account for both sampling and parameter uncertainty in the estimation of confidence intervals. The wide confidence intervals estimated in an example with high HIV prevalence indicate that it is difficult to correct statistically for the bias that may occur when a large proportion of people refuse to test.

On the Assumption of Bivariate Normality in Selection Models: A Copula Approach Applied to Estimating HIV Prevalence

Background: Heckman-type selection models have been used to control HIV prevalence estimates for selection bias when participation in HIV testing and HIV status are associated after controlling for observed variables. These models typically rely on the strong assumption that the error terms in the participation and the outcome equations that comprise the model are distributed as bivariate normal.
Methods: We introduce a novel approach for relaxing the bivariate normality assumption in selection models using copula functions. We apply this method to estimating HIV prevalence and new confidence intervals (CI) in the 2007 Zambia Demographic and Health Survey (DHS) by using interviewer identity as the selection variable that predicts participation (consent to test) but not the outcome (HIV status).
Results: We show in a simulation study that selection models can generate biased results when the bivariate normality assumption is violated. In the 2007 Zambia DHS, HIV prevalence estimates are similar irrespective of the structure of the association assumed between participation and outcome. For men, we estimate a population HIV prevalence of 21% (95% CI = 16%–25%) compared with 12% (11%–13%) among those who consented to be tested; for women, the corresponding figures are 19% (13%–24%) and 16% (15%–17%).
Conclusions: Copula approaches to Heckman-type selection models are a useful addition to the methodological toolkit of HIV epidemiology and of epidemiology in general. We develop the use of this approach to systematically evaluate the robustness of HIV prevalence estimates based on selection models, both empirically and in a simulation study.

Methods to increase response to postal and electronic questionnaires

Background: Postal and electronic questionnaires are widely used for data collection in epidemiological studies but non-response reduces the effective sample size and can introduce bias. Finding ways to increase response to postal and electronic questionnaires would improve the quality of health research. Objectives: To identify effective strategies to increase response to postal and electronic questionnaires. Search strategy: We searched 14 electronic databases to February 2008 and manually searched the reference lists of relevant trials and reviews, and all issues of two journals. We contacted the authors of all trials or reviews to ask about unpublished trials. Where necessary, we also contacted authors to confirm methods of allocation used and to clarify results presented. We assessed the eligibility of each trial using pre-defined criteria. Selection criteria: Randomised controlled trials of methods to increase response to postal or electronic questionnaires. Data collection and analysis: We extracted data on the trial participants, the intervention, the number randomised to intervention and comparison groups and allocation concealment. For each strategy, we estimated pooled odds ratios (OR) and 95% confidence intervals (CI) in a random-effects model. We assessed evidence for selection bias using Egger's weighted regression method and Begg's rank correlation test and funnel plot. We assessed heterogeneity among trial odds ratios using a Chi 2 test and the degree of inconsistency between trial results was quantified using the I 2 statistic. Main results: Postal We found 481 eligible trials.The trials evaluated 110 different ways of increasing response to postal questionnaires.We found substantial heterogeneity among trial results in half of the strategies. The odds of response were at least doubled using monetary incentives (odds ratio 1.87; 95% CI 1.73 to 2.04; heterogeneity P < 0.00001, I 2 = 84%), recorded delivery (1.76; 95% CI 1.43 to 2.18; P = 0.0001, I 2 = 71%), a teaser on the envelope - e.g. a comment suggesting to participants that they may benefit if they open it (3.08; 95% CI 1.27 to 7.44) and a more interesting questionnaire topic (2.00; 95% CI 1.32 to 3.04; P = 0.06, I 2 = 80%). The odds of response were substantially higher with pre-notification (1.45; 95% CI 1.29 to 1.63; P < 0.00001, I 2 = 89%), follow-up contact (1.35; 95% CI 1.18 to 1.55; P < 0.00001, I 2 = 76%), unconditional incentives (1.61; 1.36 to 1.89; P < 0.00001, I 2 = 88%), shorter questionnaires (1.64; 95%CI 1.43 to 1.87; P < 0.00001, I 2 = 91%), providing a second copy of the questionnaire at follow up (1.46; 95% CI 1.13 to 1.90; P < 0.00001, I 2 = 82%), mentioning an obligation to respond (1.61; 95% CI 1.16 to 2.22; P = 0.98, I 2 = 0%) and university sponsorship (1.32; 95% CI 1.13 to 1.54; P < 0.00001, I 2 = 83%). The odds of response were also increased with non-monetary incentives (1.15; 95% CI 1.08 to 1.22; P < 0.00001, I 2 = 79%), personalised questionnaires (1.14; 95% CI 1.07 to 1.22; P < 0.00001, I 2 = 63%), use of hand-written addresses (1.25; 95% CI 1.08 to 1.45; P = 0.32, I 2 = 14%), use of stamped return envelopes as opposed to franked return envelopes (1.24; 95% CI 1.14 to 1.35; P < 0.00001, I 2 = 69%), an assurance of confidentiality (1.33; 95% CI 1.24 to 1.42) and first class outward mailing (1.11; 95% CI 1.02 to 1.21; P = 0.78, I 2 = 0%). The odds of response were reduced when the questionnaire included questions of a sensitive nature (0.94; 95% CI 0.88 to 1.00; P = 0.51, I 2 = 0%). Electronic: We found 32 eligible trials. The trials evaluated 27 different ways of increasing response to electronic questionnaires. We found substantial heterogeneity among trial results in half of the strategies. The odds of response were increased by more than a half using non-monetary incentives (1.72; 95% CI 1.09 to 2.72; heterogeneity P < 0.00001, I 2 = 95%), shorter e-questionnaires (1.73; 1.40 to 2.13; P = 0.08, I 2 = 68%), including a statement that others had responded (1.52; 95% CI 1.36 to 1.70), and a more interesting topic (1.85; 95% CI 1.52 to 2.26). The odds of response increased by a third using a lottery with immediate notification of results (1.37; 95% CI 1.13 to 1.65), an offer of survey results (1.36; 95% CI 1.15 to 1.61), and using a white background (1.31; 95% CI 1.10 to 1.56). The odds of response were also increased with personalised e-questionnaires (1.24; 95% CI 1.17 to 1.32; P = 0.07, I 2 = 41%), using a simple header (1.23; 95% CI 1.03 to 1.48), using textual representation of response categories (1.19; 95% CI 1.05 to 1.36), and giving a deadline (1.18; 95% CI 1.03 to 1.34). The odds of response tripled when a picture was included in an e-mail (3.05; 95% CI 1.84 to 5.06; P = 0.27, I 2 = 19%). The odds of response were reduced when "Survey" was mentioned in the e-mail subject line (0.81; 95% CI 0.67 to 0.97; P = 0.33, I 2 = 0%), and when the e-mail included a male signature (0.55; 95% CI 0.38 to 0.80; P = 0.96, I 2 = 0%). Authors' conclusions: Health researchers using postal and electronic questionnaires can increase response using the strategies shown to be effective in this systematic review. Copyright © 2009 The Cochrane Collaboration. Published by John Wiley & Sons, Ltd.


--------------------------------------------------------------------------------

Reaxys Database Information|

--------------------------------------------------------------------------------

Smart meter devices and the effect of feedback on residential electricity consumption: Evidence from a natural experiment in Northern Ireland

Using a unique set of data and exploiting a large-scale natural experiment, we estimate the effect of real-time usage information on residential electricity consumption in Northern Ireland. Starting in April 2002, the utility replaced prepayment meters with advanced meters that allow the consumer to track usage in real-time. We rely on this event, account for the endogeneity of price and payment plan with consumption through a plan selection correction term, and find that the provision of information is associated with a decline in electricity consumption of 11-17%. We find that the reduction is robust to different specifications, selection-bias correction methods and subsamples of the original data. The advanced metering program delivers reasonably cost-effective reductions in carbon dioxide emissions, even under the most conservative usage reduction scenarios.

Exercise rehabilitation following intensive care unit discharge for recovery from critical illness

Background: Skeletal muscle wasting and weakness are significant complications of critical illness, associated with the degree of illness severity and periods of reduced mobility during mechanical ventilation. They contribute to the profound physical and functional deficits observed in survivors. These impairments may persist for many years following discharge from the intensive care unit (ICU) and may markedly influence health-related quality of life. Rehabilitation is a key strategy in the recovery of patients following critical illness. Exercise based interventions are aimed at targeting this muscle wasting and weakness. Physical rehabilitation delivered during ICU admission has been systematically evaluated and shown to be beneficial. However its effectiveness when initiated after ICU discharge has yet to be established. Objectives: To assess the effectiveness of exercise rehabilitation programmes, initiated after ICU discharge, on functional exercise capacity and health-related quality of life in adult ICU survivors who have been mechanically ventilated for more than 24 hours. Search methods:We searched the following databases: the Cochrane Central Register of Controlled Trials (CENTRAL) (The Cochrane Library), OvidSP MEDLINE, Ovid SP EMBASE, and CINAHL via EBSCO host to 15th May 2014. We used a specific search strategy for each database. This included synonyms for ICU and critical illness, exercise training and rehabilitation. We searched the reference lists of included studies and contacted primary authors to obtain further information regarding potentially eligible studies. We also searched major clinical trials registries (Clinical Trials and Current Controlled Trials) and the personal libraries of the review authors. We applied no language or publication restriction. We reran the search in February 2015. We will deal with any studies of interest when we update the review.  Selection criteria:We included randomized controlled trials (RCTs), quasi-RCTs, and controlled clinical trials (CCTs) that compared an exercise interventioninitiated after ICU discharge to any other intervention or a control or ‘usual care’ programme in adult (≥18years) survivors ofcritical illness. Data collection and analysis:We used standard methodological procedures expected by The Cochrane Collaboration. Main results:We included six trials (483 adult ICU participants). Exercise-based interventions were delivered on the ward in two studies; both onthe ward and in the community in one study; and in the community in three studies. The duration of the intervention varied according to the length of stay in hospital following ICU discharge (up to a fixed duration of 12 weeks).Risk of bias was variable for all domains across all trials. High risk of bias was evident in all studies for performance bias, although blinding of participants and personnel in therapeutic rehabilitation trials can be pragmatically challenging. Low risk of bias was at least 50% for all other domains across all trials, although high risk of bias was present in one study for random sequence generation (selection bias), incomplete outcome data (attrition bias) and other sources. Risk of bias was unclear for remaining studies across the domains.All six studies measured effect on the primary outcome of functional exercise capacity, although there was wide variability in natureof intervention, outcome measures and associated metrics, and data reporting. Overall quality of the evidence was very low. Only two studies using the same outcome measure for functional exercise capacity, had the potential for pooling of data and assessment of heterogeneity. On statistical advice, this was considered inappropriate to perform this analysis and study findings were therefore qualitatively described. Individually, three studies reported positive results in favour of the intervention. A small benefit (versus. control)was evident in anaerobic threshold in one study (mean difference, MD (95% confidence interval, CI), 1.8 mlO2/kg/min (0.4 to 3.2),P value = 0.02), although this effect was short-term, and in a second study, both incremental (MD 4.7 (95% CI 1.69 to 7.75) Watts, P value = 0.003) and endurance (MD 4.12 (95% CI 0.68 to 7.56) minutes, P value = 0.021) exercise testing demonstrated improvement.Finally self-reported physical function increased significantly following a rehabilitation manual (P value = 0.006). Remaining studies found no effect of the intervention.Similar variability in with regard findings for the primary outcome of health-related quality of life were also evident. Only two studies evaluated this outcome. Following statistical advice, these data again were considered inappropriate for pooling to determine overall effect and assessment of heterogeneity. Qualitative description of findings was therefore undertaken. Individually, neither study reported differences between intervention and control groups for health-related quality of life as a result of the intervention. Overall quality of the evidence was very low.Mortality was reported by all studies, ranging from 0% to 18.8%. Only one non-mortality adverse event was reported across all patients in all studies (a minor musculoskeletal injury). Withdrawals, reported in four studies, ranged from 0% to 26.5% in control groups,and 8.2% to 27.6% in intervention groups. Loss to follow-up, reported in all studies, ranged from 0% to 14% in control groups, and 0% to 12.5% in intervention groups. Authors’ conclusions:We are unable, at this time, to determine an overall effect on functional exercise capacity, or health-related quality of life, of an exercise based intervention initiated after ICU discharge in survivors of critical illness. Meta-analysis of findings was not appropriate. This was due to insufficient study number and data. Individual study findings were inconsistent. Some studies reported a beneficial effect of the intervention on functional exercise capacity, and others not. No effect was reported on health-related quality of life. Methodological rigour was lacking across a number of domains influencing quality of the evidence. There was also wide variability in the characteristics of interventions, outcome measures and associated metrics, and data reporting.If further trials are identified, we may be able to determine the effect of exercise-based interventions following ICU discharge, on functional exercise capacity and health-related quality of life in survivors of critical illness.

Survey of new 2007 and 2011 Cochrane reviews found 37% of prespecified outcomes not reported

OBJECTIVES: To survey the outcomes used in Cochrane Reviews, as part of our work within the Core Outcome Measures in Effectiveness Trials Initiative.

STUDY DESIGN AND SETTING: A descriptive survey of Cochrane Reviews, divided by Cochrane Review Group (CRG), published in full for the first time in 2007 and 2011. Outcomes specified in the methods section of each review and outcomes reported in the results section of each review were of interest, in this exploration of the common use of outcomes and core outcome sets (COS).

RESULTS: Seven hundred eighty-eight reviews, specifying 6,127 outcomes, were included. When we excluded specified outcomes from the 86 reviews that did not include any studies, we found that 1,996 (37%) specified outcomes were not reported. Of the 361 new reviews with studies from 2011, 113 (31%) had a "summary of findings" table (SoF). Fifteen broad outcome categories were identified and used to manage the outcome data. We found consistency in the use of these categories across CRGs but inconsistency in outcomes within these categories.

CONCLUSION: COS have been used rarely in Cochrane Reviews, but the introduction of SoF makes the development and application of COS timelier than ever.

Vaccination and All Cause Child Mortality 1985-2011: Global Evidence from the Demographic and Health Surveys

Based on models with calibrated parameters for infection, case fatality rates, and vaccine efficacy, basic childhood vaccinations have been estimated to be highly cost effective. We estimate the association of vaccination with mortality directly from survey data. Using 149 cross-sectional Demographic and Health Surveys, we determine the relationship between vaccination coverage and under five mortality at the survey cluster level. Our data include approximately one million children in 68,490 clusters in 62 countries. We consider the childhood measles, Bacille Calmette-Guérin (BCG), Diphtheria-Pertussis-Tetanus (DPT), Polio, and maternal tetanus vaccinations. Using modified Poisson regression to estimate the relative risk of child mortality in each cluster, we also adjust for selection bias caused by the vaccination status of dead children not being reported. Childhood vaccination, and in particular measles and tetanus vaccination, is associated with substantial reductions in childhood mortality. We estimate that children in clusters with complete vaccination coverage have relative risk of mortality 0.73 (95% Confidence Interval: 0.68, 0.77) that of children in a cluster with no vaccination. While widely used, basic vaccines still have coverage rates well below 100% in many countries, and our results emphasize the effectiveness of increasing their coverage rates in order to reduce child mortality.

Adjusting HIV Prevalence Estimates for Non-participation: an Application to Demographic Surveillance

Introduction: HIV testing is a cornerstone of efforts to combat the HIV epidemic, and testing conducted as part of surveillance provides invaluable data on the spread of infection and the effectiveness of campaigns to reduce the transmission of HIV. However, participation in HIV testing can be low, and if respondents systematically select not to be tested because they know or suspect they are HIV positive (and fear disclosure), standard approaches to deal with missing data will fail to remove selection bias. We implemented Heckman-type selection models, which can be used to adjust for missing data that are not missing at random, and established the extent of selection bias in a population-based HIV survey in an HIV hyperendemic community in rural South Africa.

Methods: We used data from a population-based HIV survey carried out in 2009 in rural KwaZulu-Natal, South Africa. In this survey, 5565 women (35%) and 2567 men (27%) provided blood for an HIV test. We accounted for missing data using interviewer identity as a selection variable which predicted consent to HIV testing but was unlikely to be independently associated with HIV status. Our approach involved using this selection variable to examine the HIV status of residents who would ordinarily refuse to test, except that they were allocated a persuasive interviewer. Our copula model allows for flexibility when modelling the dependence structure between HIV survey participation and HIV status.

Results: For women, our selection model generated an HIV prevalence estimate of 33% (95% CI 27–40) for all people eligible to consent to HIV testing in the survey. This estimate is higher than the estimate of 24% generated when only information from respondents who participated in testing is used in the analysis, and the estimate of 27% when imputation analysis is used to predict missing data on HIV status. For men, we found an HIV prevalence of 25% (95% CI 15–35) using the selection model, compared to 16% among those who participated in testing, and 18% estimated with imputation. We provide new confidence intervals that correct for the fact that the relationship between testing and HIV status is unknown and requires estimation.

Conclusions: We confirm the feasibility and value of adopting selection models to account for missing data in population-based HIV surveys and surveillance systems. Elements of survey design, such as interviewer identity, present the opportunity to adopt this approach in routine applications. Where non-participation is high, true confidence intervals are much wider than those generated by standard approaches to dealing with missing data suggest.

Technological aids for the rehabilitation of memory and executive functioning in children and adolescents with acquired brain injury (Review).

Background The use of technology in healthcare settings is on the increase and may represent a cost-effective means of delivering rehabilitation. Reductions in treatment time, and delivery in the home, are also thought to be benefits of this approach. Children and adolescents with brain injury often experience deficits in memory and executive functioning that can negatively affect their school work, social lives, and future occupations. Effective interventions that can be delivered at home, without the need for high-cost clinical involvement, could provide a means to address a current lack of provision. We have systematically reviewed studies examining the effects of technology-based interventions for the rehabilitation of deficits in memory and executive functioning in children and adolescents with acquired brain injury. Objectives To assess the effects of technology-based interventions compared to placebo intervention, no treatment, or other types of intervention, on the executive functioning and memory of children and adolescents with acquired brain injury. Search methods We ran the search on the 30 September 2015. We searched the Cochrane Injuries Group Specialised Register, the Cochrane Central Register of Controlled Trials (CENTRAL), Ovid MEDLINE(R), Ovid MEDLINE(R) In-Process & Other Non-Indexed Citations, Ovid MEDLINE(R) Daily and Ovid OLDMEDLINE(R), EMBASE Classic + EMBASE (OvidSP), ISI Web of Science (SCI-EXPANDED, SSCI, CPCI-S, and CPSI-SSH), CINAHL Plus (EBSCO), two other databases, and clinical trials registers. We also searched the internet, screened reference lists, and contacted authors of included studies. Selection criteria Randomised controlled trials comparing the use of a technological aid for the rehabilitation of children and adolescents with memory or executive-functioning deficits with placebo, no treatment, or another intervention. Data collection and analysis Two review authors independently reviewed titles and abstracts identified by the search strategy. Following retrieval of full-text manuscripts, two review authors independently performed data extraction and assessed the risk of bias. Main results Four studies (involving 206 participants) met the inclusion criteria for this review. Three studies, involving 194 participants, assessed the effects of online interventions to target executive functioning (that is monitoring and changing behaviour, problem solving, planning, etc.). These studies, which were all conducted by the same research team, compared online interventions against a 'placebo' (participants were given internet resources on brain injury). The interventions were delivered in the family home with additional support or training, or both, from a psychologist or doctoral student. The fourth study investigated the use of a computer program to target memory in addition to components of executive functioning (that is attention, organisation, and problem solving). No information on the study setting was provided, however a speech-language pathologist, teacher, or occupational therapist accompanied participants. Two studies assessed adolescents and young adults with mild to severe traumatic brain injury (TBI), while the remaining two studies assessed children and adolescents with moderate to severe TBI. Risk of bias We assessed the risk of selection bias as low for three studies and unclear for one study. Allocation bias was high in two studies, unclear in one study, and low in one study. Only one study (n = 120) was able to conceal allocation from participants, therefore overall selection bias was assessed as high. One study took steps to conceal assessors from allocation (low risk of detection bias), while the other three did not do so (high risk of detection bias). Primary outcome 1: Executive functioning: Technology-based intervention versus placebo Results from meta-analysis of three studies (n = 194) comparing online interventions with a placebo for children and adolescents with TBI, favoured the intervention immediately post-treatment (standardised mean difference (SMD) -0.37, 95% confidence interval (CI) -0.66 to -0.09; P = 0.62; I2 = 0%). (As there is no 'gold standard' measure in the field, we have not translated the SMD back to any particular scale.) This result is thought to represent only a small to medium effect size (using Cohen’s rule of thumb, where 0.2 is a small effect, 0.5 a medium one, and 0.8 or above is a large effect); this is unlikely to have a clinically important effect on the participant. The fourth study (n = 12) reported differences between the intervention and control groups on problem solving (an important component of executive functioning). No means or standard deviations were presented for this outcome, therefore an effect size could not be calculated. The quality of evidence for this outcome according to GRADE was very low. This means future research is highly likely to change the estimate of effect. Primary outcome 2: Memory One small study (n = 12) reported a statistically significant difference in improvement in sentence recall between the intervention and control group following an eight-week remediation programme. No means or standard deviations were presented for this outcome, therefore an effect size could not be calculated. Secondary outcomes Two studies (n = 158) reported on anxiety/depression as measured by the Child Behavior Checklist (CBCL) and were included in a meta-analysis. We found no evidence of an effect with the intervention (mean difference -5.59, 95% CI -11.46 to 0.28; I2 = 53%). The GRADE quality of evidence for this outcome was very low, meaning future research is likely to change the estimate of effect. A single study sought to record adverse events and reported none. Two studies reported on use of the intervention (range 0 to 13 and 1 to 24 sessions). One study reported on social functioning/social competence and found no effect. The included studies reported no data for other secondary outcomes (that is quality of life and academic achievement). Authors' conclusions This review provides low-quality evidence for the use of technology-based interventions in the rehabilitation of executive functions and memory for children and adolescents with TBI. As all of the included studies contained relatively small numbers of participants (12 to 120), our findings should be interpreted with caution. The involvement of a clinician or therapist, rather than use of the technology, may have led to the success of these interventions. Future research should seek to replicate these findings with larger samples, in other regions, using ecologically valid outcome measures, and reduced clinician involvement.

Work-related training and the probability of transitioning from non-permanent to permanent employment

It is widely believed that work-related training increases a worker’s probability of moving up the job-quality ladder. This is usually couched in terms of effects on wages, but it has also been argued that training increases the probability of moving from non-permanent forms of employment to more permanent employment. This hypothesis is tested using nationally representative panel data for Australia, a country where the incidence of non-permanent employment, and especially casual employment, is high by international standards. While a positive association between participation in work-related training and the subsequent probability of moving from either casual or fixed-term contract employment to permanent employment is observed among men, this is shown to be driven not by a causal impact of training on transitions but by differences between those who do and do not receive training; i.e., selection bias.

The effect of changing the built environment on physical activity: a quantitative review of the risk of bias in natural experiments.

BACKGROUND:
Evidence regarding the association of the built environment with physical activity is influencing policy recommendations that advocate changing the built environment to increase population-level physical activity. However, to date there has been no rigorous appraisal of the quality of the evidence on the effects of changing the built environment. The aim of this review was to conduct a thorough quantitative appraisal of the risk of bias present in those natural experiments with the strongest experimental designs for assessing the causal effects of the built environment on physical activity.

METHODS:
Eligible studies had to evaluate the effects of changing the built environment on physical activity, include at least one measurement before and one measurement of physical activity after changes in the environment, and have at least one intervention site and non-intervention comparison site. Given the large number of systematic reviews in this area, studies were identified from three exemplar systematic reviews; these were published in the past five years and were selected to provide a range of different built environment interventions. The risk of bias in these studies was analysed using the Cochrane Risk of Bias Assessment Tool: for Non-Randomized Studies of Interventions (ACROBAT-NRSI).

RESULTS:
Twelve eligible natural experiments were identified. Risk of bias assessments were conducted for each physical activity outcome from all studies, resulting in a total of fifteen outcomes being analysed. Intervention sites included parks, urban greenways/trails, bicycle lanes, paths, vacant lots, and a senior citizen's centre. All outcomes had an overall critical (n = 12) or serious (n = 3) risk of bias. Domains with the highest risk of bias were confounding (due to inadequate control sites and poor control of confounding variables), measurement of outcomes, and selection of the reported result.

CONCLUSIONS:
The present review focused on the strongest natural experiments conducted to date. Given this, the failure of existing studies to adequately control for potential sources of bias highlights the need for more rigorous research to underpin policy recommendations for changing the built environment to increase physical activity. Suggestions are proposed for how future natural experiments in this area can be improved.