What is the Impact of the Underserved Pathway program on Entering an Underserved Family Medicine Residency? A Comparison of Three Approaches for Estimating the Average Treatment Effect

Thesis (Master's)--University of Washington, 2016-06

Introduction: It is well known that more primary care physicians are needed in underserved areas of the U.S. While prior research has shown that medical student experiences in underserved settings helps to increase the likelihood that they will ultimately choose to become physicians serving underserved areas, none have controlled for selection bias. In addition, none have examined intermediate steps in medical training experiences, such as how specific experiences relate to residency choices (the pipeline to eventual clinical practice). One manner of strengthening causal claims includes controlling for covariates that are correlated with treatment (program) and outcome variables using regression approaches, but this can have dimensionality or multicollinearity problems. Another approach to estimating less biased treatment effects involves propensity score (PS) methods, which are reduce the problems in the regression approach by employing a single score that captures the correlations between treatment status and all covariates. The primary aim of the present study was to use a set of methods to estimate whether the University of Washington (UW) School of Medicine’s longitudinal extracurricular experience, Underserved Pathway, impacts graduates’ choice in completing their residency in an underserved area. A secondary aim was to evaluate which of the analytic methods (and results) is most defensible. Methods: Extant data from one cohort of N = 158 UW medical students from matriculation surveys conducted between 2004 and 2011 were used for this project. Three popular approaches to estimating the Underserved Pathway program’s effect on underserved setting residency choice were employed, including multiple logistic regression, PS 1:n Matching with replacement, and inverse weighted probability (IWP) regression. Results: Average treatment (program) effects from the three approaches ranged from 12.0% of Underserved Pathway graduates choosing an underserved area for their residency (using the IPW method), 17.2% (using logistic regression), to 23.4% (using PS matching); the latter two were statistically significantly greater than zero by a Wald test. Tests of the covariate balance (i.e., the extent to which the ignorability assumption held) showed that PS matching offered better covariate balance than IPW for metrical covariates but that no other differences between methodologies on covariate balance were found. Discussion: Completion of the Underserved Pathway resulted in a significant (17.2% - 23.4%) increase in program graduates matching to a residency in an underserved setting according to the logistic regression and PS matching approaches; these methods are preferred given that neither differed in covariate balance from each other, and further, PS matching was superior in balance across groups over IPW on one covariate. Additionally, given that the logistic regression approach does not delete any cases, it seems likely that the logistic regression approach is the method that is most defensible in reducing selection bias in this situation. Selection of a method should address covariate balance with simplicity of approach with robust and transparent reporting to allow for assessment of any causal claims.