Sources of selection bias in evaluating social programs: An interpretation of conventional measures and evidence on the effectiveness of matching as a program evaluation method

This paper decomposes the conventional measure of selection bias in observational studies into three components. The first two components are due to differences in the distributions of characteristics between participant and nonparticipant (comparison) group members: the first arises from differences in the supports, and the second from differences in densities over the region of common support. The third component arises from selection bias precisely defined. Using data from a recent social experiment, we find that the component due to selection bias, precisely defined, is smaller than the first two components. However, selection bias still represents a substantial fraction of the experimental impact estimate. The empirical performance of matching methods of program evaluation is also examined. We find that matching based on the propensity score eliminates some but not all of the measured selection bias, with the remaining bias still a substantial fraction of the estimated impact. We find that the support of the distribution of propensity scores for the comparison group is typically only a small portion of the support for the participant group. For values outside the common support, it is impossible to reliably estimate the effect of program participation using matching methods. If the impact of participation depends on the propensity score, as we find in our data, the failure of the common support condition severely limits matching compared with random assignment as an evaluation estimator.

Public library consumer health information pilot project: results of a National Library of Medicine evaluation

In October 1998, the National Library of Medicine (NLM) launched a pilot project to learn about the role of public libraries in providing health information to the public and to generate information that would assist NLM and the National Network of Libraries of Medicine (NN/LM) in learning how best to work with public libraries in the future. Three regional medical libraries (RMLs), eight resource libraries, and forty-one public libraries or library systems from nine states and the District of Columbia were selected for participation. The pilot project included an evaluation component that was carried out in parallel with project implementation. The evaluation ran through September 1999. The results of the evaluation indicated that participating public librarians were enthusiastic about the training and information materials provided as part of the project and that many public libraries used the materials and conducted their own outreach to local communities and groups. Most libraries applied the modest funds to purchase additional Internet-accessible computers and/or upgrade their health-reference materials. However, few of the participating public libraries had health information centers (although health information was perceived as a top-ten or top-five topic of interest to patrons). Also, the project generated only minimal usage of NLM's consumer health database, known as MEDLINEplus, from the premises of the monitored libraries (patron usage from home or office locations was not tracked). The evaluation results suggested a balanced follow-up by NLM and the NN/LM, with a few carefully selected national activities, complemented by a package of targeted activities that, as of January 2000, are being planned, developed, or implemented. The results also highlighted the importance of building an evaluation component into projects like this one from the outset, to assure that objectives were met and that evaluative information was available on a timely basis, as was the case here.