Sources of selection bias in evaluating social programs: An interpretation of conventional measures and evidence on the effectiveness of matching as a program evaluation method

This paper decomposes the conventional measure of selection bias in observational studies into three components. The first two components are due to differences in the distributions of characteristics between participant and nonparticipant (comparison) group members: the first arises from differences in the supports, and the second from differences in densities over the region of common support. The third component arises from selection bias precisely defined. Using data from a recent social experiment, we find that the component due to selection bias, precisely defined, is smaller than the first two components. However, selection bias still represents a substantial fraction of the experimental impact estimate. The empirical performance of matching methods of program evaluation is also examined. We find that matching based on the propensity score eliminates some but not all of the measured selection bias, with the remaining bias still a substantial fraction of the estimated impact. We find that the support of the distribution of propensity scores for the comparison group is typically only a small portion of the support for the participant group. For values outside the common support, it is impossible to reliably estimate the effect of program participation using matching methods. If the impact of participation depends on the propensity score, as we find in our data, the failure of the common support condition severely limits matching compared with random assignment as an evaluation estimator.

Chemical phenotype matching between a plant and its insect herbivore

Two potential outcomes of a coevolutionary interaction are an escalating arms race and stable cycling. The general expectation has been that arms races predominate in cases of polygenic inheritance of resistance traits and permanent cycling predominates in cases in which resistance is controlled by major genes. In the interaction between Depressaria pastinacella, the parsnip webworm, and Pastinaca sativa, the wild parsnip, traits for plant resistance to insect herbivory (production of defensive furanocoumarins) as well as traits for herbivore “virulence” (ability to metabolize furanocoumarins) are characterized by continuous heritable variation. Furanocoumarin production in plants and rates of metabolism in insects were compared among four midwestern populations; these traits then were classified into four clusters describing multitrait phenotypes occurring in all or most of the populations. When the frequency of plant phenotypes belonging to each of the clusters is compared with the frequency of the insect phenotypes in each of the clusters across populations, a remarkable degree of frequency matching is revealed in three of the populations. That frequencies of phenotypes vary among populations is consistent with the fact that spatial variation occurs in the temporal cycling of phenotypes; such processes contribute in generating a geographic mosaic in this coevolutionary interaction on the landscape scale. Comparisons of contemporary plant phenotype distributions with phenotypes of herbarium specimens collected 9–125 years ago from across a similar latitudinal gradient, however, suggest that for at least one resistance trait—sphondin concentration—interactions with webworms have led to escalatory change.