Sources of selection bias in evaluating social programs: An interpretation of conventional measures and evidence on the effectiveness of matching as a program evaluation method

This paper decomposes the conventional measure of selection bias in observational studies into three components. The first two components are due to differences in the distributions of characteristics between participant and nonparticipant (comparison) group members: the first arises from differences in the supports, and the second from differences in densities over the region of common support. The third component arises from selection bias precisely defined. Using data from a recent social experiment, we find that the component due to selection bias, precisely defined, is smaller than the first two components. However, selection bias still represents a substantial fraction of the experimental impact estimate. The empirical performance of matching methods of program evaluation is also examined. We find that matching based on the propensity score eliminates some but not all of the measured selection bias, with the remaining bias still a substantial fraction of the estimated impact. We find that the support of the distribution of propensity scores for the comparison group is typically only a small portion of the support for the participant group. For values outside the common support, it is impossible to reliably estimate the effect of program participation using matching methods. If the impact of participation depends on the propensity score, as we find in our data, the failure of the common support condition severely limits matching compared with random assignment as an evaluation estimator.

Avaliação de impacto do Proagro Mais: um estudo de caso

Esta pesquisa avalia o impacto do \"Programa de Garantia da Atividade Agropecuária\" para agricultores familiares, conhecido como Proagro Mais. A relevância do trabalho fundamenta-se no considerável tamanho do Programa dentro do contexto das políticas de gestão de risco agrícola no Brasil. Além disso, é a primeira pesquisa desse tipo na literatura científica do país. A amostra é formada por produtores de milho do Estado do Paraná, tendo como linha base o ano de 2003, uma vez que é o ano anterior ao lançamento do Proagro Mais, e o ano de 2005 como ano de impacto. A base de dados utilizada neste estudo foi fornecida pelo Tribunal de Contas da União (TCU), cujas variáveis relevantes incluem características da cultura e dos agricultores familiares, como área financiada, atividades agrícolas complementares, educação e rendimento esperado. Adicionalmente, a partir de outras fontes públicas, foram adicionadas variáveis meteorológicas e regionais para controlar a localização da fazenda. O objetivo da pesquisa é avaliar o impacto do Proagro Mais sobre o montante de crédito por hectare concedido aos beneficiários do Programa. As metodologias usadas incluem o Propensity Score Matching (PSM), a Diferença das Diferenças (DID) e dois estimadores condicionais do DID com PSM usando dados em painel e repeated cross-section. As estimativas econométricas mostram que o Efeito Médio do Tratamento nos Tratados (EMTT) teve sinal negativo na maioria dos modelos revelando que, após o período de perda de rendimento, o grupo de controle teve um valor médio mais elevado de crédito por hectare do que os beneficiários do Proagro Mais. Os resultados sugerem a existência de mecanismos que poderiam complementar ou substituir o Proagro Mais como instrumento de gestão de risco agrícola, mas também podem sugerir que o Programa avaliado não cubra todos os riscos do setor.

Returns from Income Strategies in Rural Poland. Factor Markets Working Document No. 64, August 2013

In order to stabilise and improve their income situation, rural households are strongly encouraged to diversify their activities both within and outside the agricultural sector. Often, however, this advice is only moderately pursued. This paper addresses issues of rural household income diversification in the case of Poland. It investigates returns from rural household income strategies using propensity score matching methods and extensive datasets spanning 1998-2008. Results suggest that returns from combining farm and off-farm activities were lower than returns from concentrating on farming or on self-employment outside agriculture. This differential is stable over time although returns from diversification have relatively improved after Poland’s accession to the EU. This is also visible in the fact that since 2006 returns from combining farm and off-farm activities have evened with returns from relying solely on hired off-farm labour, thus smoothing the difference observed before the accession. Further, over the analysed period, households pursuing the diversification strategy performed better than those relying solely on unearned income. Finally, in general, the income in households combining farm and off-farm activities was higher than in those combining two off-farm income sources.

Efecto de las TIC en el rendimiento educativo: el Programa Conectar Igualdad en la Argentina

El objetivo de este trabajo consiste en determinar cuál es el “premio”, en términos de rendimiento educativo, de los estudiantes beneficiados con el Programa Conectar Igualdad con respecto a aquellos que no participaron en él. Con este fin, se emplea la técnica de emparejamiento o método de psm (propensity score matching). Se describen los antecedentes del Programa, así como el marco teórico con que se definen los factores explicativos del rendimiento educativo y que posiblemente inciden en la probabilidad de participar en el Programa. La población bajo estudio son los estudiantes de 15 años de la Argentina. Se emplean datos del Programa Internacional de Evaluación de Estudiantes (PISA) correspondientes al año 2012. De acuerdo con los resultados obtenidos, existen diferencias estadísticamente significativas en el rendimiento educativo promedio derivadas de la participación en el Programa Conectar Igualdad.

Methods to Control for Bias in Observational Studies

Thesis (Master's)--University of Washington, 2016-06

What is the Impact of the Underserved Pathway program on Entering an Underserved Family Medicine Residency? A Comparison of Three Approaches for Estimating the Average Treatment Effect

Thesis (Master's)--University of Washington, 2016-06

Performance of small and medium enterprises and the impact of environmental variables: evidence from Vietnam

thesis is developed from a real life application of performance evaluation of small and medium-sized enterprises (SMEs) in Vietnam. The thesis presents two main methodological developments on evaluation of dichotomous environment variable impacts on technical efficiency. Taking into account the selection bias the thesis proposes a revised frontier separation approach for the seminal Data Envelopment Analysis (DEA) model which was developed by Charnes, Cooper, and Rhodes (1981). The revised frontier separation approach is based on a nearest neighbour propensity score matching pairing treated SMEs with their counterfactuals on the propensity score. The thesis develops order-m frontier conditioning on propensity score from the conditional order-m approach proposed by Cazals, Florens, and Simar (2002), advocated by Daraio and Simar (2005). By this development, the thesis allows the application of the conditional order-m approach with a dichotomous environment variable taking into account the existence of the self-selection problem of impact evaluation. Monte Carlo style simulations have been built to examine the effectiveness of the aforementioned developments. Methodological developments of the thesis are applied in empirical studies to evaluate the impact of training programmes on the performance of food processing SMEs and the impact of exporting on technical efficiency of textile and garment SMEs of Vietnam. The analysis shows that training programmes have no significant impact on the technical efficiency of food processing SMEs. Moreover, the analysis confirms the conclusion of the export literature that exporters are self selected into the sector. The thesis finds no significant impact from exporting activities on technical efficiency of textile and garment SMEs. However, large bias has been eliminated by the proposed approach. Results of empirical studies contribute to the understanding of the impact of different environmental variables on the performance of SMEs. It helps policy makers to design proper policy supporting the development of Vietnamese SMEs.

The effects of foreign acquisition on domestic and export markets dynamics in China

Using recent data from the Chinese manufacturing industry and the generalised propensity score, this paper establishes economically significant causal effects of foreign acquisition on domestic and export markets dynamics.

Employment effects of privatisation and foreign acquisition of Chinese state-owned enterprises

This paper investigates the effects of domestic privatisation or foreign acquisition of Chinese State Owned Enterprises (SOEs) on employment growth, using firm level data for China and a combination of propensity score matching and difference-in-differences in order to identify the causal effect. Our results suggest that, controlling for output growth there is some evidence that domestic privatisation leads to contemporaneous reductions in employment growth compared to firms that did not undergo an ownership change. By contrast, there is some evidence that foreign acquisitions show higher employment growth in the post acquisition period than non-acquired SOEs.

Full and partial privatization in China:the labor consequences

This paper is the first paper to present findings evaluating the consequences for employees of full and partial privatization using difference-in-differences combined with propensity score matching. We find: (1) partial privatization causes job creation in contrast to full privatization, which destroys jobs, (2) full privatization causes higher labor productivity improvement than partial privatization, (3) wage increases occur only in partially privatized firms and (4) there are small increases in labor quality investment in both cases. The results suggest partial privatization exploits market discipline to induce labor productivity whilst simultaneously providing welfare improvements for labor. This is the ‘win-win’ outcome predicted by the ‘helping hand’ theory of government. Our results suggest that governments are likely to gain wider support for a program of partial privatization rather than full privatization.

Consumer trial, continuous use, and economic benefits of a retail service innovation:the case of the personal shopping assistant

Service innovations in retailing have the potential to benefit consumers as well as retailers. This research models key factors associated with the trial and continuous use of a specific self-service technology (SST), the personal shopping assistant (PSA), and estimates retailer benefits from implementing that innovation. Based on theoretical insights from prior SST studies, diffusion of innovation literature, and the technology acceptance model (TAM), this study develops specific hypotheses and tests them on a sample of 104 actual users of the PSA and 345 nonusers who shopped at the retail store offering the PSA device. Results indicate that factors affecting initial trial are different from those affecting continuous use. More specifically, consumers' trust toward the retailer, novelty seeking, and market mavenism are positively related to trial, while technology anxiety hinders the likelihood of trying the PSA. Perceived ease of use of the device positively impacts continuous use while consumers' need for interaction in shopping environments reduces the likelihood of continuous use. Importantly, there is evidence on retailer benefits from introducing the innovation since consumers using the PSA tend to spend more during each shopping trip. However, given the high costs of technology, the payback period for recovery of investments in innovation depends largely upon continued use of the innovation by consumers. Important implications are provided for retailers considering investments in new in-store service innovations. Incorporation of technology within physical stores affords opportunities for the retailer to reduce costs, while enhancing service provided to consumers. Therefore, service innovations in retailing have the potential to benefit consumers as well as retailers. This research models key factors associated with the trial and continuous use of a specific SST in the retail context, the PSA, and estimates retailer benefits from implementing that innovation. In so doing, the study contributes to the nascent area of research on SSTs in the retail sector. Based on theoretical insights from prior SST studies, diffusion of innovation literature, and the TAM, this study develops specific hypotheses regarding the (1) antecedent effects of technological anxiety, novelty seeking, market mavenism, and trust in the retailer on trial of the service innovation; (2) the effects of ease of use, perceived waiting time, and need for interaction on continuous use of the innovation; and (3) the effect of use of innovation on consumer spending at the store. The hypotheses were tested on a sample of 104 actual users of the PSA and 345 nonusers who shopped at the retail store offering the PSA device, one of the early adopters of PSA in Germany. Data were analyzed using logistic regression (antecedents of trial), multiple regression (antecedents of continuous use), and propensity score matching (assessing retailer benefits). Results indicate that factors affecting initial trial are different from those affecting continuous use. More specifically, consumers' trust toward the retailer, novelty seeking, and market mavenism are positively related to trial, while technology anxiety hinders the likelihood of trying the PSA. Perceived ease of use of the device positively impacts continuous use, while consumers' need for interaction in shopping environments reduces the likelihood of continuous use. Importantly, there is evidence on retailer benefits from introducing the innovation since consumers using the PSA tend to spend more during each shopping trip. However, given the high costs of technology, the payback period for recovery of investments in innovation depends largely upon continued use of the innovation by consumers. Important implications are provided for retailers considering investments in new in-store service innovations. The study contributes to the literature through its (1) simultaneous examination of antecedents of trial and continuous usage of a specific SST, (2) the demonstration of economic benefits of SST introduction for the retailer, and (3) contribution to the stream of research on service innovation, as against product innovation.

Do work placements improve final year academic performance or do high-calibre students choose to do work placements?

This study investigates whether the completion of an optional sandwich work placement enhances student performance in final year examinations. Using Propensity Score Matching, our analysis departs from the literature by controlling for self-selection. Previous studies may have overestimated the impact of sandwich work placements on performance because it might be the case that high-calibre students choose to go on placement. Our results, utilising a large student data set, indicate that self-selection is present, but the effects of a placement on student performance still have an impact. This robust finding is found to be of a remarkably similar magnitude across two UK universities.

Investment liberalisation, technology take-off and export markets entry:does foreign ownership structure matter?

Before and after its accession to the WTO in 2001, China has undergone a far-reaching investment liberalisation. As part of this, existing restrictions on foreign ownership structure and mandatory export and technology transfer requirements imposed on foreign firms have been lifted in a number of industries. Against this background we identify the causal effects of foreign acquisitions on export market entry and technology take-off and evaluate whether the level of foreign ownership plays a role in stimulating these changes. Using doubly robust propensity score reweighted bivariate probit regressions to control for the selection bias associated with firm level foreign acquisition incidences, we uncover strong but heterogeneous positive effects on export activity for all types of foreign ownership structure. We also find that minority foreign owned acquisition targets experience higher likelihood of R&D, providing evidence that joint ventures can contribute positively to China's "science and technology take-off".

Wealth shocks, credit-supply shocks, and asset allocation:evidence from household and firm portfolios

We use a unique dataset with bank clients’ security holdings for all German banks to examine how macroeconomic shocks affect asset allocation preferences of households and non-financial firms. Our analysis focuses on two alternative mechanisms which can influence portfolio choice: wealth shocks, which are represented by the sovereign debt crisis in the Eurozone, and credit-supply shocks which arise from reductions in borrowing abilities during bank distress. We document het- erogeneous responses to these two types of shocks. While households with large holdings of secu- rities from stressed Eurozone countries (Greece, Ireland, Italy, Portugal, and Spain) decrease the degree of concentration in their security portfolio as a result of the Eurozone crisis, non-financial firms with similar levels of holdings from stressed Eurozone countries do not. Credit-supply shocks at the bank level (caused by bank distress) result in lower concentration, for both households and non-financial corporations. We also show that only shocks to corporate credit bear ramifications on bank clients’ portfolio concentration, while shocks in retail credit are inconsequential. Our results are robust to falsification tests, propensity score matching techniques, and instrumental variables estimation.

Firm-level productivity growth in Northern Ireland:the impact of exporting, innovation and public financial assistance

Motivated by the historically poor productivity performance of Northern Ireland firms and the longstanding productivity gap with the UK, the aim of this thesis is to examine, through the use of firm-level data, how exporting, innovation and public financial assistance impact on firm productivity growth. These particular activities are investigated due to the continued policy focus on their link to productivity growth and the theoretical claims of a direct positive relationship. In order to undertake these analyses a newly constructed dataset is used which links together cross-sectional and longitudinal data over the 1998-2008 period from the Annual Business Survey, the Manufacturing Sales and Export Survey; the Community Innovation Survey and Invest NI Selective Financial Assistance (SFA) payment data. Econometric methodologies are employed to estimate each of the relationships with regards to productivity growth, making use in particular of Heckman selection techniques and propensity score matching to take account of critical issues of endogeneity and selection bias. The results show that more productive firms self-select into exporting but there is no resulting productivity effect from starting to export; contesting the argument for learning-by-exporting. Product innovation is also found to have no impact on productivity growth over a four year period but there is evidence of a negative process innovation impact, likely to reflect temporary learning effects. Finally SFA assistance, including the amount of the payment, is found to have no short term impact on productivity growth suggesting substantial deadweight effects and/or targeting of inefficient firms. The results provide partial evidence as to why Northern Ireland has failed to narrow the productivity gap with the rest of the UK. The analyses further highlight the need for access to comprehensive firm-level data for research purposes, not least to underpin robust evidence-based policymaking.