Abstract
A frequent challenge in program impact estimation, and causal modeling more generally, is estimation of the effect of a binary endogenous variable on a binary outcome of interest. We report results from Monte Carlo experiments designed to assess the performance of estimators frequently applied in this circumstance. Many rely on an instrumental variables identification strategy and in those instances our central interest is the overidentified case. Even when identification is technically achieved by functional form, it is widely perceived that instruments generate more credible identification. Our focus is on widely used models available in the popular STATA statistical software package, but we also evaluate a semi-parametric instrumental variables random effects model not yet available in STATA. The parameters of interest in these experiments are program impact, test statistics assessing endogeneity and overidentification tests. We consider performance under alternative behavioral circumstances by varying distributional assumptions for unobservables, instrument strength levels, sample sizes, and impact magnitudes. Some models turn in a somewhat disappointing performance. Those that rely on joint normality for identification are not particularly robust to error misspecification, raising questions about whether they should be preferred to the semi-parametric estimator (regardless of comparative ease of estimation) or even to simple single equation models that ignore endogeneity. We provide examples of the methods using data from Bangladesh and Tanzania.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Notes
- 1.
The authors are currently writing STATA commands to implement this estimator.
- 2.
We did consider predictor substitution schemes as well but, as expected, they performed poorly and we do not include them in the comparisons.
- 3.
That is, the program enrollment prevalence within the sample.
- 4.
We are grateful to Stas Kolenikov for generously sharing a STATA.ado file that he wrote implementing that Vale and Maurelli (1983) procedure.
- 5.
Experimentation suggests that variation in the values assigned to these coefficient terms had very little impact on the statistics of interest in this study.
- 6.
Step 1 was actually slightly more involved. It became apparent in early rounds of experiments that some behavioral parameters, particularly instrument strength, occasionally varied across replications to a degree with which the authors were not comfortable. In particular, the various replications from experiments involving first stage χ 2 statistics with target values of 15 and 25 occasionally produced overlapping ranges for the χ 2 statistic values actually generated across the replications for the two experiments. This muddied the waters somewhat for the purposes of making inferences about estimator performance differentials as instrument strength varied. To address this, we set tolerance bands for acceptable variation of such χ 2 values around their target for a given experiment. If, on a particular replication, a draw {ε 1, ε 2} resulted in a χ 2 value outside of the tolerance range for that experiment, that draw was discarded and a new draw {ε 1, ε 2} was made. This was done to insure that the replications within an experiment conformed to an acceptable degree to the parameters of that experiment.
- 7.
As explained in Sect. 2.3, the behavioral parameters are imposed by the design of the data generating process for each experiment and included the: program effect (\(Pr(Y _{2}\vert X,Y _{1} = 1) - Pr(Y _{2}\vert X,Y _{1} = 0)\)); correlation of the errors {ε 1, ε 2}; average of the program outcome (Y 1) within the sample; average of the outcome of interest (Y 2) within the sample; first stage strength of the instruments Z to explain Y 1 (as reflected in the χ 2 statistic emerging from a test of the joint significance of those instruments); and bivariate error type (i.e. normal or a non-normal errors).
- 8.
Recall that the overidentification test statistic for the bivariate probit model is simply the χ 2 statistic for a test of the joint significance of the instruments in the marginal probit equation for Y 2 under the “just identified” specification under which the instruments appear in both marginal probit equations and identification rests on nonlinearity from functional form (i.e. joint normality) alone. The null hypothesis of such a test is that the instruments are not jointly significant regressors in marginal probit equation for Y 2 (i.e. that they are legitimately excluded from the marginal probit equation for Y 2).
- 9.
References
Anderson T, Rubin H (1950) The asymptotic properties of estimates of the parameters of a single equation in a complete system of stochastic equations. Ann Math Stat 21:570–582
Angrist J, Krueger A (2001) Instrumental variables and the search for identification: from supply and demand to natural experiments. J Econ Perspect 15:69–85
Angrist J, Pischke J (2009) Mostly harmless econometrics: an empiricist’s companion. Princeton University Press, Princeton
Babalola S (2005) Communication, ideation and contraceptive use in Burkina Faso: an application of the propensity score matching method. J Fam Plan Reprod Health Care 31:207–212
Bassman R (1960) On finite sample distributions of generalized classical linear identifiability test statistics. J Am Stat Assoc 55:650–659
Bauman K, Viadro C, Tsui A (1993) Family planning program effects in developing countries: conclusions and related considerations. The evaluation project working paper IM-03-03
Bollen K, Guilkey D, Mroz T (1995) Binary outcomes and endogenous explanatory variables: tests and solutions with an application to the demand for contraceptive use in Tunisia. Demography 32:111–131
Bound J, Jaeger D, Baker R (1995) Problems with instrumental variables estimation when the correlation between the instruments and the endogenous explanatory variable is weak. J Am Stat Assoc 90:443–450
Cappellari L, Jenkins S (2003) Multivariate probit regression using simulated maximum likelihood. STATA J 3:278–294
Chen S, Guilkey D (2003) Determinants of contraceptive method choice in rural Tanzania between 1991 and 1999. Stud Fam Plan 34:263–276
Chiburis RC, Das J, Lokshin M (2011) A practical comparison of the bivariate probit and linear IV estimators. The World Bank Policy research working paper 5601
Durbin J (1954) Errors in variables. Rev Int Stat Inst 22:23–32
Fleishman A (1978) A method for simulating nonnormal distributions. Psychometrika 43:521–532
Gourieroux C, Monfort A, Renault E, Trognon A (1987) Generalized residuals. J Econom 34:5–32
Guilkey D, Hutchinson P (2011) Overcoming methodological challenges in evaluating health communication campaigns: evidence from rural Bangladesh. Stud Fam Plan 42:93–106
Guilkey D, Mroz T, Taylor L (1992) Estimation and testing in simultaneous equations models with discrete outcomes using cross section data. UNC-CH Department of Economics working paper
Guilkey D, Hutchinson P, Lance P (2006) Cost effectiveness analysis for health communications programs. J Health Commun 11:47–67
Hansen L (1982) Large sample properties of generalized method of moments estimators. Econometrica 50:1029–1054
Hausman J (1978) Specification tests in econometrics. Econometrica 46:1251–1271
Hayashi F (2000) Econometrics. Princeton University Press, Princeton
Heckman J, Singer B (1984) A method for minimizing the impact of distributional assumptions in econometric models for duration data. Econometrica 52:271–320
Hutchinson P, Wheeler J (2006). The cost effectiveness of health communication programs: what do we know? J Health Commun 11:7–45
Imbens G, Angrist J (1994) Indentification and estimation of local average treatment effects. Econometrica 62:467–475
Kaiser H, Dickman K (1962) Sample and population score matrices and sample correlation matrices from an arbitrary population correlation matrix. Psychometrika 27:179–182
LaLonde R (1986) Evaluating the econometric evaluations of training programs with experimental data. Am Econ Rev 76:604–620
Manning W, Duan N, Rogers W (1987) Monte Carlo evidence on the choice between sample selection and two-part models. J Econom 35:59–82
Mwaikambo L, Speizer I, Schurmann A, Morgan G, Fikree F (2011) What works in family planning interventions: a systematic review. Stud Fam Plan 42:67–82
Mroz T (1999) Discrete factor approximations in simultaneous equations models: estimating the impact of a dummy endogenous variable on a continuous outcome. J Econom 92:233–274
Ngallaba S, Kapiga S, Ruyoba I, Boerma J (1993) Tanzania demographic and health survey 1991/1992. Macro International Inc., Columbia
Rivers D, Vuong Q (1988) Limited information estimators and exogeneity tests for simultaneous probit models. J Econom 39:347–366
Sargon J (1958) The estimation of economic relationships using instrumental variables. Econometrica 26:393–415
Stock J, Staiger D (1997) Instrumental variables regression with weak instruments. Econometrica 65:557–586
Terza J, Basu A, Rathouz P (2008) Two-stage residual inclusion estimation: addressing endogeneity in health econometric modelling. J Health Econ 27:531–543
Vale C, Maurelli V (1983) Simulating multivariate nonnormal distributions. Psychometrika 48:465–471
Wu D (1974) Alternative tests of independence between stochastic regressors and disturbances: finite sample results. Econometrica 42:529–546
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2014 Springer Science+Business Media New York
About this chapter
Cite this chapter
Guilkey, D.K., Lance, P.M. (2014). Program Impact Estimation with Binary Outcome Variables: Monte Carlo Results for Alternative Estimators and Empirical Examples. In: Sickles, R., Horrace, W. (eds) Festschrift in Honor of Peter Schmidt. Springer, New York, NY. https://doi.org/10.1007/978-1-4899-8008-3_2
Download citation
DOI: https://doi.org/10.1007/978-1-4899-8008-3_2
Published:
Publisher Name: Springer, New York, NY
Print ISBN: 978-1-4899-8007-6
Online ISBN: 978-1-4899-8008-3
eBook Packages: Business and EconomicsEconomics and Finance (R0)