Skip to main content
Top

17-06-2024 | Shock

Markets matter: a simulation study of the bias-variance trade-off in comparison group selection for difference-in-differences analysis

Authors: Lauren Vollmer Forrow, Jason Rotter, Laura Blue, Jake Vogler, Laura A. Hatfield

Published in: Health Services and Outcomes Research Methodology

Login to get access

Abstract

Recent research reveals many pitfalls when selecting a comparison group for difference-in-differences analyses of observational data. Recommendations from this research, although important, can be difficult to apply to a complex, real-world evaluation. In this study we explore the potential value of tailored simulation studies to inform comparison group designs for difference-in-differences analysis. We developed a simulation study mirroring the evaluation of Primary Care First (PCF), a primary care transformation model offered in select regions of the United States. We simulated primary care practices that volunteer for the intervention (intervention group) and two potential comparison groups: non-participating practices in PCF regions (within-market) and practices outside PCF regions (out-of-market). We assumed within-market comparison (i.e., non-participating) practices differ systematically from participating practices, whereas out-of-market practices are more similar to participating practices but experience different market-level outcome trends and shocks. We used Medicare fee-for-service spending data to quantify the likely magnitude of these forces. Using difference-in-differences analysis, we then estimated a hypothetical intervention’s impact relative to each comparison group. We compared the mean squared error (MSE) of these impact estimates between the two comparison groups. The simulation revealed a bias-variance trade-off: the greater variation in impact estimates for the out-of-market comparison group generally led to higher MSE than the greater systematic bias of impact estimates for the within-market comparison group. More generally, a bespoke simulation study grounded in real data can quantify forces relevant to comparison group design and offer guidance to researchers seeking the best approach for their own evaluations.
Appendix
Available only for authorised users
Literature
go back to reference Angrist, J.D., Pischke, J.-S.: The credibility revolution in empirical economics: How Better Research Design is taking the con out of Econometrics. J. Economic Perspect. 24(2), 3–30 (2010)CrossRef Angrist, J.D., Pischke, J.-S.: The credibility revolution in empirical economics: How Better Research Design is taking the con out of Econometrics. J. Economic Perspect. 24(2), 3–30 (2010)CrossRef
go back to reference Ben-Michael, E., Feller, A., Stuart, E.A.: A trial Emulation Approach for Policy evaluations with Group-Level Longitudinal Data. Epidemiology. 32(4), 533–540 (2021)CrossRefPubMedPubMedCentral Ben-Michael, E., Feller, A., Stuart, E.A.: A trial Emulation Approach for Policy evaluations with Group-Level Longitudinal Data. Epidemiology. 32(4), 533–540 (2021)CrossRefPubMedPubMedCentral
go back to reference Bilinski, A., Hatfield, L.A.: Nothing to see Here? Non-inferiority approaches to parallel trends and other Model assumptions. arXiv Preprint. (2018). arXiv:1805.03273 Bilinski, A., Hatfield, L.A.: Nothing to see Here? Non-inferiority approaches to parallel trends and other Model assumptions. arXiv Preprint. (2018). arXiv:1805.03273
go back to reference Callaway, B., Li, T.: Policy evaluation during a pandemic. J. Econ. 236(1), 105454 (2023)CrossRef Callaway, B., Li, T.: Policy evaluation during a pandemic. J. Econ. 236(1), 105454 (2023)CrossRef
go back to reference Cefalu, M., Vegetabile, B.G., Dworsky, M., Eibner, C., Girosi, F.: Reducing Bias in difference-in-differences models using Entropy Balancing. arXiv Preprint (2020). arXiv:2011.04826. Cefalu, M., Vegetabile, B.G., Dworsky, M., Eibner, C., Girosi, F.: Reducing Bias in difference-in-differences models using Entropy Balancing. arXiv Preprint (2020). arXiv:2011.04826.
go back to reference Dette, H., Schumann, M.: Testing for equivalence of Pre-trends in difference-in-differences estimation. J. Bus. Economic Stat., 1–35. (2024) Dette, H., Schumann, M.: Testing for equivalence of Pre-trends in difference-in-differences estimation. J. Bus. Economic Stat., 1–35. (2024)
go back to reference Freyaldenhoven, S., Hansen, C., Shapiro, J.M.: Pre-event trends in the Panel Event Study Design. Am. Econ. Rev. 109(9), 3307–3338 (2019)CrossRef Freyaldenhoven, S., Hansen, C., Shapiro, J.M.: Pre-event trends in the Panel Event Study Design. Am. Econ. Rev. 109(9), 3307–3338 (2019)CrossRef
go back to reference Haber, N.A., Clarke-Deelder, E., Salomon, J.A., Feller, A., Stuart, E.A.: Impact evaluation of Coronavirus Disease 2019 policy: A guide to Common Design issues. Am. J. Epidemiol. 190(11), 2474–2486 (2021)CrossRefPubMed Haber, N.A., Clarke-Deelder, E., Salomon, J.A., Feller, A., Stuart, E.A.: Impact evaluation of Coronavirus Disease 2019 policy: A guide to Common Design issues. Am. J. Epidemiol. 190(11), 2474–2486 (2021)CrossRefPubMed
go back to reference Hartman, E., Hidalgo, F.D.: An Equivalence Approach to Balance and Placebo tests. Am. J. Polit. Sci. 62(4), 1000–1013 (2018)CrossRef Hartman, E., Hidalgo, F.D.: An Equivalence Approach to Balance and Placebo tests. Am. J. Polit. Sci. 62(4), 1000–1013 (2018)CrossRef
go back to reference Kahn-Lang, A., Lang, K.: The Promise and pitfalls of differences-in-Differences: Reflections on 16 and pregnant and other applications. J. Bus. Economic Stat. 38(3), 613–620 (2020)CrossRef Kahn-Lang, A., Lang, K.: The Promise and pitfalls of differences-in-Differences: Reflections on 16 and pregnant and other applications. J. Bus. Economic Stat. 38(3), 613–620 (2020)CrossRef
go back to reference Rambachan, A., Roth, J.: A more credible Approach to parallel trends. Rev. Econ. Stud. 90(5), 2555–2591 (2023)CrossRef Rambachan, A., Roth, J.: A more credible Approach to parallel trends. Rev. Econ. Stud. 90(5), 2555–2591 (2023)CrossRef
go back to reference Roth, J.: Pretest with caution: Event-study estimates after testing for parallel trends. Am. Economic Review: Insights. 4(3), 305–322 (2022) Roth, J.: Pretest with caution: Event-study estimates after testing for parallel trends. Am. Economic Review: Insights. 4(3), 305–322 (2022)
go back to reference Ryan, A.M., Kontopantelis, E., Linden, A., Burgess, J.F. Jr.: Now trending: Coping with Non-parallel trends in difference-in-differences analysis. Stat. Methods Med. Res. 28(12), 3697–3711 (2019)CrossRefPubMed Ryan, A.M., Kontopantelis, E., Linden, A., Burgess, J.F. Jr.: Now trending: Coping with Non-parallel trends in difference-in-differences analysis. Stat. Methods Med. Res. 28(12), 3697–3711 (2019)CrossRefPubMed
go back to reference Sun, L., and S Abraham: Estimating Dynamic Treatment effects in Event studies with Heterogeneous Treatment effects. J. Econ. 225(2), 175–199 (2021)CrossRef Sun, L., and S Abraham: Estimating Dynamic Treatment effects in Event studies with Heterogeneous Treatment effects. J. Econ. 225(2), 175–199 (2021)CrossRef
go back to reference Wing, C., Simon, K., Bello-Gomez, R.A.: Designing Difference in Difference studies: Best Practices for Public Health Policy Research. Annu. Rev. Public Health 66. (2018) Wing, C., Simon, K., Bello-Gomez, R.A.: Designing Difference in Difference studies: Best Practices for Public Health Policy Research. Annu. Rev. Public Health 66. (2018)
Metadata
Title
Markets matter: a simulation study of the bias-variance trade-off in comparison group selection for difference-in-differences analysis
Authors
Lauren Vollmer Forrow
Jason Rotter
Laura Blue
Jake Vogler
Laura A. Hatfield
Publication date
17-06-2024
Publisher
Springer US
Keywords
Shock
Shock
Published in
Health Services and Outcomes Research Methodology
Print ISSN: 1387-3741
Electronic ISSN: 1572-9400
DOI
https://doi.org/10.1007/s10742-024-00332-7