Skip to main content
Top
Published in: Prevention Science 3/2019

01-04-2019

Sample Size Planning for Cluster-Randomized Interventions Probing Multilevel Mediation

Published in: Prevention Science | Issue 3/2019

Login to get access

Abstract

Multilevel mediation analyses play an essential role in helping researchers develop, probe, and refine theories of action underlying interventions and document how interventions impact outcomes. However, little is known about how to plan studies with sufficient power to detect such multilevel mediation effects. In this study, we describe how to prospectively estimate power and identify sufficient sample sizes for experiments intended to detect multilevel mediation effects. We outline a simple approach to estimate the power to detect mediation effects with individual- or cluster-level mediators using summary statistics easily obtained from empirical literature and the anticipated magnitude of the mediation effect. We draw on a running example to illustrate several different types of mediation and provide an accessible introduction to the design of multilevel mediation studies. The power formulas are implemented in the R package PowerUpR and the PowerUp software (causalevaluation​.​org).
Footnotes
1
Core assumptions include (a) stable unit treatment value assumption, (b) sequential ignorability, (c) consistency, (d) no downstream confounders, and (e) no treatment-by-mediator interaction.
 
Literature
go back to reference Cox, K. & Kelcey, B. (2018). Optimal Sample Allocation in Group-randomized Studies of Multilevel Mediation with a Group-level Mediator. Journal of Experimental Education (in press). Cox, K. & Kelcey, B. (2018). Optimal Sample Allocation in Group-randomized Studies of Multilevel Mediation with a Group-level Mediator. Journal of Experimental Education (in press).
go back to reference Dong, N., Kelcey, B., Spybrook, J., & Maynard, R. A. (2016). PowerUp!-Mediator: A tool for calculating statistical power for causally-defined mediation in cluster randomized trials. (Version 0.4) [Software]. Available from http://www.causalevaluation.org/. Accessed 1 May 2007. Dong, N., Kelcey, B., Spybrook, J., & Maynard, R. A. (2016). PowerUp!-Mediator: A tool for calculating statistical power for causally-defined mediation in cluster randomized trials. (Version 0.4) [Software]. Available from http://​www.​causalevaluation​.​org/​. Accessed 1 May 2007.
go back to reference Gottfredson, D. C., Cook, T. D., Gardner, F. E., Gorman-Smith, D., Howe, G. W., Sandler, I. N., & Zafft, K. M. (2015). Standards of evidence for efficacy, effectiveness, and scale-up research in prevention science: Next generation. Prevention Science, 16, 893–926.CrossRefPubMedPubMedCentral Gottfredson, D. C., Cook, T. D., Gardner, F. E., Gorman-Smith, D., Howe, G. W., Sandler, I. N., & Zafft, K. M. (2015). Standards of evidence for efficacy, effectiveness, and scale-up research in prevention science: Next generation. Prevention Science, 16, 893–926.CrossRefPubMedPubMedCentral
go back to reference Kelcey, B., & Phelps, G. (2013). Considerations for Designing Group Randomized Trials of Professional Development with Teacher Knowledge Outcomes. Educational Evaluation and Policy Analysis, 35, 370–390. Kelcey, B., & Phelps, G. (2013). Considerations for Designing Group Randomized Trials of Professional Development with Teacher Knowledge Outcomes. Educational Evaluation and Policy Analysis, 35, 370–390.
go back to reference Kelcey, B., Dong, N., Spybrook, J., & Cox, K. (2017). Statistical power for causally-defined indirect effects in group-randomized trials. Journal of Educational and Behavioral Statistics. Kelcey, B., Dong, N., Spybrook, J., & Cox, K. (2017). Statistical power for causally-defined indirect effects in group-randomized trials. Journal of Educational and Behavioral Statistics.
go back to reference Kelcey, B., Dong, N., Spybrook, J., & Shen, Z. (2017a). Experimental Power for Indirect Effects in Group-randomized Studies with Group-level Mediators. Multivariate Behavioral Research, 52(6), 699–719. Kelcey, B., Dong, N., Spybrook, J., & Shen, Z. (2017a). Experimental Power for Indirect Effects in Group-randomized Studies with Group-level Mediators. Multivariate Behavioral Research, 52(6), 699–719.
go back to reference Kelcey, B., Dong, N., Spybrook, J. & Cox, K. (2017b). Statistical power for causally-defined indirect effects in group-randomized trials. Journal of Educational and Behavioral Statistics. Kelcey, B., Dong, N., Spybrook, J. & Cox, K. (2017b). Statistical power for causally-defined indirect effects in group-randomized trials. Journal of Educational and Behavioral Statistics.
go back to reference Kelcey, B. & Shen, Z. (2018). Effective and Efficient Experimental Design 2-1-1 Mediation Studies. Journal of Experimental Education (in press). Kelcey, B. & Shen, Z. (2018). Effective and Efficient Experimental Design 2-1-1 Mediation Studies. Journal of Experimental Education (in press).
go back to reference Krull, J. L., & MacKinnon, D. P. (2001). Multilevel modeling of individual and cluster level mediated effects. Multivariate Behavioral Research, 36, 249–277.CrossRefPubMed Krull, J. L., & MacKinnon, D. P. (2001). Multilevel modeling of individual and cluster level mediated effects. Multivariate Behavioral Research, 36, 249–277.CrossRefPubMed
go back to reference Phelps, G., Kelcey, B., Liu, S., & Jones, N. (2016). Informing Estimates of Program Effects for Studies of Mathematics Professional Development Using Teacher Content Knowledge Outcomes. Evaluation Review, 40, 383–409.CrossRef Phelps, G., Kelcey, B., Liu, S., & Jones, N. (2016). Informing Estimates of Program Effects for Studies of Mathematics Professional Development Using Teacher Content Knowledge Outcomes. Evaluation Review, 40, 383–409.CrossRef
go back to reference Pituch, K. A., & Stapleton, L. M. (2012). Distinguishing between cross- and cluster-level mediation processes in the cluster randomized trial. Sociological Methods & Research, 41, 630–670.CrossRef Pituch, K. A., & Stapleton, L. M. (2012). Distinguishing between cross- and cluster-level mediation processes in the cluster randomized trial. Sociological Methods & Research, 41, 630–670.CrossRef
go back to reference Preacher, K. J., & Selig, J. P. (2012). Advantages of Monte Carlo confidence intervals for indirect effects. Communication Methods and Measures, 6(2), 77–98.CrossRef Preacher, K. J., & Selig, J. P. (2012). Advantages of Monte Carlo confidence intervals for indirect effects. Communication Methods and Measures, 6(2), 77–98.CrossRef
go back to reference Texas Christian University [TCU] (2005) Institute of Behavioral Research. The Organizational Readiness for Change: Treatment Staff Version (TCU ORC-S). Available at: www.ibr.tcu.edu. Texas Christian University [TCU] (2005) Institute of Behavioral Research. The Organizational Readiness for Change: Treatment Staff Version (TCU ORC-S). Available at: www.​ibr.​tcu.​edu.
go back to reference Sobel, M. E. (1982). Asymptotic confidence intervals for indirect effects in structural equation models. Sociological Methodology, 13, 290–312.CrossRef Sobel, M. E. (1982). Asymptotic confidence intervals for indirect effects in structural equation models. Sociological Methodology, 13, 290–312.CrossRef
go back to reference Schwartz, R. (2010). Motivational interviewing (patient-centered counseling) to address childhood obesity. Pediatric Annals, 39(3), 154–158.CrossRefPubMed Schwartz, R. (2010). Motivational interviewing (patient-centered counseling) to address childhood obesity. Pediatric Annals, 39(3), 154–158.CrossRefPubMed
go back to reference Spybrook, J., Shi, R., & Kelcey, B. (2016). Progress in the Past Decade: An Examination of the Precision of Cluster Randomized Trials Funded by the U.S. Institute of Education Sciences. International Journal of Research & Method in Education, 39(3), 255–267.CrossRef Spybrook, J., Shi, R., & Kelcey, B. (2016). Progress in the Past Decade: An Examination of the Precision of Cluster Randomized Trials Funded by the U.S. Institute of Education Sciences. International Journal of Research & Method in Education, 39(3), 255–267.CrossRef
go back to reference Stapleton, L., Pituch, K., & Dion, E. (2015). Standardized effect size measures for mediation analysis in clusterrandomized trials. Journal of Experimental Education, 83(4), 547–582.CrossRef Stapleton, L., Pituch, K., & Dion, E. (2015). Standardized effect size measures for mediation analysis in clusterrandomized trials. Journal of Experimental Education, 83(4), 547–582.CrossRef
go back to reference VanderWeele, T. J. (2010). Direct and indirect effects for neighborhood-based clustered and longitudinal data. Sociological Methods & Research, 38, 515–544.CrossRef VanderWeele, T. J. (2010). Direct and indirect effects for neighborhood-based clustered and longitudinal data. Sociological Methods & Research, 38, 515–544.CrossRef
go back to reference Williams, N., & Glisson, C. (2014). Testing a theory or organizational culture, climate and youth outcomes in child welfare systems: A United States national study. Child Abuse and Neglect, 38, 4.CrossRef Williams, N., & Glisson, C. (2014). Testing a theory or organizational culture, climate and youth outcomes in child welfare systems: A United States national study. Child Abuse and Neglect, 38, 4.CrossRef
go back to reference Zhang, Z., Zyphur, M., & Preacher, K. (2009). Testing multilevel mediation using hierarchical linear models: Problems and solutions. Organizational Research Methods, 12, 695–719.CrossRef Zhang, Z., Zyphur, M., & Preacher, K. (2009). Testing multilevel mediation using hierarchical linear models: Problems and solutions. Organizational Research Methods, 12, 695–719.CrossRef
Metadata
Title
Sample Size Planning for Cluster-Randomized Interventions Probing Multilevel Mediation
Publication date
01-04-2019
Published in
Prevention Science / Issue 3/2019
Print ISSN: 1389-4986
Electronic ISSN: 1573-6695
DOI
https://doi.org/10.1007/s11121-018-0921-6

Other articles of this Issue 3/2019

Prevention Science 3/2019 Go to the issue