Top

Published in:

01-10-2015

Sample Size Considerations in Prevention Research Applications of Multilevel Modeling and Structural Equation Modeling

Authors: Rick H. Hoyle, Nisha C. Gottfredson

Published in: Prevention Science | Issue 7/2015

Abstract

When the goal of prevention research is to capture in statistical models some measure of the dynamic complexity in structures and processes implicated in problem behavior and its prevention, approaches such as multilevel modeling (MLM) and structural equation modeling (SEM) are indicated. Yet the assumptions that must be satisfied if these approaches are to be used responsibly raise concerns regarding their use in prevention research involving smaller samples. In this article, we discuss in nontechnical terms the role of sample size in MLM and SEM and present findings from the latest simulation work on the performance of each approach at sample sizes typical of prevention research. For each statistical approach, we draw from extant simulation studies to establish lower bounds for sample size (e.g., MLM can be applied with as few as ten groups comprising ten members with normally distributed data, restricted maximum likelihood estimation, and a focus on fixed effects; sample sizes as small as N = 50 can produce reliable SEM results with normally distributed data and at least three reliable indicators per factor) and suggest strategies for making the best use of the modeling approach when N is near the lower bound.

Nested data may also emerge from analyses aimed at accounting for unobserved heterogeneity in outcomes such as in latent class analysis or growth mixture models with longitudinal data. Relatively little is known about sample size requirements for analyses of these types but they almost certainly require samples that are large. For that reason, those models fall outside the scope of this manuscript.

A number of software packages can handle such analyses, including (but certainly not limited to): HLM, Mplus, R, SAS, and Stata.

We omit a discussion of least squares approaches because these tend to be less efficient than maximum likelihood (Singer and Willett 2003).

We recognized that 30 groups is unrealistic for many prevention studies. This simulation study did not consider fewer than 30 groups given that the results were already somewhat problematic for that number. Findings from work considering 10 level 2 units is presented below.

However, progress is being made on this front. See Noh and Lee (2007) for an example.

Bauer and Sterba (2011) also found that increasing the number of response categories resulted in less bias and greater efficiency. This result is not surprising given the well-known consequences of dichotomization (MacCallum et al. 2002). Whenever possible, researchers should maximize the number of response categories or use a continuous response scale to maximize power.

Software packages that can estimate SEMs include, but are not limited to: EQS, LISREL, Mplus, R, and STATA.

Aitkin, M. A., & Longford, N. T. (1986). Statistical modeling issues in school effectiveness studies. Journal of the Royal Statistical Society, 149, 1–26. Retrieved from http://www.jstor.org/stable/2981882.CrossRef

Anderson, T. W. (1957). Maximum likelihood estimates for a multivariate normal distribution when some observations are missing. Journal of the American Statistical Association, 52, 200–203. Retrieved from http://www.jstor.org/stable/2280845.CrossRef

Austin, P. C. (2010). Estimating multilevel logistic regression models when the number of clusters is low: A comparison of different statistical software procedures. The International Journal of Biostatistics, 6, 1–18. doi:10.2202/1557-4679.1195.

Bandalos, D. L., & Gagné, P. (2012). Simulation methods in structural equation modeling. In R. H. Hoyle (Ed.), Handbook of structural equation modeling (pp. 92–108). New York: Guildford Press.

Bauer, D. J. (2003). Estimating multilevel linear models as structural equation models. Journal of Educational and Behavioral Statistics, 28, 135–167. doi:10.3102/10769986028002135.CrossRef

Bauer, D. J., & Sterba, S. K. (2011). Fitting multilevel models with ordinal outcomes: Performance of alternative specifications and methods of estimation. Psychological Methods, 16, 373–390. doi:10.1037/a0025813.PubMedCentralCrossRefPubMed

Bearden, W. O., Sharma, S., & Teel, J. E. (1982). Sample size effects on chi square and other statistics used in evaluating causal models. Journal of Marketing Research, 19, 425–430. doi:10.2307/3151716.CrossRef

Bentler, P. M. (1990). Comparative fit indexes in structural models. Psychological Bulletin, 107, 238–246. doi:10.1037/0033-2909.107.2.238.CrossRefPubMed

Bollen, K. A. (1990). Overall fit in covariance structure models: Two types of sample size effects. Psychological Bulletin, 107, 256–259. doi:10.1037/0033-2909.107.2.256.CrossRef

Bollen, K. A., & Curran, P. J. (2006). Latent curve models: A structural equation approach. Hoboken: Wiley.

Bollen, K. A., & Noble, M. D. (2011). Structural equation models and the quantification of behavior. Proceedings of the National Academy of Sciences of the United States of America, 108, 15639–15646. doi:10.1073/pnas.1010661108.PubMedCentralCrossRefPubMed

Breslow, N. E., & Clayton, D. G. (1993). Approximate inference in generalized linear mixed models. Journal of the American Statistical Association, 88, 9–25. Retrieved from http://www.jstor.org/stable/2290687.

Curran, P. J. (2003). Have multilevel models been structural equation models all along? Multivariate Behavioral Research, 38, 529–569. doi:10.1207/s15327906mbr3804_5.CrossRef

Curran, P. J., Lee, T., Howard, A. L., Lane, S., & MacCallum, R. (2012). Disaggregating within-person and between-person effects in multilevel and structural equation growth models. In J. R. Harring & G. R. Hancock (Eds.), Advances in longitudinal models in the social and behavioral sciences (pp. 217–253). Charlotte: Information Age Publishing.

Demidenko, E. (2004). Mixed models: Theory and applications. Hoboken: Wiley.CrossRef

Dempster, A. P., Laird, N. M., & Rubin, D. B. (1977). Maximum likelihood from incomplete data via the EM algorithm. Journal of the Royal Statistical Society, Series B, 44, 1–38. Retrieved from http://www.jstor.org/stable/2984875?origin=JSTOR-pdf.

Duncan, S. C., Duncan, T. E., & Strycker, L. A. (2002). A multilevel analysis of neighborhood context and youth alcohol and drug problems. Prevention Science, 3, 125–133. doi:10.1023/A:1015483317310.CrossRefPubMed

Fan, X., Thompson, B., & Wang, L. (1999). Effects of sample size, estimation methods, and model specification on structural equation modeling fit indexes. Structural Equation Modeling, 6, 56–83. doi:10.1080/10705519909540119.CrossRef

Gagné, P., & Hancock, G. R. (2006). Measurement model quality, sample size, and solution propriety in confirmatory factor models. Multivariate Behavioral Research, 41, 65–83. doi:10.1207/s15327906mbr4101_5.CrossRef

Gillmore, M. R., Hawkins, J. D., Catalano, R. F., Jr., Day, L. E., Moore, M., & Abbott, R. (1991). Structure of problem behaviors in preadolescence. Journal of Consulting and Clinical Psychology, 59, 499–506. doi:10.1037/0022-006X.59.4.499.CrossRefPubMed

Goldstein, H. (1986). Multilevel mixed linear model analysis using iterative generalized least squares. Biometrika, 73, 43–56. doi:10.1093/biomet/73.1.43.CrossRef

Hopkin, C. R., Hoyle, R. H., & Gottfredson, N. C. (2013). Maximizing the yield of small samples in prevention research: A review of general strategies and best practices. Manuscript submitted for publication.

Hoyle, R. H. (2011). Structural equation modeling for social and personality psychology. London: Sage Publications.

Hoyle, R. H., & Kenny, D. A. (1999). Sample size, reliability, and tests of statistical mediation. In R. H. Hoyle (Ed.), Statistical strategies for small sample research (pp. 195–222). Thousand Oaks: Sage Publications.

Hu, L.-T., & Bentler, P. M. (1999). Cutoff criteria for fit indexes in covariance structure analysis: Conventional criteria versus new alternatives. Structural Equation Modeling, 6, 1–55. doi:10.1080/10705519909540118.CrossRef

Jackson, D. L., Voth, J., & Frey, M. P. (2013). A note on sample size and solution propriety for confirmatory factor analytic models. Structural Equation Modeling, 20, 86–97. doi:10.1080/10705511.2013.742388.CrossRef

Kaplan, D. (1995). Statistical power in structural equation modeling. In R. H. Hoyle (Ed.), Structural equation modeling: Concepts, issues, and applications (pp. 100–117). Newbury Park: Sage Publications.

Kim, K. H. (2005). The relation among fit indexes, power, and sample size in structural equation modeling. Structural Equation Modeling, 12, 368–390. doi:10.1207/s15328007sem1203_2.CrossRef

Kim, S.-Y. (2012). Sample size requirements in single- and multiphase growth mixture models: A Monte Carlo simulation. Structural Equation Modeling, 19, 457–476. doi:10.1080/10705511.2012.687672.CrossRef

Kreft, I. G. G., & de Leeuw, J. (1988). Introducing multilevel modeling. Thousand Oaks: Sage Publications.

Kreft, I. G. G., de Leeuw, J., & Aiken, L. S. (1995). The effect of different forms of centering in hierarchical linear models. Multivariate Behavioral Research, 30, 1–21. doi:10.1207/s15327906mbr3001_1.CrossRef

Lüdtke, O., Marsh, H. W., Robitzsch, A., Trautwein, U., Asparouhov, T., & Muthén, B. (2008). The multilevel latent covariate model: A new, more reliable approach to group-level effects in contextual studies. Psychological Methods, 13, 203–229. doi:10.1037/a0012869.CrossRefPubMed

Maas, C. J. M., & Hox, J. J. (2005). Sufficient sample sizes for multilevel modeling. Methodology: European Journal of Research Methods for the Behavioral and Social Sciences, 1, 85–91. doi:10.1027/1614-1881.1.3.86.CrossRef

MacCallum, R. C., Browne, M. W., & Sugawara, H. M. (1996). Power analysis and determination of sample size for covariance structure modeling. Psychological Methods, 1, 130–149. Retrieved from http://doi.apa.org/journals/met/1/2/130.pdf.CrossRef

MacCallum, R. C., Zhang, S., Preacher, K. J., & Rucker, D. D. (2002). On the practice of dichotomization of quantitative variables. Psychological Methods, 7, 19–40. doi:10.1037/1082-989X.7.1.19.CrossRefPubMed

Marsh, H. W., Hau, K.-T., Balla, J. R., & Grayson, D. (1998). Is more ever too much? The number of indicators per factor in confirmatory factor analysis. Multivariate Behavioral Research, 33, 181–220. doi:10.1207/s15327906mbr3302_1.CrossRef

McArdle, J. J., & Nesselroade, J. R. (2003). Growth curve analysis in contemporary psychological research. In J. Schinka & W. Velicer (Eds.), Comprehensive handbook of psychology: Research methods in psychology (pp. 447–480). New York: Wiley. doi:10.1002/0471264385.

Meredith, W., & Tisak, J. (1990). Latent curve analysis. Psychometrika, 55, 107–122. doi:10.1007/BF02294746.CrossRef

Noh, M., & Lee, Y. (2007). REML estimation for binary data in GLMMs. Journal of Multivariate Analysis, 98, 896–915. doi:10.1016/j.jmva.2006.11.009.CrossRef

Rabe-Hesketh, S., Skrondal, A., & Pickles, A. (2002). Reliable estimation of generalized linear mixed models using adaptive quadrature. The Stata Journal, 2, 1–21. Retrieved from http://www.stata-journal.com/article.html?article=st0005.

Raudenbush, S. W. (1997). Statistical analysis and optimal design for cluster randomized trials. Psychological Methods, 2, 173–185. doi:10.1037/1082-989X.2.2.173.CrossRef

Raudenbush, S. W., & Bryk, A. S. (2002). Hierarchical linear models: Applications and data analysis methods. Thousand Oaks: Sage Publications.

Raudenbush, S. W., & Liu, X. (2000). Statistical power and optimal design for multisite randomized trials. Psychological Methods, 5, 199–213. doi:10.1037//1082-989X.5.2.199.CrossRefPubMed

Raudenbush, S. W., Bryk, A. S., & Congdon, R. (2004). HLM 6 for Windows [Computer software]. Lincolnwood: Scientific Software.

Raudenbush, S. W., Spybrook, J., Congdon, R., Liu, X., & Martinez, A. (2011). Optimal Design Software for multi-level and longitudinal research (version 3.01) [Software]. Available from www.wtgrantfoundation.org.

Rodríguez, G., & Goldman, N. (2001). Improved estimation procedures for multilevel models with binary response: A case study. Journal of the Royal Statistical Society: Series A (Statistics in Society), 164, 339–355. doi:10.1111/1467-985X.00206.CrossRef

Schabenberger, O. (2005). Introducing the GLIMMIX procedure for generalized linear mixed models. Cary: SAS Institute. Retrieved from http://www2.sas.com/proceedings/sugi30/196-30.pdf.

Schwarz, G. E. (1978). Estimating the dimension of a model. Annals of Statistics, 6, 461–464. Retrieved from http://www.jstor.org/stable/2958889.CrossRef

Shiyko, M. P., Lanza, S. T., Tan, X., Li, R., & Shiffman, S. (2012). Using the time-varying effect model (TVEM) to examine dynamic associations between negative affect and self-confidence on smoking urges: Differences between successful quitters and relapsers. Prevention Science, 13, 288–299. doi:10.1007/s11121-011-0264-z.PubMedCentralCrossRefPubMed

Simon, T. R., Ikeda, R. M., Smith, E. P., Reese, L. E., Rabiner, D. L., Miller-Johnson, S., et al. (2008). The multisite violence prevention project: Impact of a universal school-based violence prevention program on social-cognitive outcomes. Prevention Science, 9, 231–244. doi:10.1007/s11121-008-0101-1.PubMedCentralCrossRef

Singer, J. D., & Willett, J. B. (2003). Applied longitudinal data analysis: Modeling change and event occurrence. New York: Oxford University Press.CrossRef

Snijders, T. A. B., & Bosker, R. J. (2004). Multilevel analysis: An introduction to basic and advanced multilevel modeling (2nd ed.). London: Sage Publications.

Steiger, J. H., & Lind, J. C. (1980). Statistically based tests for the number of common factors. Iowa City: Paper presented at the Meeting of the Psychometric Society.

Tanaka, J. S. (1987). “How big is big enough?”: Sample size and goodness of fit in structural equation models with latent variables. Child Development, 58, 134–146. Retrieved from http://www.jstor.org/stable/1130296.CrossRef

Weston, R., & Gore, P. A., Jr. (2006). A brief guide to structural equation modeling. The Counseling Psychologist, 34, 719–751. doi:10.1177/0011000006286345.CrossRef

Whiteside, S. P., & Lynam, D. R. (2001). The five factor model and impulsivity: Using a structural model of personality to understand impulsivity. Personality and Individual Differences, 30, 669–689. doi:10.1016/S0191-8869(00)00064-7.CrossRef

Title: Sample Size Considerations in Prevention Research Applications of Multilevel Modeling and Structural Equation Modeling
Authors: Rick H. Hoyle
Nisha C. Gottfredson
Publication date: 01-10-2015
Publisher: Springer US
Published in: Prevention Science / Issue 7/2015
Print ISSN: 1389-4986
Electronic ISSN: 1573-6695
DOI: https://doi.org/10.1007/s11121-014-0489-8

Keynote webinar | Spotlight on medication adherence

Springer Medicine

Sample Size Considerations in Prevention Research Applications of Multilevel Modeling and Structural Equation Modeling

Abstract

Keynote webinar | Spotlight on medication adherence

Springer Medicine

Abstract

Please log in to get access to this content

Other articles of this Issue 7/2015

Maybe Small Is Too Small a Term: Introduction to Advancing Small Sample Prevention Science

Designs for Testing Group-Based Interventions with Limited Numbers of Social Units: The Dynamic Wait-Listed and Regression Point Displacement Designs

Bayesian Methods for Prevention Research

Clustering Methods with Qualitative Data: a Mixed-Methods Approach for Prevention Research with Small Samples

Small Sample Research: Considerations Beyond Statistical Power

Research Designs for Intervention Research with Small Samples II: Stepped Wedge and Interrupted Time-Series Designs