Top

Published in:

01-10-2015

Maximizing the Yield of Small Samples in Prevention Research: A Review of General Strategies and Best Practices

Authors: Cameron R. Hopkin, Rick H. Hoyle, Nisha C. Gottfredson

Published in: Prevention Science | Issue 7/2015

Abstract

The goal of this manuscript is to describe strategies for maximizing the yield of data from small samples in prevention research. We begin by discussing what “small” means as a description of sample size in prevention research. We then present a series of practical strategies for getting the most out of data when sample size is small and constrained. Our focus is the prototypic between-group test for intervention effects; however, we touch on the circumstance in which intervention effects are qualified by one or more moderators. We conclude by highlighting the potential usefulness of graphical methods when sample size is too small for inferential statistical methods.

Our focus on statistical power assumes a traditional null hypothesis statistical testing (NHST) approach to data analysis. We recognize the shortcomings of this approach and its frequent misuse; however, because it remains the primary approach to the analysis of data from prevention trials, it is the approach on which our analysis and recommendations focus. For readers interested in concerns about NHST and potential alternatives, Nickerson (2000) and Harlow et al. (1997) provide balanced, largely nontechnical presentations.

See von Hippel (2013) for potential problems and solutions for use of these methods with small samples.

An informative discussion of the use of covariates to increase statistical power is provided by Dennis et al. (2009).

Information about approaches to data visualization can be found in Young (1996) and a collection of papers edited by Post et al. (2003).

Bacchetti, P., Deeks, S. G., & McCune, J. M. (2011). Breaking free of sample size dogma to perform innovative translational research. Science Translational Medicine, 3, 87. doi:10.1126/scitranslmed.3001628.CrossRef

Brown, C. H., Sloboda, Z., Faggiano, F., Teasdale, B., Keller, F., Burkhart, G., Vigna-Taglianti, F., Howe, G., Masyn, K., Wang, W., Muthén, B., Stephens, P., Grey, S., & Perrino, T. (2011). Methods for synthesizing findings on moderation effects across multiple randomized trials. Prevention Science, 14, 144–156. doi:10.1007/s11121-011-0207-8.CrossRef

Carrig, M., Wirth, R. J., & Curran, P. J. (2004). A SAS macro for estimating and visualizing individual growth curves. Structural Equation Modeling: An Interdisciplinary Journal, 11, 132–149. doi:10.1207/S15328007SEM1101_9.CrossRef

Cohen, J., Cohen, P., West, S., & Aiken, L. (2003). Applied multiple regression/correlation analysis for the behavioral sciences (3rd ed.). Mahwah: Erlbaum.

Collins, L. M., Schafer, J. L., & Kam, C. (2001). A comparison of inclusive and restrictive strategies in modern missing data procedures. Psychological Methods, 6, 330–351. doi:10.1037/1082-989X.6.4.330.CrossRefPubMed

Collins, L. M., Dziak, J. J., & Li, R. (2009). Design of experiments with multiple independent variables: A resource management perspective on complete and reduced factorial designs. Psychological Methods, 14, 202–224. doi:10.1037/a0015826.PubMedCentralCrossRefPubMed

Collins, L. M., Baker, T. D., Mermelstein, R. J., Piper, M. E., Jorenby, D. E., Smith, S. S., Christiansen, B. A., Schlam, T. R., Cook, J. W., & Fiore, M. C. (2011). The multiphase optimization strategy for engineering effective tobacco use interventions. Annals of Behavioral Medicine, 41, 208–226. doi:10.1007/s12160-010-9253-x.PubMedCentralCrossRefPubMed

Curran, P. J., & Hussong, A. M. (2009). Integrative data analysis: The simultaneous analysis of multiple data sets. Psychological Methods, 14, 81–100. doi:10.1037/a0015914.PubMedCentralCrossRefPubMed

Dennis, M., Francis, D. J., Cirino, P. T., Schachar, R., Barnes, M. A., & Fletcher, J. M. (2009). Why IQ is not a covariate in cognitive studies of neurodevelopmental disorders. Journal of the International Neuropsychological Society, 15, 331–343. doi:10.1017/S1355617709090481.PubMedCentralCrossRefPubMed

Dumas, J. E., Lynch, A. M., Laughlin, J. E., Phillips Smith, E., & Prinz, R. J. (2001). Promoting intervention fidelity. Conceptual issues, methods, and preliminary results from the EARLY ALLIANCE prevention trial. American Journal of Preventive Medicine, 20, 38–47. doi:10.1016/S0749-3797(00)00272-5.CrossRefPubMed

Embretson, S. E. (1996). Item response theory models and spurious interaction effect in factorial ANOVA designs. Applied Psychological Measurement, 20, 201–212. doi:10.1177/014662169602000302.CrossRef

Enders, C. K. (2010). Applied missing data analysis. New York: Guildford Press.

Fok, C. C. T., Henry, D., & Allen, J. A (2015). Maybe small is too small a term: Introduction to advancing small sample prevention science. Prevention Science, in press.

Friendly, M. (1995). Exploratory and graphical methods of data analysis [Online short course]. Retrieved from http://www.datavis.ca/courses/eda/. Accessed 6 Aug 2013.

Graham, J. W. (2009). Missing data analysis: Making it work in the real world. Annual Review of Psychology, 60, 549–576. doi:10.1146/annurev.psych.58.110405.085530.CrossRefPubMed

Hansen, W. B., & Collins, L. M. (1994). Seven ways to increase power without increasing N. NIDA Research Monograph. In L. M. Collins & L. A. Seitz (Eds.), Advances in data analysis for prevention intervention research (NIDA Research Monograph 142, NIH Publication No. 94–3599, pp. 184–195). Rockville, MD: National Institutes of Health.

Hansen, W. B., Tobler, N. S., & Graham, J. W. (1990). Attrition in substance abuse prevention research: A meta-analysis of 85 longitudinally followed cohorts. Evaluation Review, 14, 677–685. doi:10.1177/0193841X9001400608.CrossRef

Harlow, L. L., Mulaik, S. A., & Steiger, J. H. (Eds.). (1997). What if there were no significance tests? Mahwah: Erlbaum.

Hawkins, N. G., Sanson-Fisher, R. W., Shakeshaft, A., D’Este, C., & Green, L. W. (2007). The multiple baseline design for evaluating population-based research. American Journal of Preventive Medicine, 33, 162–168. doi:10.1016/j.amepre.2007.03.020.CrossRefPubMed

Hoyle, R. H. (Ed.). (1999). Statistical strategies for small sample research‬. Thousand Oaks: Sage Publications.

Hoyle, R. H., & Gottfredson, N. C. (2015). Sample size considerations in prevention research applications of multilevel modeling and structural equation modeling. Prevention Science. doi:10.1007/s11121-014-0489-8.

Kang, S., & Waller, G. (2005). Moderated multiple regression, spurious interaction effects, and IRT. Applied Psychological Measurement, 29, 87–105. doi:10.1177/0146621604272737.CrossRef

Kratochwill, T. R., & Levin, J. R. (2010). Enhancing the scientific credibility of single-case intervention research: Randomization to the rescue. Psychological Methods, 15, 124–144. doi:10.1037/a0017736.CrossRefPubMed

Nickerson, R. S. (2000). Null hypothesis significance testing: A review of an old and continuing controversy. Psychological Methods, 5, 241–301. doi:10.1037//1082-989X.S.2.241.CrossRefPubMed

Post, F. H., Nielson, G. M., & Bonneau, G.-P. (Eds.) (2003). Data visualization: The state of the art. Boston: Kluwer Academic Publishers. Retrieved from http://www.springer.com/computer/image+processing/book/978-1-4020-7259-8?otherVersion=978-1-4613-5430-7

Raudenbush, S. W. (1997). Statistical analysis and optimal design for cluster randomized trials. Psychological Methods, 2, 173–185. doi:10.1037/1082-989X.2.2.173.CrossRef

Schafer, J. L. (1997). Analysis of incomplete multivariate data. London: Chapman & Hall.CrossRef

Supplee, L. H., Kelly, B. C., MacKinnon, D. M., & Barofsky, M. Y. (2013). Subgroup analysis in prevention and intervention science [Special issue]. Prevention Science 14(2).

von Hippel, P. T. (2013). The bias and efficiency of incomplete-data estimators in small univariate normal samples. Sociological Methods & Research, 42, 531–558. doi:10.1177/0049124113494582.CrossRef

Wang, R., & Ware, J. H. (2013). Detecting moderator effects using subgroup analyses. Prevention Science, 14, 111–120. doi:10.1007/s11121-011-0221-x.PubMedCentralCrossRefPubMed

Young, F. W. (1996). ViSta: The visual statistics system. Chapel Hill: Thurstone Psychometric Laboratory Research Memorandum 94-1(c). Retrieved from http://forrest.psych.unc.edu/research/index.html

Young, F. W., & Bann, C. M. (1996). ViSta: A visual statistics system. In R. A. Stine & J. Fox (Eds.), Statistical computing environments for social research (pp. 207–236). Thousand Oaks: Sage Publications.

Zand, D., Thomson, N. R., Dugan, M., Braun, J. A., Holterman-Hommes, P., & Hunter, P. L. (2006). Predictors of retention in an alcohol, tobacco, and other drug prevention study. Evaluation Review, 30, 209–222. doi:10.1177/0193841X05281160.CrossRefPubMed

Title: Maximizing the Yield of Small Samples in Prevention Research: A Review of General Strategies and Best Practices
Authors: Cameron R. Hopkin
Rick H. Hoyle
Nisha C. Gottfredson
Publication date: 01-10-2015
Publisher: Springer US
Published in: Prevention Science / Issue 7/2015
Print ISSN: 1389-4986
Electronic ISSN: 1573-6695
DOI: https://doi.org/10.1007/s11121-014-0542-7

Keynote webinar | Spotlight on medication adherence

Springer Medicine

Maximizing the Yield of Small Samples in Prevention Research: A Review of General Strategies and Best Practices

Abstract

Keynote webinar | Spotlight on medication adherence

Springer Medicine

Abstract

Please log in to get access to this content

Other articles of this Issue 7/2015

Why Small is Too Small a Term: Prevention Science for Health Disparities, Culturally Distinct Groups, and Community-Level Intervention

Inflection and Reflection: Design Issues that Shape the Field of Prevention Science

Bayesian Methods for Prevention Research

Sample Size Considerations in Prevention Research Applications of Multilevel Modeling and Structural Equation Modeling

Maybe Small Is Too Small a Term: Introduction to Advancing Small Sample Prevention Science

The Intersection Between Prevention Science and Evidence-Based Policy: How the SPR Evidence Standards Support Human Services Prevention Programs