Abstract
Multiple-baseline studies are prevalent in behavioral research, but questions remain about how to best analyze the resulting data. Monte Carlo methods were used to examine the utility of multilevel models for multiplebaseline data under conditions that varied in the number of participants, number of repeated observations per participant, variance in baseline levels, variance in treatment effects, and amount of autocorrelation in the Level 1 errors. Interval estimates of the average treatment effect were examined for two specifications of the Level 1 error structure (σ2 I and first-order autoregressive) and for five different methods of estimating the degrees of freedom (containment, residual, between—within, Satterthwaite, and Kenward—Roger). When the Satterthwaite or Kenward—Roger method was used and an autoregressive Level 1 error structure was specified, the interval estimates of the average treatment effect were relatively accurate. Conversely, the interval estimates of the treatment effect variance were inaccurate, and the corresponding point estimates were biased.
Article PDF
Similar content being viewed by others
Avoid common mistakes on your manuscript.
References
Barlow, D. H., & Hersen, M. (1984). Single case experimental designs: Strategies for studying behavior change (2nd ed.). New York: Pergamon.
Bock, M. A. (2007). A social—behavioral learning strategy intervention for a child with Asperger syndrome: Brief report. Remedial & Special Education, 28, 258–265. doi:10.1177/07419325070280050101
Bulté, I., & Onghena, P. (2009). Randomization tests for multiplebaseline designs: An extension of the SCRT-R package. Behavior Research Methods, 41, 477–485.
Busk, P. L., & Marascuilo, L. A. (1988). Autocorrelation in singlesubject research: A counterargument to the myth of no autocorrelation. Behavioral Assessment, 10, 229–242.
Dukes, W. F. (1965). N=1. Psychological Bulletin, 64, 74–79. doi:10.1037/ h0021964
Fai, A. H.-T., & Cornelius, P. L. (1996). Approximate F-tests of multiple degree of freedom hypotheses in generalized least squares analyses of unbalanced split-plot experiments. Journal of Statistical Computation & Simulation, 54, 363–378.
Ferron, J. [M.], Dailey, R., & Yi, Q. (2002). Effects of misspecifying the first-level error structure in two-level models of change. Multivariate Behavioral Research, 37, 379–403. doi:10.1207/ S15327906MBR3703_4
Ferron, J. [M.], & Scott, H. (2005). Multiple baseline design. In B. S. Everitt & D. C. Howell (Eds.), Encyclopedia of statistics in behavioral science (Vol. 2, pp. 941–945). Hoboken, NJ: Wiley.
Fouladi, R. T., & Shieh, Y.-Y. (2004). A comparison of two general approaches to mixed model longitudinal analyses under small sample size conditions. Communications in Statistics: Simulation & Computation, 33, 807–824. doi:10.1081/SAC-200033260
Glover, D. S., Brown, G. P., Fairburn, C. G., & Shafran, R. (2007). A preliminary evaluation of cognitive-behaviour therapy for clinical perfectionism: A case series. British Journal of Clinical Psychology, 46, 85–94. doi:10.1348/014466506X117388
Gomez, E. V., Schaalje, G. B., & Fellingham, G. W. (2005). Performance of the Kenward—Roger method when the covariance structure is selected using AIC and BIC. Communications in Statistics: Simulation & Computation, 34, 377–392. doi:10.1081/SAC-200055719
Gresham, F. M., McIntyre, L. L., Olson-Tinker, H., Dolstra, L., McLaughlin, V., & Van, M. (2004). Relevance of functional behavioral assessment research for school-based interventions and positive behavioral support. Research in Developmental Disabilities, 25, 19–37. doi:10.1016/j.ridd.2003.04.003
Hox, J. (1998). Multilevel modeling: When and why. In I. Balderjahn, R. Mathar, & M. Schader (Eds.), Classification, data analysis, and data highways (pp. 147–154). New York: Springer.
Huitema, B. E. (1985). Autocorrelation in applied behavior analysis: A myth. Behavioral Assessment, 7, 107–118.
Huitema, B. E., & McKean, J. W. (1998). Irrelevant autocorrelation in least-squares intervention models. Psychological Methods, 3, 104–116. doi:10.1037/1082-989X.3.1.104
Kenward, M. G., & Roger, J. H. (1997). Small sample inference for fixed effects from restricted maximum likelihood. Biometrics, 53, 983–997.
Koehler, M. J., & Levin, J. R. (1998). Regulated randomization: A potentially sharper analytical tool for the multiple-baseline design. Psychological Methods, 3, 206–217. doi:10.1037/1082-989X.3.2.206
Kowalchuk, R. K., Keselman, H. J., Algina, J., & Wolfinger, R. D. (2004). The analysis of repeated measurements with mixed-model adjusted F tests. Educational & Psychological Measurement, 64, 224–242. doi:10.1177/0013164403260196
Kratochwill, T. R., & Piersel, W. C. (1983). Time-series research: Contributions to empirical clinical practice. Behavioral Assessment, 5, 165–176.
Maas, C. J. M., & Hox, J. J. (2004). Robustness issues in multilevel regression analysis. Statistica Neerlandica, 58, 127–137. doi:10.1046/j.0039-0402.2003.00252.x
Mahar, M. T., Murphy, S. K., Rowe, D. A., Golden, J., Shields, A. T., & Raedeke, T. D. (2006). Effects of a classroom-based program on physical activity and on-task behavior. Medicine & Science in Sports & Exercise, 38, 2086–2094.
Marascuilo, L. A., & Busk, P. L. (1988). Combining statistics for multiple-baseline AB and replicated ABAB designs across subjects. Behavioral Assessment, 10, 1–28.
Matyas, T. A., & Greenwood, K. M. (1997). Serial dependency in single-case time series. In R. D. Franklin, D. B. Allison, & B. S. Gorman (Eds.), Design and analysis of single-case research (pp. 215–243). Mahwah, NJ: Erlbaum.
McKnight, S. D., McKean, J. W., & Huitema, B. E. (2000). A double bootstrap method to analyze linear models with autoregressive error terms. Psychological Methods, 5, 87–101. doi:10.1037/1082-989X.5.1.87
Mok, M. (1995). Sample size requirements for 2-level designs in educational research. Unpublished manuscript, Macquarie University, Sydney, Australia.
Morgan, D. L., & Morgan, R. K. (2001). Single-participant research design: Bringing science to managed care. American Psychologist, 56, 119–127. doi:10.1037/0003-066X.56.2.119
Musti-Rao, S., & Cartledge, G. (2007). Effects of a supplemental early reading intervention with at-risk urban learners. Topics in Early Childhood Special Education, 27, 70–85. doi:10.1177/02711214070270020301
Naylor, E. V., Antonuccio, D. O., Johnson, G., Spogen, D., & O’Donohue, W. (2007). A pilot study investigating behavioral prescriptions for depression. Journal of Clinical Psychology in Medical Settings, 14, 152–159. doi:10.1007/s10880-007-9064-9
Nugent, W. (1996). Integrating single-case and group-comparison designs for evaluation research. Journal of Applied Behavioral Science, 32, 209–226. doi:10.1177/0021886396322007
O’Callaghan, P. M., Allen, K. D., Powell, S., & Salama, F. (2006). The efficacy of noncontingent escape for decreasing children’s disruptive behavior during restorative dental treatment. Journal of Applied Behavior Analysis, 39, 161–171. doi:10.1901/jaba.2006.79-05
Parsonson, B. S., & Baer, D. M. (1992). The visual analysis of data, and current research into the stimuli controlling it. In T. R. Kratochwill & J. R. Levin (Eds.), Single-case research design and analysis: New directions for psychology and education (pp. 15–40). Hillsdale, NJ: Erlbaum.
Raudenbush, S. W., & Bryk, A. S. (2002). Hierarchical linear models: Applications and data analysis methods (2nd ed.). Thousand Oaks, CA: Sage.
Satterthwaite, F. E. (1941). Synthesis of variance. Psychometrika, 6, 309-316. doi:10.1007/BF02288586
Schaalje, G. B., McBride, J. B., & Fellingham, G. W. (2001). Approximations to distributions of test statistics in complex mixed linear models using SAS Proc MIXED. In Proceedings of the SAS Users Group International 26th Annual Conference (Paper 262-26). Available at support.sas.com/events/sasglobalforum/previous/index. html.
Shadish, W. R., Cook, T. D., & Campbell, D. T. (2002). Experimental and quasiexperimental designs for generalized causal inference. Boston: Houghton Mifflin.
Shadish, W. R., & Rindskopf, D. M. (2007). Methods for evidencebased practice: Quantitative synthesis of single-subject designs. New Directions for Evaluation, 113, 95–109.
Singh, N. N., Lancioni, G. E., Winton, A. S. W., Adkins, A. D., Singh, J., & Singh, A. N. (2007). Mindfulness training assists individuals with moderate mental retardation to maintain their community placements. Behavior Modification, 31, 800–814. doi:10.1177/0145445507300925
Tiger, J. H., Hanley, G. P., & Hernandez, E. (2006). An evaluation of the value of choice with preschool children. Journal of Applied Behavior Analysis, 39, 1–16. doi:10.1901/jaba.2006.158-04
Tsao, L.-L., & Odom, S. L. (2006). Sibling-mediated social interaction intervention for young children with autism. Topics in Early Childhood Special Education, 26, 106–123. doi:10.1177/02711214060260020101
Van den Noortgate, W., & Onghena, P. (2003a). Combining singlecase experimental data using hierarchical linear models. School Psychology Quarterly, 18, 325–346. doi:10.1521/scpq.18.3.325.22577
Van den Noortgate, W., & Onghena, P. (2003b). Hierarchical linear models for the quantitative integration of effect sizes in single-case research. Behavior Research Methods, Instruments, & Computers, 35, 1–10.
Velicer, W. F., & Fava, J. L. (2003). Time series analysis. In I. B. Weiner (Series Ed.) and J. Schinka & W. F. Velicer (Vol. Eds.), Handbook of psychology: Vol. 2. Research methods in psychology (pp. 581–606). Hoboken, NJ: Wiley.
Wambaugh, J. L., & Ferguson, M. (2007). Application of semantic feature analysis to retrieval of action names in aphasia. Journal of Rehabilitation Research & Development, 44, 381–394.
Author information
Authors and Affiliations
Corresponding author
Additional information
This work was supported in part by the University of South Florida, College of Education, through a minigrant awarded to J.M.F.
Rights and permissions
About this article
Cite this article
Ferron, J.M., Bell, B.A., Hess, M.R. et al. Making treatment effect inferences from multiple-baseline data: The utility of multilevel modeling approaches. Behavior Research Methods 41, 372–384 (2009). https://doi.org/10.3758/BRM.41.2.372
Received:
Accepted:
Issue Date:
DOI: https://doi.org/10.3758/BRM.41.2.372