Top

BMC Medical Research Methodology

Published in:

Open Access 01-12-2022 | Research

Evaluating the performance of Bayesian and restricted maximum likelihood estimation for stepped wedge cluster randomized trials with a small number of clusters

Authors: Kelsey L. Grantham, Jessica Kasza, Stephane Heritier, John B. Carlin, Andrew B. Forbes

Published in: BMC Medical Research Methodology | Issue 1/2022

Abstract

Background

Stepped wedge trials are an appealing and potentially powerful cluster randomized trial design. However, they are frequently implemented with a small number of clusters. Standard analysis methods for these trials such as a linear mixed model with estimation via maximum likelihood or restricted maximum likelihood (REML) rely on asymptotic properties and have been shown to yield inflated type I error when applied to studies with a small number of clusters. Small-sample methods such as the Kenward-Roger approximation in combination with REML can potentially improve estimation of the fixed effects such as the treatment effect. A Bayesian approach may also be promising for such multilevel models but has not yet seen much application in cluster randomized trials.

Methods

We conducted a simulation study comparing the performance of REML with and without a Kenward-Roger approximation to a Bayesian approach using weakly informative prior distributions on the intracluster correlation parameters. We considered a continuous outcome and a range of stepped wedge trial configurations with between 4 and 40 clusters. To assess method performance we calculated bias and mean squared error for the treatment effect and correlation parameters and the coverage of 95% confidence/credible intervals and relative percent error in model-based standard error for the treatment effect.

Results

Both REML with a Kenward-Roger standard error and degrees of freedom correction and the Bayesian method performed similarly well for the estimation of the treatment effect, while intracluster correlation parameter estimates obtained via the Bayesian method were less variable than REML estimates with different relative levels of bias.

Conclusions

The use of REML with a Kenward-Roger approximation may be sufficient for the analysis of stepped wedge cluster randomized trials with a small number of clusters. However, a Bayesian approach with weakly informative prior distributions on the intracluster correlation parameters offers a viable alternative, particularly when there is interest in the probability-based inferences permitted within this paradigm.

Available only for authorised users

Grayling MJ, Wason JMS, Mander AP. Stepped wedge cluster randomized controlled trial designs: a review of reporting quality and design features. Trials. 2017; 18(1):33. https://doi.org/10.1186/s13063-017-1783-0.CrossRefPubMedPubMedCentral

Hemming K, Haines TP, Chilton PJ, Girling AJ, Lilford RJ. The stepped wedge cluster randomised trial: rationale, design, analysis, and reporting. BMJ. 2015; 350:391. https://doi.org/10.1136/bmj.h391.CrossRef

Hussey MA, Hughes JP. Design and analysis of stepped wedge cluster randomized trials. Contemp Clin Trials. 2007; 28(2):182–91. https://doi.org/10.1016/j.cct.2006.05.007.CrossRefPubMed

Beard E, Lewis JJ, Copas A, Davey C, Osrin D, Baio G, Thompson JA, Fielding KL, Omar RZ, Ononge S, Hargreaves J, Prost A. Stepped wedge randomised controlled trials: systematic review of studies published between 2010 and 2014. Trials. 2015; 16(1). https://doi.org/10.1186/s13063-015-0839-2.

Barker D, McElduff P, D’Este C, Campbell MJ. Stepped wedge cluster randomised trials: a review of the statistical methodology used and available. BMC Med Res Methodol. 2016; 16(1). https://doi.org/10.1186/s12874-016-0176-5.

Taljaard M, Teerenstra S, Ivers NM, Fergusson DA. Substantial risks associated with few clusters in cluster randomized and stepped wedge designs. Clin Trials. 2016; 13(4):459–63. https://doi.org/10.1177/1740774516634316.CrossRefPubMed

Maas C, Hox J. Sufficient sample sizes for multilevel modeling. Methodology. 2005; 1(3):86–92. https://doi.org/10.1027/1614-2241.1.3.86.CrossRef

McNeish DM, Stapleton LM. The Effect of Small Sample Size on Two-Level Model Estimates: A Review and Illustration. Educ Psychol Rev. 2016; 28(2):295–314. https://doi.org/10.1007/s10648-014-9287-x.CrossRef

Kahan BC, Forbes G, Ali Y, Jairath V, Bremner S, Harhay MO, Hooper R, Wright N, Eldridge SM, Leyrat C. Increased risk of type I errors in cluster randomised trials with small or medium numbers of clusters: a review, reanalysis, and simulation study. Trials. 2016; 17(1):438. https://doi.org/10.1186/s13063-016-1571-2.CrossRefPubMedPubMedCentral

10.

McNeish D, Stapleton LM. Modeling Clustered Data with Very Few Clusters. Multivariate Behav Res. 2016; 51(4):495–518. https://doi.org/10.1080/00273171.2016.1167008.CrossRefPubMed

11.

Leyrat C, Morgan KE, Leurent B, Kahan BC. Cluster randomized trials with a small number of clusters: which analyses should be used?,. Int J Epidemiol. 2018; 47(1):321–31. https://doi.org/10.1093/ije/dyx169.CrossRefPubMed

12.

Nugent JR, Kleinman KP. Type I error control for cluster randomized trials under varying small sample structures. BMC Med Res Methodol. 2021; 21(1):65. https://doi.org/10.1186/s12874-021-01236-7.CrossRefPubMedPubMedCentral

13.

Kenward MG, Roger JH. Small Sample Inference for Fixed Effects from Restricted Maximum Likelihood. Biometrics. 1997; 53(3):983–97. https://doi.org/10.2307/2533558.CrossRefPubMed

14.

Kenward MG, Roger JH. An improved approximation to the precision of fixed effects from restricted maximum likelihood. Comput Stat Data Anal. 2009; 53(7):2583–95. https://doi.org/10.1016/j.csda.2008.12.013.CrossRef

15.

Baldwin SA, Fellingham GW. Bayesian methods for the analysis of small sample multilevel data with a complex variance structure. Psychol Methods. 2013; 18(2):151–64. https://doi.org/10.1037/a0030642.CrossRefPubMed

16.

Spiegelhalter DJ. Bayesian methods for cluster randomized trials with continuous responses. Stat Med. 2001; 20(3):435–52. https://doi.org/10.1002/1097-0258(20010215)20:3<435::AID-SIM804>3.0.CO;2-E.CrossRefPubMed

17.

Turner RM, Omar RZ, Thompson SG. Bayesian methods of analysis for cluster randomized trials with binary outcome data. Stat Med. 2001; 20(3):453–72. https://doi.org/10.1002/1097-0258(20010215)20:3<453::AID-SIM803>3.0.CO;2-L.CrossRefPubMed

18.

Jones BG, Streeter AJ, Baker A, Moyeed R, Creanor S. Bayesian statistics in the design and analysis of cluster randomised controlled trials and their reporting quality: a methodological systematic review. Syst Rev. 2021; 10(1):91. https://doi.org/10.1186/s13643-021-01637-1.CrossRefPubMedPubMedCentral

19.

Cunanan KM, Carlin BP, Peterson KA. A practical Bayesian stepped wedge design for community-based cluster-randomized clinical trials: The British Columbia Telehealth Trial. Clin Trials. 2016; 13(6):641–50. https://doi.org/10.1177/1740774516656583.CrossRefPubMedPubMedCentral

20.

Zhan D, Ouyang Y, Xu L, Wong H. Improving efficiency in the stepped-wedge trial design via Bayesian modeling with an informative prior for the time effects. Clin Trials. 2021; 18(3):295–302. https://doi.org/10.1177/1740774520980052.CrossRefPubMedPubMedCentral

21.

Gelman A, Carlin JB, Stern HS, Dunson DB, Vehtari A, Rubin DB. Bayesian Data Analysis, 3rd edn. London: CRC Press; 2013.CrossRef

22.

Gelman A, Hill J. Data Analysis Using Regression and Multilevel/Hierarchical Models, 1st edn. Cambridge: CRC Press; 2007.

23.

Smid SC, McNeish D, Miočević M, van de Schoot R. Bayesian Versus Frequentist Estimation for Structural Equation Models in Small Sample Contexts: A Systematic Review. Struct Equ Model A Multidiscip J. 2020; 27(1):131–61. https://doi.org/10.1080/10705511.2019.1577140.CrossRef

24.

Campbell MK, Piaggio G, Elbourne DR, Altman DG. Consort 2010 statement: extension to cluster randomised trials. BMJ Br Med J. 2012; 345:5661. https://doi.org/10.1136/bmj.e5661.CrossRef

25.

Hemming K, Taljaard M, McKenzie JE, Hooper R, Copas A, Thompson JA, Dixon-Woods M, Aldcroft A, Doussau A, Grayling M, Kristunas C, Goldstein CE, Campbell MK, Girling A, Eldridge S, Campbell MJ, Lilford RJ, Weijer C, Forbes AB, Grimshaw JM. Reporting of stepped wedge cluster randomised trials: extension of the CONSORT 2010 statement with explanation and elaboration. BMJ. 2018; 363:1614. https://doi.org/10.1136/bmj.k1614.CrossRef

26.

Turner RM, Thompson SG, Spiegelhalter DJ. Prior distributions for the intracluster correlation coefficient, based on multiple previous estimates, and their application in cluster randomized trials. Clin Trials. 2005; 2(2):108–18. https://doi.org/10.1191/1740774505cn072oa.CrossRefPubMed

27.

Hooper R, Teerenstra S, de Hoop E, Eldridge S. Sample size calculation for stepped wedge and other longitudinal cluster randomised trials. Stat Med. 2016; 35(26):4718–28. https://doi.org/10.1002/sim.7028.CrossRefPubMed

28.

Girling AJ, Hemming K. Statistical efficiency and optimal design for stepped cluster studies under linear mixed effects models. Stat Med. 2016; 35:2149–66. https://doi.org/10.1002/sim.6850.CrossRefPubMedPubMedCentral

29.

Martin J, Taljaard M, Girling A, Hemming K. Systematic review finds major deficiencies in sample size methodology and reporting for stepped-wedge cluster randomised trials. BMJ Open. 2016; 6(2). https://doi.org/10.1136/bmjopen-2015-010166.

30.

Korevaar E, Kasza J, Taljaard M, Hemming K, Haines T, Turner EL, Thompson JA, Hughes JP, Forbes AB. Intra-cluster correlations from the CLustered OUtcome Dataset bank to inform the design of longitudinal cluster trials. Clin Trials. 2021. https://doi.org/10.1177/17407745211020852.

31.

Bates D, Mächler M, Bolker B, Walker S. Fitting Linear Mixed-Effects Models Using lme4. J Stat Softw. 2015; 67(1). https://doi.org/10.18637/jss.v067.i01.

32.

R Core Team. R: A Language and Environment for Statistical Computing. Vienna: R Foundation for Statistical Computing; 2019. R Foundation for Statistical Computing. https://www.R-project.org/.

33.

Halekoh U, Højsgaard S. A Kenward-Roger approximation and parametric bootstrap methods for tests in linear mixed models - The R package pbkrtest. J Stat Software. 2014; 59(9):1–32. https://doi.org/10.18637/jss.v059.i09.CrossRef

34.

Satterthwaite FE. An Approximate Distribution of Estimates of Variance Components. Biometrics Bull. 1946; 2(6):110–14. https://doi.org/10.2307/3002019.CrossRef

35.

Lüdecke D, Ben-Shachar MS, Patil I, Makowski D. parameters: Extracting, Computing and Exploring the Parameters of Statistical Models using R. J Open Source Softw. 2020; 5(53):2445. https://doi.org/10.21105/joss.02445.CrossRef

36.

Kuznetsova A, Brockhoff P, Christensen R. lmerTest Package: Tests in Linear Mixed Effects Models. J Stat Softw. 2017; 82:1–26. https://doi.org/10.18637/JSS.V082.I13.CrossRef

37.

Gelman A. Prior distributions for variance parameters in hierarchical models (comment on article by Browne and Draper). Bayesian Anal. 2006; 1(3):515–34. https://doi.org/10.1214/06-BA117A.CrossRef

38.

Martin J, Girling A, Nirantharakumar K, Ryan R, Marshall T, Hemming K. Intra-cluster and inter-period correlation coefficients for cross-sectional cluster randomised controlled trials for type-2 diabetes in UK primary care. Trials. 2016; 17(1):402. https://doi.org/10.1186/s13063-016-1532-9.CrossRefPubMedPubMedCentral

39.

Stan Development Team. RStan: the R interface to Stan. R package version 2.21.2. 2020. http://mc-stan.org/. Accessed 14 Sept 2020.

40.

Stan Development Team. Brief Guide to Stan’s Warnings. 2020. Accessed 11 May 2021. https://mc-stan.org/misc/warnings.html#divergent-transitions-after-warmup.

41.

Vehtari A, Gelman A, Simpson D, Carpenter B, Burkner P-C. Rank-Normalization, Folding, and Localization: An Improved \(\widehat {R}\) for Assessing Convergence of MCMC. Bayesian Anal. 2020. https://doi.org/10.1214/20-BA1221.

42.

Morris TP, White IR, Crowther MJ. Using simulation studies to evaluate statistical methods. Stat Med. 2019; 38(11):2074–102. https://doi.org/10.1002/sim.8086.CrossRefPubMedPubMedCentral

43.

Goscinski WJ, McIntosh P, Felzmann U, Maksimenko A, Hall C, Gureyev T, Thompson D, Janke A, Galloway G, Killeen N, Raniga P, Kaluza O, Ng A, Poudel G, Barnes D, Nguyen T, Bonnington P, Egan G. The multi-modal Australian ScienceS Imaging and Visualization Environment (MASSIVE) high performance computing infrastructure: applications in neuroscience and neuroinformatics research. Front Neuroinformatics. 2014; 8:30. https://doi.org/10.3389/fninf.2014.00030.CrossRef

44.

Betancourt M. Diagnosing Biased Inference with Divergences. 2017. Accessed 11 May 2021. https://mc-stan.org/users/documentation/case-studies/divergences_and_bias.html.

45.

Zitzmann S, Lüdtke O, Robitzsch A, Hecht M. On the Performance of Bayesian Approaches in Small Samples: A Comment on Smid, McNeish, Miocevic, and van de Schoot (2020). Struct Equ Model A Multidiscip J. 2020:1–11. https://doi.org/10.1080/10705511.2020.1752216.

46.

Betancourt M, Girolami M. Hamiltonian Monte Carlo for hierarchical models In: Upadhyay SK, Singh U, Dey DK, Loganathan A, editors. Current Trends in Bayesian Methodology with Applications. New York: Chapman and Hall/CRC: 2015. p. 79–101.

47.

Kasza J, Hemming K, Hooper R, Matthews JNS, Forbes AB. Impact of non-uniform correlation structure on sample size and power in multiple-period cluster randomised trials. Stat Methods Med Res. 2019; 28:703–16. https://doi.org/10.1177/0962280217734981.CrossRefPubMed

48.

Grantham KL, Kasza J, Heritier S, Hemming K, Forbes AB. Accounting for a decaying correlation structure in cluster randomized trials with continuous recruitment. Stat Med. 2019; 38(11):1918–34. https://doi.org/10.1002/sim.8089.CrossRefPubMed

Title: Evaluating the performance of Bayesian and restricted maximum likelihood estimation for stepped wedge cluster randomized trials with a small number of clusters
Authors: Kelsey L. Grantham
Jessica Kasza
Stephane Heritier
John B. Carlin
Andrew B. Forbes
Publication date: 01-12-2022
Publisher: BioMed Central
Published in: BMC Medical Research Methodology / Issue 1/2022
Electronic ISSN: 1471-2288
DOI: https://doi.org/10.1186/s12874-022-01550-8

Keynote webinar | Spotlight on medication adherence

Springer Medicine

Evaluating the performance of Bayesian and restricted maximum likelihood estimation for stepped wedge cluster randomized trials with a small number of clusters

Abstract

Background

Methods

Results

Conclusions

Keynote webinar | Spotlight on medication adherence

Springer Medicine

Abstract

Background

Methods

Results

Conclusions

Please log in to get access to this content

Other articles of this Issue 1/2022

A systematic review and psychometric evaluation of resilience measurement scales for people living with dementia and their carers

Improving the accuracy of self-reported height and weight in surveys: an experimental study

Structured expert elicitation to inform long-term survival extrapolations using alternative parametric distributions: a case study of CAR T therapy for relapsed/ refractory multiple myeloma

Evaluation of multiple imputation approaches for handling missing covariate information in a case-cohort study with a binary outcome

A photograph of the researcher on the invitation letter did not affect the participation rate of a postal survey: a randomized study within a trial (SWAT)

Informed Bayesian survival analysis