Top

BMC Medical Research Methodology

Published in:

Open Access 01-12-2019 | Research article

Likelihood-based random-effects meta-analysis with few studies: empirical and simulation studies

Authors: Svenja E. Seide, Christian Röver, Tim Friede

Published in: BMC Medical Research Methodology | Issue 1/2019

Abstract

Background

Standard random-effects meta-analysis methods perform poorly when applied to few studies only. Such settings however are commonly encountered in practice. It is unclear, whether or to what extent small-sample-size behaviour can be improved by more sophisticated modeling.

Methods

We consider likelihood-based methods, the DerSimonian-Laird approach, Empirical Bayes, several adjustment methods and a fully Bayesian approach. Confidence intervals are based on a normal approximation, or on adjustments based on the Student-t-distribution. In addition, a linear mixed model and two generalized linear mixed models (GLMMs) assuming binomial or Poisson distributed numbers of events per study arm are considered for pairwise binary meta-analyses. We extract an empirical data set of 40 meta-analyses from recent reviews published by the German Institute for Quality and Efficiency in Health Care (IQWiG). Methods are then compared empirically as well as in a simulation study, based on few studies, imbalanced study sizes, and considering odds-ratio (OR) and risk ratio (RR) effect sizes. Coverage probabilities and interval widths for the combined effect estimate are evaluated to compare the different approaches.

Results

Empirically, a majority of the identified meta-analyses include only 2 studies. Variation of methods or effect measures affects the estimation results. In the simulation study, coverage probability is, in the presence of heterogeneity and few studies, mostly below the nominal level for all frequentist methods based on normal approximation, in particular when sizes in meta-analyses are not balanced, but improve when confidence intervals are adjusted. Bayesian methods result in better coverage than the frequentist methods with normal approximation in all scenarios, except for some cases of very large heterogeneity where the coverage is slightly lower. Credible intervals are empirically and in the simulation study wider than unadjusted confidence intervals, but considerably narrower than adjusted ones, with some exceptions when considering RRs and small numbers of patients per trial-arm. Confidence intervals based on the GLMMs are, in general, slightly narrower than those from other frequentist methods. Some methods turned out impractical due to frequent numerical problems.

Conclusions

In the presence of between-study heterogeneity, especially with unbalanced study sizes, caution is needed in applying meta-analytical methods to few studies, as either coverage probabilities might be compromised, or intervals are inconclusively wide. Bayesian estimation with a sensibly chosen prior for between-trial heterogeneity may offer a promising compromise.

Available only for authorised users

Turner RM, Davey J, Clarke MJ, Thompson SG, Higgins JPT. Predicting the extent of heterogeneity in meta-analysis, using empirical data from the Cochrane Database of Systematic Reviews. Int J Epidemiol. 2012; 41(3):818–27. https://doi.org/10.1093/ije/dys041.

Röver C, Knapp G, Friede T. Hartung-Knapp-Sidik-Jonkman approach and its modification for random-effects meta-analysis with few studies. BMC Med Res Methodol. 2015;15. https://doi.org/10.1186/s12874-015-0091-1.

Friede T, Röver C, Wandel S, Neuenschwander B. Meta-analysis of few small studies in orphan diseases. Res Synth Methods. 2017; 8(1):79–91. https://doi.org/10.1002/jrsm.1217.

Higgins JPT, Thompson SG, Spiegelhalter DJ. A re-evaluation of random-effects meta-analysis. Journal of the Royal Statistical Society A. 2009; 172(1):137–59. https://doi.org/10.1111/j.1467-985X.2008.00552.x.

Friede T, Röver C, Wandel S, Neuenschwander B. Meta-analysis of two studies in the presence of heterogeneity with applications in rare diseases. Biom J. 2017; 59(4):658–71. https://doi.org/10.1002/bimj.201500236.

Bender R, Friede T, Koch A, Kuss O, Schlattmann P, Schwarzer G, Skipka G. Methods for evidence synthesis in the case of very few studies. Res Synth Methods. 2018. https://doi.org/10.1002/jrsm.1297.

Böhning D, Rattanasiri S, Kuhnert R. Meta-analysis of Binary Data Using Profile Likelihood. Boca Raton: Taylor & Francis; 2008.CrossRef

Morris TP, Fisher DJ, Kenward MG, Carpenter JR. Meta-analysis of Gaussian individual patient data: two-stage or not two-stage?Stat Med. 2018. https://doi.org/10.1002/sim.7589.

Mathew T, Nordström K. Comparison of one-step and two-step analysis models using individual patient data. Biom J. 2010; 52(2):271–87. https://doi.org/10.1002/bimj.200900143.

10.

Burke DL, Ensor J, Riley RD. Meta-analysis using individual participant data: one-stage and two-stage approaches, and why they may differ. Stat Med. 2017; 36(5):855–75.CrossRef

11.

Kontopantelis E. A comparison of one-stage vs two-stage individual patient data meta-analysis methods: a simulation study. Res Synth Methods. 2018. https://doi.org/10.1002/jrsm.1303.

12.

Debray T, Moons KGM, Abo-Zaid GMA, Koffijberg H, Riley RD. Individual participant data meta-analysis for a binary outcome: one-stage or two-stage?PLoS ONE. 2013; 8(4):60650.CrossRef

13.

Jackson D, Law M, Stijnen T, Viechtbauer W, White IR. A comparison of seven random-effects models for meta-analyses that estimate the summary odds ratio. Stat Med. 2018; 37(7):1059–85. https://doi.org/10.1002/sim.7588.

14.

IntHout J, Ioannidis JPA, Borm GF. The Hartung-Knapp-Sidik-Jonkman method for random effects meta-analysis is straightforward and considerably outperforms the standard DerSimonian-Laird method. BMC Med Res Methodol. 2014; 14:25. https://doi.org/10.1186/1471-2288-14-25.

15.

Fleiss JL. The statistical basis of meta-analysis. Stat Methods Med Res. 1993; 2(2):121–45.CrossRef

16.

Hedges LV, Olkin I. Statistical Methods for Meta-analysis. San Diego: Academic Press; 1985.

17.

Hartung J, Knapp G, Sinha BK. Statistical Meta-analysis with Applications. Hoboken: Wiley; 2008.CrossRef

18.

DerSimonian R, Laird N. Meta-analysis in clinical trials. Control Clin Trials. 1986; 7(3):177–88. https://doi.org/10.1016/0197-2456(86)90046-2.

19.

Viechtbauer W. Bias and efficiency of meta-analytic variance estimators in the random-effects model. J Educ Behav Stat. 2005; 30(3):261–93. https://doi.org/10.3102/10769986030003261.

20.

Raudenbush SW. Analyzing effect sizes: random-effects models In: Cooper HM, Larry VH, Valentine JC, editors. The Handbook of Research Synthesis and Meta-Analysis. New York City: Russell Sage Foundation: 2009. p. 295–316.

21.

Morris CN. Empirical Bayes methods for combining likelihoods: comment. J Am Stat Assoc. 1996; 91(434):555–8. https://doi.org/10.2307/2291646.

22.

Paule RC, Mandel J. Consensus values and weighting factors. J Res Natl Bur Stand. 1982; 87(5):1–9. https://doi.org/10.6028/jres.087.022.

23.

Turner RM, Jackson D, Wei Y, Thompson SG, Higgins PT. Predictive distributions for between-study heterogeneity and simple methods for their application in Bayesian meta-analysis. Stat Med. 2015; 34(6):984–98. https://doi.org/10.1002/sim.6381.

24.

Dias S, Sutton AJ, Welton NJ, Ades AE. NICE DSU Technical Support Document 2: A Generalized Linear Modelling Framework for Pairwise and Network Meta-analysis of Randomized Controlled Trials. London: National Institute for Health and Clinical Excellence (NICE); 2014. National Institute for Health and Clinical Excellence (NICE). available from: http://www.nicedsu.org.uk.

25.

Röver C. Bayesian random-effects meta-analysis using the bayesmeta R package. arXiv preprint 1711.08683. 2017. http://www.arxiv.org/abs/1711.08683.

26.

Spiegelhalter DJ, Abrams KR, Myles JP. Bayesian Approaches to Clinical Trials and Health-care Evaluation. Chichester: Wiley; 2004. https://doi.org/10.1002/0470092602.

27.

Turner RM, Omar RZ, Yang M, Goldstein H, Thompson SG. A multilevel model framework for meta-analysis of clinical trials with binary outcomes. Stat Med. 2000; 19(24):3417–32.CrossRef

28.

van Houwelingen HC, Zwinderman KH, Stijnen T. A bivariate approach to meta-analysis. Stat Med. 1993; 12(24):2273–84. https://doi.org/10.1002/sim.4780122405.

29.

Stijnen T, Hamza TH, Özdemir P. Random effects meta-analysis of event outcome in the framework of the generalized linear mixed model with applications in sparse data. Stat Med. 2010; 29(29):3046–67. https://doi.org/10.1002/sim.4040.

30.

Hartung J, Knapp G. On tests of the overall treatment effect in meta-analysis with normally distributed responses. Stat Med. 2001; 20(12):1771–82. https://doi.org/10.1002/sim.791.

31.

Hartung J, Knapp G. A refined method for the meta-analysis of controlled clinical trials with binary outcome. Stat Med. 2001; 20(24):3875–89. https://doi.org/10.1002/sim.1009.

32.

Sidik K, Jonkman JN. A simple confidence interval for meta-analysis. Stat Med. 2002; 21(21):3153–9. https://doi.org/10.1002/sim.1262.

33.

Knapp G, Hartung J. Improved tests for a random effects meta-regression with a single covariate. Stat Med. 2003; 22(17):2693–710. https://doi.org/10.1002/sim.1482.

34.

Higgins JPT, Thompson SG. Quantifying heterogeneity in a meta-analysis. Stat Med. 2002; 21(11):1539–58. https://doi.org/10.1002/sim.1186.

35.

Hoaglin DC. Misunderstandings about Q and ’Cochran’s Q test’ in meta-analysis. Stat Med. 2016; 35(4):485–95. https://doi.org/10.1002/sim.6632.

36.

Borenstein M, Higgins JPT, Hedges LV, Rothstein HR. Basics of meta-analysis: I ² is not an absolute measure of heterogeneity. Res Synth Methods. 2017; 8(1):5–18. https://doi.org/10.1002/jrsm.1230.

37.

Higgins JPT, Thompson SG, Deeks JJ, Altman DG. Measuring inconsistency in meta-analyses. BMJ. 2003; 327(7414):557–60. https://doi.org/10.1136/bmj.327.7414.557.

38.

R Core Team. R: a Language and Environment for Statistical Computing. Vienna, Austria: R Foundation for Statistical Computing; 2016. R Foundation for Statistical Computing. https://www.R-project.org/.

39.

Viechtbauer W. metafor: Meta-analysis Package For R. 2009. R package. https://cran.r-project.org/package=metafor.

40.

Viechtbauer W. Conducting meta-analyses in R with the metafor package. J Stat Softw. 2010; 36(3):1–48.CrossRef

41.

Röver C. bayesmeta: Bayesian random-effects Meta-analysis. 2015. R package. https://cran.r-project.org/package=bayesmeta.

42.

Seide SE, Röver C, Friede T. Meta-analysis data extracted from IQWiG publications. Göttingen Research Online. 2018. https://doi.org/10.25625/BWYBNK.

43.

44.

Kontopantelis E, Springate DA, Reeves D. A re-analysis of the Cochrane Library data: the dangers of unobserved heterogeneity in meta-analyses. PLoS ONE. 2013; 8(7):1–14. https://doi.org/10.1371/journal.pone.0069930.

45.

Rukhin AL. Estimating heterogeneity variance in meta-analysis. J R Stat Soc Ser B (Stat Methodol). 2013; 75(3):451–69. https://doi.org/10.1111/j.1467-9868.2012.01047.x.

46.

Partlett C, Riley RD. Random effects meta-analysis: Coverage performance of 95% confidence and prediction intervals following REML estimation. Stat Med. 2017; 36(2):301–17. https://doi.org/10.1002/sim.7140.

47.

Günhan BK, Friede T, Held L. A design-by-treatment interaction model for network meta-analysis and meta-regression with integrated nested Laplace approximations. 2018; 9(2):179–94. https://doi.org/10.1002/jrsm.1285.

48.

Jackson D, White IR. When should meta-analysis avoid making hidden normality assumptions?Biom J. 2018. https://doi.org/10.1002/bimj.201800071.

49.

Veroniki AA, Jackson D, Bender R, Kuß O, Langan D, Higgins JPT, Knapp G, Salanti G. Methods to calculate uncertainty in the estimated overall effect size from a random-effects meta-analysis. 2018. https://doi.org/10.1002/jrsm.1319.

50.

Röver C, Friede T. Contribution to the discussion of “When should meta-analysis avoid making hidden normality assumptions?”: A Bayesian perspective. Biom J. 2018; 60(6):1068–70. https://doi.org/10.1002/bimj.201800179.

51.

Günhan BK, Röver S, Friede T. Meta-analysis of few studies involving rare events. arXiv preprint 1809.04407. 2018.

Title: Likelihood-based random-effects meta-analysis with few studies: empirical and simulation studies
Authors: Svenja E. Seide
Christian Röver
Tim Friede
Publication date: 01-12-2019
Publisher: BioMed Central
Published in: BMC Medical Research Methodology / Issue 1/2019
Electronic ISSN: 1471-2288
DOI: https://doi.org/10.1186/s12874-018-0618-3

At a glance: The ONWARDS insulin icodec trials

Springer Medicine

Likelihood-based random-effects meta-analysis with few studies: empirical and simulation studies

Abstract

Background

Methods

Results

Conclusions

At a glance: The ONWARDS insulin icodec trials

Springer Medicine

Abstract

Background

Methods

Results

Conclusions

Please log in to get access to this content

Other articles of this Issue 1/2019

A comparison of machine learning techniques for classification of HIV patients with antiretroviral therapy-induced mitochondrial toxicity from those without mitochondrial toxicity

Crowding in the emergency department in the absence of boarding – a transition regression model to predict departures and waiting time

A review of spline function procedures in R

Test equating sleep scales: applying the Leunbach’s model

Assessing causal treatment effect estimation when using large observational datasets

A novel approach to determine two optimal cut-points of a continuous predictor with a U-shaped relationship to hazard ratio in survival data: simulation and application