Top

BMC Medical Research Methodology

Published in:

Open Access 01-12-2021 | Research

Most published meta-regression analyses based on aggregate data suffer from methodological pitfalls: a meta-epidemiological study

Authors: Michael Geissbühler, Cesar A. Hincapié, Soheila Aghlmandi, Marcel Zwahlen, Peter Jüni, Bruno R. da Costa

Published in: BMC Medical Research Methodology | Issue 1/2021

Abstract

Background

Due to clinical and methodological diversity, clinical studies included in meta-analyses often differ in ways that lead to differences in treatment effects across studies. Meta-regression analysis is generally recommended to explore associations between study-level characteristics and treatment effect, however, three key pitfalls of meta-regression may lead to invalid conclusions. Our aims were to determine the frequency of these three pitfalls of meta-regression analyses, examine characteristics associated with the occurrence of these pitfalls, and explore changes between 2002 and 2012.

Methods

A meta-epidemiological study of studies including aggregate data meta-regression analysis in the years 2002 and 2012. We assessed the prevalence of meta-regression analyses with at least 1 of 3 pitfalls: ecological fallacy, overfitting, and inappropriate methods to regress treatment effects against the risk of the analysed outcome. We used logistic regression to investigate study characteristics associated with pitfalls and examined differences between 2002 and 2012.

Results

Our search yielded 580 studies with meta-analyses, of which 81 included meta-regression analyses with aggregated data. 57 meta-regression analyses were found to contain at least one pitfall (70%): 53 were susceptible to ecological fallacy (65%), 14 had a risk of overfitting (17%), and 5 inappropriately regressed treatment effects against the risk of the analysed outcome (6%). We found no difference in the prevalence of meta-regression analyses with methodological pitfalls between 2002 and 2012, nor any study-level characteristic that was clearly associated with the occurrence of any of the pitfalls.

Conclusion

The majority of meta-regression analyses based on aggregate data contain methodological pitfalls that may result in misleading findings.

Available only for authorised users

Davey Smith G, Egger M, Phillips AN. Meta-analysis beyond the grand mean? BMJ. 1997;315(7122):1610–4. https://doi.org/10.1136/bmj.315.7122.1610.CrossRefPubMedPubMedCentral

da Costa BR, Jüni P. Systematic reviews and meta-analyses of randomized trials: principles and pitfalls. Eur Heart J. 2014;35(47):3336–45. https://doi.org/10.1093/eurheartj/ehu424.CrossRefPubMed

Higgins JPT, Thomas J, Chandler J, et al. Cochrane handbook for systematic reviews of interventions. 2nd edn. Chichester: Wiley; 2019. https://doi.org/10.1002/9781119536604.CrossRef

Thompson SG, Higgins JPT. How should meta-regression analyses be undertaken and interpreted? Stat Med. 2002;21(11):1559–73. https://doi.org/10.1002/sim.1187.CrossRefPubMed

Berlin JA, Santanna J, Schmid CH, Szczech LA, Feldman HI. Anti-Lymphocyte antibody induction therapy study group. Individual patient- versus group-level data meta-regressions for the investigation of treatment effect modifiers: ecological bias rears its ugly head. Stat Med. 2002;21(3):371–87. https://doi.org/10.1002/sim.1023.CrossRefPubMed

Austin PC, Steyerberg EW. The number of subjects per variable required in linear regression analyses. J Clin Epidemiol. 2015;68(6):627–36. https://doi.org/10.1016/j.jclinepi.2014.12.014.CrossRefPubMed

Vittinghoff E, McCulloch CE. Relaxing the Rule of Ten Events per Variable in Logistic and Cox Regression. Am J Epidemiol. 2007;165(6):710–8. https://doi.org/10.1093/aje/kwk052.CrossRefPubMed

Senn S. Importance of trends in the interpretation of an overall odds ratio in the meta-analysis of clinical trials. Stat Med. 1994;13(3):293–6. https://doi.org/10.1002/sim.4780130310.CrossRefPubMed

Sharp SJ, Thompson SG, Altman DG. The relation between treatment benefit and underlying risk in meta-analysis. BMJ. 1996;313(7059):735–8. https://doi.org/10.1136/bmj.313.7059.735.CrossRefPubMedPubMedCentral

10.

McIntosh MW. The population risk as an explanatory variable in research synthesis of clinical trials. Stat Med. 1996;15(16):1713–28. https://doi.org/10.1002/(sici)1097-0258(19960830)15:16%3c1713::aid-sim331%3e3.0.co;2-d.

11.

Thompson SG, Smith TC, Sharp SJ. Investigating underlying risk as a source of heterogeneity in meta-analysis. Stat Med. 1997;16(23):2741–58. https://doi.org/10.1002/(sici)1097-0258(19971215)16:23%3c2741::aid-sim703%3e3.0.co;2-0.

12.

Sharp SJ, Thompson SG. Analysing the relationship between treatment effect and underlying risk in meta-analysis: comparison and development of approaches. Stat Med. 2000;19(23):3251–74. https://doi.org/10.1002/1097-0258(20001215)19:23%3c3251::aid-sim625%3e3.0.co;2-2.

13.

Arends LR, Hoes AW, Lubsen J, Grobbee DE, Stijnen T. Baseline risk as predictor of treatment benefit: three clinical meta-re-analyses. Stat Med. 2000;19(24):3497–518. https://doi.org/10.1002/1097-0258(20001230)19:24%3c3497::aid-sim830%3e3.0.co;2-h.

14.

Song F, Xiong T, Parekh-Bhurke S, et al. Inconsistency between direct and indirect comparisons of competing interventions: meta-epidemiological study. BMJ. 2011;343(aug16 2):d4909-9. https://doi.org/10.1136/bmj.d4909.CrossRefPubMedPubMedCentral

15.

Song F, Loke YK, Walsh T, Glenny AM, Eastwood AJ, Altman DG. Methodological problems in the use of indirect comparisons for evaluating healthcare interventions: survey of published systematic reviews. BMJ. 2009;338(apr03 1):b1147-7. https://doi.org/10.1136/bmj.b1147.CrossRefPubMedPubMedCentral

16.

Schmid CH, Lau J, McIntosh MW, Cappelleri JC. An empirical study of the effect of the control rate as a predictor of treatment efficacy in meta-analysis of clinical trials. Stat Med. 1998;17(17):1923–42. https://doi.org/10.1002/(sici)1097-0258(19980915)17:17%3c1923::aid-sim874%3e3.0.co;2-6.

17.

Heinze G. A comparative investigation of methods for logistic regression with separated or nearly separated data. Stat Med. 2006;25(24):4216–26. https://doi.org/10.1002/sim.2687.CrossRefPubMed

18.

Heinze G, Schemper M. A solution to the problem of separation in logistic regression. Stat Med. 2002;21(16):2409–19. https://doi.org/10.1002/sim.1047.CrossRefPubMed

19.

Cammà C, Schepis F, Orlando A, et al. Transarterial chemoembolization for unresectable hepatocellular carcinoma: meta-analysis of randomized controlled trials. Radiology. 2002;224(1):47–54. https://doi.org/10.1148/radiol.2241011262.CrossRefPubMed

20.

Najaka SS, Gottfredson DC, Wilson DB. A meta-analytic inquiry into the relationship between selected risk factors and problem behavior. Prev Sci. 2001;2(4):257–71. https://doi.org/10.1023/a:1013610115351.CrossRefPubMed

21.

Zapf A, Rauch G, Kieser M. Why do you need a biostatistician? BMC Med Res Methodol. 2020;20(1):23–6. https://doi.org/10.1186/s12874-020-0916-4.CrossRefPubMedPubMedCentral

22.

Giraudeau B, Higgins JPT, Tavernier E, Trinquart L. Sample size calculation for meta-epidemiological studies. Stat Med. 2016;35(2):239–50. https://doi.org/10.1002/sim.6627.CrossRefPubMed

23.

Lambert PC, Sutton AJ, Abrams KR, Jones DR. A comparison of summary patient-level covariates in meta-regression with individual patient data meta-analysis. J Clin Epidemiol. 2002;55(1):86–94. https://doi.org/10.1016/s0895-4356(01)00414-0.CrossRefPubMed

24.

Schmid CH, Stark PC, Berlin JA, Landais P, Lau J. Meta-regression detected associations between heterogeneous treatment effects and study-level, but not patient-level, factors. J Clin Epidemiol. 2004;57(7):683–97. https://doi.org/10.1016/j.jclinepi.2003.12.001.CrossRefPubMed

25.

Fisher DJ, Carpenter JR, Morris TP, Freeman SC, Tierney JF. Meta-analytical methods to identify who benefits most from treatments: daft, deluded, or deft approach? BMJ. 2017;356:j573. https://doi.org/10.1136/bmj.j573.CrossRefPubMedPubMedCentral

26.

Burke DL, Ensor J, Riley RD. Meta-analysis using individual participant data: one-stage and two-stage approaches, and why they may differ. Stat Med. 2017;36(5):855–75. https://doi.org/10.1002/sim.7141.CrossRefPubMed

27.

Sutton AJ, Kendrick D, Coupland CAC. Meta-analysis of individual- and aggregate-level data. Stat Med. 2008;27(5):651–69. https://doi.org/10.1002/sim.2916.CrossRefPubMed

28.

Rücker G, Schwarzer G, Carpenter JR, Schumacher M. Undue reliance on I 2 in assessing heterogeneity may mislead. BMC Med Res Methodol. 2008;8(1):841–9. https://doi.org/10.1186/1471-2288-8-79.CrossRef

Title: Most published meta-regression analyses based on aggregate data suffer from methodological pitfalls: a meta-epidemiological study
Authors: Michael Geissbühler
Cesar A. Hincapié
Soheila Aghlmandi
Marcel Zwahlen
Peter Jüni
Bruno R. da Costa
Publication date: 01-12-2021
Publisher: BioMed Central
Published in: BMC Medical Research Methodology / Issue 1/2021
Electronic ISSN: 1471-2288
DOI: https://doi.org/10.1186/s12874-021-01310-0

Keynote webinar | Spotlight on medication adherence

Springer Medicine

Most published meta-regression analyses based on aggregate data suffer from methodological pitfalls: a meta-epidemiological study

Abstract

Background

Methods

Results

Conclusion

Keynote webinar | Spotlight on medication adherence

Springer Medicine

Abstract

Background

Methods

Results

Conclusion

Please log in to get access to this content

Other articles of this Issue 1/2021

Feasibility of a hybrid clinical trial for respiratory virus detection in toddlers during the influenza season

A multiyear systematic survey of the quality of reporting for randomised trials in dentistry, neurology and geriatrics published in journals of Spain and Latin America

Predicting postoperative surgical site infection with administrative data: a random forests algorithm

International representation of authors, editors and research in neurology journals

Impact of the COVID-19 pandemic on publication dynamics and non-COVID-19 research production

Central data monitoring in the multicentre randomised SafeBoosC-III trial – a pragmatic approach