Top

Systematic Reviews

Published in:

Open Access 01-12-2024 | Research

Synthesis methods used to combine observational studies and randomised trials in published meta-analyses

Authors: Cherifa Cheurfa, Sofia Tsokani, Katerina-Maria Kontouli, Isabelle Boutron, Anna Chaimani

Published in: Systematic Reviews | Issue 1/2024

Abstract

Background

This study examined the synthesis methods used in meta-analyses pooling data from observational studies (OSs) and randomised controlled trials (RCTs) from various medical disciplines.

Methods

We searched Medline via PubMed to identify reports of systematic reviews of interventions, including and pooling data from RCTs and OSs published in 110 high-impact factor general and specialised journals between 2015 and 2019. Screening and data extraction were performed in duplicate. To describe the synthesis methods used in the meta-analyses, we considered the first meta-analysis presented in each article.

Results

Overall, 132 reports were identified with a median number of included studies of 14 [9–26]. The median number of OSs was 6.5 [3–12] and that of RCTs was 3 [1–6]. The effect estimates recorded from OSs (i.e., adjusted or unadjusted) were not specified in 82% (n = 108) of the meta-analyses. An inverse-variance common-effect model was used in 2% (n = 3) of the meta-analyses, a random-effects model was used in 55% (n = 73), and both models were used in 40% (n = 53). A Poisson regression model was used in 1 meta-analysis, and 2 meta-analyses did not report the model they used. The mean total weight of OSs in the studied meta-analyses was 57.3% (standard deviation, ± 30.3%). Only 44 (33%) meta-analyses reported results stratified by study design. Of them, the results between OSs and RCTs had a consistent direction of effect in 70% (n = 31). Study design was explored as a potential source of heterogeneity in 79% of the meta-analyses, and confounding factors were investigated in only 10% (n = 13). Publication bias was assessed in 70% (n = 92) of the meta-analyses. Tau-square was reported in 32 meta-analyses with a median of 0.07 [0–0.30].

Conclusion

The inclusion of OSs in a meta-analysis on interventions could provide useful information. However, considerations of several methodological and conceptual aspects of OSs, that are required to avoid misleading findings, were often absent or insufficiently reported in our sample.

Available only for authorised users

Ziff OJ, Lane DA, Samra M, Griffith M, Kirchhof P, Lip GYH, et al. Safety and efficacy of digoxin: systematic review and meta-analysis of observational and controlled trial data. BMJ. 2015;351: h4451. https://doi.org/10.1136/bmj.h4451.CrossRefPubMedPubMedCentral

Metelli S, Chaimani A. Challenges in meta-analyses with observational studies. Evid Based Ment Health. 2020;23:83–7. https://doi.org/10.1136/ebmental-2019-300129.CrossRefPubMedPubMedCentral

Shrier I, Boivin J-F, Steele RJ, Platt RW, Furlan A, Kakuma R, et al. Should meta-analyses of interventions include observational studies in addition to randomized controlled trials? A critical examination of underlying principles. Am J Epidemiol. 2007;166:1203–9. https://doi.org/10.1093/aje/kwm189.CrossRefPubMed

Abraham NS, Byrne CJ, Young JM, Solomon MJ. Meta-analysis of well-designed nonrandomized comparative studies of surgical procedures is as good as randomized controlled trials. J Clin Epidemiol. 2010;63:238–45. https://doi.org/10.1016/j.jclinepi.2009.04.005.CrossRefPubMed

Tinetti ME. The gap between clinical trials and the real world: extrapolating treatment effects from younger to older adults. JAMA Intern Med. 2014;174:397–8. https://doi.org/10.1001/jamainternmed.2013.13283.CrossRefPubMed

Norris SL, Atkins D, Bruening W, Fox S, Johnson E, Kane R, et al. Observational studies in systematic [corrected] reviews of comparative effectiveness: AHRQ and the Effective Health Care Program. J Clin Epidemiol. 2011;64:1178–86. https://doi.org/10.1016/j.jclinepi.2010.04.027.CrossRefPubMed

Ioannidis JP, Haidich AB, Pappa M, Pantazis N, Kokori SI, Tektonidou MG, et al. Comparison of evidence of treatment effects in randomized and nonrandomized studies. JAMA. 2001;286:821–30. https://doi.org/10.1001/jama.286.7.821.CrossRefPubMed

Kimachi M, Onishi A, Tajika A, Kimachi K, Furukawa TA. Systematic differences in effect estimates between observational studies and randomized control trials in meta-analyses in nephrology. Sci Rep. 2021;11:6088. https://doi.org/10.1038/s41598-021-85519-5.ADSCrossRefPubMedPubMedCentral

Schwingshackl L, Balduzzi S, Beyerbach J, Bröckelmann N, Werner SS, Zähringer J, et al. Evaluating agreement between bodies of evidence from randomised controlled trials and cohort studies in nutrition research: meta-epidemiological study. BMJ. 2021;374: n1864. https://doi.org/10.1136/bmj.n1864.CrossRefPubMedPubMedCentral

10.

Naudet F, Maria AS, Falissard B. Antidepressant response in major depressive disorder: a meta-regression comparison of randomized controlled trials and observational studies. PLoS ONE. 2011;6: e20811. https://doi.org/10.1371/journal.pone.0020811.ADSCrossRefPubMedPubMedCentral

11.

Shikata S, Nakayama T, Noguchi Y, Taji Y, Yamagishi H. Comparison of effects in randomized controlled trials with observational studies in digestive surgery. Ann Surg. 2006;244:668–76. https://doi.org/10.1097/01.sla.0000225356.04304.bc.CrossRefPubMedPubMedCentral

12.

Bun R-S, Scheer J, Guillo S, Tubach F, Dechartres A. Meta-analyses frequently pooled different study types together: a meta-epidemiological study. J Clin Epidemiol. 2020;118:18–28. https://doi.org/10.1016/j.jclinepi.2019.10.013.CrossRefPubMed

13.

Golder S, Loke YK, Bland M. Meta-analyses of adverse effects data derived from randomised controlled trials as compared to observational studies: methodological overview. PLoS Med. 2011;8: e1001026. https://doi.org/10.1371/journal.pmed.1001026.CrossRefPubMedPubMedCentral

14.

Beyerbach J, Stadelmaier J, Hoffmann G, Balduzzi S, Bröckelmann N, Schwingshackl L. Evaluating concordance of bodies of evidence from randomized controlled trials, dietary intake, and biomarkers of intake in cohort studies: A meta-epidemiological study. Adv Nutr Bethesda Md. 2022;13:48–65. https://doi.org/10.1093/advances/nmab095.CrossRef

15.

Wang SV, Schneeweiss S, RCT-DUPLICATE Initiative, Franklin JM, Desai RJ, Feldman W, et al. Emulation of randomized clinical trials with nonrandomized database analyses: Results of 32 clinical trials. JAMA 2023;329:1376–85. https://doi.org/10.1001/jama.2023.4221.

16.

Verde PE, Ohmann C. Combining randomized and non-randomized evidence in clinical research: a review of methods and applications. Res Synth Methods. 2015;6:45–62. https://doi.org/10.1002/jrsm.1122.CrossRefPubMed

17.

Mueller M, D’Addario M, Egger M, Cevallos M, Dekkers O, Mugglin C, et al. Methods to systematically review and meta-analyse observational studies: a systematic scoping review of recommendations. BMC Med Res Methodol. 2018;18:44. https://doi.org/10.1186/s12874-018-0495-9.CrossRefPubMedPubMedCentral

18.

Dwyer T, Couper D, Walter SD. Sources of heterogeneity in the meta-analysis of observational studies: the example of SIDS and sleeping position. J Clin Epidemiol. 2001;54:440–7. https://doi.org/10.1016/s0895-4356(00)00313-9.CrossRefPubMed

19.

Simunovic N, Sprague S, Bhandari M. Methodological issues in systematic reviews and meta-analyses of observational studies in orthopaedic research. J Bone Joint Surg Am. 2009;91(Suppl 3):87–94. https://doi.org/10.2106/JBJS.H.01576.CrossRefPubMed

20.

O’Connor AM, Sargeant JM. Meta-analyses including data from observational studies. Prev Vet Med. 2014;113:313–22. https://doi.org/10.1016/j.prevetmed.2013.10.017.CrossRefPubMed

21.

Higgins JP, Ramsay C, Reeves BC, Deeks JJ, Shea B, Valentine JC, et al. Issues relating to study design and risk of bias when including non-randomized studies in systematic reviews on the effects of interventions. Res Synth Methods. 2013;4:12–25. https://doi.org/10.1002/jrsm.1056.CrossRefPubMed

22.

Valentine JC, Thompson SG. Issues relating to confounding and meta-analysis when including non-randomized studies in systematic reviews on the effects of interventions. Res Synth Methods. 2013;4:26–35. https://doi.org/10.1002/jrsm.1064.CrossRefPubMed

23.

Thompson S, Ekelund U, Jebb S, Lindroos AK, Mander A, Sharp S, et al. A proposed method of bias adjustment for meta-analyses of published observational studies. Int J Epidemiol. 2011;40:765–77. https://doi.org/10.1093/ije/dyq248.CrossRefPubMed

24.

Cheurfa C, Tsokani S, Kontouli K-M, Boutron I, Chaimani A. Empirical evaluation of the methods used in systematic reviews including observational studies and randomized trials. J Clin Epidemiol. 2023;158:44–52. https://doi.org/10.1016/j.jclinepi.2023.02.008.CrossRefPubMed

25.

Jonathan J Deeks, Julian PT Higgins, Douglas G Altman; on behalf of the Cochrane Statistical Methods Group. Chapter 10: Analysing data and undertaking meta-analyses. In: Higgins J, Thomas J, Chandler J, Cumpston M, Li T, Page MJ, Welch VA, editors. Cochrane Handbook for Systematic Reviews of Interventions Version 6.3. Cochrane; 2022. https://training.cochrane.org/handbook/current/chapter-10 (Accessed 21 Sept 2022).

26.

R. 3.4.2, R Core Team (2014) R: A language and environment for statistical computing. R Foundation for Statistical Computing, Vienna, Austria. URL. https://www.r-project.org

27.

XLSTAT 2022.1 version, by Microsoft Excel. https://www.xlstat.com/fr/.

28.

Mariani J, Macchia A, De Abreu M, Gonzalez Villa Monte G, Tajer C. Multivessel versus Single Vessel Angioplasty in Non-ST Elevation Acute Coronary Syndromes: A Systematic Review and Meta analysis. PloS One 2016;11:e0148756. https://doi.org/10.1371/journal.pone.0148756.

29.

Sterne JA, Hernán MA, Reeves BC, Savović J, Berkman ND, Viswanathan M, et al. ROBINS-I: a tool for assessing risk of bias in non-randomised studies of interventions. BMJ. 2016;355: i4919. https://doi.org/10.1136/bmj.i4919.CrossRefPubMedPubMedCentral

30.

Schünemann HJ, Cuello C, Akl EA, Mustafa RA, Meerpohl JJ, Thayer K, et al. GRADE Working Group. GRADE guidelines: 18. How ROBINS-I and other tools to assess risk of bias in nonrandomized studies should be used to rate the certainty of a body of evidence. J Clin Epidemiol 2019;111:105–14. https://doi.org/10.1016/j.jclinepi.2018.01.012.

31.

Reeves BC, Deeks JJ, Higgins JPT, Shea B, Tugwell P, Wells GA; on behalf of the Cochrane Non-Randomized Studies of Interventions Methods Group. Chapter 24: Including non-randomized studies on intervention effects. In: Higgins J, Thomas J, Chandler J, Cumpston M, Li T, Page MJ, Welch VA, editors. Cochrane Handbook for Systematic Reviews of Interventions Version 6.3. Cochrane; 2022. https://training.cochrane.org/handbook/current/chapter-24 (Accessed 20 Sept 2021).

32.

Veroniki AA, Tsokani S, Zevgiti S, Pagkalidou I, Kontouli KM, Ambarcioglu P, et al. Do reporting guidelines have an impact? Empirical assessment of changes in reporting before and after the PRISMA extension statement for network meta-analysis. Syst Rev. 2021;10(1):246. https://doi.org/10.1186/s13643-021-01780-9.Erratum.In:SystRev.2022Jun3;11(1):112.CrossRefPubMedPubMedCentral

Title: Synthesis methods used to combine observational studies and randomised trials in published meta-analyses
Authors: Cherifa Cheurfa
Sofia Tsokani
Katerina-Maria Kontouli
Isabelle Boutron
Anna Chaimani
Publication date: 01-12-2024
Publisher: BioMed Central
Published in: Systematic Reviews / Issue 1/2024
Electronic ISSN: 2046-4053
DOI: https://doi.org/10.1186/s13643-024-02464-w

Keynote webinar | Spotlight on medication adherence

Springer Medicine

Synthesis methods used to combine observational studies and randomised trials in published meta-analyses

Abstract

Background

Methods

Results

Conclusion

Keynote webinar | Spotlight on medication adherence

Springer Medicine

Abstract

Background

Methods

Results

Conclusion

Please log in to get access to this content

Other articles of this Issue 1/2024

Factors influencing HPV vaccine implementation in South Asia: a scoping review protocol

Correlation among post-surgery recurrence of CRSwNP and TCM syndromes and tissue inflammatory cell infiltration type: a study protocol

Understanding and treating body image disturbances in eating disorders through body illusion interventions: a scoping review protocol

Climate change, biodiversity loss, and Indigenous Peoples’ health and wellbeing: a systematic umbrella review protocol

Use of digital media by adolescents for sexual and reproductive health and rights communication in sub-Saharan Africa: a protocol for systematic review

Personalised learning in higher education for health sciences: a scoping review protocol