Top

Published in:

01-01-2020 | GUIDELINES

A 24-step guide on how to design, conduct, and successfully publish a systematic review and meta-analysis in medical research

Authors: Taulant Muka, Marija Glisic, Jelena Milic, Sanne Verhoog, Julia Bohlius, Wichor Bramer, Rajiv Chowdhury, Oscar H. Franco

Published in: European Journal of Epidemiology | Issue 1/2020

Abstract

To inform evidence-based practice in health care, guidelines and policies require accurate identification, collation, and integration of all available evidence in a comprehensive, meaningful, and time-efficient manner. Approaches to evidence synthesis such as carefully conducted systematic reviews and meta-analyses are essential tools to summarize specific topics. Unfortunately, not all systematic reviews are truly systematic, and their quality can vary substantially. Since well-conducted evidence synthesis typically involves a complex set of steps, we believe formulating a cohesive, step-by-step guide on how to conduct a systemic review and meta-analysis is essential. While most of the guidelines on systematic reviews focus on how to report or appraise systematic reviews, they lack guidance on how to synthesize evidence efficiently. To facilitate the design and development of evidence syntheses, we provide a clear and concise, 24-step guide on how to perform a systematic review and meta-analysis of observational studies and clinical trials. We describe each step, illustrate it with concrete examples, and provide relevant references for further guidance. The 24-step guide (1) simplifies the methodology of conducting a systematic review, (2) provides healthcare professionals and researchers with methodologically sound tools for conducting systematic reviews and meta-analyses, and (3) it can enhance the quality of existing evidence synthesis efforts. This guide will help its readers to better understand the complexity of the process, appraise the quality of published systematic reviews, and better comprehend (and use) evidence from medical literature.

Available only for authorised users

Manchikanti L. Evidence-based medicine, systematic reviews, and guidelines in interventional pain management, part I: introduction and general considerations. Pain Physician. 2008;11(2):161–86.PubMed

Moher D, Cook DJ, Eastwood S, Olkin I, Rennie D, Stroup DF. Improving the quality of reports of meta-analyses of randomised controlled trials: the QUOROM statement. Qual Rep Meta-Anal Lancet. 1999;354(9193):1896–900.

Moher D, Liberati A, Tetzlaff J, Altman DG, Group P. Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. Ann Intern Med. 2009;151(4):264–9.CrossRef

Dekkers OM, Vandenbroucke JP, Cevallos M, Renehan AG, Altman DG, Egger M. COSMOS-E: guidance on conducting systematic reviews and meta-analyses of observational studies of etiology. PLoS Med. 2019;16(2):e1002742. https://doi.org/10.1371/journal.pmed.1002742.CrossRefPubMedPubMedCentral

Conway A, Inglis SC, Chang AM, Horton-Breshears M, Cleland JG, Clark RA. Not all systematic reviews are systematic: a meta-review of the quality of systematic reviews for non-invasive remote monitoring in heart failure. J Telemed Telecare. 2013;19(6):326–37. https://doi.org/10.1177/1357633X13503427.CrossRefPubMed

Gurevitch J, Koricheva J, Nakagawa S, Stewart G. Meta-analysis and the science of research synthesis. Nature. 2018;555(7695):175–82. https://doi.org/10.1038/nature25753.CrossRefPubMed

Nikolakopoulou A, Mavridis D, Furukawa TA, et al. Living network meta-analysis compared with pairwise meta-analysis in comparative effectiveness research: empirical study. BMJ (Clin Res). 2018;360:k585. https://doi.org/10.1136/bmj.k585.CrossRefPubMedCentral

Schardt C, Adams MB, Owens T, Keitz S, Fontelo P. Utilization of the PICO framework to improve searching PubMed for clinical questions. BMC Med Inform Decis Mak. 2007;7:16. https://doi.org/10.1186/1472-6947-7-16.CrossRefPubMedPubMedCentral

Bettany-Saltikov J. How to do a systematic literature review in nursing: a step-by-step guide. Berkshire: McGraw-Hill Education; 2012.

10.

Booth Andrew. Clear and present questions: formulating questions for evidence based practice. Library Hi Tech. 2006;24(3):355–68.CrossRef

11.

Cooke A, Smith D, Booth A. Beyond PICO: the SPIDER tool for qualitative evidence synthesis. Qual Health Res. 2012;22(10):1435–43. https://doi.org/10.1177/1049732312452938.CrossRefPubMed

12.

Haynes AS, Derrick GE, Redman S, et al. Identifying trustworthy experts: how do policymakers find and assess public health researchers worth consulting or collaborating with? PLoS ONE. 2012;7(3):e32665. https://doi.org/10.1371/journal.pone.0032665.CrossRefPubMedPubMedCentral

13.

Oxman AD, Guyatt GH. The science of reviewing research. Ann NY Acad Sci. 1993;703:125–33 discussion 33–4.CrossRef

14.

Bramer WM, Rethlefsen ML, Kleijnen J, Franco OH. Optimal database combinations for literature searches in systematic reviews: a prospective exploratory study. Syst Rev. 2017;6(1):245. https://doi.org/10.1186/s13643-017-0644-y.CrossRefPubMedPubMedCentral

15.

Rethlefsen ML, Farrell AM, Osterhaus Trzasko LC, Brigham TJ. Librarian co-authors correlated with higher quality reported search strategies in general internal medicine systematic reviews. J Clin Epidemiol. 2015;68(6):617–26. https://doi.org/10.1016/j.jclinepi.2014.11.025.CrossRefPubMed

16.

Glisic M, Kastrati N, Gonzalez-Jaramillo V, et al. Associations between phytoestrogens, glucose homeostasis, and risk of diabetes in women: a systematic review and meta-analysis. Adv Nutr. 2018;9(6):726–40. https://doi.org/10.1093/advances/nmy048.CrossRefPubMedPubMedCentral

17.

Franco OH, Chowdhury R, Troup J, et al. Use of plant-based therapies and menopausal symptoms: a systematic review and meta-analysis. JAMA. 2016;315(23):2554–63. https://doi.org/10.1001/jama.2016.8012.CrossRefPubMed

18.

Elamin MB, Flynn DN, Bassler D, et al. Choice of data extraction tools for systematic reviews depends on resources and review complexity. J Clin Epidemiol. 2009;62(5):506–10. https://doi.org/10.1016/j.jclinepi.2008.10.016.CrossRefPubMed

19.

Bramer WM, Milic J, Mast F. Reviewing retrieved references for inclusion in systematic reviews using EndNote. J Med Libr Assoc. 2017;105(1):84–7. https://doi.org/10.5195/jmla.2017.111.CrossRefPubMedPubMedCentral

20.

Mourad Ouzzani HH, Fedorowicz Zbys, Elmagarmid Ahmed. Rayyan—a web and mobile app for systematic reviews. Syst Rev. 2016;5:210. https://doi.org/10.1186/s13643-016-0384-4.CrossRefPubMedPubMedCentral

21.

DistillerSR EP, Ottawa, Canada, Available at https://www.evidencepartners.com/.

22.

Covidence. Cochrane Community. https://community.cochrane.org/help/tools-and-software/covidence. Accessed 3 Jul 2018.

23.

Higgins JPT, Savovic J, Page MJ, Hróbjartsson A, Boutron I, et al. A revised tool for assessing risk of bias in randomized trials. Cochrane Database Syst Rev. 2016;10(Suppl 1):29–31.

24.

Higgins JP, Altman DG, Gotzsche PC, et al. The cochrane collaboration’s tool for assessing risk of bias in randomised trials. BMJ (Clin Res Ed.). 2011;343:d5928. https://doi.org/10.1136/bmj.d5928.CrossRef

25.

Sterne JA, Hernan MA, Reeves BC, et al. ROBINS-I: a tool for assessing risk of bias in non-randomised studies of interventions. BMJ (Clin Res Ed). 2016. https://doi.org/10.1136/bmj.i4919.CrossRef

26.

Whiting PF, Rutjes AW, Westwood ME, et al. QUADAS-2: a revised tool for the quality assessment of diagnostic accuracy studies. Ann Intern Med. 2011;155(8):529–36. https://doi.org/10.7326/0003-4819-155-8-201110180-00009.CrossRef

27.

Hayden JA, van der Windt DA, Cartwright JL, Cote P, Bombardier C. Assessing bias in studies of prognostic factors. Ann Intern Med. 2013;158(4):280–6. https://doi.org/10.7326/0003-4819-158-4-201302190-00009.CrossRef

28.

Wolff RF, Moons KGM, Riley RD, et al. PROBAST: a tool to assess the risk of bias and applicability of prediction model studies. Ann Intern Med. 2019;170(1):51–8. https://doi.org/10.7326/M18-1376.CrossRefPubMed

29.

Luhnen M, Prediger B, Neugebauer EAM, Mathes T. Systematic reviews of health economic evaluations: a protocol for a systematic review of characteristics and methods applied. Syst Rev. 2017;6(1):238. https://doi.org/10.1186/s13643-017-0639-8.CrossRefPubMedPubMedCentral

30.

Rodgers M, Sowden A, Petticrew M, et al. Testing methodological guidance on the conduct of narrative synthesis in systematic reviews: effectiveness of interventions to promote smoke alarm ownership and function. Evaluation. 2009;15(1):49–73. https://doi.org/10.1177/1356389008097871.CrossRef

31.

Ioannidis JP, Patsopoulos NA, Evangelou E. Uncertainty in heterogeneity estimates in meta-analyses. BMJ (Clin Res ed.). 2007;335(7626):914–6. https://doi.org/10.1136/bmj.39343.408449.80.CrossRef

32.

Higgins JP, Thompson SG, Deeks JJ, Altman DG. Measuring inconsistency in meta-analyses. BMJ (Clin Res Ed.). 2003;327(7414):557–60. https://doi.org/10.1136/bmj.327.7414.557.CrossRef

33.

Huedo-Medina TB, Sanchez-Meca J, Marin-Martinez F, Botella J. Assessing heterogeneity in meta-analysis: Q statistic or I2 index? Psychol Methods. 2006;11(2):193–206. https://doi.org/10.1037/1082-989X.11.2.193.CrossRefPubMed

34.

Higgins JP, Thompson SG. Quantifying heterogeneity in a meta-analysis. Stat Med. 2002;21(11):1539–58. https://doi.org/10.1002/sim.1186.CrossRef

35.

Rucker G, Schwarzer G, Carpenter JR, Schumacher M. Undue reliance on I(2) in assessing heterogeneity may mislead. BMC Med Res Methodol. 2008;8:79. https://doi.org/10.1186/1471-2288-8-79.CrossRefPubMedPubMedCentral

36.

Ioannidis JP, Haidich AB, Pappa M, et al. Comparison of evidence of treatment effects in randomized and nonrandomized studies. JAMA. 2001;286(7):821–30. https://doi.org/10.1001/jama.286.7.821.CrossRefPubMed

37.

Reeves BCDJ, Higgins JPT. Wells GA Chapter 13: Including non-randomized studies. In: Higgins JPT, Green S, editors. Cochrane Handbook for Systematic Reviews of Interventions Version 510 [updated March 2011]. Oxford: The Cochrane Collaboration; 2011.

38.

Abraham NS, Byrne CJ, Young JM, Solomon MJ. Meta-analysis of well-designed nonrandomized comparative studies of surgical procedures is as good as randomized controlled trials. J Clin Epidemiol. 2010;63(3):238–45. https://doi.org/10.1016/j.jclinepi.2009.04.005.CrossRefPubMed

39.

Tonin FS, Rotta I, Mendes AM, Pontarolo R. Network meta-analysis: a technique to gather evidence from direct and indirect comparisons. Pharm Pract (Granada). 2017;15(1):943. https://doi.org/10.18549/PharmPract.2017.01.943.CrossRef

40.

Dias S, Caldwell DM. Network meta-analysis explained. Arch Dis Child Fetal Neonatal Ed. 2019;104(1):F8–12. https://doi.org/10.1136/archdischild-2018-315224.CrossRefPubMed

41.

Taneri PE, Kiefte-de Jong JC, Bramer WM, Daan NM, Franco OH, Muka T. Association of alcohol consumption with the onset of natural menopause: a systematic review and meta-analysis. Hum Reprod Update. 2016;22(4):516–28. https://doi.org/10.1093/humupd/dmw013.CrossRefPubMed

42.

Nikolakopoulou A, Mavridis D, Salanti G. Demystifying fixed and random effects meta-analysis. Evid-Based Mental Health. 2014;17(2):53–7. https://doi.org/10.1136/eb-2014-101795.CrossRef

43.

Harris RBM, Deeks J, et al. Metan: fixed-and random-effects meta-analysis. Stata J. 2008. https://doi.org/10.1177/1536867X0800800102.CrossRef

44.

Chaimani A, Mavridis D, Salanti G. A hands-on practical tutorial on performing meta-analysis with Stata. Evid-Based Mental Health. 2014;17(4):111–6. https://doi.org/10.1136/eb-2014-101967.CrossRef

45.

Higgins JPT GSe. Cochrane Handbook for Systematic Reviews of Interventions Version 5.1.0 [updated March 2011]. The Cochrane Collaboration, 2011. Available from www.handbook.cochrane.org.

46.

Thompson SG, Higgins JP. How should meta-regression analyses be undertaken and interpreted? Stat Med. 2002;21(11):1559–73. https://doi.org/10.1002/sim.1187.CrossRefPubMed

47.

Marty Richardsona P, Doneganb S. Interpretation of subgroup analyses in systematic reviews: a tutorial. Clin Epidemiol Glob Health. 2019;7:192–8.CrossRef

48.

Hannah R, Rothstein AJS, Borenstein M. Publication bias in meta-analysis. New York: Wiley; 2005.

49.

Mavridis D, Salanti G. Exploring and accounting for publication bias in mental health: a brief overview of methods. Evid Based Ment Health. 2014;17(1):11–5. https://doi.org/10.1136/eb-2013-101700.CrossRefPubMed

50.

Lau J, Ioannidis JP, Terrin N, Schmid CH, Olkin I. The case of the misleading funnel plot. BMJ (Clin Res Ed). 2006;333(7568):597–600. https://doi.org/10.1136/bmj.333.7568.597.CrossRef

51.

Sutton AJ, Higgins JP. Recent developments in meta-analysis. Stat Med. 2008;27(5):625–50. https://doi.org/10.1002/sim.2934.CrossRefPubMed

52.

Harbord RM, Egger M, Sterne JA. A modified test for small-study effects in meta-analyses of controlled trials with binary endpoints. Stat Med. 2006;25(20):3443–57. https://doi.org/10.1002/sim.2380.CrossRefPubMed

53.

Begg CB, Mazumdar M. Operating characteristics of a rank correlation test for publication bias. Biometrics. 1994;50(4):1088–101.CrossRef

54.

Schünemann H, Brożek J, Guyatt G, Oxman A, editors. GRADE handbook for grading quality of evidence and strength of recommendations. Updated October 2013. The GRADE Working Group, 2013. Available from guidelinedevelopment.org/handbook. 2013.

Title: A 24-step guide on how to design, conduct, and successfully publish a systematic review and meta-analysis in medical research
Authors: Taulant Muka
Marija Glisic
Jelena Milic
Sanne Verhoog
Julia Bohlius
Wichor Bramer
Rajiv Chowdhury
Oscar H. Franco
Publication date: 01-01-2020
Publisher: Springer Netherlands
Published in: European Journal of Epidemiology / Issue 1/2020
Print ISSN: 0393-2990
Electronic ISSN: 1573-7284
DOI: https://doi.org/10.1007/s10654-019-00576-5

Keynote webinar | Spotlight on sleep in brain health

Springer Medicine

A 24-step guide on how to design, conduct, and successfully publish a systematic review and meta-analysis in medical research

Abstract

Keynote webinar | Spotlight on sleep in brain health

Springer Medicine

Abstract

Please log in to get access to this content

Other articles of this Issue 1/2020

Prevalence, incidence and future projection of diabetic eye disease in Europe: a systematic review and meta-analysis

The Longitudinal Aging Study Amsterdam: cohort update 2019 and additional data collections

Risk of colorectal cancer in users of bisphosphonates: analysis of population-based electronic health records

Authors’ Reply: A comparison of different methods to handle missing data in the context of propensity score analysis

Opportunities, challenges and expectations management for translating biobank research to precision medicine

The validity of propensity score analysis using complete cases with partially observed covariates