Top

BMC Medical Research Methodology

Published in:

Open Access 01-12-2021 | Research article

Managing overlap of primary study results across systematic reviews: practical considerations for authors of overviews of reviews

Authors: Carole Lunny, Dawid Pieper, Pierre Thabet, Salmaan Kanji

Published in: BMC Medical Research Methodology | Issue 1/2021

Abstract

Background

Overviews often identify and synthesise a large number of systematic reviews on the same topic, which is likely to lead to overlap (i.e. duplication) in primary studies across the reviews. Using a primary study result multiple times in the same analysis overstates its sample size and number of events, falsely leading to greater precision in the analysis. This paper aims to: (a) describe types of overlapping data that arise from the same primary studies reported across multiple reviews, (b) describe methods to identify and explain overlap of primary study data, and (c) present six case studies illustrating different approaches to manage overlap.

Methods

We first updated the search in PubMed for methods from the MOoR framework relating to overlap of primary studies. One author screened the studies titles and abstracts, and any full-text articles retrieved, extracted methods data relating to overlap of primary studies and mapped it to the overlap methods from the MOoR framework. We also describe six case studies as examples of overviews that use specific overlap methods across the steps in the conduct of an overview. For each case study, we discuss potential methodological implications in terms of limitations, efficiency, usability, and resource use.

Results

Nine methods studies were found and mapped to the methods identified by the MOoR framework to address overlap. Overlap methods were mapped across four steps in the conduct of an overview – the eligibility criteria step, the data extraction step, the assessment of risk of bias step, and the synthesis step. Our overview case studies used multiple methods to reduce overlap at different steps in the conduct of an overview.

Conclusions

Our study underlines that there is currently no standard methodological approach to deal with overlap in primary studies across reviews. The level of complexity when dealing with overlap can vary depending on the yield, trends and patterns of the included literature and the scope of the overview question. Choosing a method might be dependent on the number of included reviews and their primary studies. Gaps in evaluation of methods to address overlap were found and further investigation in this area is needed.

Available only for authorised users

Bornmann L, Mutz R. Growth rates of modern science: a bibliometric analysis based on the number of publications and cited references. J Assoc Inf Sci Technol. 2015;66(11):2215–22 https://doi.org/10.1002/asi.23329.CrossRef

Bastian H, Glasziou P, Chalmers I. Seventy-five trials and eleven systematic reviews a day: how will we ever keep up? PLoS Med. 2010;7(9):e1000326 https://doi.org/10.1371/journal.pmed.1000326.PubMedPubMedCentralCrossRef

Ioannidis JP. The mass production of redundant, misleading, and conflicted systematic reviews and meta-analyses. Milbank Q. 2016;94(3):485–514 https://doi.org/10.1111/1468-0009.12210.PubMedPubMedCentralCrossRef

Doundoulakis I, Antza C, Apostolidou-Kiouti F, Akrivos E, Karvounis H, Kotsis V, et al. Overview of systematic reviews of non–vitamin K Oral anticoagulants in atrial fibrillation: evidence of publication overlap. Circulation. 2018;11(12):e004769 https://doi.org/10.1161/CIRCOUTCOMES.118.004769.PubMed

Lunny C, McKenzie JE, McDonald S. Retrieval of overviews of systematic reviews in MEDLINE was improved by the development of an objectively derived and validated search strategy. J Clin Epidemiol. 2016;74:107–18 https://doi.org/10.1016/j.jclinepi.2015.12.002.PubMedCrossRef

Hunt H, Pollock A, Campbell P, Estcourt L, Brunton G. An introduction to overviews of reviews: planning a relevant research question and objective for an overview. Syst Rev. 2018;7(1):39 https://doi.org/10.1186/s13643-018-0695-8.PubMedPubMedCentralCrossRef

Lunny C, Brennan SE, McDonald S, McKenzie JE. Toward a comprehensive evidence map of overview of systematic review methods: paper 1-purpose, eligibility, search and data extraction. Syst Rev. 2017;6(1):231 https://doi.org/10.1186/s13643-017-0617-1.PubMedPubMedCentralCrossRef

Lunny C, Neelakant T, Chen A, Shinger G, Stevens A, Tasnim S, Sadeghipouya S, Adams S, Zheng YW, Lin L, Yang PH. Bibliometric study of ‘overviews of systematic reviews’: evaluation of prevalence, citation and impact factor. 2021. PREPRINT (Version 1) available at Research Square. https://doi.org/10.21203/rs.3.rs-237881/v1.

Australian Government Department of Health. Review of the Australian Government Rebate on Natural Therapies for Private Health Insurance. Canberra: Communication Branch, Department of Health; 2015.

10.

Becker LA, Oxman AD. Chapter 22: Overviews of Reviews, in Cochrane Handbook for Systematic Reviews of Interventions, J.P. Higgins, Green, S., Editor. 2008, Cochrane. p. 607–631.

11.

Cooper H, Koenka AC. The overview of reviews: unique challenges and opportunities when research syntheses are the principal elements of new integrative scholarship. Am Psychol. 2012;67(6):446–62 https://doi.org/10.1037/a0027119.PubMedCrossRef

12.

Thomson D, Russell K, Becker L, Klassen T, Hartling L. The evolution of a new publication type: steps and challenges of producing overviews of reviews. Res Synth Methods. 2010;1(3–4):198–211 https://doi.org/10.1002/jrsm.30.PubMedCrossRef

13.

Pieper D, Antoine SL, Mathes T, Neugebauer EAM, Eikermann M. Systematic review finds overlapping reviews were not mentioned in every other overview. J Clin Epidemiol. 2014;67(4):368–75 https://doi.org/10.1016/j.jclinepi.2013.11.007.PubMedCrossRef

14.

Hartling L, Chisholm A, Thomson D, Dryden DM. A descriptive analysis of overviews of reviews published between 2000 and 2011. PLoS One. 2012;7(11):e49667 https://doi.org/10.1371/journal.pone.0049667. Epub 2012 Nov 15.PubMedPubMedCentralCrossRef

15.

Smith V, Devane D, Begley CM, Clarke M. Methodology in conducting a systematic review of systematic reviews of healthcare interventions. BMC Med Res Methodol. 2011;11(1):15 https://doi.org/10.1186/1471-2288-11-15.PubMedPubMedCentralCrossRef

16.

Lunny C, Brennan SE, McDonald S, McKenzie JE. Toward a comprehensive evidence map of overview of systematic review methods: paper 2-risk of bias assessment; synthesis, presentation and summary of the findings; and assessment of the certainty of the evidence. Syst Rev. 2018;7(1):159 https://doi.org/10.1186/s13643-018-0784-8.PubMedPubMedCentralCrossRef

17.

Hennessy EA, Johnson BT. Examining overlap of included studies in meta-reviews: guidance for using the corrected covered area index. Res Synth Methods. 2020;11(1):134–45 https://doi.org/10.1002/jrsm.1390.PubMedCrossRef

18.

Pollock A, Campbell P, Brunton G, Hunt H, Estcourt L. Selecting and implementing overview methods: implications from five exemplar overviews. Syst Rev. 2017;6(1):145 https://doi.org/10.1186/s13643-017-0534-3.PubMedPubMedCentralCrossRef

19.

Pollock M, Fernandes RM, Newton AS, Scott SD, Hartling L. The impact of different inclusion decisions on the comprehensiveness and complexity of overviews of reviews of healthcare interventions. Syst Rev. 2019;8(1):18 https://doi.org/10.1186/s13643-018-0914-3.PubMedPubMedCentralCrossRef

20.

Gates M, Gates A, Guitard S, Pollock M, Hartling L. Guidance for overviews of reviews continues to accumulate, but important challenges remain: a scoping review. Syst Rev. 2020;9(1):254 https://doi.org/10.1186/s13643-020-01509-0.PubMedPubMedCentralCrossRef

21.

Lunny C, Brennan SE, McDonald S, McKenzie JE. Evidence map of studies evaluating methods for conducting, interpreting and reporting overviews of systematic reviews of interventions: rationale and design. Syst Rev. 2016;5(1):4 https://doi.org/10.1186/s13643-015-0178-0.PubMedPubMedCentralCrossRef

22.

Bidonde J, Busch A, Bath B, Milosavljevic S. Exercise for adults with fibromyalgia: an umbrella systematic review with synthesis of best evidence. Curr Rheumatol Rev. 2014;10(1):45–79 https://doi.org/10.2174/1573403X10666140914155304.PubMedCrossRef

23.

Murphy LA, Harrington P, Taylor SJC, Teljeur C, Smith SM, Pinnock H, et al. Clinical-effectiveness of self-management interventions in chronic obstructive pulmonary disease: an overview of reviews. Chron Respir Dis. 2017;14(3):276–88 https://doi.org/10.1177/1479972316687208.PubMedPubMedCentralCrossRef

24.

Prousali E, Haidich AB, Fontalis A, Ziakas N, Brazitikos P, Mataftsi A. Efficacy and safety of interventions to control myopia progression in children: an overview of systematic reviews and meta-analyses. BMC Ophthalmol. 2019;19(1):106 https://doi.org/10.1186/s12886-019-1112-3.PubMedPubMedCentralCrossRef

25.

Ryan R, Santesso N, Lowe D, Hill S, Grimshaw J, Prictor M, et al. Interventions to improve safe and effective medicines use by consumers: an overview of systematic reviews. Cochrane Database Syst Rev. 2014;2014(4):CD007768 https://doi.org/10.1002/14651858.CD007768.pub3.PubMedCentral

26.

Thabet P, Joshi A, MacDonald E, Hutton B, Cheng W, Stevens A, et al. Clinical and pharmacokinetic/dynamic outcomes of prolonged infusions of beta-lactam antimicrobials: An overview of systematic reviews. PLoS One. 2021;16(1):e0244966 https://doi.org/10.1371/journal.pone.0244966.PubMedPubMedCentralCrossRef

27.

Patnode CD, et al., U.S. Preventive Services Task Force Evidence Syntheses, formerly Systematic Evidence Reviews, in Behavioral Counseling and Pharmacotherapy Interventions for Tobacco Cessation in Adults, Including Pregnant Women: A Review of Reviews for the U.S. Preventive Services Task Force. Rockville: Agency for Healthcare Research and Quality (US); 2015.

28.

Pollock, M., Fernandez, R.M., Becker, L., Pieper, D., Hartling, L., Chapter V: Overviews of reviews, in Cochrane Handbook for Systematic Reviews of Interventions version 6.0 (updated July 2019), T.J. Higgins JPT, Chandler J, Cumpston M, Li T, Page MJ, Welch VA (editors), Editor. 2019, Cochrane. Available from www.training.cochrane.org/handbook.

29.

Ballard M, Montgomery P. Risk of bias in overviews of reviews: a scoping review of methodological guidance and four-item checklist. Res Synth Methods. 2017;8(1):92–108 https://doi.org/10.1002/jrsm.1229.PubMedCrossRef

30.

Hennessy EA, Johnson BT, Keenan C. Best practice guidelines and essential methodological steps to conduct rigorous and systematic meta-reviews. Appl Psychol Health Well Being. 2019;11(3):353–81 https://doi.org/10.1111/aphw.12169.PubMedPubMedCentralCrossRef

31.

Pollock M, Fernandes RM, Newton AS, Scott SD, Hartling L. A decision tool to help researchers make decisions about including systematic reviews in overviews of reviews of healthcare interventions. Syst Rev. 2019;8(1):29 https://doi.org/10.1186/s13643-018-0768-8.PubMedPubMedCentralCrossRef

32.

Pérez-Bracchiglione J, Niño de Guzmán E, Roqué Figuls M, Urrútia G. Graphical representation of overlap degree of primary studies in systematic reviews in overviews [abstract OS29.1], in Cochrane Colloquium 2019. Santiago; 2019. https://colloquium2019.cochrane.org/abstracts/graphical-representation-overlap-degree-primary-studies-systematic-reviews-included

33.

Bougioukas KI, Vounzoulaki E, Mantsiou CD, Savvides ED, Karakosta C, Diakonidis T, et al. Methods for depicting overlap in overviews of systematic reviews: An introduction to static tabular and graphical displays. J Clin Epidemiol. 2021;132:34–45 https://doi.org/10.1016/j.jclinepi.2020.12.004. Epub 2020 Dec 9.PubMedCrossRef

34.

Mathes T, Klaßen P, Pieper D. Frequency of data extraction errors and methods to increase data extraction quality: a methodological review. BMC Med Res Methodol. 2017;17(1):152 https://doi.org/10.1186/s12874-017-0431-4.PubMedPubMedCentralCrossRef

35.

Puljak L, Riva N, Parmelli E, González-Lorenzo M, Moja L, Pieper D. Data extraction methods: an analysis of internal reporting discrepancies in single manuscripts and practical advice. J Clin Epidemiol. 2020;117:158–64 https://doi.org/10.1016/j.jclinepi.2019.09.003.PubMedCrossRef

36.

Reynders RM, Ladu L, Di Girolamo N. Contacting of authors modified crucial outcomes of systematic reviews but was poorly reported, not systematic, and produced conflicting results. J Clin Epidemiol. 2019;115:64–76 https://doi.org/10.1016/j.jclinepi.2019.07.001.CrossRef

37.

Voss PH, Rehfuess EA. Quality appraisal in systematic reviews of public health interventions: an empirical study on the impact of choice of tool on meta-analysis. J Epidemiol Community Health. 2013;67(1):98–104 https://doi.org/10.1136/jech-2011-200940.PubMedCrossRef

38.

Puljak L, Ramic I, Arriola Naharro C, Brezova J, Lin YC, Surdila AA, et al. Cochrane risk of bias tool was used inadequately in the majority of non-Cochrane systematic reviews. J Clin Epidemiol. 2020;123:114–9 https://doi.org/10.1016/j.jclinepi.2020.03.019.PubMedCrossRef

39.

Moja L, Fernandez del Rio MP, Banzi R, Cusi C, D’Amico R, Liberati A, et al. Multiple systematic reviews: methods for assessing discordances of results. Intern Emerg Med. 2012;7(6):563–8 https://doi.org/10.1007/s11739-012-0846-1.PubMedCrossRef

40.

Jadad AR, Cook DJ, Browman GP. A guide to interpreting discordant systematic reviews. Cmaj. 1997;156(10):1411–6.PubMedPubMedCentral

41.

Bolland MJ, Grey A, Reid IR. Differences in overlapping meta-analyses of vitamin D supplements and falls. J Clin Endocrinol Metab. 2014;99(11):4265–72.PubMedCrossRef

42.

Tang LL, Caudy M, Taxman F. A statistical method for synthesizing meta-analyses. Comput Math Methods Med. 2013;2013:732989.PubMedPubMedCentral

43.

Shea BJ, Grimshaw JM, Wells GA, Boers M, Andersson N, Hamel C, et al. Development of AMSTAR: a measurement tool to assess the methodological quality of systematic reviews. BMC Med Res Methodol. 2007;7(1):10 https://doi.org/10.1186/1471-2288-7-10.PubMedPubMedCentralCrossRef

44.

Effing TW, Vercoulen JH, Bourbeau J, Trappenburg J, Lenferink A, Cafarella P, et al. Definition of a COPD self-management intervention: international expert group consensus. Eur Respir J. 2016;48(1):46–54 https://doi.org/10.1183/13993003.00025-2016.PubMedCrossRef

45.

Kung J, Chiappelli F, Cajulis OO, Avezova R, Kossan G, Chew L, et al. From systematic reviews to clinical recommendations for evidence-based health care: validation of revised assessment of multiple systematic reviews (R-AMSTAR) for grading of clinical relevance. Open Dent J. 2010;4:84–91 https://doi.org/10.2174/1874210601004020084.PubMedPubMedCentral

46.

Lunny C, Brennan SE, Reid J, McDonald S, McKenzie JE. Overviews of reviews incompletely report methods for handling overlapping, discordant, and problematic data. J Clin Epidemiol. 2020;118:69–85 https://doi.org/10.1016/j.jclinepi.2019.09.025.PubMedCrossRef

47.

Pizzarello L, Abiose A, Ffytche T, Duerksen R, Thulasiraj R, Taylor H, et al. VISION 2020: the right to sight: a global initiative to eliminate avoidable blindness. Arch Ophthalmol. 2004;122(4):615–20 https://doi.org/10.1001/archopht.122.4.615.PubMedCrossRef

48.

Whiting P, Savović J, Higgins JP, Caldwell DM, Reeves BC, Shea B, et al. ROBIS: a new tool to assess risk of bias in systematic reviews was developed. J Clin Epidemiol. 2016;69:225–34 https://doi.org/10.1016/j.jclinepi.2015.06.005.PubMedPubMedCentralCrossRef

49.

Guyatt GH, Oxman AD, Vist GE, Kunz R, Falck-Ytter Y, Alonso-Coello P, et al. GRADE: an emerging consensus on rating quality of evidence and strength of recommendations. BMJ. 2008;336(7650):924–6 https://doi.org/10.1136/bmj.39489.470347.AD.PubMedPubMedCentralCrossRef

50.

NHMRC. Guideline for guidelines: Assessing certainty of the evidence: How valid and reliable are your conclusions? 2019, National Health and Medical Research Council, Australia Government. 2019. https://www.nhmrc.gov.au/guidelinesforguidelines/develop/assessing-certainty-evidence#_ENREF_8.

51.

Database of Abstracts of Reviews of Effects (DARE). Quality-assessed Reviews. 1995. Available from: https://www.ncbi.nlm.nih.gov/books/NBK285222/.

52.

Fleming PS, Seehra J, Polychronopoulou A, Fedorowicz Z, Pandis N. Cochrane and non-Cochrane systematic reviews in leading orthodontic journals: a quality paradigm? Eur J Orthod. 2013;35(2):244–8 https://doi.org/10.1093/ejo/cjs016.PubMedCrossRef

53.

Lundh A, Knijnenburg SL, Jørgensen AW, van Dalen EC, Kremer LCM. Quality of systematic reviews in pediatric oncology--a systematic review. Cancer Treat Rev. 2009;35(8):645–52 https://doi.org/10.1016/j.ctrv.2009.08.010.PubMedCrossRef

54.

Moseley AM, Elkins MR, Herbert RD, Maher CG, Sherrington C. Cochrane reviews used more rigorous methods than non-Cochrane reviews: survey of systematic reviews in physiotherapy. J Clin Epidemiol. 2009;62(10):1021–30 https://doi.org/10.1016/j.jclinepi.2008.09.018.PubMedCrossRef

55.

Windsor B, Popovich I, Jordan V, Showell M, Shea B, Farquhar C. Methodological quality of systematic reviews in subfertility: a comparison of Cochrane and non-Cochrane systematic reviews in assisted reproductive technologies. Hum Reprod. 2012;27(12):3460–6 https://doi.org/10.1093/humrep/des342.PubMedCrossRef

Title: Managing overlap of primary study results across systematic reviews: practical considerations for authors of overviews of reviews
Authors: Carole Lunny
Dawid Pieper
Pierre Thabet
Salmaan Kanji
Publication date: 01-12-2021
Publisher: BioMed Central
Published in: BMC Medical Research Methodology / Issue 1/2021
Electronic ISSN: 1471-2288
DOI: https://doi.org/10.1186/s12874-021-01269-y

Keynote webinar | Spotlight on medication adherence

Springer Medicine

Managing overlap of primary study results across systematic reviews: practical considerations for authors of overviews of reviews

Abstract

Background

Methods

Results

Conclusions

Keynote webinar | Spotlight on medication adherence

Springer Medicine

Abstract

Background

Methods

Results

Conclusions

Please log in to get access to this content

Other articles of this Issue 1/2021

Multiple imputation to quantify misclassification in observational studies of the cognitively impaired: an application for pain assessment in nursing home residents

Absolute mortality risk assessment of COVID-19 patients: the Khorshid COVID Cohort (KCC) study

Recruitment principles and strategies for supportive care research in pediatric oncology

The effect of home visits as an additional recruitment step on the composition of the final sample: a cross-sectional analysis in two study centers of the German National Cohort (NAKO)

Inspection plan for COVID-19 patients for Weibull distribution using repetitive sampling under indeterminacy

A data visualisation method for assessing exposure misclassification in case-crossover studies: the example of tricyclic antidepressants and the risk of hip fracture in older people