Top

Systematic Reviews

Published in:

Open Access 01-12-2024 | Research

Comparative analysis of Cochrane and non-Cochrane reviews over three decades

Authors: Mikkel Zola Andersen, Philine Zeinert, Jacob Rosenberg, Siv Fonnes

Published in: Systematic Reviews | Issue 1/2024

Abstract

Background

Systematic reviews are viewed as the best study design to guide clinical decision-making as they are the least biased publications assuming they are well-conducted and include well-designed studies. Cochrane was initiated in 1993 with an aim of conducting high-quality systematic reviews. We aimed to examine the publication rates of non-Cochrane systematic reviews (henceforth referred to simply as “systematic reviews”) and Cochrane reviews produced throughout Cochrane’s existence and characterize changes throughout the period.

Methods

This observational study collected data on systematic reviews published between 1993 and 2022 in PubMed. Identified Cochrane reviews were linked to data from the Cochrane Database of Systematic Reviews via their Digital Object Identifier. Systematic reviews and Cochrane reviews were analyzed separately. Two authors screened a random sample of records to validate the overall sample, providing a precision of 98%.

Results

We identified 231,602 (94%) systematic reviews and 15,038 (6%) Cochrane reviews. Publication of systematic reviews has continuously increased with a median yearly increase rate of 26%, while publication of Cochrane reviews has decreased since 2015. From 1993 to 2002, Cochrane reviews constituted 35% of all systematic reviews in PubMed compared with 3.5% in 2013–2022. Systematic reviews consistently had fewer authors than Cochrane reviews, but the number of authors increased over time for both. Chinese first authors conducted 15% and 4% of systematic reviews published from 2013–2022 and 2003–2012, respectively. Most Cochrane reviews had first authors from the UK (36%). The native English-speaking countries the USA, the UK, Canada, and Australia produced a large share of systematic reviews (42%) and Cochrane reviews (62%). The largest publishers of systematic reviews in the last 10 years were gold open access journals.

Conclusions

Publication of systematic reviews is increasing rapidly, while fewer Cochrane reviews have been published through the last decade. Native English-speaking countries produced a large proportion of both types of systematic reviews. Gold open access journals and Chinese first authors dominated the publication of systematic reviews for the past 10 years. More research is warranted examining why fewer Cochrane reviews are being published. Additionally, examining these systematic reviews for research waste metrics may provide a clearer picture of their utility.

Available only for authorised users

Bornmann L, Haunschild R, Mutz R. Growth rates of modern science: a latent piecewise growth curve approach to model publication numbers from established and new literature databases. Humanit Soc Sci Commun. 2021;8:224. https://doi.org/10.1057/s41599-021-00903-w.CrossRef

Sackett DL, Rosenberg WMC, Gray JAM, Haynes RB, Richardson WS. Evidence based medicine: what it is and what it isn’t. BMJ. 1996;312:71. https://doi.org/10.1136/bmj.312.7023.71.CrossRefPubMedPubMedCentral

Murad MH, Asi N, Alsawas M, Alahdab F. New evidence pyramid. BMJ Evid Based Med. 2016;21:125–7. https://doi.org/10.1136/ebmed-2016-110401.CrossRef

Murad MH. Clinical practice guidelines: a primer on development and dissemination. Mayo Clin Proc. 2017;92:423–33. https://doi.org/10.1016/j.mayocp.2017.01.001.CrossRefPubMed

Qaseem A, Forland F, Macbeth F, Ollenschläger G, Phillips S, van der Wees P, et al. Guidelines International Network: toward international standards for clinical practice guidelines. Ann Intern Med. 2012;156:525–31. https://doi.org/10.7326/0003-4819-156-7-201204030-00009.CrossRefPubMed

Ault A. Climbing a medical everest. Science. 2003;300:2024–5. https://doi.org/10.1126/science.300.5628.2024.CrossRefPubMed

Clarke M, Chalmers I. Reflections on the history of systematic reviews. BMJ Evidence-Based Med. 2018;23:121–2. https://doi.org/10.1136/BMJEBM-2018-110968.CrossRef

Lind J. A Treatise of the scurvy in three parts. Edinburgh: Sands, Murray and Cochran; 1753. https://www.jameslindlibrary.org/lind-j-1753/.

Higgins JPT, Thomas J, Chandler J, Cumpston M, Li T, Page MJ, et al. Cochrane Handbook of Systematic Reviews of Interventions version 6.3 (updated February 2022). Cochrane 2022. https://training.cochrane.org/handbook (accessed 19 Apr 2023).

10.

Higgins JPT, Lasserson T, Thomas J, Flemyng E, Churchill R. Methodological Expectations of Cochrane Intervention Reviews. London: Cochrane; 2023. https://community.cochrane.org/mecir-manual/key-points-and-introduction/how-cite-mecir-standards.

11.

Stroup DF, Berlin JA, Morton SC, Olkin I, Williamson GD, Rennie D, et al. Meta-analysis of observational studies in epidemiology: a proposal for reporting. JAMA. 2000;283:2008–12. https://doi.org/10.1001/jama.283.15.2008.CrossRefPubMed

12.

Page MJ, McKenzie JE, Bossuyt PM, Boutron I, Hoffmann TC, Mulrow CD, et al. The PRISMA 2020 statement: an updated guideline for reporting systematic reviews. BMJ. 2021;372: n71. https://doi.org/10.1136/bmj.n71.CrossRefPubMedPubMedCentral

13.

Home Page. JBI 2023. https://jbi.global/. Accessed 13 Sept 2023.

14.

Benchimol EI, Smeeth L, Guttmann A, Harron K, Moher D, Petersen I, et al. The Reporting of studies conducted using observational routinely-collected health data (RECORD) statement. PLOS Med. 2015;12:e1001885. https://doi.org/10.1371/journal.pmed.1001885.CrossRefPubMedPubMedCentral

15.

Buciek JH, Andersen MZ, Fonnes S, Andresen K, Rosenberg J. The historical group of withdrawn Cochrane reviews should be distinguished from retracted papers. J Evid Based Med. 2021;14:269–71. https://doi.org/10.1111/JEBM.12449.CrossRefPubMed

16.

Bramer WM, Giustini D, Kramer BM, Anderson PF. The comparative recall of Google Scholar versus PubMed in identical searches for biomedical systematic reviews: a review of searches used in systematic reviews. Syst Rev. 2013;2:115. https://doi.org/10.1186/2046-4053-2-115.CrossRefPubMedPubMedCentral

17.

PubMed. NLM 2023. https://pubmed.ncbi.nlm.nih.gov/. Accessed 13 Sept 2023.

18.

Cochrane Database of Systematic Reviews. Cochrane 2023. https://www.cochranelibrary.com/cdsr/reviews. Accessed 13 Sept 2023.

19.

Halladay CW, Trikalinos TA, Schmid IT, Schmid CH, Dahabreh IJ. Using data sources beyond PubMed has a modest impact on the results of systematic reviews of therapeutic interventions. J Clin Epidemiol. 2015;68:1076–84. https://doi.org/10.1016/J.JCLINEPI.2014.12.017.CrossRefPubMed

20.

United Nations. Eleventh United Nations conference on the standardization of geographical names. United Nations; 2018. https://doi.org/10.18356/823a24b2-en.

21.

Andersen MZ, Gülen S, Fonnes S, Andresen K, Rosenberg J. Half of Cochrane reviews were published more than 2 years after the protocol. J Clin Epidemiol. 2020;124:85–93. https://doi.org/10.1016/j.jclinepi.2020.05.011.CrossRefPubMed

22.

Tricco AC, Brehaut J, Chen MH, Moher D. Following 411 Cochrane protocols to completion: a retrospective cohort study. PLoS ONE. 2008;3:e3684. https://doi.org/10.1371/journal.pone.0003684.CrossRefPubMedPubMedCentral

23.

Runjic E, Behmen D, Pieper D, Mathes T, Tricco AC, Moher D, et al. Following Cochrane review protocols to completion 10 years later: a retrospective cohort study and author survey. J Clin Epidemiol. 2019;111:41–8. https://doi.org/10.1016/j.jclinepi.2019.03.006.CrossRefPubMed

24.

Borah R, Brown AW, Capers PL, Kaiser KA. Analysis of the time and workers needed to conduct systematic reviews of medical interventions using data from the PROSPERO registry. BMJ Open. 2017;7:e012545. https://doi.org/10.1136/bmjopen-2016-012545.CrossRefPubMedPubMedCentral

25.

Tsujimoto H, Tsujimoto Y, Kataoka Y. Unpublished systematic reviews and financial support: a meta-epidemiological study. BMC Res Notes. 2017;10:703. https://doi.org/10.1186/s13104-017-3043-5.CrossRefPubMedPubMedCentral

26.

Andersen MZ, Fonnes S, Andresen K, Rosenberg J. Most published meta-analyses were made available within two years of protocol registration. Eur J Integr Med. 2021;44:101342. https://doi.org/10.1016/j.eujim.2021.101342.CrossRef

27.

Collier A, Heilig L, Schilling L, Williams H, Dellavalle RP. Cochrane Skin Group systematic reviews are more methodologically rigorous than other systematic reviews in dermatology. Br J Dermatol. 2006;155:1230–5. https://doi.org/10.1111/j.1365-2133.2006.07496.x.CrossRefPubMed

28.

Windsor B, Popovich I, Jordan V, Showell M, Shea B, Farquhar C. Methodological quality of systematic reviews in subfertility: a comparison of Cochrane and non-Cochrane systematic reviews in assisted reproductive technologies. Hum Reprod. 2012;27:3460–6. https://doi.org/10.1093/humrep/des342.CrossRefPubMed

29.

Fleming PS, Seehra J, Polychronopoulou A, Fedorowicz Z, Pandis N. Cochrane and non-Cochrane systematic reviews in leading orthodontic journals: a quality paradigm? Eur J Orthod. 2013;35:244–8. https://doi.org/10.1093/ejo/cjs016.CrossRefPubMed

30.

Moher D, Tetzlaff J, Tricco AC, Sampson M, Altman DG. Epidemiology and reporting characteristics of systematic reviews. PLoS Med. 2007;4:e78. https://doi.org/10.1371/journal.pmed.0040078.CrossRefPubMedPubMedCentral

31.

Wen J, Ren Y, Wang L, Li Y, Liu Y, Zhou M, et al. The reporting quality of meta-analyses improves: a random sampling study. J Clin Epidemiol. 2008;61:770–5. https://doi.org/10.1016/j.jclinepi.2007.10.008.CrossRefPubMed

32.

Delaney A, Bagshaw SM, Ferland A, Laupland K, Manns B, Doig C. The quality of reports of critical care meta-analyses in the Cochrane Database of Systematic Reviews: an independent appraisal. Crit Care Med. 2007;35:589–94. https://doi.org/10.1097/01.CCM.0000253394.15628.FD.CrossRefPubMed

33.

Bunn F, Trivedi D, Alderson P, Hamilton L, Martin A, Iliffe S. The impact of Cochrane Systematic Reviews: a mixed method evaluation of outputs from Cochrane Review Groups supported by the UK National Institute for Health Research. Syst Rev. 2014;3:125. https://doi.org/10.1186/2046-4053-3-125.CrossRefPubMedPubMedCentral

34.

Champion C. Cochrane Author Satisfaction Survey 2019. London, UK: 2020. https://community.cochrane.org/sites/default/files/uploads/inline-files/Cochrane%20Author%20Satisfaction%20Survey%202019%20Report.pdf.

35.

Cochrane’s new production model. Cochrane 2023. https://futurecochrane.org/new-production-model. Accessed 13 Sept 2023.

36.

Key projects and milestones. Cochrane 2023. https://futurecochrane.org/key-projects-and-milestones-nav. Accessed 13 Sept 2023.

37.

Tian J, Zhang J, Ge L, Yang K, Song F. The methodological and reporting quality of systematic reviews from China and the USA are similar. J Clin Epidemiol. 2017;85:50–8. https://doi.org/10.1016/J.JCLINEPI.2016.12.004.CrossRefPubMed

38.

Tennant JP, Waldner F, Jacques DC, Masuzzo P, Collister LB, Hartgerink CHJ. The academic, economic and societal impacts of Open Access: an evidence-based review [version 3; peer review: 4 approved, 1 approved with reservations]. F1000Research. 2016;5:632. https://doi.org/10.12688/f1000research.8460.3.CrossRefPubMedPubMedCentral

39.

Khoo SY-S. Article processing charge hyperinflation and price insensitivity: an open access sequel to the serials crisis. Lib Q. 2019;29:1–18. https://doi.org/10.18352/lq.10280.CrossRef

40.

Fuchs C, Sandoval M. The diamond model of open access publishing: why policy makers, scholars, universities, libraries, labour unions and the publishing world need to take non-commercial, non-profit open access serious. TripleC: Communication, capitalism & critique. 2013;11:428–43. https://doi.org/10.31269/triplec.v11i2.502.CrossRef

41.

Glasziou P, Chalmers I. Research waste is still a scandal—an essay by Paul Glasziou and Iain Chalmers. BMJ. 2018;363:k4645. https://doi.org/10.1136/BMJ.K4645.CrossRef

42.

Altman DG. The scandal of poor medical research. BMJ. 1994;308:283–4. https://doi.org/10.1136/BMJ.308.6924.283.CrossRefPubMedPubMedCentral

43.

Puljak L, Lund H. Definition, harms, and prevention of redundant systematic reviews. Syst Rev. 2023;12:63. https://doi.org/10.1186/S13643-023-02191-8.CrossRefPubMedPubMedCentral

44.

Grainger MJ, Bolam FC, Stewart GB, Nilsen EB. Evidence synthesis for tackling research waste. Nat Ecol Evol. 2020;4:495–7. https://doi.org/10.1038/s41559-020-1141-6.CrossRefPubMed

45.

Gülen S, Fonnes S, Andresen K, Rosenberg J. Increasing number of authors in Cochrane reviews. J Evid Based Med. 2020;13:34–41. https://doi.org/10.1111/jebm.12371.CrossRefPubMed

46.

Tilak G, Prasad V, Jena AB. Authorship inflation in medical publications. Inquiry. 2015;52:0046958015598311. https://doi.org/10.1177/0046958015598311.CrossRefPubMedPubMedCentral

47.

Vale RD. Accelerating scientific publication in biology. Proc Natl Acad Sci U S A. 2015;112:13439–46. https://doi.org/10.1073/pnas.1511912112.CrossRefPubMedPubMedCentral

48.

Papatheodorou SI, Trikalinos TA, Ioannidis JPA. Inflated numbers of authors over time have not been just due to increasing research complexity. J Clin Epidemiol. 2008;61:546–51. https://doi.org/10.1016/J.JCLINEPI.2007.07.017.CrossRefPubMed

49.

Bachmann LM, Estermann P, Kronenberg C, Ter Riet G. Identifying diagnostic accuracy studies in EMBASE. J Med Libr Assoc. 2003;91:341–6.PubMedPubMedCentral

50.

Hoffmeyer BD, Andersen MZ, Fonnes S, Rosenberg J. Most Cochrane reviews have not been updated for more than 5 years. J Evid Based Med. 2021;14:181–4. https://doi.org/10.1111/JEBM.12447.CrossRefPubMed

Title: Comparative analysis of Cochrane and non-Cochrane reviews over three decades
Authors: Mikkel Zola Andersen
Philine Zeinert
Jacob Rosenberg
Siv Fonnes
Publication date: 01-12-2024
Publisher: BioMed Central
Published in: Systematic Reviews / Issue 1/2024
Electronic ISSN: 2046-4053
DOI: https://doi.org/10.1186/s13643-024-02531-2

Keynote webinar | Spotlight on medication adherence

Springer Medicine

Comparative analysis of Cochrane and non-Cochrane reviews over three decades

Abstract

Background

Methods

Results

Conclusions

Keynote webinar | Spotlight on medication adherence

Springer Medicine

Abstract

Background

Methods

Results

Conclusions

Please log in to get access to this content

Other articles of this Issue 1/2024

Understanding and treating body image disturbances in eating disorders through body illusion interventions: a scoping review protocol

Self-reporting of psychoneurophysical (PNP) symptoms in adults with four chronic diseases: a protocol for a scoping review

Does arthroscopic or open washout in native knee septic arthritis result in superior post-operative function? A systematic review and meta-analysis of randomised controlled trials and observational studies

Palliative care needs of people and/or their families with serious and/or chronic health conditions in low- or middle-income country (LMIC) humanitarian settings—a systematic scoping review protocol

Psychometric properties of stigma and discrimination measurement tools for persons living with HIV: a systematic review using the COSMIN methodology

Synthesis methods used to combine observational studies and randomised trials in published meta-analyses