Top

Published in:

Open Access 01-12-2021 | Protocol

The REPRISE project: protocol for an evaluation of REProducibility and Replicability In Syntheses of Evidence

Authors: Matthew J. Page, David Moher, Fiona M. Fidler, Julian P. T. Higgins, Sue E. Brennan, Neal R. Haddaway, Daniel G. Hamilton, Raju Kanukula, Sathya Karunananthan, Lara J. Maxwell, Steve McDonald, Shinichi Nakagawa, David Nunan, Peter Tugwell, Vivian A. Welch, Joanne E. McKenzie

Published in: Systematic Reviews | Issue 1/2021

Abstract

Background

Investigations of transparency, reproducibility and replicability in science have been directed largely at individual studies. It is just as critical to explore these issues in syntheses of studies, such as systematic reviews, given their influence on decision-making and future research. We aim to explore various aspects relating to the transparency, reproducibility and replicability of several components of systematic reviews with meta-analysis of the effects of health, social, behavioural and educational interventions.

Methods

The REPRISE (REProducibility and Replicability In Syntheses of Evidence) project consists of four studies. We will evaluate the completeness of reporting and sharing of review data, analytic code and other materials in a random sample of 300 systematic reviews of interventions published in 2020 (Study 1). We will survey authors of systematic reviews to explore their views on sharing review data, analytic code and other materials and their understanding of and opinions about replication of systematic reviews (Study 2). We will then evaluate the extent of variation in results when we (a) independently reproduce meta-analyses using the same computational steps and analytic code (if available) as used in the original review (Study 3), and (b) crowdsource teams of systematic reviewers to independently replicate a subset of methods (searches for studies, selection of studies for inclusion, collection of outcome data, and synthesis of results) in a sample of the original reviews; 30 reviews will be replicated by 1 team each and 2 reviews will be replicated by 15 teams (Study 4).

Discussion

The REPRISE project takes a systematic approach to determine how reliable systematic reviews of interventions are. We anticipate that results of the REPRISE project will inform strategies to improve the conduct and reporting of future systematic reviews.

Available only for authorised users

Baker M, Penny D. 1,500 scientists lift the lid on reproducibility. Nature. 2016;533(7604):452–4. https://doi.org/10.1038/533452a.CrossRefPubMed

National Academies of Sciences Engineering and Medicine. Reproducibility and replicability in science. Washington, DC: The National Academies Press; 2019.

Nuijten MB, Bakker M, Maassen E, Wicherts JM. Verify original results through reanalysis before replicating. Behav Brain Sci. 2018;41:e143. https://doi.org/10.1017/S0140525X18000791.CrossRefPubMed

Lakens D, Hilgard J, Staaks J. On the reproducibility of meta-analyses: six practical recommendations. BMC Psychol. 2016;4(1):24. https://doi.org/10.1186/s40359-016-0126-3.CrossRefPubMedPubMedCentral

Nosek BA, Errington TM. What is replication? PLoS Biol. 2020;18(3):e3000691. https://doi.org/10.1371/journal.pbio.3000691.CrossRefPubMedPubMedCentral

Machery E. What is a replication? Philos Sci. 2020;87(4):545-67. https://doi.org/10.1086/709701.

Vachon B, Curran JA, Karunananthan S, Brehaut J, Graham ID, Moher D, et al. Replication Research Series-Paper 1 : A concept analysis and meta-narrative review established a comprehensive theoretical definition of replication research to improve its use. J Clin Epidemiol. 2021;129:176–87. https://doi.org/10.1016/j.jclinepi.2020.07.006.CrossRefPubMed

Wallach JD, Boyack KW, Ioannidis JPA. Reproducible research practices, transparency, and open access data in the biomedical literature, 2015-2017. PLoS Biol. 2018;16(11):e2006930. https://doi.org/10.1371/journal.pbio.2006930.CrossRefPubMedPubMedCentral

Hardwicke TE, Wallach JD, Kidwell MC, Bendixen T, Cruwell S, Ioannidis JPA. An empirical assessment of transparency and reproducibility-related research practices in the social sciences (2014-2017). R Soc Epen Sci. 2020;7(2):190806. https://doi.org/10.1098/rsos.190806.CrossRef

10.

Open Science Collaboration. PSYCHOLOGY. Estimating the reproducibility of psychological science. Science (New York, NY). 2015;349(6251):aac4716.CrossRef

11.

Camerer CF, Dreber A, Forsell E, Ho TH, Huber J, Johannesson M, et al. Evaluating replicability of laboratory experiments in economics. Science (New York, NY). 2016;351(6280):1433–6.CrossRef

12.

Camerer CF, Dreber A, Holzmeister F, Ho T-H, Huber J, Johannesson M, et al. Evaluating the replicability of social science experiments in Nature and Science between 2010 and 2015. Nat Hum Behav. 2018;2(9):637–44. https://doi.org/10.1038/s41562-018-0399-z.CrossRefPubMed

13.

Begley CG, Ellis LM. Drug development: raise standards for preclinical cancer research. Nature. 2012;483(7391):531–3. https://doi.org/10.1038/483531a.CrossRefPubMed

14.

Errington TM, Iorns E, Gunn W, Tan FE, Lomax J, Nosek BA. An open investigation of the reproducibility of cancer biology research. eLife. 2014;3:e04333. https://doi.org/10.7554/eLife.04333.CrossRefPubMedCentral

15.

Chang AC, Li P. Is Economics Research Replicable? Sixty Published Papers from Thirteen Journals Say “Usually Not”. In: Finance and Economics Discussion Series 2015-083 Washington: Board of Governors of the Federal Reserve System; 2015. https://doi.org/10.17016/FEDS2015083.CrossRef

16.

Naudet F, Sakarovitch C, Janiaud P, Cristea I, Fanelli D, Moher D, et al. Data sharing and reanalysis of randomized controlled trials in leading biomedical journals with a full data sharing policy: survey of studies published in The BMJ and PLOS Medicine. BMJ. 2018;360:k400.CrossRef

17.

Fidler F, Wilcox J. Reproducibility of Scientific Results. In: The Stanford Encyclopedia of Philosophy (Winter 2018 Edition); 2018. https://plato.stanford.edu/entries/scientific-reproducibility/.

18.

Tugwell P, Welch VA, Karunananthan S, Maxwell LJ, Akl EA, Avey MT, et al. When to replicate systematic reviews of interventions: consensus checklist. BMJ. 2020;370:m2864.CrossRef

19.

Wanous JP, Sullivan SE, Malinak J. The role of judgment calls in meta-analysis. J Appl Psychol. 1989;74(2):259–64. https://doi.org/10.1037/0021-9010.74.2.259.CrossRef

20.

Tendal B, Higgins JP, Juni P, Hrobjartsson A, Trelle S, Nuesch E, et al. Disagreements in meta-analyses using outcomes measured on continuous or rating scales: observer agreement study. BMJ. 2009;339(aug13 2):b3128. https://doi.org/10.1136/bmj.b3128.CrossRefPubMedPubMedCentral

21.

Palpacuer C, Hammas K, Duprez R, Laviolle B, Ioannidis JPA, Naudet F. Vibration of effects from diverse inclusion/exclusion criteria and analytical choices: 9216 different ways to perform an indirect comparison meta-analysis. BMC Med. 2019;17(1):174. https://doi.org/10.1186/s12916-019-1409-3.CrossRefPubMedPubMedCentral

22.

Haddaway NR, Rytwinski T. Meta-analysis is not an exact science: Call for guidance on quantitative synthesis decisions. Environ Int. 2018;114:357–9. https://doi.org/10.1016/j.envint.2018.02.018.CrossRefPubMed

23.

Hacke C, Nunan D. Discrepancies in meta-analyses answering the same clinical question were hard to explain: a meta-epidemiological study. J Clin Epidemiol. 2020;119:47–56. https://doi.org/10.1016/j.jclinepi.2019.11.015.CrossRefPubMed

24.

Page MJ, Welch VA, Haddaway NR, Karunananthan S, Maxwell LJ, Tugwell P. "One more time": why replicating some syntheses of evidence relevant to COVID-19 makes sense. J Clin Epidemiol. 2020;125:179–82. https://doi.org/10.1016/j.jclinepi.2020.05.024.CrossRefPubMedPubMedCentral

25.

Page MJ, Altman DG, Shamseer L, McKenzie JE, Ahmadzai N, Wolfe D, et al. Reproducible research practices are underused in systematic reviews of biomedical interventions. J Clin Epidemiol. 2018;94:8–18. https://doi.org/10.1016/j.jclinepi.2017.10.017.CrossRefPubMed

26.

Wayant C, Page MJ, Vassar M. Evaluation of reproducible research practices in oncology systematic reviews with meta-analyses referenced by national comprehensive cancer network guidelines. JAMA Oncol. 2019;5(11):1550–5. https://doi.org/10.1001/jamaoncol.2019.2564.CrossRefPubMedCentral

27.

Maggio LA, Tannery NH, Kanter SL. Reproducibility of literature search reporting in medical education reviews. Acad Med. 2011;86(8):1049–54. https://doi.org/10.1097/ACM.0b013e31822221e7.CrossRefPubMed

28.

Koffel JB, Rethlefsen ML. Reproducibility of search strategies is poor in systematic reviews published in high-impact pediatrics, cardiology and surgery journals: a cross-sectional study. PLoS One. 2016;11(9):e0163309. https://doi.org/10.1371/journal.pone.0163309.CrossRefPubMedPubMedCentral

29.

Faggion CM Jr, Huivin R, Aranda L, Pandis N, Alarcon M. The search and selection for primary studies in systematic reviews published in dental journals indexed in MEDLINE was not fully reproducible. J Clin Epidemiol. 2018;98:53–61. https://doi.org/10.1016/j.jclinepi.2018.02.011.CrossRefPubMed

30.

Polanin JR, Hennessy EA, Tsuji S. Transparency and Reproducibility of Meta-Analyses in Psychology: A Meta-Review. Perspect Psychol Sci. 2020;15(4):1026–41. https://doi.org/10.1177/1745691620906416.CrossRefPubMed

31.

Gotzsche PC, Hrobjartsson A, Maric K, Tendal B. Data extraction errors in meta-analyses that use standardized mean differences. JAMA. 2007;298(4):430–7. https://doi.org/10.1001/jama.298.4.430.CrossRefPubMed

32.

Lakens D, Page-Gould E, van Assen MA, Spellman B, Schönbrodt FD, Hasselman F, et al. Examining the reproducibility of meta-analyses in psychology: a preliminary report; 2017. https://doi.org/10.31222/osf.io/xfbjf.CrossRef

33.

Maassen E, van Assen MALM, Nuijten MB, Olsson-Collentine A, Wicherts JM. Reproducibility of individual effect sizes in meta-analyses in psychology. PLoS One. 2020;15(5):e0233107. https://doi.org/10.1371/journal.pone.0233107.CrossRefPubMedPubMedCentral

34.

Stegeman I, Leeflang MMG. Meta-analyses of diagnostic test accuracy could not be reproduced. J Clin Epidemiol. 2020;127:161–6. https://doi.org/10.1016/j.jclinepi.2020.06.033.CrossRefPubMed

35.

Thompson R, Bandera E, Burley V, Cade J, Forman D, Freudenheim J, et al. Reproducibility of systematic literature reviews on food, nutrition, physical activity and endometrial cancer. Public Health Nutr. 2008;11(10):1006–14. https://doi.org/10.1017/S1368980007001334.CrossRefPubMed

36.

Low J, Ross JS, Ritchie JD, Gross CP, Lehman R, Lin H, et al. Comparison of two independent systematic reviews of trials of recombinant human bone morphogenetic protein-2 (rhBMP-2): the Yale Open Data Access Medtronic Project. Syst Rev. 2017;6(1):28. https://doi.org/10.1186/s13643-017-0422-x.CrossRefPubMedPubMedCentral

37.

Basic documents: forty-ninth edition (including amendments adopted up to 31 May 2019). Geneva: World Health Organisation; 2020.

38.

Covidence systematic review software, Veritas Health Innovation, Melbourne, Australia. Available at www.covidence.org. Accessed 13 Apr 2021.

39.

Page MJ, Shamseer L, Altman DG, Tetzlaff J, Sampson M, Tricco AC, et al. Epidemiology and reporting characteristics of systematic reviews of biomedical research: a cross-sectional study. PLoS Med. 2016;13(5):e1002028. https://doi.org/10.1371/journal.pmed.1002028.CrossRefPubMedPubMedCentral

40.

Moher D, Liberati A, Tetzlaff J, Altman DG, The PRISMA Group. Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. PLoS Med. 2009;6:e1000097.CrossRef

41.

van Panhuis WG, Paul P, Emerson C, Grefenstette J, Wilder R, Herbst AJ, et al. A systematic review of barriers to data sharing in public health. BMC Public Health. 2014;14(1):1144. https://doi.org/10.1186/1471-2458-14-1144.CrossRefPubMedPubMedCentral

42.

Van den Eynden V, Knight G, Vlad A, Radler B, Tenopir C, Leon D, et al. Survey of Wellcome researchers and their attitudes to open research. Wellcome Trust J Contrib. 2016. https://doi.org/10.6084/m9figshare4055448v1.

43.

Ali-Khan SE, Harris LW, Gold ER. Motivating participation in open science by examining researcher incentives. eLife. 2017;6:e29319. https://doi.org/10.7554/eLife.29319.CrossRefPubMedPubMedCentral

44.

Allen C, Mehler DMA. Open science challenges, benefits and tips in early career and beyond. PLoS Biol. 2019;17(5):e3000246. https://doi.org/10.1371/journal.pbio.3000246.CrossRefPubMedPubMedCentral

45.

Houtkoop BL, Chambers C, Macleod M, Bishop DVM, Nichols TE, Wagenmakers E-J. Data sharing in psychology: a survey on barriers and preconditions. Adv Methods Pract Psychol Sci. 2018;1(1):70–85. https://doi.org/10.1177/2515245917751886.CrossRef

46.

Perrier L, Blondal E, MacDonald H. The views, perspectives, and experiences of academic researchers with data sharing and reuse: A meta-synthesis. PLoS One. 2020;15(2):e0229182. https://doi.org/10.1371/journal.pone.0229182.CrossRefPubMedPubMedCentral

47.

Chawinga WD, Zinn S. Global perspectives of research data sharing: A systematic literature review. Libr Inf Sci Res. 2019;41(2):109–22. https://doi.org/10.1016/j.lisr.2019.04.004.CrossRef

48.

Fraser H, Barnett A, Parker TH, Fidler F. The role of replication studies in ecology. Ecol Evol. 2020;10(12):5197–207. https://doi.org/10.1002/ece3.6330.CrossRefPubMedPubMedCentral

49.

Ahadi A, Hellas A, Ihantola P, Korhonen A, Petersen A. Replication in computing education research: researcher attitudes and experiences. In: Proceedings of the 16th Koli Calling International Conference on Computing Education Research. Koli, Finland: Association for Computing Machinery; 2016. p. 2–11.CrossRef

50.

Vachon B, Curran JA, Karunananthan S, Brehaut J, Graham ID, Moher D, et al. Changing research culture toward more use of replication research: a narrative review of barriers and strategies. J Clin Epidemiol. 2021;129:21–30. https://doi.org/10.1016/j.jclinepi.2020.09.027.CrossRefPubMed

51.

Thomas DR. A general inductive approach for analyzing qualitative evaluation data. Am J Eval. 2006;27(2):237–46. https://doi.org/10.1177/1098214005283748.CrossRef

52.

Turner SL, Karahalios A, Forbes AB, Taljaard M, Grimshaw JM, McKenzie JE. Comparison of six statistical methods for interrupted time series studies: empirical evaluation of 190 published series. 07 December 2020, PREPRINT (Version 1) available at Research Square. https://doi.org/10.21203/rs.3.rs-118335/v1.

53.

Viechtbauer W. Conducting Meta-Analyses in R with the metafor Package. J Stat Softw. 2010;36(3):48.CrossRef

54.

Bland JM, Altman DG. Statistical methods for assessing agreement between two methods of clinical measurement. Lancet. 1986;1(8476):307–10.CrossRef

55.

Nama N, Barrowman N, O'Hearn K, Sampson M, Zemek R, McNally JD. Quality control for crowdsourcing citation screening: the importance of assessment number and qualification set size. J Clin Epidemiol. 2020;122:160–2. https://doi.org/10.1016/j.jclinepi.2020.02.009.CrossRefPubMed

56.

Nama N, Sampson M, Barrowman N, Sandarage R, Menon K, Macartney G, et al. Crowdsourcing the citation screening process for systematic reviews: validation study. J Med Internet Res. 2019;21(4):e12953. https://doi.org/10.2196/12953.CrossRefPubMedPubMedCentral

57.

Mortensen ML, Adam GP, Trikalinos TA, Kraska T, Wallace BC. An exploration of crowdsourcing citation screening for systematic reviews. Res Synth Methods. 2017;8(3):366–86. https://doi.org/10.1002/jrsm.1252.CrossRefPubMedPubMedCentral

58.

Barba LA. Terminologies for reproducible research. arXiv. 2018:1802.03311.

59.

Klein RA, Ratliff KA, Vianello M, Adams RB, Bahník Š, Bernstein MJ, et al. Investigating variation in replicability. Soc Psychol. 2014;45(3):142–52. https://doi.org/10.1027/1864-9335/a000178.CrossRef

60.

Silberzahn R, Uhlmann EL, Martin DP, Anselmi P, Aust F, Awtrey E, et al. Many analysts, one data set: making transparent how variations in analytic choices affect results. Adv Methods Pract Psychol Sci. 2018;1(3):337–56. https://doi.org/10.1177/2515245917747646.CrossRef

61.

Zwaan RA, Etz A, Lucas RE, Donnellan MB. Making replication mainstream. Behav Brain Sci. 2018;41:e120. https://doi.org/10.1017/S0140525X17001972.CrossRef

62.

Page MJ, McKenzie JE, Bossuyt PM, Boutron I, Hoffmann TC, Mulrow CD, et al. The PRISMA 2020 statement: an updated guideline for reporting systematic reviews. Syst Rev. 2021;10:89. https://doi.org/10.1186/s13643-021-01626-4.

63.

Botvinik-Nezer R, Holzmeister F, Camerer CF, Dreber A, Huber J, Johannesson M, et al. Variability in the analysis of a single neuroimaging dataset by many teams. Nature. 2020;582(7810):84–8. https://doi.org/10.1038/s41586-020-2314-9.CrossRefPubMedPubMedCentral

64.

Landis JR, Koch GG. The measurement of observer agreement for categorical data. Biometrics. 1977;33(1):159–74. https://doi.org/10.2307/2529310.CrossRefPubMed

65.

Borah R, Brown AW, Capers PL, Kaiser KA. Analysis of the time and workers needed to conduct systematic reviews of medical interventions using data from the PROSPERO registry. BMJ Open. 2017;7(2):e012545. https://doi.org/10.1136/bmjopen-2016-012545.CrossRefPubMedPubMedCentral

66.

Haddaway NR, Westgate MJ. Predicting the time needed for environmental systematic reviews and systematic maps. Conserv Biol. 2019;33(2):434–43. https://doi.org/10.1111/cobi.13231.CrossRefPubMed

67.

Niforatos JD, Weaver M, Johansen ME. Assessment of publication trends of systematic reviews and randomized clinical trials, 1995 to 2017. JAMA Intern Med. 2019;179(11):1593–4. https://doi.org/10.1001/jamainternmed.2019.3013.CrossRefPubMedCentral

68.

Gurevitch J, Koricheva J, Nakagawa S, Stewart G. Meta-analysis and the science of research synthesis. Nature. 2018;555(7695):175–82. https://doi.org/10.1038/nature25753.CrossRefPubMed

69.

Nakagawa S, Koricheva J, Macleod M, Viechtbauer W. Introducing our series: research synthesis and meta-research in biology. BMC Biol. 2020;18(1):20. https://doi.org/10.1186/s12915-020-0755-0.CrossRefPubMedPubMedCentral

70.

Karunananthan S, Maxwell LJ, Welch V, Petkovic J, Pardo Pardo J, Rader T, et al. When and how to replicate systematic reviews. Cochrane Database Syst Rev. 2020;(2):MR000052.

Title: The REPRISE project: protocol for an evaluation of REProducibility and Replicability In Syntheses of Evidence
Authors: Matthew J. Page
David Moher
Fiona M. Fidler
Julian P. T. Higgins
Sue E. Brennan
Neal R. Haddaway
Daniel G. Hamilton
Raju Kanukula
Sathya Karunananthan
Lara J. Maxwell
Steve McDonald
Shinichi Nakagawa
David Nunan
Peter Tugwell
Vivian A. Welch
Joanne E. McKenzie
Publication date: 01-12-2021
Publisher: BioMed Central
Published in: Systematic Reviews / Issue 1/2021
Electronic ISSN: 2046-4053
DOI: https://doi.org/10.1186/s13643-021-01670-0

At a glance: The STEP trials

Springer Medicine

The REPRISE project: protocol for an evaluation of REProducibility and Replicability In Syntheses of Evidence

Abstract

Background

Methods

Discussion

At a glance: The STEP trials

Springer Medicine

Abstract

Background

Methods

Discussion

Please log in to get access to this content

Other articles of this Issue 1/2021

Protocol for a systematic review of outcomes from microsurgical free-tissue transfer performed on short-term collaborative surgical trips in low-income and middle-income countries

Retention in opioid agonist treatment: a rapid review and meta-analysis comparing observational studies and randomized controlled trials

Social marketing-based interventions to promote healthy nutrition behaviors: a systematic review protocol

Evidence for implementation of interventions to promote mental health in the workplace: a systematic scoping review protocol

Mapping evidence on management of cervical cancer in sub-Saharan Africa: scoping review protocol

The effect of preoperative stoma site marking on risk of stoma-related complications in patients with intestinal ostomy—protocol of a systematic review and meta-analysis