Top

Insights into Imaging

Published in:

01-12-2020 | Original Article

An analysis of key indicators of reproducibility in radiology

Authors: Bryan D. Wright, Nam Vo, Johnny Nolan, Austin L. Johnson, Tyler Braaten, Daniel Tritz, Matt Vassar

Published in: Insights into Imaging | Issue 1/2020

Abstract

Background

Given the central role of radiology in patient care, it is important that radiological research is grounded in reproducible science. It is unclear whether there is a lack of reproducibility or transparency in radiologic research.

Purpose

To analyze published radiology literature for the presence or lack of key indicators of reproducibility.

Methods

This cross-sectional retrospective study was performed by conducting a search of the National Library of Medicine (NLM) for publications contained within journals in the field of radiology. Our inclusion criteria were being MEDLINE indexed, written in English, and published from January 1, 2014, to December 31, 2018. We randomly sampled 300 publications for this study. A pilot-tested Google form was used to record information from the publications regarding indicators of reproducibility. Following peer-review, we extracted data from an additional 200 publications in an attempt to reproduce our initial results. The additional 200 publications were selected from the list of initially randomized publications.

Results

Our initial search returned 295,543 records, from which 300 were randomly selected for analysis. Of these 300 records, 294 met inclusion criteria and 6 did not. Among the empirical publications, 5.6% (11/195, [3.0–8.3]) contained a data availability statement, 0.51% (1/195) provided clear documented raw data, 12.0% (23/191, [8.4–15.7]) provided a materials availability statement, 0% provided analysis scripts, 4.1% (8/195, [1.9–6.3]) provided a pre-registration statement, 2.1% (4/195, [0.4–3.7]) provided a protocol statement, and 3.6% (7/195, [1.5–5.7]) were pre-registered. The validation study of the 5 key indicators of reproducibility—availability of data, materials, protocols, analysis scripts, and pre-registration—resulted in 2 indicators (availability of protocols and analysis scripts) being reproduced, as they fell within the 95% confidence intervals for the proportions from the original sample. However, materials’ availability and pre-registration proportions from the validation sample were lower than what was found in the original sample.

Conclusion

Our findings demonstrate key indicators of reproducibility are missing in the field of radiology. Thus, the ability to reproduce studies contained in radiology publications may be problematic and may have potential clinical implications.

Jackson WL (2014) Imaging utilization trends and reimbursement. Diagn Imaging.

Rosenkrantz AB, Pinnamaneni N, Babb JS, Doshi AM (2016) Most common publication types in radiology journals: what is the level of evidence? Acad Radiol 23(5):628–633CrossRef

Pitcher RD (2019) The role of radiology in global health. In: Mollura DJ, Culp MP, Lungren MP (eds) Radiology in Global Health: Strategies, Implementation, and Applications. Springer International Publishing, Cham, pp 157–174CrossRef

WHO | Medical imaging. February 2017. https://www.who.int/diagnostic_imaging/en/. Accessed June 27, 2019.

Baker M (2016) 1,500 scientists lift the lid on reproducibility. Nature 533(7604):452–454CrossRef

Aerts HJWL (2018) Data science in radiology: a path forward. Clin Cancer Res 24(3):532–534CrossRef

Hardwicke TE, Wallach JD, Kidwell MC, Bendixen T, Crüwell S, Ioannidis JPA (2019) An empirical assessment of transparency and reproducibility-related research practices in the social sciences (2014-2017). R Soc Open Sci 7(2):190806

Electronic Code of Federal Regulations-US Department of Health and Human Services’ Code of Federal Regulation 45 CFR 46.102(d). https://www.ecfr.gov/cgi-bin/retrieveECFR?gp=&SID=83cd09e1c0f5c6937cd97513160fc3f&pitd=20180719&n=pt45.1.46&r=PART&ty=HTML#se45.1.46_1102 in effect July 19, 2018.

Murad MH, Wang Z (2017) Guidelines for reporting meta-epidemiological methodology research. Evid Based Med 22(4):139–142CrossRef

10.

Liberati A, Altman DG, Tetzlaff J et al (2009) The PRISMA statement for reporting systematic reviews and meta-analyses of studies that evaluate health care interventions: explanation and elaboration. J Clin Epidemiol 62(10):e1–e34CrossRef

11.

Wallach JD, Boyack KW, Ioannidis JPA (2018) Reproducible research practices, transparency, and open access data in the biomedical literature, 2015–2017. PLoS Biol 16(11):e2006930CrossRef

12.

Iqbal SA, Wallach JD, Khoury MJ, Schully SD, Ioannidis JPA (2016) Reproducible research practices and transparency across the biomedical literature. PLoS Biol 14(1):e1002333CrossRef

13.

Read Metadata from DICOM Files - MATLAB & Simulink. https://www.mathworks.com/help/images/read-metadata-from-dicom-files.html. Accessed August 4, 2019.

14.

Doran SJ, d’Arcy J, Collins DJ et al (2012) Informatics in radiology: development of a research PACS for analysis of functional imaging data in clinical research and clinical trials. Radiographics 32(7):2135–2150CrossRef

15.

Sardanelli F, Alì M, Hunink MG, Houssami N, Sconfienza LM, Di Leo G (2018) To share or not to share? Expected pros and cons of data sharing in radiological research. Eur Radiol 28(6):2328–2335CrossRef

16.

Warren E (2016) Strengthening research through data sharing. N Engl J Med. 375(5):401–403CrossRef

17.

Naudet F, Sakarovitch C, Janiaud P et al (2018) Data sharing and reanalysis of randomized controlled trials in leading biomedical journals with a full data sharing policy: survey of studies published inThe BMJandPLOS Medicine. BMJ:k400. https://doi.org/10.1136/bmj.k400

18.

Federer LM, Belter CW, Joubert DJ et al (2018) Data sharing in PLOS ONE: an analysis of data availability statements. PLoS One. 13(5):e0194768CrossRef

19.

0000-0003-1953-, 0000-0002-7378-. Making progress toward open data: reflections on data sharing at PLOS ONE | EveryONE: The PLOS ONE blog. EveryONE. https://blogs.plos.org/everyone/2017/05/08/making-progress-toward-open-data/. Published May 8, 2017. Accessed June 20, 2019.

20.

ICMJE | Journals stating that they follow the ICMJE Recommendations. http://www.icmje.org/journals-following-the-icmje-recommendations/. Accessed August 12, 2019.

21.

Bosserdt M, Hamm B, Dewey M (February 2019) Clinical trials in radiology and data sharing: results from a survey of the European Society of Radiology (ESR) research committee. Eur Radiol. https://doi.org/10.1007/s00330-019-06105-y

22.

Piccolo SR, Frampton MB (2016) Tools and techniques for computational reproducibility. Gigascience. 5(1):30CrossRef

23.

Garijo D, Kinnings S, Xie L et al (2013) Quantifying reproducibility in computational biology: the case of the tuberculosis drugome. PLoS One. 8(11):e80278CrossRef

24.

Gronenschild EHBM, Habets P, Jacobs HIL et al (2012) The effects of FreeSurfer version, workstation type, and Macintosh operating system version on anatomical volume and cortical thickness measurements. PLoS One. 7(6):e38234CrossRef

25.

Parmar C, Barry JD, Hosny A, Quackenbush J, Aerts HJWL (2018) Data analysis strategies in medical imaging. Clin Cancer Res. 24(15):3492–3499CrossRef

26.

Traverso A, Wee L, Dekker A, Gillies R (2018) Repeatability and reproducibility of radiomic features: a systematic review. Int J Radiat Oncol Biol Phys. 102(4):1143–1158CrossRef

27.

Carp J (2012) On the plurality of (methodological) worlds: estimating the analytic flexibility of FMRI experiments. Front Neurosci. 6:149CrossRef

28.

Poldrack RA, Gorgolewski KJ, Varoquaux G (2019) Computational and informatic advances for reproducible data analysis in neuroimaging. Annu Rev Biomed Data Sci. 2(1):119–138CrossRef

29.

Gorgolewski KJ, Poldrack RA (2016) A practical guide for improving transparency and reproducibility in neuroimaging research. PLoS Biol. 14(7):e1002506CrossRef

30.

Triphan S, Biederer J, Burmester K, et al. Raw data and analysis scripts for “Design and application of an MR reference phantom for multicentre lung imaging trials.” 2018. https://doi.org/10.11588/DATA/FHOCRZ

31.

Triphan SMF, Biederer J, Burmester K et al (2018) Design and application of an MR reference phantom for multicentre lung imaging trials. PLoS One. 13(7):e0199148CrossRef

32.

Cohen JF, Korevaar DA, Altman DG et al (2016) STARD 2015 guidelines for reporting diagnostic accuracy studies: explanation and elaboration. BMJ Open. 6(11):e012799CrossRef

33.

The CARE guidelines: consensus-based clinical case reporting guideline development | The EQUATOR Network. http://www.equator-network.org/reporting-guidelines/care/. Accessed August 13, 2019.

34.

Reporting guidelines | The EQUATOR Network. http://www.equator-network.org/reporting-guidelines/. Accessed August 13, 2019.

35.

Guidelines for reporting reliability and agreement studies (GRRAS) were proposed | The EQUATOR Network. https://www.equator-network.org/reporting-guidelines/guidelines-for-reporting-reliability-and-agreement-studies-grras-were-proposed/. Accessed August 13, 2019.

36.

Kottner J, Audigé L, Brorson S et al (2011) Guidelines for Reporting Reliability and Agreement Studies (GRRAS) were proposed. J Clin Epidemiol. 64(1):96–106CrossRef

37.

Gerke O, Möller S, Debrabant B, Halekoh U (2018) Odense Agreement Working Group. Experience applying the Guidelines for Reporting Reliability and Agreement Studies (GRRAS) indicated five questions should be addressed in the planning phase from a statistical point of view. Diagnostics (Basel) 8(4):69. https://doi.org/10.3390/diagnostics8040069

38.

Cronin P, Rawson JV (2016) Review of research reporting guidelines for radiology researchers. Acad Radiol. 23(5):537–558CrossRef

39.

What is a reporting guideline? | The EQUATOR Network. http://www.equator-network.org/about-us/what-is-a-reporting-guideline/. Accessed August 13, 2019.

40.

Oster NV, Carney PA, Allison KH et al (2013) Development of a diagnostic test set to assess agreement in breast pathology: practical application of the Guidelines for Reporting Reliability and Agreement Studies (GRRAS). BMC Women’s Health 13(1). https://doi.org/10.1186/1472-6874-13-3

41.

Open Science Collaboration (2015) Estimating the reproducibility of psychological science. Science 349(6251):aac4716CrossRef

42.

APS: Leading the Way in Replication and Open Science. Association for Psychological Science. https://www.psychologicalscience.org/publications/observer/obsonline/aps-reproducibility-and-replication-initiatives.html. Accessed June 30, 2019.

43.

Günel Karadeniz P, Uzabacı E, Atış Kuyuk S et al (2019) Statistical errors in articles published in radiology journals. Diagn Interv Radiol 25(2):102–108CrossRef

44.

Stat Reviewer. http://www.statreviewer.com/. Accessed August 1, 2019.

45.

Klein RA, Ratliff K, Vianello M et al (2014) Investigating variation in replicability: a “many labs” replication project. Open Science Framework.

46.

Klein RA, Vianello M, Hasselman F et al (2018) Many Labs 2: investigating variation in replicability across samples and settings. Advances in Methods and Practices in Psychological Science 1(4):443–490CrossRef

47.

Munafò MR, Nosek BA, DVM B et al (2017) A manifesto for reproducible science. Nature Human Behaviour 1(1). https://doi.org/10.1038/s41562-016-0021

48.

Higgins JPT, Green S (2011) Cochrane Handbook for Systematic Reviews of Interventions. John Wiley & Sons

Title: An analysis of key indicators of reproducibility in radiology
Authors: Bryan D. Wright
Nam Vo
Johnny Nolan
Austin L. Johnson
Tyler Braaten
Daniel Tritz
Matt Vassar
Publication date: 01-12-2020
Publisher: Springer Berlin Heidelberg
Published in: Insights into Imaging / Issue 1/2020
Electronic ISSN: 1869-4101
DOI: https://doi.org/10.1186/s13244-020-00870-x

Keynote webinar | Spotlight on medication adherence

Springer Medicine

An analysis of key indicators of reproducibility in radiology

Abstract

Background

Purpose

Methods

Results

Conclusion

Keynote webinar | Spotlight on medication adherence

Springer Medicine

Abstract

Background

Purpose

Methods

Results

Conclusion

Please log in to get access to this content

Other articles of this Issue 1/2020

Persistent left superior vena cava: clinical importance and differential diagnoses

Redefining the structure of structured reporting in radiology

Diagnostic Reference Levels based on clinical indications in computed tomography: a literature review

Imaging of mandibular fractures: a pictorial review

Electronic cleansing of tagged residue in CT colonography: what radiologists need to know

Imaging diagnosis of metastatic breast cancer