Top

22-09-2023 | Artificial Intelligence | Imaging Informatics and Artificial Intelligence

Self-reporting with checklists in artificial intelligence research on medical imaging: a systematic review based on citations of CLAIM

Authors: Burak Kocak, Ali Keles, Tugba Akinci D’Antonoli

Abstract

Objective

To evaluate the usage of a well-known and widely adopted checklist, Checklist for Artificial Intelligence in Medical imaging (CLAIM), for self-reporting through a systematic analysis of its citations.

Methods

Google Scholar, Web of Science, and Scopus were used to search for citations (date, 29 April 2023). CLAIM’s use for self-reporting with proof (i.e., filled-out checklist) and other potential use cases were systematically assessed in research papers. Eligible papers were evaluated independently by two readers, with the help of automatic annotation. Item-by-item confirmation analysis on papers with checklist proof was subsequently performed.

Results

A total of 391 unique citations were identified from three databases. Of the 118 papers included in this study, 12 (10%) provided a proof of self-reported CLAIM checklist. More than half (70; 59%) only mentioned some sort of adherence to CLAIM without providing any proof in the form of a checklist. Approximately one-third (36; 31%) cited the CLAIM for reasons unrelated to their reporting or methodological adherence. Overall, the claims on 57 to 93% of the items per publication were confirmed in the item-by-item analysis, with a mean and standard deviation of 81% and 10%, respectively.

Conclusion

Only a small proportion of the publications used CLAIM as checklist and supplied filled-out documentation; however, the self-reported checklists may contain errors and should be approached cautiously. We hope that this systematic citation analysis would motivate artificial intelligence community about the importance of proper self-reporting, and encourage researchers, journals, editors, and reviewers to take action to ensure the proper usage of checklists.

Clinical relevance statement

Only a small percentage of the publications used CLAIM for self-reporting with proof (i.e., filled-out checklist). However, the filled-out checklist proofs may contain errors, e.g., false claims of adherence, and should be approached cautiously. These may indicate inappropriate usage of checklists and necessitate further action by authorities.

Key Points

• Of 118 eligible papers, only 12 (10%) followed the CLAIM checklist for self-reporting with proof (i.e., filled-out checklist). More than half (70; 59%) only mentioned some kind of adherence without providing any proof.

• Overall, claims on 57 to 93% of the items were valid in item-by-item confirmation analysis, with a mean and standard deviation of 81% and 10%, respectively.

• Even with the checklist proof, the items declared may contain errors and should be approached cautiously.

Available only for authorised users

Glasziou P, Altman DG, Bossuyt P et al (2014) Reducing waste from incomplete or unusable reports of biomedical research. Lancet 383:267–276. https://doi.org/10.1016/S0140-6736(13)62228-X

Pouwels KB, Widyakusuma NN, Groenwold RHH, Hak E (2016) Quality of reporting of confounding remained suboptimal after the STROBE guideline. J Clin Epidemiol 69:217–224. https://doi.org/10.1016/j.jclinepi.2015.08.009 CrossRefPubMed

Begley CG, Ellis LM (2012) Raise standards for preclinical cancer research. Nature 483:531–533. https://doi.org/10.1038/483531a CrossRefPubMed

Prinz F, Schlange T, Asadullah K (2011) Believe it or not: how much can we rely on published data on potential drug targets? Nat Rev Drug Discov 10:712–712. https://doi.org/10.1038/nrd3439-c1 CrossRefPubMed

Landis SC, Amara SG, Asadullah K et al (2012) A call for transparent reporting to optimize the predictive value of preclinical research. Nature 490:187–191. https://doi.org/10.1038/nature11556 CrossRefPubMedPubMedCentral

Blanco D, Biggane AM, Cobo E, MiRoR network, (2018) Are CONSORT checklists submitted by authors adequately reflecting what information is actually reported in published papers? Trials 19:80. https://doi.org/10.1186/s13063-018-2475-0 CrossRefPubMedPubMedCentral

Library | EQUATOR Network. https://www.equator-network.org/library/. Accessed 16 May 2023

Hopewell S, Dutton S, Yu L-M, et al (2010) The quality of reports of randomised trials in 2000 and 2006: comparative study of articles indexed in PubMed. BMJ 340:c723. https://doi.org/10.1136/bmj.c723

Moher D, Jones A, Lepage L, CONSORT Group (Consolidated Standards for Reporting of Trials) (2001) Use of the CONSORT statement and quality of reports of randomized trials: a comparative before-and-after evaluation. JAMA 285:1992–1995. https://doi.org/10.1001/jama.285.15.1992

10.

Turner L, Shamseer L, Altman DG et al (2012) Does use of the CONSORT Statement impact the completeness of reporting of randomised controlled trials published in medical journals? A Cochrane Review. Syst Rev 1:60. https://doi.org/10.1186/2046-4053-1-60

11.

Kocak B, Baessler B, Cuocolo R et al (2023) Trends and statistics of artificial intelligence and radiomics research in Radiology, Nuclear Medicine, and Medical Imaging: bibliometric analysis. Eur Radiol. https://doi.org/10.1007/s00330-023-09772-0 CrossRefPubMed

12.

Klontzas ME, Gatti AA, Tejani AS, Kahn CE (2023) AI reporting guidelines: how to select the best one for your research. Radiol Artif Intell 5:e230055. https://doi.org/10.1148/ryai.230055 CrossRefPubMedPubMedCentral

13.

Kocak B, Baessler B, Bakas S et al (2023) CheckList for EvaluAtion of Radiomics research (CLEAR): a step-by-step reporting guideline for authors and reviewers endorsed by ESR and EuSoMII. Insights Imaging 14:75. https://doi.org/10.1186/s13244-023-01415-8 CrossRefPubMedPubMedCentral

14.

Zrubka Z, Gulácsi L, Péntek M (2022) Time to start using checklists for reporting artificial intelligence in health care and biomedical research: a rapid review of available tools. In: 2022 IEEE 26th International Conference on Intelligent Engineering Systems (INES). pp 000015–000020

15.

Zwanenburg A (2023) Standardisation and harmonisation efforts in quantitative imaging. Eur Radiol. https://doi.org/10.1007/s00330-023-09921-5 CrossRefPubMed

16.

Mongan J, Moy L, Kahn CE (2020) Checklist for Artificial Intelligence in Medical Imaging (CLAIM): a guide for authors and reviewers. Radiol Artif Intell 2:e200029. https://doi.org/10.1148/ryai.2020200029 CrossRefPubMedPubMedCentral

17.

Altman DG (1996) Better reporting of randomised controlled trials: the CONSORT statement. BMJ 313:570–571. https://doi.org/10.1136/bmj.313.7057.570 CrossRefPubMedPubMedCentral

18.

Cartledge PT, Hopkinson D, Nsanzabaganwa C, Bassat Q (2019) Using a Reporting Guideline (Checklist). J Trop Pediatr 65:521–525. https://doi.org/10.1093/tropej/fmz073 CrossRefPubMed

19.

McEvoy NL, Tume LN, Trapani J (2022) What are publication reporting checklists and why are they so important? Nurs Crit Care 27:291–293. https://doi.org/10.1111/nicc.12771 CrossRefPubMed

20.

Kocak B, Chepelev LL, Chu LC et al (2023) Assessment of RadiomIcS rEsearch (ARISE): a brief guide for authors, reviewers, and readers from the Scientific Editorial Board of European Radiology. Eur Radiol. https://doi.org/10.1007/s00330-023-09768-w CrossRefPubMed

21.

Page MJ, McKenzie JE, Bossuyt PM et al (2021) The PRISMA 2020 statement: an updated guideline for reporting systematic reviews. BMJ 372:n71. https://doi.org/10.1136/bmj.n71 CrossRefPubMedPubMedCentral

22.

Zhong J, Zhang C, Hu Y et al (2022) Automated prediction of the neoadjuvant chemotherapy response in osteosarcoma with deep learning and an MRI-based radiomics nomogram. Eur Radiol 32:6196–6206. https://doi.org/10.1007/s00330-022-08735-1 CrossRefPubMed

23.

Zhang J, Zhan C, Zhang C et al (2023) Fully automatic classification of breast lesions on multi-parameter MRI using a radiomics model with minimal number of stable, interpretable features. Radiol Med 128:160–170. https://doi.org/10.1007/s11547-023-01594-w

24.

Zech JR, Carotenuto G, Igbinoba Z et al (2023) Detecting pediatric wrist fractures using deep-learning-based object detection. Pediatr Radiol. https://doi.org/10.1007/s00247-023-05588-8 CrossRefPubMed

25.

Wu Y, Gao R, Lin P et al (2022) An endorectal ultrasound-based radiomics signature for preoperative prediction of lymphovascular invasion of rectal cancer. BMC Med Imaging 22:84. https://doi.org/10.1186/s12880-022-00813-6 CrossRefPubMedPubMedCentral

26.

Wang K, George-Jones NA, Chen L, Hunter JB, Wang J (2022) Joint vestibular schwannoma enlargement prediction and segmentation using a deep multi‐task model. Laryngoscope 133(10):2754–2760. https://doi.org/10.1002/lary.30516

27.

Wang C, Shao J, Xu X et al (2022) DeepLN: a multi-task ai tool to predict the imaging characteristics. Front Oncol 12:683792. https://doi.org/10.3389/fonc.2022.683792 CrossRefPubMedPubMedCentral

28.

Shiri I, Salimi Y, Pakbin M et al (2022) COVID-19 prognostic modeling using CT radiomic features and machine learning algorithms: analysis of a multi-institutional dataset of 14,339 patients. Comput Biol Med 145:105467. https://doi.org/10.1016/j.compbiomed.2022.105467 CrossRefPubMedPubMedCentral

29.

Mao B, Zhang L, Ning P et al (2020) Preoperative prediction for pathological grade of hepatocellular carcinoma via machine learning–based radiomics. Eur Radiol 30:6924–6932. https://doi.org/10.1007/s00330-020-07056-5 CrossRefPubMed

30.

La Greca Saint-Esteven A, Bogowicz M, Konukoglu E et al (2022) A 2.5D convolutional neural network for HPV prediction in advanced oropharyngeal cancer. Comput Biol Med 142:105215. https://doi.org/10.1016/j.compbiomed.2022.105215

31.

Ibrahim A, Vaidyanathan A, Primakov S et al (2023) Deep learning based identification of bone scintigraphies containing metastatic bone disease foci. Cancer Imaging 23:12. https://doi.org/10.1186/s40644-023-00524-3 CrossRefPubMedPubMedCentral

32.

Bretzner M, Bonkhoff AK, Schirmer MD et al (2023) Radiomics-derived brain age predicts functional outcome after acute ischemic stroke. Neurology 100:e822–e833. https://doi.org/10.1212/WNL.0000000000201596 CrossRefPubMedPubMedCentral

33.

Albiol A, Albiol F, Paredes R et al (2022) A comparison of Covid-19 early detection between convolutional neural networks and radiologists. Insights Imaging 13:122. https://doi.org/10.1186/s13244-022-01250-3 CrossRefPubMedPubMedCentral

34.

Yusuf M, Atal I, Li J et al (2020) Reporting quality of studies using machine learning models for medical diagnosis: a systematic review. BMJ Open 10:e034568. https://doi.org/10.1136/bmjopen-2019-034568 CrossRefPubMedPubMedCentral

35.

Zhong J, Hu Y, Zhang G et al (2022) An updated systematic review of radiomics in osteosarcoma: utilizing CLAIM to adapt the increasing trend of deep learning application in radiomics. Insights Imaging 13:138. https://doi.org/10.1186/s13244-022-01277-6 CrossRefPubMedPubMedCentral

36.

Si L, Zhong J, Huo J et al (2022) Deep learning in knee imaging: a systematic review utilizing a Checklist for Artificial Intelligence in Medical Imaging (CLAIM). Eur Radiol 32:1353–1361. https://doi.org/10.1007/s00330-021-08190-4 CrossRefPubMed

37.

Bhandari A, Scott L, Weilbach M et al (2023) Assessment of artificial intelligence (AI) reporting methodology in glioma MRI studies using the Checklist for AI in Medical Imaging (CLAIM). Neuroradiology 65:907–913. https://doi.org/10.1007/s00234-023-03126-9 CrossRefPubMedPubMedCentral

38.

Belue MJ, Harmon SA, Lay NS et al (2023) The low rate of adherence to checklist for artificial intelligence in medical imaging criteria among published prostate MRI artificial intelligence algorithms. J Am Coll Radiol 20:134–145. https://doi.org/10.1016/j.jacr.2022.05.022

Title: Self-reporting with checklists in artificial intelligence research on medical imaging: a systematic review based on citations of CLAIM
Authors: Burak Kocak
Ali Keles
Tugba Akinci D’Antonoli
Publication date: 22-09-2023
Publisher: Springer Berlin Heidelberg
Keyword: Artificial Intelligence
Published in: European Radiology
Print ISSN: 0938-7994
Electronic ISSN: 1432-1084
DOI: https://doi.org/10.1007/s00330-023-10243-9

Keynote Webinar | Spotlight on Diet and Diabetes

Springer Medicine

Self-reporting with checklists in artificial intelligence research on medical imaging: a systematic review based on citations of CLAIM

Abstract

Objective

Methods

Results

Conclusion

Clinical relevance statement

Key Points

Keynote Webinar | Spotlight on Diet and Diabetes

Springer Medicine

Abstract

Objective

Methods

Results

Conclusion

Clinical relevance statement

Key Points

Please log in to get access to this content