Skip to main content
Top
Published in:

Open Access 17-05-2024 | Artificial Intelligence | Chest

Performance of AI to exclude normal chest radiographs to reduce radiologists’ workload

Authors: Steven Schalekamp, Kicky van Leeuwen, Erdi Calli, Keelin Murphy, Matthieu Rutten, Bram Geurts, Liesbeth Peters-Bax, Bram van Ginneken, Mathias Prokop

Published in: European Radiology | Issue 11/2024

Login to get access

Abstract

Introduction

This study investigates the performance of a commercially available artificial intelligence (AI) system to identify normal chest radiographs and its potential to reduce radiologist workload.

Methods

Retrospective analysis included consecutive chest radiographs from two medical centers between Oct 1, 2016 and Oct 14, 2016. Exclusions comprised follow-up exams within the inclusion period, bedside radiographs, incomplete images, imported radiographs, and pediatric radiographs. Three chest radiologists categorized findings into normal, clinically irrelevant, clinically relevant, urgent, and critical. A commercial AI system processed all radiographs, scoring 10 chest abnormalities on a 0–100 confidence scale. AI system performance was evaluated using the area under the ROC curve (AUC), assessing the detection of normal radiographs. Sensitivity was calculated for the default and a conservative operating point. the detection of negative predictive value (NPV) for urgent and critical findings, as well as the potential workload reduction, was calculated.

Results

A total of 2603 radiographs were acquired in 2141 unique patients. Post-exclusion, 1670 radiographs were analyzed. Categories included 479 normal, 332 clinically irrelevant, 339 clinically relevant, 501 urgent, and 19 critical findings. The AI system achieved an AUC of 0.92. Sensitivity for normal radiographs was 92% at default and 53% at the conservative operating point. At the conservative operating point, NPV was 98% for urgent and critical findings, and could result in a 15% workload reduction.

Conclusion

A commercially available AI system effectively identifies normal chest radiographs and holds the potential to lessen radiologists’ workload by omitting half of the normal exams from reporting.

Clinical relevance statement

The AI system is able to detect half of all normal chest radiographs at a clinically acceptable operating point, thereby potentially reducing the workload for the radiologists by 15%.

Key Points

  • The AI system reached an AUC of 0.92 for the detection of normal chest radiographs.
  • Fifty-three percent of normal chest radiographs were identified with a NPV of 98% for urgent findings. AI can reduce the workload of chest radiography reporting by 15%.
Literature
1.
go back to reference Raoof S, Feigin D, Sung A, Raoof S, Irugulpati L, Rosenow 3rd EC (2012) Interpretation of plain chest roentgenogram. Chest 141:545–558CrossRefPubMed Raoof S, Feigin D, Sung A, Raoof S, Irugulpati L, Rosenow 3rd EC (2012) Interpretation of plain chest roentgenogram. Chest 141:545–558CrossRefPubMed
3.
go back to reference Calli E, Sogancioglu E, van Ginneken B, van Leeuwen KG, Murphy K (2021) Deep learning for chest X-ray analysis: a survey. Med Image Anal 72:102125CrossRefPubMed Calli E, Sogancioglu E, van Ginneken B, van Leeuwen KG, Murphy K (2021) Deep learning for chest X-ray analysis: a survey. Med Image Anal 72:102125CrossRefPubMed
4.
go back to reference Chassagnon G, Vakalopoulou M, Paragios N, Revel MP (2020) Artificial intelligence applications for thoracic imaging. Eur J Radiol 123:108774CrossRefPubMed Chassagnon G, Vakalopoulou M, Paragios N, Revel MP (2020) Artificial intelligence applications for thoracic imaging. Eur J Radiol 123:108774CrossRefPubMed
5.
go back to reference Fanni SC, Marcucci A, Volpi F, Valentino S, Neri E, Romei C (2023) Artificial intelligence-based software with CE mark for chest X-ray interpretation: opportunities and challenges. Diagnostics (Basel) 13:2020 Fanni SC, Marcucci A, Volpi F, Valentino S, Neri E, Romei C (2023) Artificial intelligence-based software with CE mark for chest X-ray interpretation: opportunities and challenges. Diagnostics (Basel) 13:2020
6.
go back to reference Ahn JS, Ebrahimian S, McDermott S et al (2022) Association of artificial intelligence-aided chest radiograph interpretation with reader performance and efficiency. JAMA Netw Open 5:e2229289CrossRefPubMedPubMedCentral Ahn JS, Ebrahimian S, McDermott S et al (2022) Association of artificial intelligence-aided chest radiograph interpretation with reader performance and efficiency. JAMA Netw Open 5:e2229289CrossRefPubMedPubMedCentral
7.
go back to reference Homayounieh F, Digumarthy S, Ebrahimian S et al (2021) An artificial intelligence-based chest X-ray model on human nodule detection accuracy from a multicenter study. JAMA Netw Open 4:e2141096CrossRefPubMedPubMedCentral Homayounieh F, Digumarthy S, Ebrahimian S et al (2021) An artificial intelligence-based chest X-ray model on human nodule detection accuracy from a multicenter study. JAMA Netw Open 4:e2141096CrossRefPubMedPubMedCentral
8.
go back to reference Hwang EJ, Nam JG, Lim WH et al (2019) Deep learning for chest radiograph diagnosis in the emergency department. Radiology 293:573–580CrossRefPubMed Hwang EJ, Nam JG, Lim WH et al (2019) Deep learning for chest radiograph diagnosis in the emergency department. Radiology 293:573–580CrossRefPubMed
9.
go back to reference Hwang EJ, Park S, Jin K-N et al (2019) Development and validation of a deep learning-based automated detection algorithm for major thoracic diseases on chest radiographs. JAMA Netw Open 2:e191095CrossRefPubMedPubMedCentral Hwang EJ, Park S, Jin K-N et al (2019) Development and validation of a deep learning-based automated detection algorithm for major thoracic diseases on chest radiographs. JAMA Netw Open 2:e191095CrossRefPubMedPubMedCentral
10.
go back to reference Schalekamp S, van Ginneken B, Koedam E et al (2014) Computer-aided detection improves detection of pulmonary nodules in chest radiographs beyond the support by bone-suppressed images. Radiology 272:252–261CrossRefPubMed Schalekamp S, van Ginneken B, Koedam E et al (2014) Computer-aided detection improves detection of pulmonary nodules in chest radiographs beyond the support by bone-suppressed images. Radiology 272:252–261CrossRefPubMed
11.
go back to reference Shin HJ, Lee S, Kim S, Son NH, Kim EK (2023) Hospital-wide survey of clinical experience with artificial intelligence applied to daily chest radiographs. PLoS One 18:e0282123CrossRefPubMedPubMedCentral Shin HJ, Lee S, Kim S, Son NH, Kim EK (2023) Hospital-wide survey of clinical experience with artificial intelligence applied to daily chest radiographs. PLoS One 18:e0282123CrossRefPubMedPubMedCentral
12.
go back to reference Annarumma M, Withey SJ, Bakewell RJ, Pesce E, Goh V, Montana G (2019) Automated triaging of adult chest radiographs with deep artificial neural networks. Radiology 291:272CrossRefPubMed Annarumma M, Withey SJ, Bakewell RJ, Pesce E, Goh V, Montana G (2019) Automated triaging of adult chest radiographs with deep artificial neural networks. Radiology 291:272CrossRefPubMed
13.
go back to reference Dunnmon JA, Yi D, Langlotz CP, Re C, Rubin DL, Lungren MP (2019) Assessment of convolutional neural networks for automated classification of chest radiographs. Radiology 290:537–544CrossRefPubMed Dunnmon JA, Yi D, Langlotz CP, Re C, Rubin DL, Lungren MP (2019) Assessment of convolutional neural networks for automated classification of chest radiographs. Radiology 290:537–544CrossRefPubMed
14.
go back to reference Nabulsi Z, Sellergren A, Jamshy S et al (2021) Deep learning for distinguishing normal versus abnormal chest radiographs and generalization to two unseen diseases tuberculosis and COVID-19. Sci Rep 11:15523CrossRefPubMedPubMedCentral Nabulsi Z, Sellergren A, Jamshy S et al (2021) Deep learning for distinguishing normal versus abnormal chest radiographs and generalization to two unseen diseases tuberculosis and COVID-19. Sci Rep 11:15523CrossRefPubMedPubMedCentral
15.
go back to reference Khurana A, Patel B, Sharpe Jr R (2022) Geographic variations in growth of radiologists and medicare enrollees from 2012 to 2019. J Am Coll Radiol 19:1006–1014CrossRefPubMed Khurana A, Patel B, Sharpe Jr R (2022) Geographic variations in growth of radiologists and medicare enrollees from 2012 to 2019. J Am Coll Radiol 19:1006–1014CrossRefPubMed
17.
go back to reference Zhang X, Lin D, Pforsich H, Lin VW (2020) Physician workforce in the United States of America: forecasting nationwide shortages. Hum Resour Health 18:8CrossRefPubMedPubMedCentral Zhang X, Lin D, Pforsich H, Lin VW (2020) Physician workforce in the United States of America: forecasting nationwide shortages. Hum Resour Health 18:8CrossRefPubMedPubMedCentral
18.
go back to reference Bruls RJM, Kwee RM (2020) Workload for radiologists during on-call hours: dramatic increase in the past 15 years. Insights Imaging 11:121 Bruls RJM, Kwee RM (2020) Workload for radiologists during on-call hours: dramatic increase in the past 15 years. Insights Imaging 11:121
19.
go back to reference Lantsman CD, Barash Y, Klang E, Guranda L, Konen E, Tau N (2022) Trend in radiologist workload compared to number of admissions in the emergency department. Eur J Radiol 149:110195 Lantsman CD, Barash Y, Klang E, Guranda L, Konen E, Tau N (2022) Trend in radiologist workload compared to number of admissions in the emergency department. Eur J Radiol 149:110195
20.
go back to reference Kwee TC, Kwee RM (2021) Workload of diagnostic radiologists in the foreseeable future based on recent scientific advances: growth expectations and role of artificial intelligence. Insights Imaging 12:88 Kwee TC, Kwee RM (2021) Workload of diagnostic radiologists in the foreseeable future based on recent scientific advances: growth expectations and role of artificial intelligence. Insights Imaging 12:88
21.
go back to reference Dyer T, Dillard L, Harrison M et al (2021) Diagnosis of normal chest radiographs using an autonomous deep-learning algorithm. Clin Radiol 76:473.e479–473.e415CrossRef Dyer T, Dillard L, Harrison M et al (2021) Diagnosis of normal chest radiographs using an autonomous deep-learning algorithm. Clin Radiol 76:473.e479–473.e415CrossRef
23.
go back to reference Plesner LL, Muller FC, Nybing JD et al (2023) Autonomous chest radiograph reporting using AI: estimation of clinical impact. Radiology 307:e222268CrossRefPubMed Plesner LL, Muller FC, Nybing JD et al (2023) Autonomous chest radiograph reporting using AI: estimation of clinical impact. Radiology 307:e222268CrossRefPubMed
24.
go back to reference Bossuyt, Reitsma PM, Bruns DE JB et al (2015) STARD 2015: an updated list of essential items for reporting diagnostic accuracy studies. Radiology 277:826–832CrossRefPubMed Bossuyt, Reitsma PM, Bruns DE JB et al (2015) STARD 2015: an updated list of essential items for reporting diagnostic accuracy studies. Radiology 277:826–832CrossRefPubMed
25.
go back to reference Kaviani P, Kalra MK, Digumarthy SR et al (2022) Frequency of missed findings on chest radiographs (CXRs) in an International, Multicenter Study: application of AI to reduce missed findings. Diagnostics (Basel) 12:2382 Kaviani P, Kalra MK, Digumarthy SR et al (2022) Frequency of missed findings on chest radiographs (CXRs) in an International, Multicenter Study: application of AI to reduce missed findings. Diagnostics (Basel) 12:2382
26.
go back to reference Bradley SH, Abraham S, Callister ME et al (2019) Sensitivity of chest X-ray for detecting lung cancer in people presenting with symptoms: a systematic review. Br J Gen Pract 69:e827–e835CrossRefPubMedPubMedCentral Bradley SH, Abraham S, Callister ME et al (2019) Sensitivity of chest X-ray for detecting lung cancer in people presenting with symptoms: a systematic review. Br J Gen Pract 69:e827–e835CrossRefPubMedPubMedCentral
27.
go back to reference Dillon DG, Rodriguez RM (2021) Screening performance of the chest X-ray in adult blunt trauma evaluation: Is it effective and what does it miss? Am J Emerg Med 49:310–314CrossRefPubMed Dillon DG, Rodriguez RM (2021) Screening performance of the chest X-ray in adult blunt trauma evaluation: Is it effective and what does it miss? Am J Emerg Med 49:310–314CrossRefPubMed
28.
go back to reference Quekel LG, Kessels AG, Goei R, van Engelshoven JM (1999) Miss rate of lung cancer on the chest radiograph in clinical practice. Chest 115:720–724CrossRefPubMed Quekel LG, Kessels AG, Goei R, van Engelshoven JM (1999) Miss rate of lung cancer on the chest radiograph in clinical practice. Chest 115:720–724CrossRefPubMed
Metadata
Title
Performance of AI to exclude normal chest radiographs to reduce radiologists’ workload
Authors
Steven Schalekamp
Kicky van Leeuwen
Erdi Calli
Keelin Murphy
Matthieu Rutten
Bram Geurts
Liesbeth Peters-Bax
Bram van Ginneken
Mathias Prokop
Publication date
17-05-2024
Publisher
Springer Berlin Heidelberg
Published in
European Radiology / Issue 11/2024
Print ISSN: 0938-7994
Electronic ISSN: 1432-1084
DOI
https://doi.org/10.1007/s00330-024-10794-5

Other articles of this Issue 11/2024

European Radiology 11/2024 Go to the issue