Top

Published in:

Open Access 01-12-2023 | Artificial Intelligence | Original Article

Retrospective batch analysis to evaluate the diagnostic accuracy of a clinically deployed AI algorithm for the detection of acute pulmonary embolism on CTPA

Authors: Eline Langius-Wiffen, Pim A. de Jong, Firdaus A. Mohamed Hoesein, Lisette Dekker, Andor F. van den Hoven, Ingrid M. Nijholt, Martijn F. Boomsma, Wouter B. Veldhuis

Published in: Insights into Imaging | Issue 1/2023

Abstract

Purpose

To generate and extend the evidence on the clinical validity of an artificial intelligence (AI) algorithm to detect acute pulmonary embolism (PE) on CT pulmonary angiography (CTPA) of patients suspected of PE and to evaluate the possibility of reducing the risk of missed findings in clinical practice with AI-assisted reporting.

Methods

Consecutive CTPA scan data of 3316 patients referred because of suspected PE between 24-2-2018 and 31-12-2020 were retrospectively analysed by a CE-certified and FDA-approved AI algorithm. The output of the AI was compared with the attending radiologists’ report. To define the reference standard, discordant findings were independently evaluated by two readers. In case of disagreement, an experienced cardiothoracic radiologist adjudicated.

Results

According to the reference standard, PE was present in 717 patients (21.6%). PE was missed by the AI in 23 patients, while the attending radiologist missed 60 PE. The AI detected 2 false positives and the attending radiologist 9. The sensitivity for the detection of PE by the AI algorithm was significantly higher compared to the radiology report (96.8% vs. 91.6%, p < 0.001). Specificity of the AI was also significantly higher (99.9% vs. 99.7%, p = 0.035). NPV and PPV of the AI were also significantly higher than the radiology report.

Conclusion

The AI algorithm showed a significantly higher diagnostic accuracy for the detection of PE on CTPA compared to the report of the attending radiologist. This finding indicates that missed positive findings could be prevented with the implementation of AI-assisted reporting in daily clinical practice.

Critical relevance statement

Missed positive findings on CTPA of patients suspected of pulmonary embolism can be prevented with the implementation of AI-assisted care.

Key points

The AI algorithm showed excellent diagnostic accuracy detecting PE on CTPA.
Accuracy of the AI was significantly higher compared to the attending radiologist.
Highest diagnostic accuracy can likely be achieved by radiologists supported by AI.
Our results indicate that implementation of AI-assisted reporting could reduce the number of missed positive findings.

Graphical abstract

Bruls RJM, Kwee RM (2020) Workload for radiologists during on-call hours: Dramatic increase in the past 15 years. Insights Imaging 11:121. https://doi.org/10.1186/s13244-020-00925-zCrossRefPubMedPubMedCentral

Kwee TC, Kwee RM (2021) Workload of diagnostic radiologists in the foreseeable future based on recent scientific advances: growth expectations and role of artificial intelligence. Insights Imaging 12:88–94. https://doi.org/10.1186/s13244-021-01031-4CrossRefPubMedPubMedCentral

Ruutiainen AT, Durand DJ, Scanlon MH, Itri JN (2013) Increased error rates in preliminary reports issued by radiology residents working more than 10 consecutive hours overnight. Acad Radiol 20:305–311. https://doi.org/10.1016/j.acra.2012.09.028CrossRefPubMed

Hanna TN, Lamoureux C, Krupinski EA, Weber S, Johnson J (2018) Effect of shift, schedule, and volume on interpretive accuracy: a retrospective analysis of 2.9 million radiologic examinations. Radiology 287:205–212. https://doi.org/10.1148/radiol.2017170555CrossRefPubMed

Harolds JA, Parikh JR, Bluth EI, Dutton SC, Recht MP (2016) Burnout of radiologists: frequency, risk factors, and remedies: a report of the ACR commission on human resources. J Am Coll Radiol 13:411–416. https://doi.org/10.1016/j.jacr.2015.11.003CrossRefPubMed

Kligerman S, Lahiji K, Weihe E et al (2015) Detection of pulmonary embolism on computed tomography: Improvement using a model-based iterative reconstruction algorithm compared with filtered back projection and iterative reconstruction algorithms. J Thorac Imaging 30:60–68. https://doi.org/10.1097/RTI.0000000000000122CrossRefPubMed

Konstantinides SV, Torbicki A, Agnelli G et al (2014) 2014 ESC guidelines on the diagnosis and management of acute pulmonary embolism: the task force for the diagnosis and management of acute pulmonary embolism of the european society of cardiology (ESC)endorsed by the european respiratory society (ERS). Eur Heart J 35:3033–3080. https://doi.org/10.1093/eurheartj/ehu283CrossRefPubMed

Neri E, de Souza N, Brady A et al (2019) What the radiologist should know about artificial intelligence—an ESR white paper. Insights Imaging 10:44. https://doi.org/10.1186/s13244-019-0738-2CrossRef

Daye D, Wiggins WF, Lungren MP et al (2022) Implementation of clinical artificial intelligence in radiology: Who decides and how? Radiology 305:555–563. https://doi.org/10.1148/radiol.212151CrossRefPubMed

10.

Vasey B, Nagendran M, Campbell B et al (2022) Reporting guideline for the early-stage clinical evaluation of decision support systems driven by artificial intelligence: DECIDE-AI. Nat Med 28:924–933. https://doi.org/10.1038/s41591-022-01772-9CrossRefPubMed

11.

Liu Z, Yuan H, Wang H (2022) CAM-wnet: an effective solution for accurate pulmonary embolism segmentation. Med Phys 49:5294–5303. https://doi.org/10.1002/mp.15719CrossRefPubMed

12.

Weikert T, Winkel DJ, Bremerich J et al (2020) Automated detection of pulmonary embolism in CT pulmonary angiograms using an AI-powered algorithm. Eur Radiol 30:6545–6553. https://doi.org/10.1007/s00330-020-06998-0CrossRefPubMed

13.

Huang S, Kothari T, Banerjee I et al (2020) PENet—a scalable deep-learning model for automated diagnosis of pulmonary embolism using volumetric CT imaging. NPJ Digit Med 3:61. https://doi.org/10.1038/s41746-020-0266-yCrossRefPubMedPubMedCentral

14.

Buls N, Watté N, Nieboer K, Ilsen B, de Mey J (2021) Performance of an artificial intelligence tool with real-time clinical workflow integration—detection of intracranial hemorrhage and pulmonary embolism. Phys Med 83:154–160. https://doi.org/10.1016/j.ejmp.2021.03.015CrossRefPubMed

15.

Cheikh AB, Gorincour G, Nivet H et al (2022) How artificial intelligence improves radiological interpretation in suspected pulmonary embolism. Eur Radiol 32:5831–5842. https://doi.org/10.1007/s00330-022-08645-2CrossRefPubMedPubMedCentral

16.

Huhtanen H, Nyman M, Mohsen T, Virkki A, Karlsson A, Hirvonen J (2022) Automated detection of pulmonary embolism from CT-angiograms using deep learning. BMC Med Imaging 22:43. https://doi.org/10.1186/s12880-022-00763-zCrossRefPubMedPubMedCentral

17.

Soffer S, Klang E, Shimon O et al (2021) Deep learning for pulmonary embolism detection on computed tomography pulmonary angiogram: a systematic review and meta-analysis. Sci Rep 11:15814–15823. https://doi.org/10.1038/s41598-021-95249-3CrossRefPubMedPubMedCentral

18.

Yu AC, Mohajer B, Eng J (2022) External validation of deep learning algorithms for radiologic diagnosis: a systematic review. Radiol Artif Intell 4:e210064. https://doi.org/10.1148/ryai.210064CrossRefPubMedPubMedCentral

19.

Petry M, Lansky C, Chodakiewitz Y, Maya M, Pressman B (2022) Decreased hospital length of stay for ICH and PE after adoption of an artificial intelligence-augmented radiological worklist triage system. Radiol Res Pract 2022:2141839. https://doi.org/10.1155/2022/2141839CrossRefPubMedPubMedCentral

20.

Romero-Martín S, Elías-Cabot E, Raya-Povedano J, Gubern-Mérida A, Rodríguez-Ruiz A, Álvarez-Benito M (2022) Stand-alone use of artificial intelligence for digital mammography and digital breast tomosynthesis screening: a retrospective evaluation. Radiology 302:535–542. https://doi.org/10.1148/radiol.211590CrossRefPubMed

21.

Geis JR, Brady AP, Wu CC et al (2019) Ethics of artificial intelligence in radiology: summary of the joint European and North American multisociety statement. Radiology 293(2):436–440. https://doi.org/10.1148/radiol.2019191586CrossRefPubMed

22.

Eskandari A, Narayanasamy S, Ward C et al (2022) Prevalence and significance of incidental findings on computed tomography pulmonary angiograms: a retrospective cohort study. Am J Emerg Med 54:232–237. https://doi.org/10.1016/j.ajem.2022.01.064CrossRefPubMed

23.

Hall WB, Truitt SG, Scheunemann LP et al (2009) The prevalence of clinically relevant incidental findings on chest computed tomographic angiograms ordered to diagnose pulmonary embolism. Arch Internal Med 169:1961–1965. https://doi.org/10.1001/archinternmed.2009.360CrossRef

24.

Richman PB, Courtney DM, Friese J et al (2004) Prevalence and significance of nonthromboembolic findings on chest computed tomography angiography performed to rule out pulmonary embolism: a multicenter study of 1025 emergency department patients. Acad Emerg Med 11:642–647CrossRefPubMed

25.

Devaraj A, Sayer C, Sheard S, Grubnic S, Nair A, Vlahos I (2015) Diagnosing acute pulmonary embolism with computed tomography: Imaging update. J Thorac Imaging 30:176–192. https://doi.org/10.1097/RTI.0000000000000146CrossRefPubMed

26.

van der Hulle T, Cheung WY, Kooij S et al (2017) Simplified diagnostic management of suspected pulmonary embolism (the YEARS study): a prospective, multicentre, cohort study. Lancet 390:289–297. https://doi.org/10.1016/S0140-6736(17)30885-1CrossRefPubMed

27.

van der Pol LM, Dronkers CEA, van der Hulle T et al (2018) The YEARS algorithm for suspected pulmonary embolism: shorter visit time and reduced costs at the emergency department. J Thromb Haemost 16:725–733. https://doi.org/10.1111/jth.13972CrossRefPubMed

28.

Hutchinson BD, Navin P, Marom EM, Truong MT, Bruzzi JF (2015) Overdiagnosis of pulmonary embolism by pulmonary CT angiography. AJR Am J Roentgenol 205:271–277. https://doi.org/10.2214/AJR.14.13938CrossRefPubMed

29.

Park SH, Han K, Jang HY et al (2022) Methods for clinical evaluation of artificial intelligence algorithms for medical diagnosis. Radiology. https://doi.org/10.1148/radiol.220182CrossRefPubMed

30.

Bossuyt PMM, Reitsma JB, Linnet K, Moons KGM (2012) Beyond diagnostic accuracy: the clinical utility of diagnostic tests. Clin Chem 58:1636–1643. https://doi.org/10.1373/clinchem.2012.182576CrossRefPubMed

31.

Grutters JPC, Govers T, Nijboer J, Tummers M, van der Wilt G, Jan RMM (2019) Problems and promises of health technologies: the role of early health economic modeling. Int J Health Policy Manag 8:575–582. https://doi.org/10.15171/ijhpm.2019.36CrossRefPubMedPubMedCentral

32.

Wolff J, Pauling J, Keck A, Baumbach J (2020) The economic impact of artificial intelligence in health care: systematic review. J Med Internet Res 22:e16866. https://doi.org/10.2196/16866CrossRefPubMedPubMedCentral

Title: Retrospective batch analysis to evaluate the diagnostic accuracy of a clinically deployed AI algorithm for the detection of acute pulmonary embolism on CTPA
Authors: Eline Langius-Wiffen
Pim A. de Jong
Firdaus A. Mohamed Hoesein
Lisette Dekker
Andor F. van den Hoven
Ingrid M. Nijholt
Martijn F. Boomsma
Wouter B. Veldhuis
Publication date: 01-12-2023
Publisher: Springer Vienna
Keywords: Artificial Intelligence
Acute Pulmonary Embolism
Acute Pulmonary Embolism
Acute Pulmonary Embolism
Published in: Insights into Imaging / Issue 1/2023
Electronic ISSN: 1869-4101
DOI: https://doi.org/10.1186/s13244-023-01454-1

Keynote webinar | Spotlight on medication adherence

Springer Medicine

Retrospective batch analysis to evaluate the diagnostic accuracy of a clinically deployed AI algorithm for the detection of acute pulmonary embolism on CTPA

Abstract

Purpose

Methods

Results

Conclusion

Critical relevance statement

Key points

Graphical abstract

Keynote webinar | Spotlight on medication adherence

Springer Medicine

Abstract

Purpose

Methods

Results

Conclusion

Critical relevance statement

Key points

Graphical abstract

Please log in to get access to this content

Other articles of this Issue 1/2023

Correlation of androgen receptor with ultrasound, clinicopathological features and clinical outcomes in breast cancer

American College of Radiology Appropriateness Criteria®: a bibliometric analysis of panel members

Whole-body low-dose CT can be of value in prostate cancer decision-making: a retrospective study on 601 patients

The diagnostic performance and clinical value of deep learning-based nodule detection system concerning influence of location of pulmonary nodule

Feasibility and effectiveness of automatic deep learning network and radiomics models for differentiating tumor stroma ratio in pancreatic ductal adenocarcinoma

Association of sex-specific abdominal adipose tissue with WHO/ISUP grade in clear cell renal cell carcinoma