Top

Journal of Translational Medicine

Published in:

Open Access 01-12-2021 | COVID-19 | Research

Self-supervised deep learning model for COVID-19 lung CT image segmentation highlighting putative causal relationship among age, underlying disease and COVID-19

Authors: Daryl L. X. Fung, Qian Liu, Judah Zammit, Carson Kai-Sang Leung, Pingzhao Hu

Published in: Journal of Translational Medicine | Issue 1/2021

Abstract

Background

Coronavirus disease 2019 (COVID-19) is very contagious. Cases appear faster than the available Polymerase Chain Reaction test kits in many countries. Recently, lung computerized tomography (CT) has been used as an auxiliary COVID-19 testing approach. Automatic analysis of the lung CT images is needed to increase the diagnostic efficiency and release the human participant. Deep learning is successful in automatically solving computer vision problems. Thus, it can be introduced to the automatic and rapid COVID-19 CT diagnosis. Many advanced deep learning-based computer vison techniques were developed to increase the model performance but have not been introduced to medical image analysis.

Methods

In this study, we propose a self-supervised two-stage deep learning model to segment COVID-19 lesions (ground-glass opacity and consolidation) from chest CT images to support rapid COVID-19 diagnosis. The proposed deep learning model integrates several advanced computer vision techniques such as generative adversarial image inpainting, focal loss, and lookahead optimizer. Two real-life datasets were used to evaluate the model’s performance compared to the previous related works. To explore the clinical and biological mechanism of the predicted lesion segments, we extract some engineered features from the predicted lung lesions. We evaluate their mediation effects on the relationship of age with COVID-19 severity, as well as the relationship of underlying diseases with COVID-19 severity using statistic mediation analysis.

Results

The best overall F1 score is observed in the proposed self-supervised two-stage segmentation model (0.63) compared to the two related baseline models (0.55, 0.49). We also identified several CT image phenotypes that mediate the potential causal relationship between underlying diseases with COVID-19 severity as well as the potential causal relationship between age with COVID-19 severity.

Conclusions

This work contributes a promising COVID-19 lung CT image segmentation model and provides predicted lesion segments with potential clinical interpretability. The model could automatically segment the COVID-19 lesions from the raw CT images with higher accuracy than related works. The features of these lesions are associated with COVID-19 severity through mediating the known causal of the COVID-19 severity (age and underlying diseases).

Available only for authorised users

Disease outbreak news. WHO|Novel coronavirus—China. WHO. 2020 [cited 2020 Sep 22]. https://www.who.int/csr/don/12-january-2020-novel-coronavirus-china/en/

Dong E, Du H, Gardner L. An interactive web-based dashboard to track COVID-19 in real time. Lancet Infect Dis. 2020;20:533–4.CrossRef

Aleta A, Martín-Corral D, y Piontti AP, Ajelli M, Litvinova M, Chinazzi M, et al. Modelling the impact of testing, contact tracing and household quarantine on second waves of COVID-19. Nat Hum Behav. 2020;4:964–71. https://doi.org/10.1038/s41562-020-0931-9.CrossRefPubMed

Harmon SA, Sanford TH, Xu S, Turkbey EB, Roth H, Xu Z, et al. Artificial intelligence for the detection of COVID-19 pneumonia on chest CT using multinational datasets. Nat Commun. 2020;11:1–7. https://doi.org/10.1038/s41467-020-17971-2.CrossRef

Gozes O, Frid-Adar M, Greenspan H, Browning PD, Zhang H, Ji W, et al. Rapid AI development cycle for the coronavirus (COVID-19) pandemic: initial results for automated detection & patient monitoring using deep learning CT image analysis. 2020. http://arxiv.org/abs/2003.05037

Wynants L, Van Calster B, Collins GS, Riley RD, Heinze G, Schuit E, et al. Prediction models for diagnosis and prognosis of covid-19: Systematic review and critical appraisal. BMJ. 2020;369:18.

Yan L, Zhang H-T, Goncalves J, Xiao Y, Wang M, Guo Y, et al. An interpretable mortality prediction model for COVID-19 patients. Nat Mach Intell. 2020;2:283–8. https://doi.org/10.1038/s42256-020-0180-7.CrossRef

Nikpouraghdam M, Jalali Farahani A, Alishiri GH, Heydari S, Ebrahimnia M, Samadinia H, et al. Epidemiological characteristics of coronavirus disease 2019 (COVID-19) patients in IRAN: a single center study. J Clin Virol. 2020;127:104378.CrossRef

Banerjee A, Pasea L, Harris S, Gonzalez-Izquierdo A, Torralbo A, Shallcross L, et al. Estimating excess 1-year mortality associated with the COVID-19 pandemic according to underlying conditions and age: a population-based cohort study. Lancet. 2020;395:1715–25.CrossRef

10.

Remy-Jardin M, Tillie-Leblond I, Szapiro D, Ghaye B, Cotte L, Mastora I, et al. CT angiography of pulmonary embolism in patients with underlying respiratory disease: impact of multislice CT on image quality and negative predictive value. Eur Radiol. 2002;12:1971–8. https://doi.org/10.1007/s00330-002-1485-0.CrossRefPubMed

11.

Harris A, Kamishima T, Hao HY, Kato F, Omatsu T, Onodera Y, et al. Splenic volume measurements on computed tomography utilizing automatically contouring software and its relationship with age, gender, and anthropometric parameters. Eur J Radiol Elsevier. 2010;75:e97-101.CrossRef

12.

MacKinnon DP, Fairchild AJ, Fritz MS. Mediation analysis. Annu Rev Psychol. 2007;58:593–614.CrossRef

13.

Chen, X., Yao, L., Zhou, T., Dong, J. & Zhang, Y. Momentum contrastive learning for few-shot COVID-19 diagnosis from chest CT images. Pattern Recogn 2021;113:107826. https://doi.org/10.1016/j.patcog.2021.107826.

14.

Li Y, et al. Efficient and Effective Training of COVID-19 Classification Networks With Self-Supervised Dual-Track Learning to Rank. IEEE J Biomed Health Info 2020; 24(10):2787-2797. https://doi.org/10.1109/JBHI.2020.3018181

15.

Shahinfar S, Meek P, Falzon G. “How many images do I need?” Understanding how sample size per class affects deep learning model performance metrics for balanced designs in autonomous wildlife monitoring. Ecol Inform. 2020;57:101085.CrossRef

16.

Voulodimos A, Protopapadakis E, Katsamenis I, Doulamis A, Doulamis N. A few-shot U-net deep learning model for COVID-19 infected area segmentation in CT images. Sensors. 2021;21(6):2215.CrossRef

17.

Voulodimos A, Protopapadakis E, Katsamenis I, Doulamis A, Doulamis N. Deep learning models for COVID-19 infected area segmentation in CT images. medRxiv. 2020. https://doi.org/10.1101/2020.05.08.20094664.CrossRef

18.

Ma J, Wang Y, An X, Ge C, Yu Z, Chen J, et al. Toward data-efficient learning: a benchmark for COVID-19 CT lung and infection segmentation. Med Phys. 2021;48:1197–210.CrossRef

19.

Lizancos Vidal P, de Moura J, Novo J, Ortega M. Multi-stage transfer learning for lung segmentation using portable X-ray devices for patients with COVID-19. Expert Syst Appl. 2021;173:114677.CrossRef

20.

Aslan MF, Unlersen MF, Sabanci K, Durdu A. CNN-based transfer learning–BiLSTM network: a novel approach for COVID-19 infection detection. Appl Soft Comput. 2021;98:106912.CrossRef

21.

Katsamenis I, Protopapadakis E, Voulodimos A, Doulamis A, Doulamis N. Transfer learning for COVID-19 pneumonia detection and classification in chest X-ray images. In: Proceedings of the PCI. 2020. p. 170–174. https://doi.org/10.1145/3437120.3437300.

22.

Fan D-P, Zhou T, Ji G-P, Zhou Y, Chen G, Fu H, et al. Inf-net: automatic COVID-19 Lung infection segmentation from CT images. IEEE Trans Med Imaging 2020;39:2626–37. https://doi.org/10.1109/TMI.2020.2996645.CrossRef

23.

Yao Q, Xiao L, Liu P, Kevin Zhou S. Label-free segmentation of COVID-19 lesions in lung CT. IEEE Trans Med Imaging 2021. https://doi.org/10.1109/TMI.2021.3066161.CrossRefPubMedPubMedCentral

24.

Wang Y, Zhang J, Kan M, Shan S, Chen X. Self-supervised equivariant attention mechanism for weakly supervised semantic segmentation. In: Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR). 2020. pp. 12272–12281.

25.

Bertalmio M, Sapiro G, Caselles V, Ballester C. Image inpainting. In: Proceedings of the SIGGRAPH. 2000. pp. 417–424. https://doi.org/10.1145/344779.344972.

26.

Pathak D, Krahenbuhl P, Donahue J, Darrell T, Efros AA. Context encoders: feature learning by inpainting. In: Proceedings of the IEEE conference on computer vision and pattern recognition. 2016. pp. 2536–2544.

27.

Liu G, Reda FA, Shih KJ, Wang T-C, Tao A, Catanzaro B. Image inpainting for irregular holes using partial convolutions. Proc. ECCV 2018;89–105. https://doi.org/10.1007/978-3-030-01252-6_6.CrossRef

28.

Xie J, Xu L, Chen E. Image denoising and inpainting with deep neural networks. In: NIPS. 2021. pp. 350–358. https://proceedings.neurips.cc/paper/2012/file/6cdd60ea0045eb7a6ec44c54d29ed402-Paper.pdf.

29.

Hassan ET, Abbas HM, Mohamed HK. Image inpainting based on image segmentation and segment classification. In: Proceedings of ICCSCE 2013. 2013. p. 28–33. https://doi.org/10.1109/ICCSCE.2013.6719927

30.

Yu J, Lin Z, Yang J, Shen X, Lu X, Huang TS. Generative image inpainting with contextual attention. openaccess.thecvf.com. 2010;4:34–40. https://github.com/

31.

Goodfellow IJ, Pouget-Abadie J, Mirza M, Xu B, Warde-Farley D, Ozair S, et al. Generative adversarial networks. 2014;1–9. http://arxiv.org/abs/1406.2661

32.

Lavanya M, Kannan PM. Lung lesion detection in CT scan images using the Fuzzy Local Information Cluster Means (FLICM) automatic segmentation algorithm and back propagation network classification. Asian Pacific J Cancer Prev. 2017;18:3395–9.

33.

Collins J, Stern EJ. Chest radiology: the essentials. Lippincott Williams & Wilkins; 2008.

34.

Dahnert WF. Radiology review manual, 8e. Lippincott Williams & Wilkins; 2017. ISBN 9781496360694.

35.

Robin Smithuis O van D and CS-P. The radiology assistant : basic interpretation. https://radiologyassistant.nl/chest/hrct/basic-interpretation

36.

COVID-19—Medical segmentation. http://medicalsegmentation.com/covid19/

37.

Lin T-Y, Goyal P, Girshick R, He K, Dollár P. Focal Loss for Dense Object Detection. IEEE Trans Pattern Anal Mach Intell 2017;42:318–27. https://doi.org/10.1109/TPAMI.2018.2858826

38.

Bottou L, Curtis FE, Nocedal J. Optimization methods for large-scale machine learning. SIAM Rev. 2018;60(2):223–311.CrossRef

39.

Zhang MR, Lucas J, Hinton G, Ba J. Lookahead optimizer: k Steps forward, 1 step back. Adv Neural Inf Process Syst. 2019;32:1–19.

40.

Ning W, Lei S, Yang J, Cao Y et al.. Open resource of clinical data from patients with pneumonia for the prediction of COVID-19 outcomes via deep learning. Nat Biomed Eng 2020;4:1197–1207. https://doi.org/10.1038/s41551-020-00633-5CrossRefPubMedPubMedCentral

41.

Van Griethuysen JJM, Fedorov A, Parmar C, Hosny A, Aucoin N, Narayan V, et al. Computational radiomics system to decode the radiographic phenotype. Cancer Res. 2017;77:e104–7.CrossRef

42.

Wei T, Simko V. R package “corrplot”: visualization of a correlation matrix. 2017.

43.

Rosseel Y. Lavaan: An R package for structural equation modeling. J Stat Softw 2012;48:1–36. https://doi.org/10.18637/jss.v048.i02.CrossRef

44.

Gunzler D, Chen T, Wu P, Zhang H. Introduction to mediation analysis with structural equation modeling. Shanghai Arch Psychiatry. 2013;25:390–4.PubMedPubMedCentral

45.

Muthen B. Applications of causally defined direct and indirect effects in mediation analysis using SEM in Mplus. Reliab Risk Saf Back to Futur. 2010;106–13.

46.

Hooda R, Mittal A, Sofat S. Lung segmentation in chest radiographs using fully convolutional networks. Turkish J Electr Eng Comput Sci 2019;27:710–22. https://doi.org/10.3906/elk-1710-157.CrossRef

47.

Zhang K, Liu X, Shen J, Li Z, Sang Y, Wu X, et al. Clinically applicable AI system for accurate diagnosis, quantitative measurements, and prognosis of COVID-19 pneumonia using computed tomography. Cell Cell Press. 2020;181:1423-1433.e11.

48.

Ronneberger O, Fischer P, Brox T. U-Net: Convolutional networks for biomedical image segmentation. In: Proc. MICCAI 2015. 2015. pp. 1–8. https://doi.org/10.1007/978-3-319-24574-4_28.

49.

Zeng Z, Xie W, Zhang Y, Access YL-I, 2019 U. RIC-Unet: An improved neural network based on Unet for nuclei segmentation in histology images. IEEE Access 2019;7:21420–8. https://doi.org/10.1109/ACCESS.2019.2896920.CrossRef

50.

Wang C, Macgillivray T, Macnaught G, Yang G, Newby D. A two-stage 3D Unet framework for multi-class segmentation on full resolution image. In: Proceedings of STACOM 2018. 2018. pp. 191–199. https://doi.org/10.1007/978-3-030-12029-0_21.

51.

Weng Y, Zhou T, Li Y, Access XQ-I, 2019 U. Nas-unet: Neural architecture search for medical image segmentation. IEEE Access 2019;7:44247–57. https://doi.org/10.1109/ACCESS.2019.2908991.CrossRef

52.

Li X, Chen H, Qi X, Dou Q, Fu C-W, Heng P-A. H-DenseUNet: Hybrid densely connected UNet for liver and tumor segmentation from CT volumes. IEEE Trans Med Imaging 2018;37:2663–74. https://doi.org/10.1109/TMI.2018.2845918.

53.

Chen W, Liu B, Peng S, Sun J, Qiao X. S3D-UNET: Separable 3D U-Net for brain tumor segmentation. In: Proceedings of BrainLes 2018. 2018. pp. 358–368. https://doi.org/10.1007/978-3-030-11726-9_32.

54.

Dudewicz EJ, van der Meulen EC. Entropy-based tests of uniformity. J Am Stat Assoc. 1981;76:967.CrossRef

55.

van Timmeren JE, Leijenaar RTH, van Elmpt W, Reymen B, Lambin P. Feature selection methodology for longitudinal cone-beam CT radiomics. Acta Oncol (Madr). 2017;56:1537–43. https://doi.org/10.1080/0284186X.2017.1350285.CrossRef

56.

Lee SH, Cho HH, Lee HY, Park H. Clinical impact of variability on CT radiomics and suggestions for suitable feature selection: a focus on lung cancer. Cancer Imaging. 2019;19:54. https://doi.org/10.1186/s40644-019-0239-z.CrossRefPubMedPubMedCentral

57.

Ai T, Yang Z, Hou H, Zhan C, Chen C, Lv W, et al. Correlation of chest CT and RT-PCR testing for coronavirus disease 2019 (COVID-19) in China: a report of 1014 cases. Radiology. 2020;296:E32-40. https://doi.org/10.1148/radiol.2020200642.CrossRefPubMed

58.

Dai WC, Zhang HW, Yu J, Xu HJ, Chen H, Luo SP, et al. CT Imaging and differential diagnosis of COVID-19. Can Assoc Radiol J. 2020;71:195–200.CrossRef

59.

Shen C, Mark R, Kagetsu NJ, Becker AS, Bar-Yam Y. Combining PCR and CT testing for COVID. 2020. http://arxiv.org/abs/2006.02140

60.

Simpson S, Kay FU, Abbara S, Bhalla S, Chung JH, Chung M, et al. Radiological society of north America expert consensus statement on reporting chest CT findings related to COVID-19. Endorsed by the society of thoracic Radiology, the American College of Radiology, and RSNA. Radiol Cardiothorac Imaging. 2020;2:e200152. https://doi.org/10.1148/ryct.2020200152.CrossRefPubMedPubMedCentral

61.

Bai HX, Hsieh B, Xiong Z, Halsey K, Choi JW, Tran TML, et al. Performance of radiologists in differentiating COVID-19 from non-COVID-19 viral pneumonia at chest CT. Radiology. 2020;296:E46-54.CrossRef

62.

Wen Z, Chi Y, Zhang L, Liu H, Du K, Li Z, et al. Coronavirus disease 2019: initial detection on chest CT in a retrospective multicenter study of 103 Chinese subjects. Radiol Cardiothorac Imaging. 2020;2:e200092. https://doi.org/10.1148/ryct.2020200092.CrossRefPubMedPubMedCentral

63.

Fang Y, Zhang H, Xie J, Lin M, Ying L, Pang P, et al. Sensitivity of chest CT for COVID-19: comparison to RT-PCR. Radiology. 2020;296(2):E115–7. https://doi.org/10.1148/radiol.2020200432.CrossRefPubMed

64.

Inui S, Fujikawa A, Jitsu M, Kunishima N, Watanabe S, Suzuki Y, et al. Chest CT findings in cases from the cruise ship “Diamond Princess” with coronavirus disease 2019 (COVID-19). Radiol Cardiothorac Imaging. 2020;2:e200110. https://doi.org/10.1148/ryct.2020200110.CrossRefPubMedPubMedCentral

65.

Singh N, Fratesi J. Chest CT imaging of an early Canadian case of COVID-19 in a 28-year-old man. CMAJ 2020;192:E455. https://doi.org/10.1503/cmaj.200431

66.

Rubin GD. Costing in radiology and health care: rationale, relativity, rudiments, and realities. Radiology. 2017;282:333–47. https://doi.org/10.1148/radiol.2016160749.CrossRefPubMed

67.

Yuan M, Pu X-H, Xu X-Q, Zhang Y-D, Zhong Y, Li H, et al. Lung adenocarcinoma: assessment of epidermal growth factor receptor mutation status based on extended models of diffusion-weighted image. J Magn Reson Imaging. 2017;46:281–9. https://doi.org/10.1002/jmri.25572.CrossRefPubMed

68.

Weiss GJ, Ganeshan B, Miles KA, Campbell DH, Cheung PY, Frank S, et al. Noninvasive image texture analysis differentiates K-ras mutation from pan-wildtype NSCLC and is prognostic. PLoS One 2014;9:e100244. https://doi.org/10.1371/journal.pone.0100244.

69.

Koyama H, Ohno Y, Yamazaki Y, Nogami M, Kusaka A, Murase K, et al. Quantitatively assessed CT imaging measures of pulmonary interstitial pneumonia: effects of reconstruction algorithms on histogram parameters. Eur J Radiol. 2010;74:142–6.CrossRef

70.

Schofield R, Ganeshan B, Fontana M, Nasis A, Castelletti S, Rosmini S, et al. Texture analysis of cardiovascular magnetic resonance cine images differentiates aetiologies of left ventricular hypertrophy. Clin Radiol. 2019;74:140–9.CrossRef

71.

Baek HJ, Kim HS, Kim N, Choi YJ, Kim YJ. Percent change of perfusion skewness and kurtosis: a potential imaging biomarker for early treatment response in patients with newly diagnosed glioblastomas. Radiology 2012;264:834–43. https://doi.org/10.1148/radiol.12112120.

Title: Self-supervised deep learning model for COVID-19 lung CT image segmentation highlighting putative causal relationship among age, underlying disease and COVID-19
Authors: Daryl L. X. Fung
Qian Liu
Judah Zammit
Carson Kai-Sang Leung
Pingzhao Hu
Publication date: 01-12-2021
Publisher: BioMed Central
Keyword: COVID-19
Published in: Journal of Translational Medicine / Issue 1/2021
Electronic ISSN: 1479-5876
DOI: https://doi.org/10.1186/s12967-021-02992-2

Live Webinar | 27-06-2024 | 18:00 (CEST)

Keynote webinar | Spotlight on medication adherence

Reserve your place

Live: Thursday 27th June 2024, 18:00-19:30 (CEST)

WHO estimates that half of all patients worldwide are non-adherent to their prescribed medication. The consequences of poor adherence can be catastrophic, on both the individual and population level.

Join our expert panel to discover why you need to understand the drivers of non-adherence in your patients, and how you can optimize medication adherence in your clinics to drastically improve patient outcomes.

Prof. Kevin Dolgin

Prof. Florian Limbourg

Prof. Anoop Chauhan

Developed by: Springer Medicine

Obesity Clinical Trial Summary

At a glance: The STEP trials

A round-up of the STEP phase 3 clinical trials evaluating semaglutide for weight loss in people with overweight or obesity.

Developed by: Springer Medicine

Highlights from the ACC 2024 Congress

Year in Review: Pediatric cardiology

Pediatric Cardiology Video

Watch Dr. Anne Marie Valente present the last year's highlights in pediatric and congenital heart disease in the official ACC.24 Year in Review session.

Year in Review: Pulmonary vascular disease

Cardiopulmonary Comorbidity Video

The last year's highlights in pulmonary vascular disease are presented by Dr. Jane Leopold in this official video from ACC.24.

Year in Review: Valvular heart disease

Valvular Heart Disease Video

Watch Prof. William Zoghbi present the last year's highlights in valvular heart disease from the official ACC.24 Year in Review session.

Year in Review: Heart failure and cardiomyopathies

Heart Failure Video

Watch this official video from ACC.24. Dr. Biykem Bozkurt discusses last year's major advances in heart failure and cardiomyopathies.

ACC 2024 Congress Coverage

Year in Review: Heart failure and cardiomyopathies

Heart Failure Video

Watch this official video from ACC.24. Dr. Biykem Bozkurt discusses last year's major advances in heart failure and cardiomyopathies.

Watch now

Semaglutide benefits people with type 2 diabetes and obesity-related heart failure

16-04-2024 Type 2 Diabetes News

Incentivised to exercise

11-04-2024 Lifestyle Modifications News

New intervention for STEMI-related cardiogenic shock

10-04-2024 Myocardial Infarction News

» View more highlights on the ACC 2024 congress hub page

Image Credits

Illustration of figure on mountain summit/© Orapun / Stock.adobe.com, STEP AAG HeroTeaserCollection image/© Tarek / Generated with AI / Stock.adobe.com, ACC 2024 Pediatric and Congenital Heart Disease: Year in Review/© 2024 American College of Cardiology, ACC 2024 Pulmonary Vascular Disease: Year in Review/© 2024 American College of Cardiology, ACC 2024 Valvular Heart Disease: Year in Review/© 2024 American College of Cardiology, ACC 2024 HF and Cardiomyopathies: Year in Review/© 2024 American College of Cardiology, Woman out walking/© Seventyfour / stock.adobe.com (symbolic image with model), Woman checking smartwatch /© saltdium / stock.adobe.com (symbolic image with model), Cardiac catheterization/© Damian / stock.adobe.com

Keynote webinar | Spotlight on medication adherence

Springer Medicine

Abstract

Background

Methods

Results

Conclusions

Please log in to get access to this content

Other articles of this Issue 1/2021

Analysis of viral load in different specimen types and serum antibody levels of COVID-19 patients

Retraction Note to: Intra-articular injection in the knee of adipose derived stromal cells (stromal vascular fraction) and platelet rich plasma for osteoarthritis

The tumor immune microenvironment of primary and metastatic HER2− positive breast cancers utilizing gene expression and spatial proteomic profiling

Knockdown of long noncoding RNA GAS5 reduces vascular smooth muscle cell apoptosis by inactivating EZH2-mediated RIG-I signaling pathway in abdominal aortic aneurysm

Correction to: Serum CCL20 combined with IL-17A as early diagnostic and prognostic biomarkers for human colorectal cancer

Inhibition of DKC1 induces telomere-related senescence and apoptosis in lung adenocarcinoma

Live: Thursday 27th June 2024, 18:00-19:30 (CEST)

Highlights from the ACC 2024 Congress

ACC 2024 Congress Coverage