Top

Published in:

01-04-2018 | Review

Deep learning with convolutional neural network in radiology

Authors: Koichiro Yasaka, Hiroyuki Akai, Akira Kunimatsu, Shigeru Kiryu, Osamu Abe

Published in: Japanese Journal of Radiology | Issue 4/2018

Abstract

Deep learning with a convolutional neural network (CNN) is gaining attention recently for its high performance in image recognition. Images themselves can be utilized in a learning process with this technique, and feature extraction in advance of the learning process is not required. Important features can be automatically learned. Thanks to the development of hardware and software in addition to techniques regarding deep learning, application of this technique to radiological images for predicting clinically useful information, such as the detection and the evaluation of lesions, etc., are beginning to be investigated. This article illustrates basic technical knowledge regarding deep learning with CNNs along the actual course (collecting data, implementing CNNs, and training and testing phases). Pitfalls regarding this technique and how to manage them are also illustrated. We also described some advanced topics of deep learning, results of recent clinical studies, and the future directions of clinical application of deep learning techniques.

LeCun Y, Bengio Y, Hinton G. Deep learning. Nature. 2015;521:436–44.CrossRefPubMed

Fukushima K, Miyake S. Neocognitron: a new algorithm for pattern recognition tolerant of deformations and shifts in position. Pattern Recognit. 1982;15:455–69.CrossRef

Hubel DH, Wiesel TN. Receptive fields, binocular interaction and functional architecture in the cat’s visual cortex. J Physiol. 1962;160:106–54.CrossRefPubMedPubMedCentral

Krizhevsky A, Sutskever I, Hinton G. ImageNet classification with deep convolutional neural networks. Advances in Neural Information Processing System 25 (NIPS 2012). 2012. https://papers.nips.cc/paper/4824-imagenet-classification-with-deep-convolutional-neural-networks:Published. Accessed 14 Dec 2017.

Kahn CE Jr. From images to actions: opportunities for artificial intelligence in radiology. Radiology. 2017;285:719–20.CrossRefPubMed

Dreyer KJ, Geis JR. When machines think: radiology’s next frontier. Radiology. 2017;285:713–8.CrossRefPubMed

Gillies RJ, Kinahan PE, Hricak H. Radiomics: images are more than pictures, they are data. Radiology. 2016;278:563–77.CrossRefPubMed

Skogen K, Ganeshan B, Good C, Critchley G, Miles K. Measurements of heterogeneity in gliomas on computed tomography relationship to tumour grade. J Neurooncol. 2013;111:213–9.CrossRefPubMed

Lubner MG, Stabo N, Abel EJ, Del Rio AM, Pickhardt PJ. CT textural analysis of large primary renal cell carcinomas: pretreatment tumor heterogeneity correlates with histologic findings and clinical outcomes. AJR Am J Roentgenol. 2016;207:96–105.CrossRefPubMed

10.

Yasaka K, Akai H, Nojima H, Shinozaki-Ushiku A, Fukayama M, Nakajima J, et al. Quantitative computed tomography texture analysis for estimating histological subtypes of thymic epithelial tumors. Eur J Radiol. 2017;92:84–92.CrossRefPubMed

11.

Kickingereder P, Burth S, Wick A, Gotz M, Eidel O, Schlemmer HP, et al. Radiomic profiling of glioblastoma: identifying an imaging predictor of patient survival with improved performance over established clinical and radiologic risk models. Radiology. 2016;280:880–9.CrossRefPubMed

12.

Huang Y, Liu Z, He L, Chen X, Pan D, Ma Z, et al. Radiomics signature: a potential biomarker for the prediction of disease-free survival in early-stage (I or II) non-small cell lung cancer. Radiology. 2016;281:947–57.CrossRefPubMed

13.

Yip C, Landau D, Kozarski R, Ganeshan B, Thomas R, Michaelidou A, et al. Primary esophageal cancer: heterogeneity as potential prognostic biomarker in patients treated with definitive chemotherapy and radiation therapy. Radiology. 2014;270:141–8.CrossRefPubMed

14.

Ng F, Ganeshan B, Kozarski R, Miles KA, Goh V. Assessment of primary colorectal cancer heterogeneity by using whole-tumor texture analysis: contrast-enhanced CT texture as a biomarker of 5-year survival. Radiology. 2013;266:177–84.CrossRefPubMed

15.

Kiryu S, Akai H, Nojima M, Hasegawa K, Shinkawa H, Kokudo N, et al. Impact of hepatocellular carcinoma heterogeneity on computed tomography as a prognostic indicator. Sci Rep. 2017;7:12689.CrossRefPubMedPubMedCentral

16.

Li H, Zhu Y, Burnside ES, Drukker K, Hoadley KA, Fan C, et al. MR imaging radiomics signatures for predicting the risk of breast cancer recurrence as given by research versions of MammaPrint, Oncotype DX, and PAM50 gene assays. Radiology. 2016;281:382–91.CrossRefPubMedPubMedCentral

17.

Goh V, Ganeshan B, Nathan P, Juttla JK, Vinayan A, Miles KA. Assessment of response to tyrosine kinase inhibitors in metastatic renal cell cancer: CT texture as a predictive biomarker. Radiology. 2011;261:165–71.CrossRefPubMed

18.

Le QV, Ranzato M, Monga R, Devin M, Chen K, Corrado GS, et al. Building high-level features using large scale unsupervised learning. International Conference on Machine Learning. 2012. http://icml.cc/2012/papers. Accessed 14 Dec 2017.

19.

Nakao T, Hanaoka S, Nomura Y, Sato I, Nemoto M, Miki S, et al. Deep neural network-based computer-assisted detection of cerebral aneurysms in MR angiography. J Magn Reson Imaging. 2017. https://doi.org/10.1002/jmri.25842.PubMed

20.

Gonzalez G, Ash SY, Vegas Sanchez-Ferrero G, Onieva Onieva J, Rahaghi FN, Ross JC, et al. Disease staging and prognosis in smokers using deep learning in chest computed tomography. Am J Respir Crit Care Med. 2018;197:193–203.CrossRefPubMed

21.

Nair V, Hinton G. Rectified linear units improve restricted Boltzmann machines. International Conference on Machine Learning. 2010. Accessed 14 Dec 2017.

22.

Srivastava N, Hinton GE, Krizhevsky A, Sutskever I, Salakhutdinov R. Dropout: a simple way to prevent neural networks from overfitting. J Mach Learn Res. 2014;15:1929–58.

23.

Ioffe S, Szegedy C. Batch normalization: accelerating deep network training by reducing internal covariate shift. Cornell University Library. 2015. http://arxiv.org/abs/1502.03167. Accessed 30 April 2017.

24.

Banerjee I, Crawley A, Bhethanabotla M, Daldrup-Link HE, Rubin DL. Transfer learning on fused multiparametric MR images for classifying histopathological subtypes of rhabdomyosarcoma. Comput Med Imaging Graph. 2017. https://doi.org/10.1016/j.compmedimag.2017.05.002.

25.

Duchi J, Hazan E, Singer Y. Adaptive subgradient methods for online learning and stochastic optimization. J Mach Learn Res. 2011;12:2121–59.

26.

Kingma DP, Ba JL. Adam: a method for stochastic optimization. Cornell University Library. 2014. http://arxiv.org/abs/1412.6980. Accessed 30 April 2017.

27.

Szegedy C, Liu W, Jia Y, Sermanet P, Reed S, Anguelov D, et al. Going deeper with convolutions. Cornell University Library. 2014. https://arxiv.org/abs/1409.4842. Accessed 14 Dec 2017.

28.

He K, Zhang X, Ren S, Sun J. Deep residual learning for image recognition. Cornell University Library. 2015. https://arxiv.org/abs/1512.03385. Accessed 14 Dec 2017.

29.

Korfiatis P, Kline TL, Lachance DH, Parney IF, Buckner JC, Erickson BJ. Residual deep convolutional neural network predicts MGMT methylation status. J Digit Imaging. 2017;30:622–8.CrossRefPubMed

30.

Andrearczyk V, Whelan PF. Using filter banks in convolutional neural networks for texture classification. Cornell University Library. 2016. https://arxiv.org/abs/1601.02919. Accessed 30 April 2017.

31.

Long J, Shelhamer E, Darrell T. Fully convolutional networks for semantic segmentation. 2015. http://ieeexplore.ieee.org/document/7298965/?reload=true. Accessed 14 Dec 2017.

32.

Ronneberger O, Fischer P, Brox T. U-Net: convolutional networks for biomedical image segmentation. Cornell University Library. 2015. https://arxiv.org/abs/1505.04597. Accessed 14 Dec 2017.

33.

Badrinarayanan V, Kendall A, Cipolla R. SegNet: a deep convolutional encoder-decoder architecture for image segmentation. Cornell University Library. 2015. https://arxiv.org/pdf/1511.00561. Accessed 14 Dec 2017.

34.

Mao X, Shen C, Yang Y. Image restoration using very deep convolutional encoder-decoder networks with symmetric skip connections. Cornell University Library. 2016. https://arxiv.org/abs/1603.09056. Accessed 14 Dec 2017.

35.

Lakhani P, Sundaram B. Deep learning at chest radiography: automated classification of pulmonary tuberculosis by using convolutional neural networks. Radiology. 2017;284:574–82.CrossRefPubMed

36.

Caruana R. Multitask learning. Mach Learn. 1997;28:41–75.CrossRef

37.

Bengio Y. Deep learning of representations for unsupervised and transfer learning. In: JMLR: Workshop and Conference Proceedings. 2012;17–37.

38.

Mohamed AA, Berg WA, Peng H, Luo Y, Jankowitz RC, Wu S. A deep learning method for classifying mammographic breast density categories. Med Phys. 2018;45:314–21.CrossRefPubMed

39.

Larson DB, Chen MC, Lungren MP, Halabi SS, Stence NV, Langlotz CP. Performance of a deep-learning neural network model in assessing skeletal maturity on pediatric hand radiographs. Radiology. 2017. https://doi.org/10.1148/radiol.2017170236.

40.

Cicero M, Bilbily A, Colak E, Dowdell T, Gray B, Perampaladas K, et al. Training and validating a deep convolutional neural network for computer-aided detection and classification of abnormalities on frontal chest radiographs. Invest Radiol. 2017;52:281–7.CrossRefPubMed

41.

Becker AS, Marcon M, Ghafoor S, Wurnig MC, Frauenfelder T, Boss A. Deep learning in mammography: diagnostic accuracy of a multipurpose image analysis software in the detection of breast cancer. Invest Radiol. 2017;52:434–40.CrossRefPubMed

42.

Prevedello LM, Erdal BS, Ryu JL, Little KJ, Demirer M, Qian S, et al. Automated critical test findings identification and online notification system using artificial intelligence in imaging. Radiology. 2017;285:923–31.CrossRefPubMed

43.

Wang X, Yang W, Weinreb J, Han J, Li Q, Kong X, et al. Searching for prostate cancer by fully automated magnetic resonance imaging classification: deep learning versus non-deep learning. Sci Rep. 2017;7:15415.CrossRefPubMedPubMedCentral

44.

Ghafoorian M, Karssemeijer N, Heskes T, Bergkamp M, Wissink J, Obels J, et al. Deep multi-scale location-aware 3D convolutional neural networks for automated detection of lacunes of presumed vascular origin. Neuroimage Clin. 2017;14:391–9.CrossRefPubMedPubMedCentral

45.

Jiang H, Ma H, Qian W, Gao M, Li Y. An automatic detection system of lung nodule based on multi-group patch-based deep learning network. IEEE J Biomed Health Inform. 2017. https://doi.org/10.1109/JBHI.2017.2725903.

46.

Parsons DW, Jones S, Zhang X, Lin JC, Leary RJ, Angenendt P, et al. An integrated genomic analysis of human glioblastoma multiforme. Science. 2008;321:1807–12.CrossRefPubMedPubMedCentral

47.

Yan H, Parsons DW, Jin G, McLendon R, Rasheed BA, Yuan W, et al. IDH1 and IDH2 mutations in gliomas. N Engl J Med. 2009;360:765–73.CrossRefPubMedPubMedCentral

48.

Hegi ME, Diserens AC, Gorlia T, Hamou MF, de Tribolet N, Weller M, et al. MGMT gene silencing and benefit from temozolomide in glioblastoma. N Engl J Med. 2005;352:997–1003.CrossRefPubMed

49.

Chang K, Bai HX, Zhou H, Su C, Bi WL, Agbodza E, et al. Residual convolutional neural network for determination of IDH status in low- and high-grade gliomas from MR imaging. Clin Cancer Res. 2017. https://doi.org/10.1158/1078-0432.CCR-17-2236.

50.

Chi J, Walia E, Babyn P, Wang J, Groot G, Eramian M. Thyroid nodule classification in ultrasound images by fine-tuning deep convolutional neural network. J Digit Imaging. 2017;30:477–86.CrossRefPubMedPubMedCentral

51.

Malempati S, Hawkins DS. Rhabdomyosarcoma: review of the Children’s Oncology Group (COG) Soft-Tissue Sarcoma Committee experience and rationale for current COG studies. Pediatr Blood Cancer. 2012;59:5–10.CrossRefPubMedPubMedCentral

52.

Anthimopoulos M, Christodoulidis S, Ebner L, Christe A, Mougiakakou S. Lung pattern classification for interstitial lung diseases using a deep convolutional neural network. IEEE Trans Med Imaging. 2016;35:1207–16.CrossRefPubMed

53.

Song Q, Zhao L, Luo X, Dou X. Using deep learning for classification of lung nodules on computed tomography images. J Healthc Eng. 2017;2017:8314740.CrossRefPubMedPubMedCentral

54.

Nibali A, He Z, Wollersheim D. Pulmonary nodule classification with deep residual networks. Int J Comput Assist Radiol Surg. 2017;12:1799–808.CrossRefPubMed

55.

Yasaka K, Akai H, Kunimatsu A, Abe O, Kiryu S. Liver fibrosis: deep convolutional neural network for staging by using gadoxetic acid-enhanced hepatobiliary phase MR images. Radiology. 2017. https://doi.org/10.1148/radiol.2017171928.

56.

Yasaka K, Akai H, Abe O, Kiryu S. Deep learning with convolutional neural network for differentiation of liver masses at dynamic contrast-enhanced CT: a preliminary study. Radiology. 2018;286:887–96.CrossRefPubMed

57.

Ben-Cohen A, Klang E, Diamant I, Rozendorn N, Raskin SP, Konen E, et al. CT image-based decision support system for categorization of Liver metastases into primary cancer sites: initial results. Acad Radiol. 2017;24:1501–9.CrossRefPubMed

58.

Lao J, Chen Y, Li ZC, Li Q, Zhang J, Liu J, et al. A deep learning-based radiomics model for prediction of survival in glioblastoma multiforme. Sci Rep. 2017;7:10353.CrossRefPubMedPubMedCentral

59.

Ibragimov B, Xing L. Segmentation of organs-at-risks in head and neck CT images using convolutional neural networks. Med Phys. 2017;44:547–57.CrossRefPubMedPubMedCentral

60.

Ibragimov B, Toesca D, Chang D, Koong A, Xing L. Combining deep learning with anatomy analysis for segmentation of portal vein for liver SBRT planning. Phys Med Biol. 2017;62:8943–58.PubMed

61.

Men K, Dai J, Li Y. Automatic segmentation of the clinical target volume and organs at risk in the planning CT for rectal cancer using deep dilated convolutional neural networks. Med Phys. 2017;44:6377–89.CrossRefPubMed

62.

Liu F, Jang H, Kijowski R, Bradshaw T, McMillan AB. Deep learning MR imaging-based attenuation correction for PET/MR imaging. Radiology. 2018;286:676–84.CrossRefPubMed

63.

Leynes AP, Yang J, Wiesinger F, Kaushik SS, Shanbhag DD, Seo Y, et al. Direct PseudoCT generation for pelvis PET/MRI attenuation correction using deep convolutional neural networks with multi-parametric MRI: zero echo-time and dixon deep pseudoCT (ZeDD-CT). J Nucl Med. 2017. https://doi.org/10.2967/jnumed.117.198051.PubMed

64.

Yasaka K, Katsura M, Akahane M, Sato J, Matsuda I, Ohtomo K. Model-based iterative reconstruction for reduction of radiation dose in abdominopelvic CT: comparison to adaptive statistical iterative reconstruction. Springerplus. 2013;2:209.CrossRefPubMedPubMedCentral

65.

Katsura M, Matsuda I, Akahane M, Sato J, Akai H, Yasaka K, et al. Model-based iterative reconstruction technique for radiation dose reduction in chest CT: comparison with the adaptive statistical iterative reconstruction technique. Eur Radiol. 2012;22:1613–23.CrossRefPubMed

66.

Pickhardt PJ, Lubner MG, Kim DH, Tang J, Ruma JA, del Rio AM, et al. Abdominal CT with model-based iterative reconstruction (MBIR): initial results of a prospective trial comparing ultralow-dose with standard-dose imaging. AJR Am J Roentgenol. 2012;199:1266–74.CrossRefPubMedPubMedCentral

67.

Yamada Y, Jinzaki M, Tanami Y, Shiomi E, Sugiura H, Abe T, et al. Model-based iterative reconstruction technique for ultralow-dose computed tomography of the lung: a pilot study. Invest Radiol. 2012;47:482–9.CrossRefPubMed

68.

Deak Z, Grimm JM, Treitl M, Geyer LL, Linsenmaier U, Korner M, et al. Filtered back projection, adaptive statistical iterative reconstruction, and a model-based iterative reconstruction in abdominal CT: an experimental clinical study. Radiology. 2013;266:197–206.CrossRefPubMed

69.

Yasaka K, Katsura M, Hanaoka S, Sato J, Ohtomo K. High-resolution CT with new model-based iterative reconstruction with resolution preference algorithm in evaluations of lung nodules: comparison with conventional model-based iterative reconstruction and adaptive statistical iterative reconstruction. Eur J Radiol. 2016;85:599–606.CrossRefPubMed

70.

Yasaka K, Furuta T, Kubo T, Maeda E, Katsura M, Sato J, et al. Full and hybrid iterative reconstruction to reduce artifacts in abdominal CT for patients scanned without arm elevation. Acta Radiol. 2017;58:1085–93.CrossRefPubMed

71.

Chen H, Zhang Y, Zhang W, Liao P, Li K, Zhou J, et al. Low-dose CT via convolutional neural network. Biomed Opt Express. 2017;8:679–94.CrossRefPubMedPubMedCentral

72.

Chen H, Zhang Y, Kalra MK, Lin F, Chen Y, Liao P, et al. Low-dose CT with a residual encoder-decoder convolutional neural network. IEEE Trans Med Imaging. 2017;36:2524–35.CrossRefPubMed

73.

Mackin D, Fave X, Zhang L, Fried D, Yang J, Taylor B, et al. Measuring computed tomography scanner variability of radiomics features. Invest Radiol. 2015;50:757–65.CrossRefPubMedPubMedCentral

74.

Yasaka K, Akai H, Mackin D, Court L, Moros E, Ohtomo K, et al. Precision of quantitative computed tomography texture analysis using image filtering: a phantom study for scanner variability. Medicine (Baltimore). 2017;96:e6993.CrossRefPubMedPubMedCentral

Title: Deep learning with convolutional neural network in radiology
Authors: Koichiro Yasaka
Hiroyuki Akai
Akira Kunimatsu
Shigeru Kiryu
Osamu Abe
Publication date: 01-04-2018
Publisher: Springer Japan
Published in: Japanese Journal of Radiology / Issue 4/2018
Print ISSN: 1867-1071
Electronic ISSN: 1867-108X
DOI: https://doi.org/10.1007/s11604-018-0726-3

At a glance: The ONWARDS insulin icodec trials

Springer Medicine

Deep learning with convolutional neural network in radiology

Abstract

At a glance: The ONWARDS insulin icodec trials

Springer Medicine

Abstract

Please log in to get access to this content

Other articles of this Issue 4/2018

Correction to: Global and Japanese regional variations in radiologist potential workload for computed tomography and magnetic resonance imaging examinations

Pathology and images of radiation-induced hepatitis: a review article

Utility of second-generation single-energy metal artifact reduction in helical lung computed tomography for patients with pulmonary arteriovenous malformation after coil embolization

Correction to: The sternalis muscle: radiologic findings on MDCT

Global and Japanese regional variations in radiologist potential workload for computed tomography and magnetic resonance imaging examinations