Top

European Journal of Nuclear Medicine and Molecular Imaging

Published in:

Open Access 01-10-2021 | Glioma | Original Article

Real-time intraoperative glioma diagnosis using fluorescence imaging and deep convolutional neural networks

Authors: Biluo Shen, Zhe Zhang, Xiaojing Shi, Caiguang Cao, Zeyu Zhang, Zhenhua Hu, Nan Ji, Jie Tian

Published in: European Journal of Nuclear Medicine and Molecular Imaging | Issue 11/2021

Abstract

Purpose

Surgery is the predominant treatment modality of human glioma but suffers difficulty on clearly identifying tumor boundaries in clinic. Conventional practice involves neurosurgeon’s visual evaluation and intraoperative histological examination of dissected tissues using frozen section, which is time-consuming and complex. The aim of this study was to develop fluorescent imaging coupled with artificial intelligence technique to quickly and accurately determine glioma in real-time during surgery.

Methods

Glioma patients (N = 23) were enrolled and injected with indocyanine green for fluorescence image–guided surgery. Tissue samples (N = 1874) were harvested from surgery of these patients, and the second near-infrared window (NIR-II, 1000–1700 nm) fluorescence images were obtained. Deep convolutional neural networks (CNNs) combined with NIR-II fluorescence imaging (named as FL-CNN) were explored to automatically provide pathological diagnosis of glioma in situ in real-time during patient surgery. The pathological examination results were used as the gold standard.

Results

The developed FL-CNN achieved the area under the curve (AUC) of 0.945. Comparing to neurosurgeons’ judgment, with the same level of specificity >80%, FL-CNN achieved a much higher sensitivity (93.8% versus 82.0%, P < 0.001) with zero time overhead. Further experiments demonstrated that FL-CNN corrected >70% of the errors made by neurosurgeons. FL-CNN was also able to rapidly predict grade and Ki-67 level (AUC 0.810 and 0.625) of tumor specimens intraoperatively.

Conclusion

Our study demonstrates that deep CNNs are better at capturing important information from fluorescence images than surgeons’ evaluation during patient surgery. FL-CNN is highly promising to provide pathological diagnosis intraoperatively and assist neurosurgeons to obtain maximum resection safely.

Trial registration

ChiCTR ChiCTR2000029402. Registered 29 January 2020, retrospectively registered

Available only for authorised users

Lapointe S, Perry A, Butowski NA. Primary brain tumours in adults. Lancet. 2018;392:432–46.CrossRef

Ostrom QT, Gittleman H, Liao P, Vecchione-Koval T, Wolinsky Y, Kruchko C, et al. CBTRUS statistical report: primary brain and other central nervous system tumors diagnosed in the United States in 2010–2014. Neuro-Oncol. 2017;19:v1–v88.CrossRef

Hollon TC, Pandian B, Adapa AR, Urias E, Save AV, Khalsa SSS, et al. Near real-time intraoperative brain tumor diagnosis using stimulated Raman histology and deep neural networks. Nat Med. 2020;26:52–8. https://doi.org/10.1038/s41591-019-0715-9.CrossRefPubMedPubMedCentral

Campanella G, Hanna MG, Geneslaw L, Miraflor A, Werneck Krauss Silva V, Busam KJ, et al. Clinical-grade computational pathology using weakly supervised deep learning on whole slide images. Nat Med. 2019;25:1301–9. https://doi.org/10.1038/s41591-019-0508-1.CrossRefPubMedPubMedCentral

Kather JN, Pearson AT, Halama N, Jäger D, Krause J, Loosen SH, et al. Deep learning can predict microsatellite instability directly from histology in gastrointestinal cancer. Nat Med. 2019;25:1054–6. https://doi.org/10.1038/s41591-019-0462-y.CrossRefPubMedPubMedCentral

Coudray N, Ocampo PS, Sakellaropoulos T, Narula N, Snuderl M, Fenyö D, et al. Classification and mutation prediction from non-small cell lung cancer histopathology images using deep learning. Nat Med. 2018;24:1559–67. https://doi.org/10.1038/s41591-018-0177-5.CrossRefPubMed

Hannun AY, Rajpurkar P, Haghpanahi M, Tison GH, Bourn C, Turakhia MP, et al. Cardiologist-level arrhythmia detection and classification in ambulatory electrocardiograms using a deep neural network. Nat Med. 2019;25:65–9. https://doi.org/10.1038/s41591-018-0268-3.CrossRefPubMedPubMedCentral

Lee H, Yune S, Mansouri M, Kim M, Tajmir SH, Guerrier CE, et al. An explainable deep-learning algorithm for the detection of acute intracranial haemorrhage from small datasets. Nat Biomed Eng. 2019;3:173–82. https://doi.org/10.1038/s41551-018-0324-9.CrossRefPubMed

Dong D, Tang L, Li ZY, Fang MJ, Gao JB, Shan XH, Ying XJ, Sun YS, Fu J, Wang XX, Li LM, Li ZH, Zhang DF, Zhang Y, Li ZM, Shan F, Bu ZD, Tian J, Ji JF. Development and validation of an individualized nomogram to identify occult peritoneal metastasis in patients with advanced gastric cancer. Ann Oncol. 2019;30(3):431–438. https://doi.org/10.1093/annonc/mdz001.

10.

Dong D, Fang MJ, Tang L, Shan XH, Gao JB, Giganti F, Wang RP, Chen X, Wang XX, Palumbo D, Fu J, Li WC, Li J, Zhong LZ, De Cobelli F, Ji JF, Liu ZY, Tian J. Deep learning radiomic nomogram can predict the number of lymph node metastasis in locally advanced gastric cancer: an international multicenter study. Ann Oncol. 2020;31(7):912–920. https://doi.org/10.1016/j.annonc.2020.04.003.

11.

Yang Y, Yan LF, Zhang X, Han Y, Nan HY, Hu YC, et al. Glioma grading on conventional MR images: a deep learning study with transfer learning. Front Neurosci. 2018;12:804. https://doi.org/10.3389/fnins.2018.00804.CrossRefPubMedPubMedCentral

12.

Li Z, Wang Y, Yu J, Guo Y, Cao W. Deep learning based radiomics (DLR) and its usage in noninvasive IDH1 prediction for low grade glioma. Sci Rep. 2017;7:5467. https://doi.org/10.1038/s41598-017-05848-2.CrossRefPubMedPubMedCentral

13.

Yonekura A, Kawanaka H, Prasath VBS, Aronow BJ, Takase H. Automatic disease stage classification of glioblastoma multiforme histopathological images using deep convolutional neural network. Biomed Eng Lett. 2018;8:321–7. https://doi.org/10.1007/s13534-018-0077-0.CrossRefPubMedPubMedCentral

14.

Ker J, Bai Y, Lee HY, Rao J, Wang L. Automated brain histology classification using machine learning. J Clin Neurosci. 2019;66:239–45. https://doi.org/10.1016/j.jocn.2019.05.019.CrossRefPubMed

15.

Mobadersany P, Yousefi S, Amgad M, Gutman DA, Barnholtz-Sloan JS, Velázquez Vega JE, et al. Predicting cancer outcomes from histology and genomics using convolutional networks. Proc Natl Acad Sci U S A. 2018;115:E2970–E9. https://doi.org/10.1073/pnas.1717139115.CrossRefPubMedPubMedCentral

16.

Hu Z, Fang C, Li B, Zhang Z, Cao C, Cai M, et al. First-in-human liver-tumour surgery guided by multispectral fluorescence imaging in the visible and near-infrared-I/II windows. Nat Biomed Eng. 2020;4:259–71. https://doi.org/10.1038/s41551-019-0494-0.CrossRefPubMed

17.

Qin C, Zhong J, Hu Z, Yang X, Tian J. Recent advances in Cerenkov luminescence and tomography imaging. IEEE J Sel Top Quantum Electron. 2011;18:1084–93. https://doi.org/10.1109/JSTQE.2011.2161757.CrossRef

18.

Hongbo G, Xiaowei H, Muhan L, Zeyu Z, Zhenhua H, Jie T. Weight multispectral reconstruction strategy for enhanced reconstruction accuracy and stability with Cerenkov luminescence tomography. IEEE Trans Med Imaging. 2017;36:1337–46. https://doi.org/10.1109/tmi.2017.2658661.CrossRef

19.

Hu Z, Zhao M, Qu Y, Zhang X, Zhang M, Liu M, et al. In vivo 3-dimensional radiopharmaceutical-excited fluorescence tomography. J Nucl Med. 2017;58:169–74. https://doi.org/10.2967/jnumed.116.180596.CrossRefPubMed

20.

Liu M, Zheng S, Zhang X, Guo H, Shi X, Kang X, et al. Cerenkov luminescence imaging on evaluation of early response to chemotherapy of drug-resistant gastric cancer. Nanomedicine. 2018;14:205–13. https://doi.org/10.1016/j.nano.2017.10.001.CrossRefPubMed

21.

Song T, Liu X, Qu Y, Liu H, Bao C, Leng C, Hu Z, Wang K, Tian J. A Novel Endoscopic Cerenkov Luminescence Imaging System for Intraoperative Surgical Navigation. Mol Imaging. 2015;14:443–9.

22.

Hu Z, Chi C, Liu M, Guo H, Zhang Z, Zeng C, Ye J, Wang J, Tian J, Yang W, Xu W. Nanoparticle-mediated radiopharmaceutical-excited fluorescence molecular imaging allows precise image-guided tumor-removal surgery. Nanomedicine. 2017;13(4):1323–1331. https://doi.org/10.1016/j.nano.2017.01.005.

23.

Lee JY, Thawani JP, Pierce J, Zeh R, Martinez-Lage M, Chanin M, et al. Intraoperative near-infrared optical imaging can localize gadolinium-enhancing gliomas during surgery. Neurosurgery. 2016;79:856–71.CrossRef

24.

Spyratos F, Ferrero-Poüs M, Trassard M, Hacène K, Phillips E, Tubiana-Hulin M, et al. Correlation between MIB-1 and other proliferation markers: clinical implications of the MIB-1 cutoff value. Cancer. 2002;94:2151–9. https://doi.org/10.1002/cncr.10458.CrossRefPubMed

25.

Paszke A, Gross S, Massa F, Lerer A, Bradbury J, Chanan G, et al. PyTorch: an imperative style, high-performance deep learning library. 31st Conference on Neural Information Processing Systems; 2017.

26.

Robin X, Turck N, Hainard A, Tiberti N, Lisacek F, Sanchez JC, et al. pROC: an open-source package for R and S+ to analyze and compare ROC curves. BMC Bioinformatics. 2011;12:77. https://doi.org/10.1186/1471-2105-12-77.CrossRefPubMedPubMedCentral

27.

Pedregosa F, Varoquaux G, Gramfort A, Michel V, Thirion B, Grisel O, et al. Scikit-learn: machine learning in Python. J Mach Learn Res. 2011;12:2825–30.

28.

Hunter JD. Matplotlib: a 2D graphics environment. Comput Sci Eng. 2007;9:90–5. https://doi.org/10.1109/MCSE.2007.55.CrossRef

29.

Tan M, Le Q. EfficientNet: rethinking model scaling for convolutional neural networks. Proceedings of the 36th International Conference on Machine Learning; 2019. p. 6105–14.

30.

Glorot X, Bengio Y. Understanding the difficulty of training deep feedforward neural networks. Proceedings of the 33th International Conference on Artificial Intelligence and Statistics; 2010. p. 249–56.

31.

Srivastava N, Hinton GE, Krizhevsky A, Sutskever I, Salakhutdinov R. Dropout: a simple way to prevent neural networks from overfitting. J Mach Learn Res. 2014;15:1929–58.

32.

Loshchilov I, Hutter F. SGDR: stochastic gradient descent with warm restarts. International Conference on Learning Representations; 2017.

33.

Laws ER, Parney IF, Huang W, Anderson F, Morris AM, Asher A, et al. Survival following surgery and prognostic factors for recently diagnosed malignant glioma: data from the Glioma Outcomes Project. J Neurosurg. 2003;99:467–73. https://doi.org/10.3171/jns.2003.99.3.0467.CrossRefPubMed

34.

Carr JA, Franke D, Caram JR, Perkinson CF, Saif M, Askoxylakis V, et al. Shortwave infrared fluorescence imaging with the clinically approved near-infrared dye indocyanine green. Proc Natl Acad Sci U S A. 2018;115:4465–70. https://doi.org/10.1073/pnas.1718917115.CrossRefPubMedPubMedCentral

Title: Real-time intraoperative glioma diagnosis using fluorescence imaging and deep convolutional neural networks
Authors: Biluo Shen
Zhe Zhang
Xiaojing Shi
Caiguang Cao
Zeyu Zhang
Zhenhua Hu
Nan Ji
Jie Tian
Publication date: 01-10-2021
Publisher: Springer Berlin Heidelberg
Keywords: Glioma
Glioma
Glioma
Published in: European Journal of Nuclear Medicine and Molecular Imaging / Issue 11/2021
Print ISSN: 1619-7070
Electronic ISSN: 1619-7089
DOI: https://doi.org/10.1007/s00259-021-05326-y

At a glance: The STEP trials

Springer Medicine

Real-time intraoperative glioma diagnosis using fluorescence imaging and deep convolutional neural networks

Abstract

Purpose

Methods

Results

Conclusion

Trial registration

At a glance: The STEP trials

Springer Medicine

Abstract

Purpose

Methods

Results

Conclusion

Trial registration

Please log in to get access to this content

Other articles of this Issue 11/2021

Segmentation of white matter hyperintensities on 18F-FDG PET/CT images with a generative adversarial network

[18F]-C-SNAT4: an improved caspase-3-sensitive nanoaggregation PET tracer for imaging of tumor responses to chemo- and immunotherapies

A novel deep-learning–based approach for automatic reorientation of 3D cardiac SPECT images

Specific and non-specific binding of a tracer for the translocator-specific protein in schizophrenia: an [11C]-PBR28 blocking study

Convolutional neural networks for PET functional volume fully automatic segmentation: development and validation in a multi-center setting

Multivariate radiomics models based on 18F-FDG hybrid PET/MRI for distinguishing between Parkinson’s disease and multiple system atrophy