Top

European Journal of Nuclear Medicine and Molecular Imaging

Published in:

Open Access 22-12-2023 | Original Article

Automated identification of uncertain cases in deep learning-based classification of dopamine transporter SPECT to improve clinical utility and acceptance

Authors: Thomas Budenkotte, Ivayla Apostolova, Roland Opfer, Julia Krüger, Susanne Klutmann, Ralph Buchert

Published in: European Journal of Nuclear Medicine and Molecular Imaging | Issue 5/2024

Abstract

Purpose

Deep convolutional neural networks (CNN) are promising for automatic classification of dopamine transporter (DAT)-SPECT images. Reporting the certainty of CNN-based decisions is highly desired to flag cases that might be misclassified and, therefore, require particularly careful inspection by the user. The aim of the current study was to design and validate a CNN-based system for the identification of uncertain cases.

Methods

A network ensemble (NE) combining five CNNs was trained for binary classification of [¹²³I]FP-CIT DAT-SPECT images as “normal” or “neurodegeneration-typical reduction” with high accuracy (NE for classification, NEfC). An uncertainty detection module (UDM) was obtained by combining two additional NE, one trained for detection of “reduced” DAT-SPECT with high sensitivity, the other with high specificity. A case was considered “uncertain” if the “high sensitivity” NE and the “high specificity” NE disagreed. An internal “development” dataset of 1740 clinical DAT-SPECT images was used for training (n = 1250) and testing (n = 490). Two independent datasets with different image characteristics were used for testing only (n = 640, 645). Three established approaches for uncertainty detection were used for comparison (sigmoid, dropout, model averaging).

Results

In the test data from the development dataset, the NEfC achieved 98.0% accuracy. 4.3% of all test cases were flagged as “uncertain” by the UDM: 2.5% of the correctly classified cases and 90% of the misclassified cases. NEfC accuracy among “certain” cases was 99.8%. The three comparison methods were less effective in labelling misclassified cases as “uncertain” (40–80%). These findings were confirmed in both additional test datasets.

Conclusion

The UDM allows reliable identification of uncertain [¹²³I]FP-CIT SPECT with high risk of misclassification. We recommend that automatic classification of [¹²³I]FP-CIT SPECT images is combined with an UDM to improve clinical utility and acceptance. The proposed UDM method (“high sensitivity versus high specificity”) might be useful also for DAT imaging with other ligands and for other binary classification tasks.

Available only for authorised users

Acton PD, Newberg A. Artificial neural network classifier for the diagnosis of Parkinson’s disease using [Tc-99m] TRODAT-1 and SPECT. Phys Med Biol. 2006;51:3057–66. https://doi.org/10.1088/0031-9155/51/12/004.CrossRefPubMed

Chien CY, Hsu SW, Lee TL, Sung PS, Lin CC. Using artificial neural network to discriminate Parkinson’s disease from other parkinsonisms by focusing on putamen of dopamine transporter SPECT images. Biomedicines. 2021;9:ARTN 12. https://doi.org/10.3390/biomedicines9010012.CrossRef

Choi H, Ha S, Im HJ, Paek SH, Lee DS. Refining diagnosis of Parkinson’s disease with deep learning-based interpretation of dopamine transporter imaging. Neuroimage Clin. 2017;16:586–94. https://doi.org/10.1016/j.nicl.2017.09.010.CrossRefPubMedPubMedCentral

Hathaliya J, Parekh R, Patel N, Gupta R, Tanwar S, Alqahtani F, et al. Convolutional neural network-based Parkinson disease classification using SPECT imaging data. Mathematics-Basel. 2022;10:ARTN 2566. https://doi.org/10.3390/math10152566.CrossRef

Jothi S, Anita S, Sivakumar S. Modified exigent features block in JAN Net for analysing SPECT scan images to diagnose early-stage Parkinson’s disease. Curr Med Imaging. 2023. https://doi.org/10.2174/1573405620666230605092654.CrossRefPubMed

Kim DH, Wit H, Thurston M. Artificial intelligence in the diagnosis of Parkinson’s disease from ioflupane-123 single-photon emission computed tomography dopamine transporter scans using transfer learning. Nucl Med Commun. 2018;39:887–93. https://doi.org/10.1097/MNM.0000000000000890.CrossRefPubMed

Kurmi A, Biswas S, Sen S, Sinitca A, Kaplun D, Sarkar R. An Ensemble of CNN models for Parkinson’s disease detection using DaTscan images. Diagnostics. 2022;12:ARTN 1173. https://doi.org/10.3390/diagnostics12051173.CrossRef

Magesh PR, Myloth RD, Tom RJ. An explainable machine learning model for early detection of Parkinson’s disease using LIME on DaTSCAN imagery. Comput Biol Med. 2020;126:104041. https://doi.org/10.1016/j.compbiomed.2020.104041.CrossRefPubMed

Martinez-Murcia FJ, Gorriz JM, Ramirez J, Ortiz A. Convolutional neural networks for neuroimaging in Parkinson’s disease: is preprocessing needed? Int J Neural Syst. 2018;28:Artn 1850035. https://doi.org/10.1142/S0129065718500351.CrossRef

10.

Nazari M, Kluge A, Apostolova I, Klutmann S, Kimiaei S, Schroeder M, et al. Data-driven identification of diagnostically useful extrastriatal signal in dopamine transporter SPECT using explainable AI. Sci Rep. 2021;11:22932. https://doi.org/10.1038/s41598-021-02385-x.CrossRefPubMedPubMedCentral

11.

Nazari M, Kluge A, Apostolova I, Klutmann S, Kimiaei S, Schroeder M, et al. Explainable AI to improve acceptance of convolutional neural networks for automatic classification of dopamine transporter SPECT in the diagnosis of clinically uncertain parkinsonian syndromes. Eur J Nucl Med Mol Imaging. 2022;49:1176–86. https://doi.org/10.1007/s00259-021-05569-9.CrossRefPubMed

12.

Ortiz A, Munilla J, Martinez-Ibanez M, Gorriz JM, Ramirez J, Salas-Gonzalez D. Parkinson’s disease detection using isosurfaces-based features and convolutional neural networks. Front Neuroinform. 2019;13:ARTN 48. https://doi.org/10.3389/fninf.2019.00048.CrossRef

13.

Ozsahin I, Sekeroglu B, Pwavodi PC, Mok GSP. High-accuracy automated diagnosis of Parkinson’s disease. Current Medical Imaging. 2020;16:688–94. https://doi.org/10.2174/1573405615666190620113607.CrossRefPubMed

14.

Palumbo B, Fravolini ML, Nuvoli S, Spanu A, Paulus KS, Schillaci O, et al. Comparison of two neural network classifiers in the differential diagnosis of essential tremor and Parkinson’s disease by I-123-FP-CIT brain SPECT. Eur J Nucl Med Mol. 2010;I(37):2146–53. https://doi.org/10.1007/s00259-010-1481-6.CrossRef

15.

Pianpanit T, Lolak S, Sawangjai P, Sudhawiyangkul T, Wilaiprasitporn T. Parkinson’s disease recognition using SPECT image and interpretable AI: a tutorial. Ieee Sens J. 2021;21:22304–16. https://doi.org/10.1109/Jsen.2021.3077949.CrossRef

16.

Piccardo A, Cappuccio R, Bottoni G, Cecchin D, Mazzella L, Cirone A, et al. The role of the deep convolutional neural network as an aid to interpreting brain [F-18]DOPA PET/CT in the diagnosis of Parkinson’s disease. Eur Radiol. 2021;31:7003–11. https://doi.org/10.1007/s00330-021-07779-z.CrossRefPubMed

17.

Thakur M, Kuresan H, Dhanalakshmi S, Lai KW, Wu X. Soft attention based DenseNet model for Parkinson’s disease classification using SPECT images. Front Aging Neurosci. 2022;14:ARTN 908143. https://doi.org/10.3389/fnagi.2022.908143.CrossRef

18.

Wenzel M, Milletari F, Krueger J, Lange C, Schenk M, Apostolova I, et al. Automatic classification of dopamine transporter SPECT: deep convolutional neural networks can be trained to be robust with respect to variable image characteristics. Eur J Nucl Med Mol. 2019;I(46):2800–11. https://doi.org/10.1007/s00259-019-04502-5.CrossRef

19.

Zhao Y, Wu P, Wu JJ, Brendel M, Lu JY, Ge JJ, et al. Decoding the dopamine transporter imaging for the differential diagnosis of parkinsonism using deep learning. Eur J Nucl Med Mol. 2022;I(49):2798–811. https://doi.org/10.1007/s00259-022-05804-x.CrossRef

20.

Verger A, Grimaldi S, Ribeiro MJ, Frismand S, Guedj E. Single photon emission computed tomography/positron emission tomography molecular imaging for parkinsonism: a fast-developing field. Ann Neurol. 2021;90:711–9. https://doi.org/10.1002/ana.26187.CrossRefPubMedPubMedCentral

21.

Makinen E, Joutsa J, Johansson J, Maki M, Seppanen M, Kaasinen V. Visual versus automated analysis of [I-123]FP-CIT SPECT scans in parkinsonism. J Neural Transm. 2016;123:1309–18. https://doi.org/10.1007/s00702-016-1586-6.CrossRefPubMed

22.

Albert NL, Unterrainer M, Diemling M, Xiong GM, Bartenstein P, Koch W, et al. Implementation of the European multicentre database of healthy controls for [I-123]FP-CIT SPECT increases diagnostic accuracy in patients with clinically uncertain parkinsonian syndromes. Eur J Nucl Med Mol. 2016;I(43):1315–22. https://doi.org/10.1007/s00259-015-3304-2.CrossRef

23.

Ulmer D, Cinà G. Know your limits: uncertainty estimation with ReLU classifiers fails at reliable OOD detection (version 4). arXiv. 2021. https://doi.org/10.48550/arXiv.2012.05329.

24.

Guo C, Pleiss G, Sun Y, Weinberger KQ. On calibration of modern neural networks (version 2). arXiv 2017. https://doi.org/10.48550/arXiv.1706.04599.

25.

Karimi D, Gholipour A. Improving calibration and out-of-distribution detection in medical image segmentation with convolutional neural networks (version 3). arXiv 2022. https://doi.org/10.48550/arXiv.2004.06569.

26.

Schiebler T, Apostolova I, Mathies FL, Lange C, Klutmann S, Buchert R. No impact of attenuation and scatter correction on the interpretation of dopamine transporter SPECT in patients with clinically uncertain parkinsonian syndrome. Eur J Nucl Med Mol I. 2023. https://doi.org/10.1007/s00259-023-06293-2.CrossRef

27.

Darcourt J, Booij J, Tatsch K, Varrone A, Borght TV, Kapucu OL, et al. EANM procedure guidelines for brain neurotransmission SPECT using I-123-labelled dopamine transporter ligands, version 2. Eur J Nucl Med Mol. 2010;I(37):443–50. https://doi.org/10.1007/s00259-009-1267-x.CrossRef

28.

Djang DS, Janssen MJ, Bohnen N, Booij J, Henderson TA, Herholz K, et al. SNM practice guideline for dopamine transporter imaging with 123I-ioflupane SPECT 1.0. J Nucl Med. 2012;53:154–63. https://doi.org/10.2967/jnumed.111.100784.CrossRefPubMed

29.

Hudson HM, Larkin RS. Accelerated image-reconstruction using ordered subsets of projection data. Ieee T Med Imaging. 1994;13:601–9. https://doi.org/10.1109/42.363108.CrossRef

30.

Diemling M. HERMES camera correction for the ENCDAT database using DaTscan (White Paper). Hermes Medical Solution. 2021.

31.

Sohlberg AO, Kajaste MT. Fast Monte Carlo-simulator with full collimator and detector response modelling for SPECT. Ann Nucl Med. 2012;26:92–8. https://doi.org/10.1007/s12149-011-0550-7.CrossRefPubMed

32.

Hermes Medical Solutions. HybridRecon (White Paper).

33.

Kangasmaa TS, Constable C, Hippelainen E, Sohlberg AO. Multicenter evaluation of single-photon emission computed tomography quantification with third-party reconstruction software. Nucl Med Commun. 2016;37:983–7. https://doi.org/10.1097/Mnm.0000000000000538.CrossRefPubMed

34.

Tossici-Bolt L, Dickson JC, Sera T, de Nijs R, Bagnara MC, Jonsson C, et al. Calibration of gamma camera systems for a multicentre European I-123-FP-CIT SPECT normal database. Eur J Nucl Med Mol. 2011;I(38):1529–40. https://doi.org/10.1007/s00259-011-1801-5.CrossRef

35.

Dickson JC, Tossici-Bolt L, Sera T, Erlandsson K, Varrone A, Tatsch K, et al. The impact of reconstruction method on the quantification of DaTSCAN images. Eur J Nucl Med Mol. 2010;I(37):23–35. https://doi.org/10.1007/s00259-009-1212-z.CrossRef

36.

Varrone A, Dickson JC, Tossici-Bolt L, Sera T, Asenbaum S, Booij J, et al. European multicentre database of healthy controls for [123I]FP-CIT SPECT (ENC-DAT): age-related effects, gender differences and evaluation of different methods of analysis. Eur J Nucl Med Mol Imaging. 2013;40:213–27. https://doi.org/10.1007/s00259-012-2276-8.CrossRefPubMed

37.

Tossici-Bolt L, Dickson JC, Sera T, Booij J, Asenbaun-Nan S, Bagnara MC, et al. [(123)I]FP-CIT ENC-DAT normal database: the impact of the reconstruction and quantification methods. EJNMMI Phys. 2017;4:8. https://doi.org/10.1186/s40658-017-0175-6.CrossRefPubMedPubMedCentral

38.

Dickson JC, Tossici-Bolt L, Sera T, de Nijs R, Booij J, Bagnara MC, et al. Proposal for the standardisation of multi-centre trials in nuclear medicine imaging: prerequisites for a European I-123-FP-CIT SPECT database. Eur J Nucl Med Mol. 2012;I(39):188–97. https://doi.org/10.1007/s00259-011-1884-z.CrossRef

39.

Mathies F, Apostolova I, Dierck L, Jacobi J, Kuen K, Sauer M, et al. Multiple-pinhole collimators improve intra- and between-rater agreement and the certainty of the visual interpretation in dopamine transporter SPECT. Ejnmmi Res. 2022;12:ARTN 51. https://doi.org/10.1186/s13550-022-00923-w.CrossRef

40.

Tecklenburg K, Forgacs A, Apostolova I, Lehnert W, Klutmann S, Csirik J, et al. Performance evaluation of a novel multi-pinhole collimator for dopamine transporter SPECT. Phys Med Biol. 2020;65:ARTN 165015. https://doi.org/10.1088/1361-6560/ab9067.CrossRef

41.

Magdics M, Szirmay-Kalos L, Szlavecz Á, Hesz G, Benyó B, Cserkaszky Á, et al. TeraTomo project: a fully 3D GPU based reconstruction code for exploiting the imaging capability of the NanoPET™/CT system. Mol Imaging Biol. 2010;12(2):S1407.

42.

Parkinson Progression Marker I. The Parkinson Progression Marker Initiative (PPMI). Prog Neurobiol. 2011;95:629–35. https://doi.org/10.1016/j.pneurobio.2011.09.005.

43.

Apostolova I, Schiebler T, Lange C, Mathies FL, Lehnert W, Klutmann S, et al. Stereotactical normalization with multiple templates representative of normal and Parkinson-typical reduction of striatal uptake improves the discriminative power of automatic semi-quantitative analysis in dopamine transporter SPECT. EJNMMI Phys. 2023;10:25. https://doi.org/10.1186/s40658-023-00544-9.CrossRefPubMedPubMedCentral

44.

Kupitz D, Apostolova I, Lange C, Ulrich G, Amthauer H, Brenner W, et al. Global scaling for semi-quantitative analysis in FP-CIT SPECT. Nuklearmedizin. 2014;53:234–41. https://doi.org/10.3413/Nukmed-0659-14-04.CrossRefPubMed

45.

Buchert R, Berding G, Wilke F, Martin B, von Borczyskowski D, Mester J, et al. IBZM tool: a fully automated expert system for the evaluation of IBZM SPECT studies. Eur J Nucl Med Mol Imaging. 2006;33:1073–83. https://doi.org/10.1007/s00259-006-0067-9.CrossRefPubMed

46.

Ioffe S, Szegedy C. Batch normalization: accelerating deep network training by reducing internal covariate shift. Pr Mach Learn Res. 2015;37:448–56.

47.

He K, Zhang X, Ren S, Sun J. Deep residual learning for image recognition. IEEE Conference on Computer Vision and Pattern Recognition. Las Vegas, NV, USA; 2016: 770–8.

48.

He KM, Zhang XY, Ren SQ, Sun J. Delving deep into rectifiers: surpassing human-level performance on imagenet classification. arXiv. 2015. https://doi.org/10.48550/arXiv.1502.01852

49.

Sutskever I, Martens J, Dahl G, Hinton G. On the importance of initialization and momentum in deep learning. International Conference on Machine Learning; 2013: 1139–47.

50.

Isensee F, Jaeger PF, Kohl SAA, Petersen J, Maier-Hein KH. nnU-Net: a self-configuring method for deep learning-based biomedical image segmentation. Nat Methods. 2021;18:203–11. https://doi.org/10.1038/s41592-020-01008-z.CrossRefPubMed

51.

Buddenkotte T, Escudero Sanchez L, Crispin-Ortuzar M, Woitek R, McCague C, Brenton JD, et al. Calibrating ensembles for scalable uncertainty quantification in deep learning-based medical image segmentation. Comput Biol Med. 2023;163: 107096. https://doi.org/10.1016/j.compbiomed.2023.107096.CrossRefPubMed

52.

Yeung M, Sala E, Schonlieb CB, Rundo L. Unified focal loss: generalising dice and cross entropy-based losses to handle class imbalanced medical image segmentation. Comput Med Imaging Graph. 2022;95:102026. https://doi.org/10.1016/j.compmedimag.2021.102026.CrossRefPubMedPubMedCentral

53.

Apostolova I, Taleb DS, Lipp A, Galazky I, Kupitz D, Lange C, et al. Utility of follow-up dopamine transporter SPECT with 123I-FP-CIT in the diagnostic workup of patients with clinically uncertain parkinsonian syndrome. Clin Nucl Med. 2017;42:589–94. https://doi.org/10.1097/RLU.0000000000001696.CrossRefPubMed

54.

Morbelli S, Esposito G, Arbizu J, Barthel H, Boellaard R, Bohnen NI, et al. EANM practice guideline/SNMMI procedure standard for dopaminergic imaging in Parkinsonian syndromes 1.0. Eur J Nucl Med Mol Imaging. 2020;47:1885–912. https://doi.org/10.1007/s00259-020-04817-8.CrossRefPubMedPubMedCentral

55.

Funke E, Kupsch A, Buchert R, Brenner W, Plotkin M. Impact of subcortical white matter lesions on dopamine transporter SPECT. J Neural Transm (Vienna). 2013;120:1053–60. https://doi.org/10.1007/s00702-013-0977-1.CrossRefPubMed

56.

Kordower JH, Olanow CW, Dodiya HB, Chu Y, Beach TG, Adler CH, et al. Disease duration and the integrity of the nigrostriatal system in Parkinson’s disease. Brain. 2013;136:2419–31. https://doi.org/10.1093/brain/awt192.CrossRefPubMedPubMedCentral

57.

Walker Z, Costa DC, Walker RW, Lee L, Livingston G, Jaros E, et al. Striatal dopamine transporter in dementia with Lewy bodies and Parkinson disease: a comparison. Neurology. 2004;62:1568–72. https://doi.org/10.1212/01.wnl.0000123248.39847.1d.CrossRefPubMed

58.

Booij J, Kemp P. Dopamine transporter imaging with [(123)I]FP-CIT SPECT: potential effects of drugs. Eur J Nucl Med Mol Imaging. 2008;35:424–38. https://doi.org/10.1007/s00259-007-0621-0.CrossRefPubMed

59.

Booij J, Tissingh G, Boer GJ, Speelman JD, Stoof JC, Janssen AGM, et al. [I-123]FP-CIT SPECT shows a pronounced decline of striatal dopamine transporter labelling in early and advanced Parkinson’s disease. J Neurol Neurosur Ps. 1997;62:133–40. https://doi.org/10.1136/jnnp.62.2.133.CrossRef

60.

Scherfler C, Seppi K, Mair KJ, Donnemiller E, Virgolini I, Wenning GK, et al. Left hemispheric predominance of nigrostriatal dysfunction in Parkinson’s disease. Brain. 2012;135:3348–54. https://doi.org/10.1093/brain/aws253.CrossRefPubMed

Title: Automated identification of uncertain cases in deep learning-based classification of dopamine transporter SPECT to improve clinical utility and acceptance
Authors: Thomas Budenkotte
Ivayla Apostolova
Roland Opfer
Julia Krüger
Susanne Klutmann
Ralph Buchert
Publication date: 22-12-2023
Publisher: Springer Berlin Heidelberg
Published in: European Journal of Nuclear Medicine and Molecular Imaging / Issue 5/2024
Print ISSN: 1619-7070
Electronic ISSN: 1619-7089
DOI: https://doi.org/10.1007/s00259-023-06566-w

At a glance: The STEP trials

Springer Medicine

Automated identification of uncertain cases in deep learning-based classification of dopamine transporter SPECT to improve clinical utility and acceptance

Abstract

Purpose

Methods

Results

Conclusion

At a glance: The STEP trials

Springer Medicine

Abstract

Purpose

Methods

Results

Conclusion

Please log in to get access to this content

Other articles of this Issue 5/2024

Letter to the Editor: FDG Liver Biodistribution

After one year of ChatGPT’s launch: reflections on artificial intelligence in scientific writing

Expanding the clinical indications of [18F]fluorocholine PET/CT in primary hyperparathyroidism: the evidence cannot be evaded

The clinical acceptability of short versus long duration acquisitions for head and neck cancer using long-axial field-of-view PET/CT: a retrospective evaluation

A. Zaheer, S.P. Raman, M.L. Rosado-de-Christenson, S. Martínez-Jiménez, S.H. Garrana, G. Fananapazir, D. Rogers, B.R. Foster: Imaging Anatomy: Chest, Abdomen, Pelvis, 3rd Edition

Beyond stillness: the importance of tackling patient’s motion for reliable parametric imaging