Top

International Journal of Computer Assisted Radiology and Surgery

Open Access 03-02-2025 | Transcatheter Aortic Valve Implantation | Original Article

Multi-modal dataset creation for federated learning with DICOM-structured reports

Authors: Malte Tölle, Lukas Burger, Halvar Kelm, Florian André, Peter Bannas, Gerhard Diller, Norbert Frey, Philipp Garthe, Stefan Groß, Anja Hennemuth, Lars Kaderali, Nina Krüger, Andreas Leha, Simon Martin, Alexander Meyer, Eike Nagel, Stefan Orwat, Clemens Scherer, Moritz Seiffert, Jan Moritz Seliger, Stefan Simm, Tim Friede, Tim Seidler, Sandy Engelhardt

Published in: International Journal of Computer Assisted Radiology and Surgery

Abstract

Purpose Federated training is often challenging on heterogeneous datasets due to divergent data storage options, inconsistent naming schemes, varied annotation procedures, and disparities in label quality. This is particularly evident in the emerging multi-modal learning paradigms, where dataset harmonization including a uniform data representation and filtering options are of paramount importance.

Methods DICOM-structured reports enable the standardized linkage of arbitrary information beyond the imaging domain and can be used within Python deep learning pipelines with highdicom. Building on this, we developed an open platform for data integration with interactive filtering capabilities, thereby simplifying the process of creation of patient cohorts over several sites with consistent multi-modal data.

Results In this study, we extend our prior work by showing its applicability to more and divergent data types, as well as streamlining datasets for federated training within an established consortium of eight university hospitals in Germany. We prove its concurrent filtering ability by creating harmonized multi-modal datasets across all locations for predicting the outcome after minimally invasive heart valve replacement. The data include imaging and waveform data (i.e., computed tomography images, electrocardiography scans) as well as annotations (i.e., calcification segmentations, and pointsets), and metadata (i.e., prostheses and pacemaker dependency).

Conclusion Structured reports bridge the traditional gap between imaging systems and information systems. Utilizing the inherent DICOM reference system arbitrary data types can be queried concurrently to create meaningful cohorts for multi-centric data analysis. The graphical interface as well as example structured report templates are available at https://github.com/Cardio-AI/fl-multi-modal-dataset-creation.

Available only for authorised users

https://gdpr.eu/.

https://hl7.org/fhir/.

https://openehr.org/.

https://www.medizininformatik-initiative.de/en/start.

https://www.kaapana.ai.

https://opensearch.org/.

https://vega.github.io/vega/.

https://dzhk.de/.

TID3802 Cardiovascular PatientHistory, TID3829 Problem Properties, TID3830 Procedure Properties, TID3831 Medical Device Use, TID 3704 Patient Characteristics for ECG, TID3702 Prior ECG Study, TID3708 ECG Waveform Information, TID3715 ECG Measurement Source, TID3713 ECG Global Measurements, TID3714 ECG Lead Measurements, TID3717 ECG Qualitative Analysis, TID3719 ECG Summary, TID3700 ECG Report, TID2002 Report Narrative Code, TID2000 Basic Diagnostic Imaging Report.

https://dclunie.com/dicom3tools/dciodvfy.html.

https://www.w3.org/TR/REC-xml/.

https://dicom.nema.org/medical/dicom/current/output/chtml/part17/sect_C.7.html.

https://docs.hdfgroup.org/hdf5/v1_14/_intro_h_d_f5.html.

https://dclunie.com/dicom3tools/dciodvfy.html.

Bridge C, Gorman C, Pieper S, Doyle S, Lennerz J, Kalpathy-Cramer J, Clunie D, Fedorov A, Herrmann M (2022) Highdicom: a python library for standardized encoding of image annotations and machine learning model outputs in pathology and radiology. J Digit Imaging 35(6):1719–1737. https://doi.org/10.1007/s10278-022-00683-yCrossRefPubMedPubMedCentral

Chambon P, Bluethgen C, Delbrouck JB, Van der Sluijs R, Polacin M, Chaves JMZ, Abraham TM, Purohit S, Langlotz CP, Chaudhari A (2022) Roentgen: vision-language foundation model for chest X-ray generation. https://doi.org/10.48550/arxiv.2211.12737

Clunie DA (2000) DICOM Structured Reporting. PixelMed. http://www.dclunie.com/pixelmed/DICOMSR.book.pdf

Clunie DA (2007) Dicom structured reporting and cancer clinical trials results. Cancer Inf 4:33–56. https://doi.org/10.4137/CIN.S37032CrossRef

Fedorov A, Longabaugh WJR, Pot D, Clunie DA, Pieper SD, Gibbs DL, Bridge C, Herrmann MD, Homeyer A, Lewis R, Aerts HJWL, Krishnaswamy D, Thiriveedhi VK, Ciausu C, Schacherer DP, Bontempi D, Pihl T, Wagner U, Farahani K, Kim E et al (2023) National cancer institute imaging data commons: toward transparency, reproducibility, and scalability in imaging artificial intelligence. Radiographics. https://doi.org/10.1148/rg.230180CrossRefPubMed

Gazzarata R, Almeida J, Lindsköld L, Cangioli G, Gaeta E, Fico G, Chronaki CE (2024) Hl7 fast healthcare interoperability resources (hl7 fhir) in digital healthcare ecosystems for chronic disease management: Scoping review. Int J Med In. https://doi.org/10.1016/j.ijmedinf.2024.105507CrossRef

Généreux P, Head SJ, Wood DA, Kodali SK, Williams MR, Paradis JM, Spaziano M, Kappetein AP, Webb JG, Cribier A, Leon MB (2012) Transcatheter aortic valve implantation 10-year anniversary: review of current evidence and clinical implications. Eur Heart J 33(19):2388–2398. https://doi.org/10.1093/eurheartj/ehs220CrossRefPubMed

Kavur AE, Gezer NS, Barış M, Aslan S, Conze PH, Groza V, Pham DD, Chatterjee S, Ernst P, Özkan S, Baydar B, Lachinov D, Han S, Pauli J, Isensee F, Perkonigg M, Sathish R, Rajan R, Sheet D, Dovletov G et al (2021) CHAOS challenge-combined (CT-MR) healthy abdominal organ segmentation. Med Image Anal. https://doi.org/10.1016/j.media.2020.101950CrossRefPubMed

Krishnaswamy D, Bontempi D, Thiriveedhi VK, Punzo D, Clunie D, Bridge CP, Aerts HJWL, Kikinis R, Fedorov A (2024) Enrichment of lung cancer computed tomography collections with ai-derived annotations. Sci Data 11(1):25. https://doi.org/10.1038/s41597-023-02864-yCrossRefPubMedPubMedCentral

10.

Landman B, Xu Z, Igelsias J, Styner M, Langerak T, Klein A (2015) Multi-atlas labeling beyond the cranial vault-workshop and challenge. In: MICCAI multi-atlas labeling beyond the cranial vault-workshop and challenge. https://doi.org/10.7303/SYN3193805

11.

Lazarova E, Mora S, Maggi N, Ruggiero C, Vitale AC, Rubartelli P, Giacomini M (2022) An interoperable electronic health record system for clinical cardiology. Informatics. https://doi.org/10.3390/informatics9020047

12.

Martín-Isla C, Campello VM, Izquierdo C, Kushibar K, Sendra-Balcells C, Gkontra P, Sojoudi A, Fulton MJ, Arega TW, Punithakumar K, Li L, Sun X, Al Khalil Y, Liu D, Jabbar S, Queirós S, Galati F, Mazher M, Gao Z, Beetz M et al (2023) Deep learning segmentation of the right ventricle in cardiac MRI: the M &Ms challenge. IEEE J Biomed Health Inform 27(7):3302–3313. https://doi.org/10.1109/JBHI.2023.3267857CrossRefPubMed

13.

Nobel M, van Geel K, Robben S (2022) Structured reporting in radiology. Eur Radiol 32:2837–2854. https://doi.org/10.1007/s00330-021-08327-5CrossRefPubMed

14.

Noumeir R (2006) Benefits of the DICOM structured report. J Digit Imaging 16(4):295–306. https://doi.org/10.1007/s10278-006-0631-7CrossRef

15.

Oquab M, Darcet T, Moutakanni T, Vo HV, Szafraniec M, Khalidov V, Fernandez P, HAZIZA D, Massa F, El-Nouby A, Assran M, Ballas N, Galuba W, Howes R, Huang PY, Li SW, Misra I, Rabbat M, Sharma V, Synnaeve G, et al (2024) DINOv2: learning robust visual features without supervision. Trans Mach Learn Res. https://doi.org/10.48550/arXiv.2304.07193

16.

Pei X, Zuo K, Li Y, Pang Z (2023) A review of the application of multi-modal deep learning in medicine: Bibliometrics and future directions. Int J Comput Intell Syst 16:1. https://doi.org/10.1007/s44196-023-00225-6CrossRef

17.

Rieke N, Hancox J, Li W, Milletarì F, Roth HR, Albarqouni S, Bakas S, Galtier MN, Landman BA, Maier-Hein K, Ourselin S, Sheller M, Summers RM, Trask A, Xu D, Baust M, Cardoso MJ (2020) The future of digital health with federated learning. npj Dig Med 3(119):2398–6352. https://doi.org/10.1038/s41746-020-00323-1CrossRef

18.

Riepenhausen S, Blumenstock M, Niklas C, Hegselmann S, Neuhaus P, Meidt A, Püttmann C, Storck M, Ganzinger M, Varghese J, Dugas M (2024) Europe’s largest research infrastructure for curated medical data models with semantic annotations. Methods Inf Med EFirst. https://doi.org/10.1055/s-0044-1786839CrossRef

19.

Roth CJ, Lannum LM, Persons KR (2016) A foundation for enterprise imaging: Himss–Siim collaborative white paper. J Digit Imaging 29(5):530–538. https://doi.org/10.1007/s10278-016-9882-0CrossRefPubMedPubMedCentral

20.

Saltz J, Saltz M, Prasanna P, Moffitt R, Hajagos J, Bremer E, Balsamo J, Kurc T (2021) Stony Brook University COVID-19 positive cases. Cancer Imaging Arch. https://doi.org/10.7937/TCIA.BBAG-2923CrossRef

21.

Scherer J, Nolden M, Kleesiek J, Metzger J, Kades K, Schneider V, Bach M, Sedlaczek O, Bucher AM, Vogl TJ, Grünwald F, Kühn JP, Hoffmann RT, Kotzerke J, Bethge O, Schimmöller L, Antoch G, Müller HW, Daul A, Nikolaou K et al (2020) Joint imaging platform for federated clinical data analytics. JCO Clin Cancer Inf 4:1027–1038. https://doi.org/10.1200/CCI.20.00045CrossRef

22.

Seidler T, Tölle M, André F, Bannas P, Frey N, Friedrich S, Groß S, Hennemuth A, Krüger N, Leha A, Martin S, Meyer A, Nagel E, Orwat S, Scherer C, Simm TStefan Friede, Engelhardt S, (2022) Federated learning of TAVI outcomes (FLOTO)—a collaborative multi-center deep learning initiative. Clin Res Cardiol. https://doi.org/10.1007/s00392-022-02002-5

23.

Thrasher J, Devkota A, Siwakotai P, Chivukula R, Poudel P, Hu C, Bhattarai B, Gyawali P (2024) Multimodal federated learning in healthcare: a review. https://doi.org/10.48550/arXiv.2310.09650

24.

Tölle M, Burger L, Kelm H, Engelhardt S (2024) Towards unified multi-modal dataset creation for deep learning utilizing structured reports. Bildverarbeitung für die Medizin 2024:130–135. https://doi.org/10.1007/978-3-658-44037-4_39

25.

Tölle M, Garthe P, Scherer C, Seliger JM, Leha A, Krüger N, Simm S, Martin S, Eble S, Kelm H, Bednorz M, André F, Bannas P, Diller G, Frey N, Groß S, Hennemuth A, Kaderali L, Meyer A, Nagel E, et al (2024) Real World Federated Learning with a Knowledge Distilled Transformer for Cardiac CT Imaging.npj Digital Medicine https://doi.org/10.1038/s41746-025-01434-3

26.

Wasserthal J, Breit HC, Meyer MT, Pradella M, Hinck D, Sauter AW, Heye T, Boll D, Cyriac J, Yang S, Bach M, Segeroth M (2023) TotalSegmentator: robust segmentation of 104 anatomical structures in CT images. Radiol Artif Intell. https://doi.org/10.1148/ryai.230024

Title: Multi-modal dataset creation for federated learning with DICOM-structured reports
Authors: Malte Tölle
Lukas Burger
Halvar Kelm
Florian André
Peter Bannas
Gerhard Diller
Norbert Frey
Philipp Garthe
Stefan Groß
Anja Hennemuth
Lars Kaderali
Nina Krüger
Andreas Leha
Simon Martin
Alexander Meyer
Eike Nagel
Stefan Orwat
Clemens Scherer
Moritz Seiffert
Jan Moritz Seliger
Stefan Simm
Tim Friede
Tim Seidler
Sandy Engelhardt
Publication date: 03-02-2025
Publisher: Springer International Publishing
Keyword: Transcatheter Aortic Valve Implantation
Published in: International Journal of Computer Assisted Radiology and Surgery
Print ISSN: 1861-6410
Electronic ISSN: 1861-6429
DOI: https://doi.org/10.1007/s11548-025-03327-y

Keynote series | Spotlight on menopause

Springer Medicine

Multi-modal dataset creation for federated learning with DICOM-structured reports

Abstract

Keynote series | Spotlight on menopause

Springer Medicine

Abstract

Please log in to get access to this content