Generalizability of prostate MRI deep learning: does one size fit all data?

Excerpt

In a recent bibliometric analysis study on artificial intelligence in radiology, the article entitled “A survey on deep learning in medical image analysis” was found to be the most cited publication, both overall and per year, which confirms the overwhelming enthusiasm of the scientific community toward deep learning applications for medical imaging [1, 2]. The authors of this insightful paper keenly draw the attention to a number of unique challenges for successful deep learning modeling in the field of radiology, underlining how lesion annotation faces the issues of both object detection and substructure segmentation. Among others, data-related problems are presented, such as the lack of large expert-labeled datasets, class imbalance, and the simplification behind dichotomous classifications (e.g., a retention cyst and an ectopic benign prostatic hyperplasia nodule in the peripheral zone have characteristic appearances at prostate MRI and differ from normal benign background, but a model needs to recognize them as belonging to the same class). However, the terms “validation,” “reproducibility,” “generalizability,” and “transferability” are quite surprisingly never mentioned in the whole document. Broadly, the ability of reproducing results is pivotal in research but often problematic in radiology, including deep learning applications [3]. Despite its importance, there is a significant inconsistency in the use of this terminology which is at least partly due to the different backgrounds of the involved professionals (e.g., artificial intelligence experts use the term “validation” to refer to the tuning step and physicians/radiologists commonly use it to indicate the final model testing) [4]. To simplify, we could say that deep learning modeling (the model building stage) ends after training and tuning are completed (validation); to ensure that model’s performance is dependable and assessing its generalizability, testing on external independent datasets is required. In this light, to gain trust in deep learning models and facilitate their adoption into clinical practice, we need to address the question “Is this model reproducible or does it detect patterns merely valid in the training dataset?” In the overabundant sea of newly trained and developed deep learning models proposed for a plethora of different tasks, the paper published in this issue of European Radiology joins a small group of refreshing and much-needed generalizability studies. Indeed, in the study by Netzer and colleagues, one can identify several points of strength that should serve as an example for future external test studies [5]. The authors have assessed the performance of a previously trained model (UNETM) in the task of automatically detecting and annotating csPCa lesions on biparametric prostate MRI scans [6]. Of note, the test dataset can be considered of high quality both in terms of overall size (640 cases in total) and diversification of sources (two separate institutions and a public dataset, multiple scanner vendors). Furthermore, acquisition protocols were not homogenous across all sites, which better reflects real-world practices in prostate MRI [7]. In particular, the inclusion of a public dataset, from the PROSTATEx challenge, is especially important to allow comparability of results with future similar research efforts. …