Top

BMC Medical Informatics and Decision Making

Published in:

Open Access 01-12-2021 | Prostate Cancer | Research

Machine learning with asymmetric abstention for biomedical decision-making

Authors: Mariem Gandouz, Hajo Holzmann, Dominik Heider

Published in: BMC Medical Informatics and Decision Making | Issue 1/2021

Abstract

Machine learning and artificial intelligence have entered biomedical decision-making for diagnostics, prognostics, or therapy recommendations. However, these methods need to be interpreted with care because of the severe consequences for patients. In contrast to human decision-making, computational models typically make a decision also with low confidence. Machine learning with abstention better reflects human decision-making by introducing a reject option for samples with low confidence. The abstention intervals are typically symmetric intervals around the decision boundary. In the current study, we use asymmetric abstention intervals, which we demonstrate to be better suited for biomedical data that is typically highly imbalanced. We evaluate symmetric and asymmetric abstention on three real-world biomedical datasets and show that both approaches can significantly improve classification performance. However, asymmetric abstention rejects as many or fewer samples compared to symmetric abstention and thus, should be used in imbalanced data.

Available only for authorised users

Alvarsson J, McShane SA, Norinder U, Spjuth O. Predicting with confidence: using conformal prediction in drug discovery. J Pharm Sci. 2021;110:42–9. https://doi.org/10.1016/j.xphs.2020.09.055.CrossRefPubMed

Ayres de Campos D, Bernardes J, Garrido A, de sá JM, Pereira-leite L. Sisporto 2.0: a program for automated analysis of cardiotocograms. J Matern Fetal Med. 2000;5:311–8. https://doi.org/10.3109/14767050009053454.CrossRef

Bartlett PL, Wegkamp MH. Classification with a reject option using a hinge loss. J Mach Learn Res (JMLR). 2008;9:1823–40.

Bibault J-E, Giraud P, Burgun A. Big data and machine learning in radiation oncology: state of the art and future prospects. Cancer Lett. 2016;382(1):110–7. https://doi.org/10.1016/j.canlet.2016.05.033.CrossRefPubMed

Campagner A, Cabitza F, Ciucci D. The three-way-in and three-way-out framework to treat and exploit ambiguity in data. Int J Approx Reason. 2020;119:292–312. https://doi.org/10.1016/j.ijar.2020.01.010.CrossRef

Chen P, Pan C. Diabetes classification model based on boosting algorithms. BMC Bioinform. 2018;19:109. https://doi.org/10.1186/s12859-018-2090-9.CrossRef

Chicco D, Jurman G. The advantages of the Matthews correlation coefficient (MCC) over f1 score and accuracy in binary classification evaluation. BMC Genomics. 2020;21:6.CrossRef

Chow C. An optimum character recognition system using decision functions. IRE Trans Electron Comput. 1957;EC–6:247–54.CrossRef

Chow C. On optimum recognition error and reject tradeoff. IEEE Trans Inf Theory. 1970;16:41–6. https://doi.org/10.1109/tit.1970.1054406.CrossRef

10.

Coudray N, Ocampo PS, Sakellaropoulos T, Narula N, Snuderl M, Fenyö D, Moreira AL, Razavian N, Tsirigos A. Classification and mutation prediction from non-small cell lung cancer histopathology images using deep learning. Nat Med. 2018;24:1559–67. https://doi.org/10.1038/s41591-018-0177-5.CrossRefPubMed

11.

Dua D, Graff C. UCI machine learning repository (2017). http://archive.ics.uci.edu/ml.

12.

Fumera G, Roli F, Giacinto G. Reject option with multiple thresholds. Pattern Recogn. 2000;33:2099–101. https://doi.org/10.1016/s0031-3203(00)00059-5.CrossRef

13.

Hauschild A-C, Eick L, Wienbeck J, Heider D. Fostering reproducibility, reusability, and technology transfer in health informatics. Science. 2021;24(7):102803–1.

14.

Heider D, Dybowski JN, Wilms C, Hoffmann D. A simple structure-based model for the prediction of HIV-1 co-receptor tropism. BioData Min. 2014. https://doi.org/10.1186/1756-0381-7-14.CrossRefPubMedPubMedCentral

15.

Herbei R, Wegkamp MH. Classification with reject option. Can J Stat. 2006;34(4):709–21. https://doi.org/10.1002/cjs.5550340410.CrossRef

16.

Hernandez-Boussard T, Bozkurt S, Ioannidis JPA, Shah NH. MINIMAR (MINimum information for medical AI reporting): developing reporting standards for artificial intelligence in health care. J Am Med Inform Assoc. 2020;27(12):2011–5.CrossRef

17.

Hosmer DW, Lemeshow S. Applied logistic regression. New York: Wiley; 2000.CrossRef

18.

Landsheer JA. The clinical relevance of methods for handling inconclusive medical test results: quantification of uncertainty in medical decision-making and screening. Diagnostics. 2018;8(2):325. https://doi.org/10.3390/diagnostics8020032.CrossRef

19.

Lengauer T, Sing T. Bioinformatics-assisted anti-HIV therapy. Nat Rev Microb. 2006;4:790–7. https://doi.org/10.1038/nrmicro1477.CrossRef

20.

Libbrecht MW, Noble WS. Machine learning applications in genetics and genomics. Nat Rev Genet. 2015;16(6):321–32.CrossRef

21.

Madabhushi A, Lee G. Image analysis and machine learning in digital pathology: challenges and opportunities. Med Image Anal. 2016;33:170–5. https://doi.org/10.1016/j.media.2016.06.037.CrossRefPubMedPubMedCentral

22.

Mortier T, Wydmuch M, Dembczynski K, Hüllermeier E, Waegeman W. Efficient set-valued prediction in multi-class classification. Data Min Knowl Discov. 2021;35(4):1435–69. https://doi.org/10.1007/s10618-021-00751-x.CrossRef

23.

Neumann U, Riemenschneider M, Sowa J-P, Baars T, Kälsch J, Canbay A, Heider D. Compensation of feature selection biases accompanied with improved predictive performance for binary classification by using a novel ensemble feature selection approach. BioData Min. 2016;9:36. https://doi.org/10.1186/s13040-016-0114-4.CrossRefPubMedPubMedCentral

24.

Nguyen V-L, Destercke S, Masson M-H, Hülermeier E. Reliable multi-class classification based on pairwise epistemic and aleatoric uncertainty. In: Proceedings of the twenty-seventh international joint conference on artificial intelligence (IJCAI-18); 2018. p. 5089–95. https://doi.org/10.24963/ijcai.2018/706

25.

Riemenschneider M, Wienbeck J, Scherag A, Heider D. Data science for molecular diagnostics applications: from academia to clinic to industry. Syst Med. 2018;1:13–7. https://doi.org/10.1089/sysm.2018.0002.CrossRef

26.

Schwarz J, Heider D. Guess: projecting machine learning scores to well-calibrated probability estimates for clinical decision making. Bioinformatics. 2019;35:2458–65.CrossRef

27.

Spänig S, Emberger-Klein A, Sowa J-P, Canbay A, Menrad K, Heider D. The virtual doctor: an interactive clinical-decision-support system based on deep learning for non-invasive prediction of diabetes. Artif Intell Med. 2019;100:101706. https://doi.org/10.1016/j.artmed.2019.101706.CrossRefPubMed

28.

Spänig S, Heider D. Encodings and models for antimicrobial peptide classification for multi-resistant pathogens. BioData Min. 2019;12:7. https://doi.org/10.1186/s13040-019-0196-x.CrossRefPubMedPubMedCentral

29.

Stekhoven DJ, Bühlmann P. Missforest—nonparametric missing value imputation for mixed-type data. Bioinformatics. 2012;28(1):112–8. https://doi.org/10.1093/bioinformatics/btr597.CrossRefPubMed

30.

Tortorella F. An optimal reject rule for binary classifiers. In: Joint IAPR international workshops on statistical techniques in pattern recognition (SPR) and structural and syntactic pattern recognition (SSPR); 2000. p. 611–20. https://doi.org/10.1007/3-540-44522-6_63.

31.

Tortorella F. Reducing the classification cost of support vector classifiers through an roc-based reject rule. Pattern Anal Appl. 2004;7(2):128–43. https://doi.org/10.1007/s10044-004-0209-2.CrossRef

32.

Tsybakov AB. Optimal aggregation of classifiers in statistical learning. Ann Stat. 2004;32(1):135–66. https://doi.org/10.1214/aos/1079120131.CrossRef

33.

Wolberg WH, Mangasarian OL. Multisurface method of pattern separation for medical diagnosis applied to breast cytology. Proc Natl Acad Sci U S A. 1990;87(23):9193–6. https://doi.org/10.1073/pnas.87.23.9193.CrossRefPubMedPubMedCentral

34.

Wu E, Wu K, Daneshjou R, Ouyang D, Ho DE, Zou J. How medical AI devices are evaluated: limitations and recommendations from an analysis of FDA approvals. Nat Med. 2021;27(4):582–4. https://doi.org/10.1038/s41591-021-01312-x.CrossRefPubMed

35.

Yala A, Barzilay R, Salama L, Griffin M, Sollender G, Bardia A, Lehman C, Buckley JM, Coopey SB, Polubriaginof F, Garber JE, Smith BL, Gadd MA, Specht MC, Gudewicz TM, Guidi AJ, Taghian A, Hughes KS. Using machine learning to parse breast pathology reports. Breast Cancer Res Treat. 2017;161:203–11. https://doi.org/10.1007/s10549-016-4035-1.CrossRefPubMed

Title: Machine learning with asymmetric abstention for biomedical decision-making
Authors: Mariem Gandouz
Hajo Holzmann
Dominik Heider
Publication date: 01-12-2021
Publisher: BioMed Central
Keywords: Prostate Cancer
Prostate Cancer
Published in: BMC Medical Informatics and Decision Making / Issue 1/2021
Electronic ISSN: 1472-6947
DOI: https://doi.org/10.1186/s12911-021-01655-y

At a glance: The STEP trials

Springer Medicine

Machine learning with asymmetric abstention for biomedical decision-making

Abstract

At a glance: The STEP trials

Springer Medicine

Abstract

Please log in to get access to this content

Other articles of this Issue 1/2021

Planning a holistic summative eHealth evaluation in an interdisciplinary and multi-national setting: a case study and propositions for guideline development

Tongue image quality assessment based on a deep convolutional neural network

Identifying and evaluating clinical subtypes of Alzheimer’s disease in care electronic health records using unsupervised machine learning

Predicting unplanned medical visits among patients with diabetes: translation from machine learning to clinical implementation

Electronic clinical decision support for children with minor head trauma and intracranial injuries: a sociotechnical analysis

Use of multidimensional item response theory methods for dementia prevalence prediction: an example using the Health and Retirement Survey and the Aging, Demographics, and Memory Study