Top

BMC Medical Informatics and Decision Making

Published in:

Open Access 01-12-2023 | COVID-19 | Research

Impact of the Covid-19 pandemic on the performance of machine learning algorithms for predicting perioperative mortality

Authors: D. I. Andonov, B. Ulm, M. Graessner, A. Podtschaske, M. Blobner, B. Jungwirth, S. M. Kagerbauer

Published in: BMC Medical Informatics and Decision Making | Issue 1/2023

Abstract

Background

Machine-learning models are susceptible to external influences which can result in performance deterioration. The aim of our study was to elucidate the impact of a sudden shift in covariates, like the one caused by the Covid-19 pandemic, on model performance.

Methods

After ethical approval and registration in Clinical Trials (NCT04092933, initial release 17/09/2019), we developed different models for the prediction of perioperative mortality based on preoperative data: one for the pre-pandemic data period until March 2020, one including data before the pandemic and from the first wave until May 2020, and one that covers the complete period before and during the pandemic until October 2021. We applied XGBoost as well as a Deep Learning neural network (DL). Performance metrics of each model during the different pandemic phases were determined, and XGBoost models were analysed for changes in feature importance.

Results

XGBoost and DL provided similar performance on the pre-pandemic data with respect to area under receiver operating characteristic (AUROC, 0.951 vs. 0.942) and area under precision-recall curve (AUPR, 0.144 vs. 0.187). Validation in patient cohorts of the different pandemic waves showed high fluctuations in performance from both AUROC and AUPR for DL, whereas the XGBoost models seemed more stable. Change in variable frequencies with onset of the pandemic were visible in age, ASA score, and the higher proportion of emergency operations, among others. Age consistently showed the highest information gain. Models based on pre-pandemic data performed worse during the first pandemic wave (AUROC 0.914 for XGBoost and DL) whereas models augmented with data from the first wave lacked performance after the first wave (AUROC 0.907 for XGBoost and 0.747 for DL). The deterioration was also visible in AUPR, which worsened by over 50% in both XGBoost and DL in the first phase after re-training.

Conclusions

A sudden shift in data impacts model performance. Re-training the model with updated data may cause degradation in predictive accuracy if the changes are only transient. Too early re-training should therefore be avoided, and close model surveillance is necessary.

Available only for authorised users

Turley L, Mahon J, Sheehan E. “Out of hours” orthopaedics in an Irish regional trauma unit and the impact of COVID-19. Ir J Med Sci. 2022. https://doi.org/10.1007/s11845-022-03135-2.CrossRefPubMedPubMedCentral

Duckworth C, Chmiel FP, Burns DK, Zlatev ZD, White NM, Daniels TWV, Kiuber M, Boniface MJ. Using explainable machine learning to characterise data drift and detect emergent health risks for emergency department admissions during COVID-19. Sci Rep. 2021;11(1):23017. https://doi.org/10.1038/s41598-021-02481-y.CrossRefPubMedPubMedCentral

Chi S, Tian Y, Wang F, Zhou T, Jin S, Li J. A novel lifelong machine learning-based method to eliminate calibration drift in clinical prediction models. Artif Intell Med. 2022;125:102256. https://doi.org/10.1016/j.artmed.2022.102256.CrossRefPubMed

Celik B, Vanschoren J. Adaptation Strategies for Automated Machine Learning on Evolving Data. IEEE Trans Pattern Anal Mach Intell. 2021;43(9):3067–78. https://doi.org/10.1109/TPAMI.2021.3062900.CrossRefPubMed

Das S: Best Practices for Dealing With Concept Drift [https://neptune.ai/blog/concept-drift-best-practices] last Accessed 14 Nov 2022

Kumar S: Should a machine learning model be retrained each time new observations are available? [https://www.quora.com/Should-a-machine-learning-model-be-retrained-each-time-new-observations-are-available] last Accessed 14 Nov 2022

Collins GS, Reitsma JB, Altman DG, Moons KG. Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis (TRIPOD). Ann Intern Med. 2015;162(10):735–6. https://doi.org/10.7326/L15-5093-2.CrossRefPubMed

Chen TQ, Guestrin C. XGBoost: A Scalable Tree Boosting System. In: Kdd’16: Proceedings of the 22nd Acm Sigkdd International Conference on Knowledge Discovery and Data Mining. 2016. p. 785–94.CrossRef

Saito T, Rehmsmeier M. The Precision-Recall Plot Is More Informative than the ROC Plot When Evaluating Binary Classifiers on Imbalanced Datasets. Plos One. 2015;10(3):ARTN e0118432. https://doi.org/10.1371/journal.pone.0118432.CrossRef

10.

Bellini V, Valente M, Bertorelli G, Pifferi B, Craca M, Mordonini M, Lombardo G, Bottani E, Del Rio P, Bognami E. Machine learning in perioperative medicine: a systematic review. J Anesth Analg Crit Care. 2022;2(2):2–13. https://doi.org/10.1186/s44158-022-00033-y.CrossRef

11.

Islam MA, Majumder MZH, Hussein MA. Chronic kidney disease prediction based on machine learning algorithms. J Pathol Inform. 2023;14:100189. https://doi.org/10.1016/j.jpi.2023.100189.CrossRefPubMedPubMedCentral

12.

Shickel B, Loftus TJ, Ruppert M, Upchurch GR Jr, Ozrazgat-Baslanti T, Rashidi P, Bihorac A. Dynamic predictions of postoperative complications from explainable, uncertainty-aware, and multi-task deep neural networks. Sci Rep. 2023;13(1):1224. https://doi.org/10.1038/s41598-023-27418-5.CrossRefPubMedPubMedCentral

13.

Davis SE, Lasko TA, Chen G, Matheny ME. Calibration Drift Among Regression and Machine Learning Models for Hospital Mortality. AMIA Annu Symp Proc. 2017;2017:625–34.PubMed

14.

Davis SE, Lasko TA, Chen G, Siew ED, Matheny ME. Calibration drift in regression and machine learning models for acute kidney injury. J Am Med Inform Assoc. 2017;24(6):1052–61. https://doi.org/10.1093/jamia/ocx030.CrossRefPubMedPubMedCentral

15.

Vela D, Sharp A, Zhang R, Nguyen T, Hoang A, Pianykh OS. Temporal quality degradation in AI models. Sci Rep. 2022;12(1):11654. https://doi.org/10.1038/s41598-022-15245-z.CrossRefPubMedPubMedCentral

16.

Tsymbal A. The problem of concept drift: definitions and related work. 2004. https://citeseerx.ist.psu.edu/document?repid=rep1&type=pdf&doi=30eac73e9b482bc28b5b68cd585557de48d0618f . Accessed 09 Feb 2023.

17.

Kazakova SV, Baggs J, Parra G, Yusuf H, Romano SD, Ko JY, Harris AM, Wolford H, Rose A, Reddy SC, et al. Declines in the utilization of hospital-based care during COVID-19 pandemic. J Hosp Med. 2022. https://doi.org/10.1002/jhm.12955.CrossRefPubMed

18.

Grieco M, Galiffa G, Marcellinaro R, Santoro E, Persiani R, Mancini S, Di Paola M, Santoro R, Stipa F, Crucitti A, et al. Impact of the COVID-19 Pandemic on Enhanced Recovery After Surgery (ERAS) Application and Outcomes: Analysis in the “Lazio Network” Database. World J Surg. 2022;46(10):2288–96. https://doi.org/10.1007/s00268-022-06694-8.CrossRefPubMedPubMedCentral

19.

Pogorzelski D, McKay P, Weaver MJ, Jaeblon T, Hymes RA, Gaski GE, Fraifogl J, Ahn JS, Bzovsky S, Slobogean G, et al. The impact of COVID-19 restrictions on participant enrollment in the PREPARE trial. Contemp Clin Trials Commun. 2022;29:100973. https://doi.org/10.1016/j.conctc.2022.100973.CrossRefPubMedPubMedCentral

20.

Wolf M, Landgraeber S, Maass W, Orth P. Impact of Covid-19 on the global orthopaedic research output. Front Surg. 2022;9:962844. https://doi.org/10.3389/fsurg.2022.962844.CrossRefPubMedPubMedCentral

21.

Abdolalizadeh P, Kashkouli MB, Jafarpour S, Rezaei S, Ghanbari S, Akbarian S. Impact of COVID-19 on the patient referral pattern and conversion rate in the university versus private facial plastic surgery centers. Int Ophthalmol. 2022. https://doi.org/10.1007/s10792-022-02469-1.CrossRefPubMedPubMedCentral

22.

Mehta A, Awuah WA, Ng JC, Kundu M, Yarlagadda R, Sen M, Nansubuga EP, Abdul-Rahman T, Hasan MM. Elective surgeries during and after the COVID-19 pandemic: Case burden and physician shortage concerns. Ann Med Surg (Lond). 2022;81:104395. https://doi.org/10.1016/j.amsu.2022.104395.CrossRefPubMed

23.

Bertram A, Hyam D, Hapangama N. Out-of-hours maxillofacial trauma surgery: a risk factor for complications? Int J Oral Maxillofac Surg. 2013;42(2):214–7. https://doi.org/10.1016/j.ijom.2012.11.001.CrossRefPubMed

24.

Moll M, Qiao D, Regan EA, Hunninghake GM, Make BJ, Tal-Singer R, McGeachie MJ, Castaldi PJ, San Jose Estepar R, Washko GR, et al. Machine Learning and Prediction of All-Cause Mortality in COPD. Chest. 2020;158(3):952–64. https://doi.org/10.1016/j.chest.2020.02.079.CrossRefPubMedPubMedCentral

25.

Le Manach Y, Collins G, Rodseth R, Le Bihan-Benjamin C, Biccard B, Riou B, Devereaux PJ, Landais P. Preoperative Score to Predict Postoperative Mortality (POSPOM): Derivation and Validation. Anesthesiology. 2016;124(3):570–9. https://doi.org/10.1097/ALN.0000000000000972.CrossRefPubMed

26.

Hackett NJ, De Oliveira GS, Jain UK, Kim JY. ASA class is a reliable independent predictor of medical complications and mortality following surgery. Int J Surg. 2015;18:184–90. https://doi.org/10.1016/j.ijsu.2015.04.079.CrossRefPubMed

27.

Lacson R, Eskian M, Licaros A, Kapoor N, Khorasani R. Machine Learning Model Drift: Predicting Diagnostic Imaging Follow-Up as a Case Example. J Am Coll Radiol. 2022;19(10):1162–9. https://doi.org/10.1016/j.jacr.2022.05.030.CrossRefPubMed

28.

McCoy M, Touchet N, Chapple AG, Cohen-Rosenblum A: Total Joint Arthroplasty Patient Demographics Before and after COVID-19 Elective Surgery Restrictions. Arthroplast Today 2023:101081. https://doi.org/10.1016/j.artd.2022.101081.

Title: Impact of the Covid-19 pandemic on the performance of machine learning algorithms for predicting perioperative mortality
Authors: D. I. Andonov
B. Ulm
M. Graessner
A. Podtschaske
M. Blobner
B. Jungwirth
S. M. Kagerbauer
Publication date: 01-12-2023
Publisher: BioMed Central
Keyword: COVID-19
Published in: BMC Medical Informatics and Decision Making / Issue 1/2023
Electronic ISSN: 1472-6947
DOI: https://doi.org/10.1186/s12911-023-02151-1

Keynote webinar | Spotlight on medication adherence

Springer Medicine

Impact of the Covid-19 pandemic on the performance of machine learning algorithms for predicting perioperative mortality

Abstract

Background

Methods

Results

Conclusions

Keynote webinar | Spotlight on medication adherence

Springer Medicine

Abstract

Background

Methods

Results

Conclusions

Please log in to get access to this content

Other articles of this Issue 1/2023

Developing a mobile health application for wound telemonitoring: a pilot study on abdominal surgeries post-discharge care

Early prediction of in-hospital mortality utilizing multivariate predictive modelling of electronic medical records and socio-determinants of health of the first day of hospitalization

Proposal of neural network model for neurocognitive rehabilitation and its comparison with fuzzy expert system model

Associations between blood pressure control and clinical events suggestive of nutrition care documented in electronic health records of patients with hypertension

A deep neural network framework to derive interpretable decision rules for accurate traumatic brain injury identification of infants

Fat mass: a novel digital biomarker for remote monitoring that may indicate risk for malnutrition and new complications in decompensated cirrhosis