Skip to main content
Top
Published in: BMC Cardiovascular Disorders 1/2024

Open Access 01-12-2024 | Research

A machine learning-based prediction model for postoperative delirium in cardiac valve surgery using electronic health records

Authors: Qiuying Li, Jiaxin Li, Jiansong Chen, Xu Zhao, Jian Zhuang, Guoping Zhong, Yamin Song, Liming Lei

Published in: BMC Cardiovascular Disorders | Issue 1/2024

Login to get access

Abstract

Background

Previous models for predicting delirium after cardiac surgery remained inadequate. This study aimed to develop and validate a machine learning-based prediction model for postoperative delirium (POD) in cardiac valve surgery patients.

Methods

The electronic medical information of the cardiac surgical intensive care unit (CSICU) was extracted from a tertiary and major referral hospital in southern China over 1 year, from June 2019 to June 2020. A total of 507 patients admitted to the CSICU after cardiac valve surgery were included in this study. Seven classical machine learning algorithms (Random Forest Classifier, Logistic Regression, Support Vector Machine Classifier, K-nearest Neighbors Classifier, Gaussian Naive Bayes, Gradient Boosting Decision Tree, and Perceptron.) were used to develop delirium prediction models under full (q = 31) and selected (q = 19) feature sets, respectively.

Result

The Random Forest classifier performs exceptionally well in both feature datasets, with an Area Under the Curve (AUC) of 0.92 for the full feature dataset and an AUC of 0.86 for the selected feature dataset. Additionally, it achieves a relatively lower Expected Calibration Error (ECE) and the highest Average Precision (AP), with an AP of 0.80 for the full feature dataset and an AP of 0.73 for the selected feature dataset. To further evaluate the best-performing Random Forest classifier, SHAP (Shapley Additive Explanations) was used, and the importance matrix plot, scatter plots, and summary plots were generated.

Conclusions

We established machine learning-based prediction models to predict POD in patients undergoing cardiac valve surgery. The random forest model has the best predictive performance in prediction and can help improve the prognosis of patients with POD.
Literature
1.
go back to reference Jones D, Hodgson CL, Shehabi Y, Reade MC. Reducing confusion about post-cardiotomy delirium. Crit Care Resusc. 2017;19(1):5–8.PubMed Jones D, Hodgson CL, Shehabi Y, Reade MC. Reducing confusion about post-cardiotomy delirium. Crit Care Resusc. 2017;19(1):5–8.PubMed
8.
go back to reference Goff DJ, Lloyd-Jones DM, Bennett G, Coady S, D’Agostino RB, Gibbons R, Greenland P, Lackland DT, Levy D, O’Donnell CJ, Robinson JG, Schwartz JS, Shero ST, Smith SJ, Sorlie P, Stone NJ, Wilson PW, Jordan HS, Nevo L, Wnek J, Anderson JL, Halperin JL, Albert NM, Bozkurt B, Brindis RG, Curtis LH, Demets D, Hochman JS, Kovacs RJ, Ohman EM, Pressler SJ, Sellke FW, Shen WK, Smith SJ, Tomaselli GF. 2013 acc/aha guideline on the assessment of cardiovascular risk: a report of the American college of cardiology/american heart association task force on practice guidelines. Circulation. 2014;129(25 Suppl 2):49–S73. https://doi.org/10.1161/01.cir.0000437741.48606.98.CrossRef Goff DJ, Lloyd-Jones DM, Bennett G, Coady S, D’Agostino RB, Gibbons R, Greenland P, Lackland DT, Levy D, O’Donnell CJ, Robinson JG, Schwartz JS, Shero ST, Smith SJ, Sorlie P, Stone NJ, Wilson PW, Jordan HS, Nevo L, Wnek J, Anderson JL, Halperin JL, Albert NM, Bozkurt B, Brindis RG, Curtis LH, Demets D, Hochman JS, Kovacs RJ, Ohman EM, Pressler SJ, Sellke FW, Shen WK, Smith SJ, Tomaselli GF. 2013 acc/aha guideline on the assessment of cardiovascular risk: a report of the American college of cardiology/american heart association task force on practice guidelines. Circulation. 2014;129(25 Suppl 2):49–S73. https://​doi.​org/​10.​1161/​01.​cir.​0000437741.​48606.​98.CrossRef
11.
go back to reference Otto CM, Nishimura RA, Bonow RO, Carabello BA, Erwin JR, Gentile F, Jneid H, Krieger EV, Mack M, Mcleod C, O’Gara PT, Rigolin VH, Sundt TR, Thompson A, Toly C. 2020 acc/aha guideline for the management of patients with valvular heart disease: executive summary: a report of the American college of cardiology/american heart association joint committee on clinical practice guidelines. Circulation. 2021;143(5):e35–e71. https://doi.org/10.1161/CIR.0000000000000932.CrossRefPubMed Otto CM, Nishimura RA, Bonow RO, Carabello BA, Erwin JR, Gentile F, Jneid H, Krieger EV, Mack M, Mcleod C, O’Gara PT, Rigolin VH, Sundt TR, Thompson A, Toly C. 2020 acc/aha guideline for the management of patients with valvular heart disease: executive summary: a report of the American college of cardiology/american heart association joint committee on clinical practice guidelines. Circulation. 2021;143(5):e35–e71. https://​doi.​org/​10.​1161/​CIR.​0000000000000932​.CrossRefPubMed
14.
go back to reference Racine AM, Tommet D, D’Aquila ML, Fong TG, Gou Y, Tabloski PA, Metzger ED, Hshieh TT, Schmitt EM, Vasunilashorn SM, Kunze L, Vlassakov K, Abdeen A, Lange J, Earp B, Dickerson BC, Marcantonio ER, Steingrimsson J, Travison TG, Inouye SK, Jones RN. Machine learning to develop and internally validate a predictive model for post-operative delirium in a prospective, observational clinical cohort study of older surgical patients. J Gen Intern Med. 2021;36(2):265–73. https://doi.org/10.1007/s11606-020-06238-7.CrossRefPubMed Racine AM, Tommet D, D’Aquila ML, Fong TG, Gou Y, Tabloski PA, Metzger ED, Hshieh TT, Schmitt EM, Vasunilashorn SM, Kunze L, Vlassakov K, Abdeen A, Lange J, Earp B, Dickerson BC, Marcantonio ER, Steingrimsson J, Travison TG, Inouye SK, Jones RN. Machine learning to develop and internally validate a predictive model for post-operative delirium in a prospective, observational clinical cohort study of older surgical patients. J Gen Intern Med. 2021;36(2):265–73. https://​doi.​org/​10.​1007/​s11606-020-06238-7.CrossRefPubMed
19.
go back to reference Devlin JW, Skrobik Y, Gelinas C, Needham DM, Slooter A, Pandharipande PP, Watson PL, Weinhouse GL, Nunnally ME, Rochwerg B, Balas MC, van den Boogaard M, Bosma KJ, Brummel NE, Chanques G, Denehy L, Drouot X, Fraser GL, Harris JE, Joffe AM, Kho ME, Kress JP, Lanphere JA, Mckinley S, Neufeld KJ, Pisani MA, Payen JF, Pun BT, Puntillo KA, Riker RR, Robinson B, Shehabi Y, Szumita PM, Winkelman C, Centofanti JE, Price C, Nikayin S, Misak CJ, Flood PD, Kiedrowski K, Alhazzani W. Clinical practice guidelines for the prevention and management of pain, agitation/sedation, delirium, immobility, and sleep disruption in adult patients in the ICU, Crit. Care Med. 2018;46(9):e825–73. https://doi.org/10.1097/CCM.0000000000003299.CrossRef Devlin JW, Skrobik Y, Gelinas C, Needham DM, Slooter A, Pandharipande PP, Watson PL, Weinhouse GL, Nunnally ME, Rochwerg B, Balas MC, van den Boogaard M, Bosma KJ, Brummel NE, Chanques G, Denehy L, Drouot X, Fraser GL, Harris JE, Joffe AM, Kho ME, Kress JP, Lanphere JA, Mckinley S, Neufeld KJ, Pisani MA, Payen JF, Pun BT, Puntillo KA, Riker RR, Robinson B, Shehabi Y, Szumita PM, Winkelman C, Centofanti JE, Price C, Nikayin S, Misak CJ, Flood PD, Kiedrowski K, Alhazzani W. Clinical practice guidelines for the prevention and management of pain, agitation/sedation, delirium, immobility, and sleep disruption in adult patients in the ICU, Crit. Care Med. 2018;46(9):e825–73. https://​doi.​org/​10.​1097/​CCM.​0000000000003299​.CrossRef
30.
go back to reference Harris CR, Millman KJ, van der Walt SJ, Gommers R, Virtanen P, Cournapeau D, Wieser E, Taylor J, Berg S, Smith NJ, Kern R, Picus M, Hoyer S, van Kerkwijk MH, Brett M, Haldane A, Del RJ, Wiebe M, Peterson P, Gerard-Marchant P, Sheppard K, Reddy T, Weckesser W, Abbasi H, Gohlke C, Oliphant TE. Array programming with numpy. Nature. 2020;585(7825):357–62. https://doi.org/10.1038/s41586-020-2649-2.CrossRefPubMedPubMedCentral Harris CR, Millman KJ, van der Walt SJ, Gommers R, Virtanen P, Cournapeau D, Wieser E, Taylor J, Berg S, Smith NJ, Kern R, Picus M, Hoyer S, van Kerkwijk MH, Brett M, Haldane A, Del RJ, Wiebe M, Peterson P, Gerard-Marchant P, Sheppard K, Reddy T, Weckesser W, Abbasi H, Gohlke C, Oliphant TE. Array programming with numpy. Nature. 2020;585(7825):357–62. https://​doi.​org/​10.​1038/​s41586-020-2649-2.CrossRefPubMedPubMedCentral
44.
go back to reference Kalafi EY, Nor N, Taib NA, Ganggayah MD, Town C, Dhillon SK. Machine learning and deep learning approaches in breast cancer survival prediction using clinical data. Folia Biol (Praha). 2019;65(5–6):212–20.CrossRefPubMed Kalafi EY, Nor N, Taib NA, Ganggayah MD, Town C, Dhillon SK. Machine learning and deep learning approaches in breast cancer survival prediction using clinical data. Folia Biol (Praha). 2019;65(5–6):212–20.CrossRefPubMed
45.
go back to reference Lundberg Sm LS. A unified approach to interpreting model predictions. In advance, in Proceedings of the 31st International Conference on Neural Information Processing Systems, 2017. Lundberg Sm LS. A unified approach to interpreting model predictions. In advance, in Proceedings of the 31st International Conference on Neural Information Processing Systems, 2017.
46.
go back to reference Naeini CGHM, Mp. Obtaining well calibrated probabilities using bayesian binning., in Proceedings of the Twenty-Ninth AAAI Conference on Artificial Intelligence, University Texas at Austin, 2015. Naeini CGHM, Mp. Obtaining well calibrated probabilities using bayesian binning., in Proceedings of the Twenty-Ninth AAAI Conference on Artificial Intelligence, University Texas at Austin, 2015.
51.
go back to reference Sanders KM, Stern TA. Management of delirium associated with the use of the intra-aortic balloon pump. Am J Crit Care. 1993;2(5):371–7.CrossRefPubMed Sanders KM, Stern TA. Management of delirium associated with the use of the intra-aortic balloon pump. Am J Crit Care. 1993;2(5):371–7.CrossRefPubMed
56.
go back to reference Siokas V, Fleischmann R, Feil K, Liampas I, Kowarik MC, Bai Y, Stefanou MI, Poli S, Ziemann U, Dardiotis E, Mengel A. The role of vascular risk factors in post-stroke delirium: a systematic review and meta-analysis. J Clin Med. 2022;11(19). https://doi.org/10.3390/jcm11195835. Siokas V, Fleischmann R, Feil K, Liampas I, Kowarik MC, Bai Y, Stefanou MI, Poli S, Ziemann U, Dardiotis E, Mengel A. The role of vascular risk factors in post-stroke delirium: a systematic review and meta-analysis. J Clin Med. 2022;11(19). https://​doi.​org/​10.​3390/​jcm11195835.
58.
go back to reference Davis GM. J, The relationship between precision-recall and roc curves., in Proceedings of the 23rd International Conference on Machine Learning, Pittsburgh, Pennsylvania, 2006. Davis GM. J, The relationship between precision-recall and roc curves., in Proceedings of the 23rd International Conference on Machine Learning, Pittsburgh, Pennsylvania, 2006.
62.
go back to reference A.F.G.K. MR, Berthold. Advances in intelligent data analysis xviii. Springer International Publishing; 2020. A.F.G.K. MR, Berthold. Advances in intelligent data analysis xviii. Springer International Publishing; 2020.
63.
go back to reference Bilotta F, Lauretta MP, Borozdina A, Mizikov VM, Rosa G. Postoperative delirium: risk factors, diagnosis and perioperative care. Minerva Anestesiol. 2013;79(9):1066–76.PubMed Bilotta F, Lauretta MP, Borozdina A, Mizikov VM, Rosa G. Postoperative delirium: risk factors, diagnosis and perioperative care. Minerva Anestesiol. 2013;79(9):1066–76.PubMed
67.
go back to reference Nomi T, Kaibori M, Hirokawa F, Ueno M, Hokuto D, Noda T, Nakai T, Ikoma H, Iida H, Kubo S. Effect of laparoscopic liver resection on postoperative delirium in elderly patients with hepatocellular carcinoma. J Hepato-Biliary-Pancreat Sci. 2023;30(9):1111–8. https://doi.org/10.1002/jhbp.1341.CrossRef Nomi T, Kaibori M, Hirokawa F, Ueno M, Hokuto D, Noda T, Nakai T, Ikoma H, Iida H, Kubo S. Effect of laparoscopic liver resection on postoperative delirium in elderly patients with hepatocellular carcinoma. J Hepato-Biliary-Pancreat Sci. 2023;30(9):1111–8. https://​doi.​org/​10.​1002/​jhbp.​1341.CrossRef
Metadata
Title
A machine learning-based prediction model for postoperative delirium in cardiac valve surgery using electronic health records
Authors
Qiuying Li
Jiaxin Li
Jiansong Chen
Xu Zhao
Jian Zhuang
Guoping Zhong
Yamin Song
Liming Lei
Publication date
01-12-2024
Publisher
BioMed Central
Published in
BMC Cardiovascular Disorders / Issue 1/2024
Electronic ISSN: 1471-2261
DOI
https://doi.org/10.1186/s12872-024-03723-3

Other articles of this Issue 1/2024

BMC Cardiovascular Disorders 1/2024 Go to the issue