Top

BMC Medical Informatics and Decision Making

Published in:

Open Access 01-12-2024 | Research

Development and validation of ‘Patient Optimizer’ (POP) algorithms for predicting surgical risk with machine learning

Authors: Gideon Kowadlo, Yoel Mittelberg, Milad Ghomlaghi, Daniel K. Stiglitz, Kartik Kishore, Ranjan Guha, Justin Nazareth, Laurence Weinberg

Published in: BMC Medical Informatics and Decision Making | Issue 1/2024

Abstract

Background

Pre-operative risk assessment can help clinicians prepare patients for surgery, reducing the risk of perioperative complications, length of hospital stay, readmission and mortality. Further, it can facilitate collaborative decision-making and operational planning.

Objective

To develop effective pre-operative risk assessment algorithms (referred to as Patient Optimizer or POP) using Machine Learning (ML) that predict the development of post-operative complications and provide pilot data to inform the design of a larger prospective study.

Methods

After institutional ethics approval, we developed a base model that encapsulates the standard manual approach of combining patient-risk and procedure-risk. In an automated process, additional variables were included and tested with 10-fold cross-validation, and the best performing features were selected. The models were evaluated and confidence intervals calculated using bootstrapping. Clinical expertise was used to restrict the cardinality of categorical variables (e.g. pathology results) by including the most clinically relevant values. The models were created with logistic regression (LR) and extreme gradient-boosted trees using XGBoost (Chen and Guestrin, 2016). We evaluated performance using the area under the receiver operating characteristic curve (AUROC) and the area under the precision-recall curve (AUPRC). Data was obtained from a metropolitan university teaching hospital from January 2015 to July 2020. Data collection was restricted to adult patients undergoing elective surgery.

Results

A total of 11,475 adult admissions were included. The performance of XGBoost and LR was very similar across endpoints and metrics. For predicting the risk of any post-operative complication, kidney failure and length-of-stay (LOS), POP with XGBoost achieved an AUROC (95%CI) of 0.755 (0.744, 0.767), 0.869 (0.846, 0.891) and 0.841 (0.833, 0.847) respectively and AUPRC of 0.651 (0.632, 0.669), 0.336 (0.282, 0.390) and 0.741 (0.729, 0.753) respectively. For 30-day readmission and in-patient mortality, POP with XGBoost achieved an AUROC (95%CI) of 0.610 (0.587, 0.635) and 0.866 (0.777, 0.943) respectively and AUPRC of 0.116 (0.104, 0.132) and 0.031 (0.015, 0.072) respectively.

Conclusion

The POP algorithms effectively predicted any post-operative complication, kidney failure and LOS in the sample population. A larger study is justified to improve the algorithm to better predict complications and length of hospital stay. A larger dataset may also improve the prediction of additional specific complications, readmission and mortality.

Available only for authorised users

False Positive Rate is FP/(FP+TN). If FP is high, but TN is very large, the denominator remains high and the rate low.

Swarna SR, Boyapati S, Dutt V, Bajaj K. Deep learning in dynamic modeling of medical imaging: a review study. Proceedings of the 3rd International Conference on Intelligent Sustainable Systems, ICISS 2020. 2020. p. 745–749. https://doi.org/10.1109/ICISS49785.2020.9315990.

Suzuki K. Overview of deep learning in medical imaging. Radiol Phys Technol. 2017;10(3):257–73. https://doi.org/10.1007/S12194-017-0406-5.CrossRefPubMed

Reilly JR, Gabbe BJ, Brown WA, Hodgson CL, Myles PS. Systematic review of perioperative mortality risk prediction models for adults undergoing inpatient non-cardiac surgery. ANZ J Surg. 2021;91(5):860–70. https://doi.org/10.1111/ans.16255.CrossRefPubMed

Story DA. Postoperative complications in Australia and New Zealand (the REASON study). Perioper Med. 2013;2(1):2–4. https://doi.org/10.1186/2047-0525-2-16.MathSciNetCrossRef

Bihorac A, Ozrazgat-Baslanti T, Ebadi A, Motaei A, Madkour M, Pardalos PM, et al. MySurgeryRisk: development and validation of a machine-learning risk algorithm for major complications and death after surgery. Ann Surg. 2019;269(4):652–62. https://doi.org/10.1097/SLA.0000000000002706.CrossRefPubMed

Weiser TG, Regenbogen SE, Thompson KD, Haynes AB, Lipsitz SR, Berry WR, et al. An estimation of the global volume of surgery: a modelling strategy based on available data. Lancet. 2008;372(9633):139–44. https://doi.org/10.1016/S0140-6736(08)60878-8.CrossRefPubMed

Duckett S, Jorm C, Moran G, Parsonage H. Safer care saves money. 2018. http://www.grattan.edu.au/. Accessed 2 Jan 2022.

Wijeysundera DN. Predicting outcomes: Is there utility in risk scores? Can J Anaesth. 2016;63(2):148–58. https://doi.org/10.1007/S12630-015-0537-2.CrossRefPubMed

Swart M, Carlisle JB, Goddard J. Using predicted 30 day mortality to plan postoperative colorectal surgery care: A cohort study. Br J Anaesth. 2017;118(1):100–4. https://doi.org/10.1093/bja/aew402.CrossRefPubMed

10.

Khuri SF. The NSQIP: A new frontier in surgery. Surgery. 2005;138(5):837–43. https://doi.org/10.1016/J.SURG.2005.08.016.CrossRefPubMed

11.

Protopapa KL, Simpson JC, Smith NCE, Moonesinghe SR. Development and validation of the Surgical Outcome Risk Tool (SORT). Br J Surg. 2014;101(13):1774–83. https://doi.org/10.1002/bjs.9638.CrossRefPubMed

12.

Wolters U, Wolf T, Stützer H, Schröder T. ASA classification and perioperative variables as predictors of postoperative outcome. Br J Anaesth. 1996;77(2):217–22. https://doi.org/10.1093/BJA/77.2.217.CrossRefPubMed

13.

Hill BL, Brown R, Gabel E, Rakocz N, Lee C, Cannesson M, et al. An automated machine learning-based model predicts postoperative mortality using readily-extractable preoperative electronic health record data. Br J Anaesth. 2019;123(6):877–86. https://doi.org/10.1016/j.bja.2019.07.030.CrossRefPubMedPubMedCentral

14.

Chiew CJ, Liu N, Wong TH, Sim YE, Abdullah HR. Utilizing Machine Learning Methods for Preoperative Prediction of Postsurgical Mortality and Intensive Care Unit Admission. Ann Surg. 2020;272(6):1133–9. https://doi.org/10.1097/SLA.0000000000003297.CrossRefPubMed

15.

Corey KM, Kashyap S, Lorenzi E, Lagoo-Deenadayalan SA, Heller K, Whalen K, et al. Development and validation of machine learning models to identify high-risk surgical patients using automatically curated electronic health record data (Pythia): A retrospective, single-site study. PLoS Med. 2018;15(11):e1002701. https://doi.org/10.1371/JOURNAL.PMED.1002701.CrossRefPubMedPubMedCentral

16.

Brennan M, Puri S, Ozrazgat-Baslanti T, Feng Z, Ruppert M, Hashemighouchani H, et al. Comparing clinical judgment with the MySurgeryRisk algorithm for preoperative risk assessment: A pilot usability study. Surgery. 2019;165(5):1035–45. https://doi.org/10.1016/J.SURG.2019.01.002.CrossRefPubMed

17.

Rajkomar A, Oren E, Chen K, Dai AM, Hajaj N, Hardt M, et al. Scalable and accurate deep learning with electronic health records. NPJ Digit Med. 2018;1(1):1–10. https://doi.org/10.1038/s41746-018-0029-1.CrossRef

18.

Wang L, Tong L, Davis D, Arnold T, Esposito T. The application of unsupervised deep learning in predictive models using electronic health records. BMC Med Res Methodol. 2020;20(1):1–9. https://doi.org/10.1186/s12874-020-00923-1.CrossRef

19.

Campbell D, Boyle L, Soakell-Ho M, Hider P, Wilson L, Koea J, et al. National risk prediction model for perioperative mortality in non-cardiac surgery. Br J Surg. 2019;106(11):1549–57. https://doi.org/10.1002/BJS.11232.CrossRefPubMed

20.

Le Manach Y, Collins G, Rodseth R, Le Bihan-Benjamin C, Biccard B, Riou B, et al. Preoperative Score to Predict Postoperative Mortality (POSPOM)Derivation and Validation. Anesthesiology. 2016;124(3):570–9. https://doi.org/10.1097/ALN.0000000000000972.CrossRefPubMed

21.

Flaks-Manov N, Topaz M, Hoshen M, Balicer RD, Shadmi E. Identifying patients at highest-risk: the best timing to apply a readmission predictive model. BMC Med Inform Decis Mak. 2019;19(1):1–9. https://doi.org/10.1186/s12911-019-0836-6.CrossRef

22.

Flaks-Manov N, Srulovici E, Yahalom R, Perry-Mezre H, Balicer R, Shadmi E. Preventing hospital readmissions: healthcare providers’ perspectives on “impactibility’’ beyond EHR 30-day readmission risk prediction. J Gen Intern Med. 2020;35(5):1484–9. https://doi.org/10.1007/s11606-020-05739-9.CrossRefPubMedPubMedCentral

23.

Mišić VV, Gabel E, Hofer I, Rajaram K, Mahajan A. Machine Learning Prediction of Postoperative Emergency Department Hospital Readmission. Anesthesiology. 2020;132(5):968–80. https://doi.org/10.1097/ALN.0000000000003140.CrossRefPubMed

24.

Flaks-Manov N, Shadmi E, Yahalom R, Perry-Mezre H, Balicer RD, Srulovici E. Identification of elderly patients at risk for 30-day readmission: clinical insight beyond big data prediction. J Nurs Manag. 2021:1–11. https://doi.org/10.1111/jonm.13495.

25.

Xue B, Li D, Lu C, King CR, Wildes T, Avidan MS, et al. Use of Machine Learning to Develop and Evaluate Models Using Preoperative and Intraoperative Data to Identify Risks of Postoperative Complications. JAMA Netw Open. 2021;4(3):e212240. https://doi.org/10.1001/jamanetworkopen.2021.2240.

26.

Hospitals at a glance 2017–18, Surgery in Australia’s hospitals. 2019. https://www.aihw.gov.au/reports/hospitals/hospitals-at-a-glance-2017-18/contents/surgery-in-australia-s-hospitals. Accessed 29 Jan 2024.

27.

Collins GS, Reitsma JB, Altman DG, Moons KGM. Transparent reporting of a multivariable prediction model for individual prognosis or diagnosis (TRIPOD): The TRIPOD Statement. BMC Med. 2015;13(1):1–10. https://doi.org/10.1186/S12916-014-0241-Z/TABLES/1.CrossRefPubMedPubMedCentral

28.

Pedregosa F, Varoquaux G, Gramfort A, Michel V, Thirion B, Grisel O, et al. Scikit-learn: Machine Learning in Python. J Mach Learn Res. 2011;12:2825–30.MathSciNet

29.

Chen T, Guestrin C. XGBoost: a scalable tree boosting system. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. New York: ACM; 2016. https://doi.org/10.1145/2939672. http://dx.doi.org/10.1145/2939672.2939785.

30.

Multicollinearity Alin A. Wiley Interdiscip Rev. Comput Stat. 2010;2(3):370–4. https://doi.org/10.1002/wics.84.CrossRef

31.

Midi H, Sarkar SK, Rana S. Collinearity diagnostics of binary logistic regression model. 2013;13(3):253–67. https://doi.org/10.1080/09720502.2010.10700699.CrossRef

32.

Kuhn M, Johnson K. Feature engineering and selection: a practical approach for predictive models. 2019. http://www.feat.engineering/index.html. Accessed 13 Mar 2022.

33.

Saito T, Rehmsmeier M. The Precision-Recall Plot Is More Informative than the ROC Plot When Evaluating Binary Classifiers on Imbalanced Datasets. PLoS ONE. 2015;10(3):e0118432. https://doi.org/10.1371/JOURNAL.PONE.0118432.CrossRefPubMedPubMedCentral

34.

Štrumbelj E, Kononenko I. Explaining prediction models and individual predictions with feature contributions. Knowl Inf Syst. 2014;41(3):647–65. https://doi.org/10.1007/S10115-013-0679-X.CrossRef

35.

Qu WF, Zhou PY, Liu WR, Tian MX, Jin L, Jiang XF, et al. Age-adjusted Charlson Comorbidity Index predicts survival in intrahepatic cholangiocarcinoma patients after curative resection. Ann Transl Med. 2020;8:487. https://doi.org/10.21037/ATM.2020.03.23.

36.

Walsh C, Hripcsak G. The Effects of Data Sources, Cohort Selection, and Outcome Definition on a Predictive Model of Risk of Thirty-Day Hospital Readmissions. J Biomed Inform. 2014;52:418. https://doi.org/10.1016/J.JBI.2014.08.006.CrossRefPubMedPubMedCentral

37.

Meesters MI, von Heymann C. Optimizing Perioperative Blood and Coagulation Management During Cardiac Surgery. Anesthesiol Clin. 2019;37:713–28. https://doi.org/10.1016/j.anclin.2019.08.006.CrossRefPubMed

38.

Lang Z, Wu Y, Bao M. Coagulation Status and Surgical Approach as Predictors of Postoperative Anemia in Patients Undergoing Thoracic Surgery: A Retrospective Study. Front Surg. 2021;8:416. https://doi.org/10.3389/FSURG.2021.744810/XML/NLM.ADSCrossRef

39.

Musallam KM, Tamim HM, Richards T, Spahn DR, Rosendaal FR, Habbal A, et al. Preoperative anaemia and postoperative outcomes in non-cardiac surgery: a retrospective cohort study. Lancet. 2011;378:1396–407. https://doi.org/10.1016/S0140-6736(11)61381-0.CrossRefPubMed

40.

Gombotz H. Patient blood management is key before elective surgery. Lancet. 2011;378:1362–3. https://doi.org/10.1016/S0140-6736(11)61552-3.CrossRefPubMed

41.

Morawski K, Dvorkis Y, Monsen CB. Predicting hospitalizations from electronic health record data. Am J Manage Care. 2020;26(1). https://doi.org/10.37765/ajmc.2020.42147.

42.

Lee CK, Samad M, Hofer I, Cannesson M, Baldi P. Development and validation of an interpretable neural network for prediction of postoperative in-hospital mortality. NPJ Dig Med. 2021;4(1):1–9. https://doi.org/10.1038/s41746-020-00377-1.CrossRef

43.

Suresh H, Hunt N, Johnson A, Celi LA, Szolovits P, Ghassemi M. Clinical Intervention Prediction and Understanding with Deep Neural Networks. In: Doshi-Velez F, Fackler J, Kale D, Ranganath R, Wallace B, Wiens J, editors. Proceedings of the 2nd Machine Learning for Healthcare Conference. Proceedings of Machine Learning Research, vol. 68. PMLR; 2017. p. 322–337. https://proceedings.mlr.press/v68/suresh17a.html.

44.

Akselrod-Ballin A, Chorev M, Shoshan Y, Spiro A, Hazan A, Melamed R, et al. Predicting breast cancer by applying deep learning to linked health records and mammograms. Radiology. 2019;292:331–42. https://doi.org/10.1148/radiol.2019182622.CrossRefPubMed

45.

Mannion AF, Bianchi G, Mariaux F, Fekete TF, Reitmeir R, Moser B, et al. Can the Charlson Comorbidity Index be used to predict the ASA grade in patients undergoing spine surgery? Eur Spine J. 2020;29(12):2941–2952. https://doi.org/10.1007/s00586-020-06595-1.

Title: Development and validation of ‘Patient Optimizer’ (POP) algorithms for predicting surgical risk with machine learning
Authors: Gideon Kowadlo
Yoel Mittelberg
Milad Ghomlaghi
Daniel K. Stiglitz
Kartik Kishore
Ranjan Guha
Justin Nazareth
Laurence Weinberg
Publication date: 01-12-2024
Publisher: BioMed Central
Published in: BMC Medical Informatics and Decision Making / Issue 1/2024
Electronic ISSN: 1472-6947
DOI: https://doi.org/10.1186/s12911-024-02463-w

Keynote webinar | Spotlight on medication adherence

Springer Medicine

Development and validation of ‘Patient Optimizer’ (POP) algorithms for predicting surgical risk with machine learning

Abstract

Background

Objective

Methods

Results

Conclusion

Keynote webinar | Spotlight on medication adherence

Springer Medicine

Abstract

Background

Objective

Methods

Results

Conclusion

Please log in to get access to this content

Other articles of this Issue 1/2024

Prediction the prognosis of the poisoned patients undergoing hemodialysis using machine learning algorithms

Healthcare insurance fraud detection using data mining

A novel generative adversarial networks modelling for the class imbalance problem in high dimensional omics data

Development and application of Chinese medical ontology for diabetes mellitus

Machine learning algorithms for the prediction of adverse prognosis in patients undergoing peritoneal dialysis

Individuals’ attitudes toward digital mental health apps and implications for adoption in Portugal: web-based survey