Skip to main content
Top
Published in: Journal of General Internal Medicine 1/2020

01-01-2020 | Triage

A Gradient Boosting Machine Learning Model for Predicting Early Mortality in the Emergency Department Triage: Devising a Nine-Point Triage Score

Authors: Maximiliano Klug, MD, Yiftach Barash, MD, Sigalit Bechler, M.Sc, Yehezkel S. Resheff, Ph.D, Talia Tron, Ph.D, Avi Ironi, MD, Shelly Soffer, MD, Eyal Zimlichman, MD, Eyal Klang, MD

Published in: Journal of General Internal Medicine | Issue 1/2020

Login to get access

ABSTRACT

Background

Emergency departments (ED) are becoming increasingly overwhelmed, increasing poor outcomes. Triage scores aim to optimize the waiting time and prioritize the resource usage. Artificial intelligence (AI) algorithms offer advantages for creating predictive clinical applications.

Objective

Evaluate a state-of-the-art machine learning model for predicting mortality at the triage level and, by validating this automatic tool, improve the categorization of patients in the ED.

Design

An institutional review board (IRB) approval was granted for this retrospective study. Information of consecutive adult patients (ages 18–100) admitted at the emergency department (ED) of one hospital were retrieved (January 1, 2012–December 31, 2018). Features included the following: demographics, admission date, arrival mode, referral code, chief complaint, previous ED visits, previous hospitalizations, comorbidities, home medications, vital signs, and Emergency Severity Index (ESI). The following outcomes were evaluated: early mortality (up to 2 days post ED registration) and short-term mortality (2–30 days post ED registration). A gradient boosting model was trained on data from years 2012–2017 and examined on data from the final year (2018). The area under the curve (AUC) for mortality prediction was used as an outcome metric. Single-variable analysis was conducted to develop a nine-point triage score for early mortality.

Key Results

Overall, 799,522 ED visits were available for analysis. The early and short-term mortality rates were 0.6% and 2.5%, respectively. Models trained on the full set of features yielded an AUC of 0.962 for early mortality and 0.923 for short-term mortality. A model that utilized the nine features with the highest single-variable AUC scores (age, arrival mode, chief complaint, five primary vital signs, and ESI) yielded an AUC of 0.962 for early mortality.

Conclusion

The gradient boosting model shows high predictive ability for screening patients at risk of early mortality utilizing data available at the time of triage in the ED.
Appendix
Available only for authorised users
Literature
1.
go back to reference Carter EJ, Pouch SM, Larson EL. The relationship between emergency department crowding and patient outcomes: a systematic review. J Nurs Scholarsh. 2014;46(2):106–15.CrossRef Carter EJ, Pouch SM, Larson EL. The relationship between emergency department crowding and patient outcomes: a systematic review. J Nurs Scholarsh. 2014;46(2):106–15.CrossRef
2.
go back to reference Johnson KD, Winkelman C. The effect of emergency department crowding on patient outcomes: a literature review. Adv Emerg Nurs J. 2011;33(1):39–54.CrossRef Johnson KD, Winkelman C. The effect of emergency department crowding on patient outcomes: a literature review. Adv Emerg Nurs J. 2011;33(1):39–54.CrossRef
3.
go back to reference Pines JM, Iyer S, Disbot M, Hollander JE, Shofer FS, Datner EM. The effect of emergency department crowding on patient satisfaction for admitted patients. Acad Emerg Med. 2008;15(9):825–31.CrossRef Pines JM, Iyer S, Disbot M, Hollander JE, Shofer FS, Datner EM. The effect of emergency department crowding on patient satisfaction for admitted patients. Acad Emerg Med. 2008;15(9):825–31.CrossRef
4.
go back to reference Sun BC, Hsia RY, Weiss RE, Zingmond D, Liang L-J, Han W, et al. Effect of emergency department crowding on outcomes of admitted patients. Ann Emerg Med.. 2013;61(6):605–11. e6.CrossRef Sun BC, Hsia RY, Weiss RE, Zingmond D, Liang L-J, Han W, et al. Effect of emergency department crowding on outcomes of admitted patients. Ann Emerg Med.. 2013;61(6):605–11. e6.CrossRef
5.
go back to reference Chiu I-M, Lin Y-R, Syue Y-J, Kung C-T, Wu K-H, Li C-J. The influence of crowding on clinical practice in the emergency department. Am J Emerg Med. 2018;36(1):56–60.CrossRef Chiu I-M, Lin Y-R, Syue Y-J, Kung C-T, Wu K-H, Li C-J. The influence of crowding on clinical practice in the emergency department. Am J Emerg Med. 2018;36(1):56–60.CrossRef
6.
go back to reference Farrohknia N, Castren M, Ehrenberg A, Lind L, Oredsson S, Jonsson H, et al. Emergency department triage scales and their components: a systematic review of the scientific evidence. Scand J Trauma Resuscitation Emerg Med.. 2011;19:42.CrossRef Farrohknia N, Castren M, Ehrenberg A, Lind L, Oredsson S, Jonsson H, et al. Emergency department triage scales and their components: a systematic review of the scientific evidence. Scand J Trauma Resuscitation Emerg Med.. 2011;19:42.CrossRef
7.
go back to reference Christ M, Grossmann F, Winter D, Bingisser R, Platz E. Modern triage in the emergency department. Deutsches Arzteblatt Int. 2010;107(50):892–8. Christ M, Grossmann F, Winter D, Bingisser R, Platz E. Modern triage in the emergency department. Deutsches Arzteblatt Int. 2010;107(50):892–8.
8.
go back to reference McHugh M, Tanabe P, McClelland M, Khare RK. More Patients Are Triaged Using the Emergency Severity Index Than Any Other Triage Acuity System in the United States. Acad Emerg Med. 2012;19(1):106–9.CrossRef McHugh M, Tanabe P, McClelland M, Khare RK. More Patients Are Triaged Using the Emergency Severity Index Than Any Other Triage Acuity System in the United States. Acad Emerg Med. 2012;19(1):106–9.CrossRef
9.
go back to reference Torabi M, Moeinaddini S, Mirafzal A, Rastegari A, Sadeghkhani N. Shock index, modified shock index, and age shock index for prediction of mortality in Emergency Severity Index level 3. Am J Emerg Med. 2016;34(11):2079–83.CrossRef Torabi M, Moeinaddini S, Mirafzal A, Rastegari A, Sadeghkhani N. Shock index, modified shock index, and age shock index for prediction of mortality in Emergency Severity Index level 3. Am J Emerg Med. 2016;34(11):2079–83.CrossRef
10.
go back to reference Levin S, Toerper M, Hamrock E, Hinson JS, Barnes S, Gardner H, et al. Machine-Learning-Based Electronic Triage More Accurately Differentiates Patients With Respect to Clinical Outcomes Compared With the Emergency Severity Index. Ann Emerg Med. 2018;71(5):565–74 e2.CrossRef Levin S, Toerper M, Hamrock E, Hinson JS, Barnes S, Gardner H, et al. Machine-Learning-Based Electronic Triage More Accurately Differentiates Patients With Respect to Clinical Outcomes Compared With the Emergency Severity Index. Ann Emerg Med. 2018;71(5):565–74 e2.CrossRef
11.
go back to reference Torabi M, Mirafzal A, Rastegari A, Sadeghkhani N. Association of triage time Shock Index, Modified Shock Index, and Age Shock Index with mortality in Emergency Severity Index level 2 patients. Am J Emerg Med. 2016;34(1):63–8.CrossRef Torabi M, Mirafzal A, Rastegari A, Sadeghkhani N. Association of triage time Shock Index, Modified Shock Index, and Age Shock Index with mortality in Emergency Severity Index level 2 patients. Am J Emerg Med. 2016;34(1):63–8.CrossRef
12.
go back to reference Arya R, Wei G, McCoy JV, Crane J, Ohman-Strickland P, Eisenstein RM. Decreasing Length of Stay in the Emergency Department With a Split Emergency Severity Index 3 Patient Flow Model. Acad Emerg Med. 2013;20(11):1171–9.CrossRef Arya R, Wei G, McCoy JV, Crane J, Ohman-Strickland P, Eisenstein RM. Decreasing Length of Stay in the Emergency Department With a Split Emergency Severity Index 3 Patient Flow Model. Acad Emerg Med. 2013;20(11):1171–9.CrossRef
13.
go back to reference Rajkomar A, Oren E, Chen K, Dai AM, Hajaj N, Hardt M, et al. Scalable and accurate deep learning with electronic health records. npj Digital Med.. 2018;1(1):18.CrossRef Rajkomar A, Oren E, Chen K, Dai AM, Hajaj N, Hardt M, et al. Scalable and accurate deep learning with electronic health records. npj Digital Med.. 2018;1(1):18.CrossRef
14.
go back to reference Stewart J, Sprivulis P, Dwivedi G. Artificial intelligence and machine learning in emergency medicine. Emerg Med Australas. 0(0). Stewart J, Sprivulis P, Dwivedi G. Artificial intelligence and machine learning in emergency medicine. Emerg Med Australas. 0(0).
15.
go back to reference Coslovsky M, Takala J, Exadaktylos AK, Martinolli L, Merz TM. A clinical prediction model to identify patients at high risk of death in the emergency department. Intensive Care Med. 2015;41(6):1029–36.CrossRef Coslovsky M, Takala J, Exadaktylos AK, Martinolli L, Merz TM. A clinical prediction model to identify patients at high risk of death in the emergency department. Intensive Care Med. 2015;41(6):1029–36.CrossRef
16.
go back to reference Pearl A, Bar-Or R, Bar-Or D. An artificial neural network derived trauma outcome prediction score as an aid to triage for non-clinicians. Stud Health Technol Inform. 2008;136:253.PubMed Pearl A, Bar-Or R, Bar-Or D. An artificial neural network derived trauma outcome prediction score as an aid to triage for non-clinicians. Stud Health Technol Inform. 2008;136:253.PubMed
17.
go back to reference Dugas AF, Kirsch TD, Toerper M, Korley F, Yenokyan G, France D, et al. An Electronic Emergency Triage System to Improve Patient Distribution by Critical Outcomes. J Emerg Med. 2016;50(6):910–8.CrossRef Dugas AF, Kirsch TD, Toerper M, Korley F, Yenokyan G, France D, et al. An Electronic Emergency Triage System to Improve Patient Distribution by Critical Outcomes. J Emerg Med. 2016;50(6):910–8.CrossRef
18.
go back to reference Teubner DJ, Considine J, Hakendorf P, Kim S, Bersten AD. Model to predict inpatient mortality from information gathered at presentation to an emergency department: The Triage Information Mortality Model (TIMM). Emerg Med Australas. 2015;27(4):300–6.CrossRef Teubner DJ, Considine J, Hakendorf P, Kim S, Bersten AD. Model to predict inpatient mortality from information gathered at presentation to an emergency department: The Triage Information Mortality Model (TIMM). Emerg Med Australas. 2015;27(4):300–6.CrossRef
19.
go back to reference Schuetz P, Hausfater P, Amin D, Haubitz S, Fassler L, Grolimund E, et al. Optimizing triage and hospitalization in adult general medical emergency patients: the triage project. BMC Emerg Med. 2013;13:12.CrossRef Schuetz P, Hausfater P, Amin D, Haubitz S, Fassler L, Grolimund E, et al. Optimizing triage and hospitalization in adult general medical emergency patients: the triage project. BMC Emerg Med. 2013;13:12.CrossRef
20.
go back to reference Barak-Corren Y, Israelit SH, Reis BY. Progressive prediction of hospitalisation in the emergency department: uncovering hidden patterns to improve patient flow. Emerg Med J. 2017;34(5):308–14.CrossRef Barak-Corren Y, Israelit SH, Reis BY. Progressive prediction of hospitalisation in the emergency department: uncovering hidden patterns to improve patient flow. Emerg Med J. 2017;34(5):308–14.CrossRef
21.
go back to reference Sun Y, Heng BH, Tay SY, Seow E. Predicting hospital admissions at emergency department triage using routine administrative data. Acad Emerg Med. 2011;18(8):844–50.CrossRef Sun Y, Heng BH, Tay SY, Seow E. Predicting hospital admissions at emergency department triage using routine administrative data. Acad Emerg Med. 2011;18(8):844–50.CrossRef
22.
go back to reference Barak-Corren Y, Fine AM, Reis BY. Early Prediction Model of Patient Hospitalization From the Pediatric Emergency Department. Pediatrics. 2017;139(5).CrossRef Barak-Corren Y, Fine AM, Reis BY. Early Prediction Model of Patient Hospitalization From the Pediatric Emergency Department. Pediatrics. 2017;139(5).CrossRef
23.
go back to reference Chen T, Guestrin C, editors. Xgboost: A scalable tree boosting system. Proceedings of the 22nd acm sigkdd international conference on knowledge discovery and data mining; 2016: ACM. Chen T, Guestrin C, editors. Xgboost: A scalable tree boosting system. Proceedings of the 22nd acm sigkdd international conference on knowledge discovery and data mining; 2016: ACM.
25.
go back to reference Biau G, Scornet E. A random forest guided tour. Test. 2016;25(2):197–227.CrossRef Biau G, Scornet E. A random forest guided tour. Test. 2016;25(2):197–227.CrossRef
26.
go back to reference Hong WS, Haimovich AD, Taylor RA. Predicting hospital admission at emergency department triage using machine learning. PLoS One. 2018;13(7):e0201016.CrossRef Hong WS, Haimovich AD, Taylor RA. Predicting hospital admission at emergency department triage using machine learning. PLoS One. 2018;13(7):e0201016.CrossRef
27.
go back to reference Qiao Z, Sun N, Li X, Xia E, Zhao S, Qin Y. Using Machine Learning Approaches for Emergency Room Visit Prediction Based on Electronic Health Record Data. Stud Health Technol Inform. 2018;247:111–5.PubMed Qiao Z, Sun N, Li X, Xia E, Zhao S, Qin Y. Using Machine Learning Approaches for Emergency Room Visit Prediction Based on Electronic Health Record Data. Stud Health Technol Inform. 2018;247:111–5.PubMed
28.
go back to reference Goto T, Camargo Jr CA, Faridi MK, Yun BJ, Hasegawa K. Machine learning approaches for predicting disposition of asthma and COPD exacerbations in the ED. Am J Emerg Med. 2018;36(9):1650–4.CrossRef Goto T, Camargo Jr CA, Faridi MK, Yun BJ, Hasegawa K. Machine learning approaches for predicting disposition of asthma and COPD exacerbations in the ED. Am J Emerg Med. 2018;36(9):1650–4.CrossRef
30.
go back to reference Ho EL, Tan I, Lee I, Wu P, Chong H. Predicting Readmission at Early Hospitalization Using Electronic Health Data: A Customized Model Development. Int J Integrated Care. 2017;17(5). Ho EL, Tan I, Lee I, Wu P, Chong H. Predicting Readmission at Early Hospitalization Using Electronic Health Data: A Customized Model Development. Int J Integrated Care. 2017;17(5).
31.
go back to reference Taylor RA, Moore CL, Cheung K-H, Brandt C. Predicting urinary tract infections in the emergency department with machine learning. PLoS One. 2018;13(3):e0194085.CrossRef Taylor RA, Moore CL, Cheung K-H, Brandt C. Predicting urinary tract infections in the emergency department with machine learning. PLoS One. 2018;13(3):e0194085.CrossRef
32.
go back to reference Hill B, Brown RP, Gabel E, Lee C, Cannesson M, Loohuis LO, et al. Preoperative predictions of in-hospital mortality using electronic medical record data. bioRxiv. 2018:329813. Hill B, Brown RP, Gabel E, Lee C, Cannesson M, Loohuis LO, et al. Preoperative predictions of in-hospital mortality using electronic medical record data. bioRxiv. 2018:329813.
33.
go back to reference Maali Y, Perez-Concha O, Coiera E, Roffe D, Day RO, Gallego B. Predicting 7-day, 30-day and 60-day all-cause unplanned readmission: a case study of a Sydney hospital. BMC Med Inform Decis Mak. 2018;18(1):1.CrossRef Maali Y, Perez-Concha O, Coiera E, Roffe D, Day RO, Gallego B. Predicting 7-day, 30-day and 60-day all-cause unplanned readmission: a case study of a Sydney hospital. BMC Med Inform Decis Mak. 2018;18(1):1.CrossRef
34.
go back to reference Awad A, Bader-El-Den M, McNicholas J, Briggs J. Early hospital mortality prediction of intensive care unit patients using an ensemble learning approach. Int J Med Inform. 2017;108:185–95.CrossRef Awad A, Bader-El-Den M, McNicholas J, Briggs J. Early hospital mortality prediction of intensive care unit patients using an ensemble learning approach. Int J Med Inform. 2017;108:185–95.CrossRef
Metadata
Title
A Gradient Boosting Machine Learning Model for Predicting Early Mortality in the Emergency Department Triage: Devising a Nine-Point Triage Score
Authors
Maximiliano Klug, MD
Yiftach Barash, MD
Sigalit Bechler, M.Sc
Yehezkel S. Resheff, Ph.D
Talia Tron, Ph.D
Avi Ironi, MD
Shelly Soffer, MD
Eyal Zimlichman, MD
Eyal Klang, MD
Publication date
01-01-2020
Publisher
Springer US
Keyword
Triage
Published in
Journal of General Internal Medicine / Issue 1/2020
Print ISSN: 0884-8734
Electronic ISSN: 1525-1497
DOI
https://doi.org/10.1007/s11606-019-05512-7

Other articles of this Issue 1/2020

Journal of General Internal Medicine 1/2020 Go to the issue