Skip to main content
Top
Published in: Scandinavian Journal of Trauma, Resuscitation and Emergency Medicine 1/2020

Open Access 01-12-2020 | Artificial Intelligence | Original research

Artificial intelligence algorithm to predict the need for critical care in prehospital emergency medical services

Authors: Da-Young Kang, Kyung-Jae Cho, Oyeon Kwon, Joon-myoung Kwon, Ki-Hyun Jeon, Hyunho Park, Yeha Lee, Jinsik Park, Byung-Hee Oh

Published in: Scandinavian Journal of Trauma, Resuscitation and Emergency Medicine | Issue 1/2020

Login to get access

Abstract

Background

In emergency medical services (EMSs), accurately predicting the severity of a patient’s medical condition is important for the early identification of those who are vulnerable and at high-risk. In this study, we developed and validated an artificial intelligence (AI) algorithm based on deep learning to predict the need for critical care during EMS.

Methods

We conducted a retrospective observation cohort study. The algorithm was established using development data from the Korean national emergency department information system, which were collected during visits in real time from 151 emergency departments (EDs). We validated the algorithm using EMS run sheets from two EDs. The study subjects comprised adult patients who visited EDs. The endpoint was critical care, and we used age, sex, chief complaint, symptom onset to arrival time, trauma, and initial vital signs as the predicted variables.

Results

The number of patients in the development data was 8,981,181, and the validation data comprised 2604 EMS run sheets from two hospitals. The area under the receiver operating characteristic curve of the algorithm to predict the critical care was 0.867 (95% confidence interval, [0.864–0.871]). This result outperformed the Emergency Severity Index (0.839 [0.831–0.846]), Korean Triage and Acuity System (0.824 [0.815–0.832]), National Early Warning Score (0.741 [0.734–0.748]), and Modified Early Warning Score (0.696 [0.691–0.699]).

Conclusions

The AI algorithm accurately predicted the need for the critical care of patients using information during EMS and outperformed the conventional triage tools and early warning scores.
Appendix
Available only for authorised users
Literature
1.
go back to reference Moore L. Measuring quality and effectiveness of prehospital ems. Prehospital Emerg Care. 1999;3:325–31..CrossRef Moore L. Measuring quality and effectiveness of prehospital ems. Prehospital Emerg Care. 1999;3:325–31..CrossRef
2.
go back to reference Baxt WG, Jones G, Fortlage D. The trauma triage rule: a new, resource-based approach to the prehospital identification of major trauma victims. Ann Emerg Med. 1990;19:1401–6.CrossRef Baxt WG, Jones G, Fortlage D. The trauma triage rule: a new, resource-based approach to the prehospital identification of major trauma victims. Ann Emerg Med. 1990;19:1401–6.CrossRef
3.
go back to reference Lidal IB, Holte HH, Vist GE. Triage systems for pre-hospital emergency medical services - a systematic review. Scand J Trauma Resusc Emerg Med. 2013;21:28.CrossRef Lidal IB, Holte HH, Vist GE. Triage systems for pre-hospital emergency medical services - a systematic review. Scand J Trauma Resusc Emerg Med. 2013;21:28.CrossRef
4.
go back to reference Seymour CW. Prediction of critical illness during out-of-hospital emergency care. JAMA. 2010;304:747.CrossRef Seymour CW. Prediction of critical illness during out-of-hospital emergency care. JAMA. 2010;304:747.CrossRef
5.
go back to reference Kahn JM, Branas CC, Schwab CW, Asch DA. Regionalization of medical critical care: what can we learn from the trauma experience? Crit Care Med. 2008;36:3085–8.CrossRef Kahn JM, Branas CC, Schwab CW, Asch DA. Regionalization of medical critical care: what can we learn from the trauma experience? Crit Care Med. 2008;36:3085–8.CrossRef
6.
go back to reference Evans SM, Murray A, Patrick I, Fitzgerald M, Smith S, Andrianopoulos N, et al. Assessing clinical handover between paramedics and the trauma team. Injury. 2010;41:460–4.CrossRef Evans SM, Murray A, Patrick I, Fitzgerald M, Smith S, Andrianopoulos N, et al. Assessing clinical handover between paramedics and the trauma team. Injury. 2010;41:460–4.CrossRef
7.
go back to reference Dojmi Di Delupis F, Pisanelli P, Di Luccio G, Kennedy M, Tellini S, Nenci N, et al. Communication during handover in the pre-hospital/hospital interface in Italy: from evaluation to implementation of multidisciplinary training through high-fidelity simulation. Intern Emerg Med. 2014;9:575–82.CrossRef Dojmi Di Delupis F, Pisanelli P, Di Luccio G, Kennedy M, Tellini S, Nenci N, et al. Communication during handover in the pre-hospital/hospital interface in Italy: from evaluation to implementation of multidisciplinary training through high-fidelity simulation. Intern Emerg Med. 2014;9:575–82.CrossRef
8.
go back to reference Gulshan V, Peng L, Coram M, Stumpe MC, Wu D, Narayanaswamy A, et al. Development and validation of a deep learning algorithm for detection of diabetic retinopathy in retinal fundus photographs. Jama. 2016;304:649–56. Gulshan V, Peng L, Coram M, Stumpe MC, Wu D, Narayanaswamy A, et al. Development and validation of a deep learning algorithm for detection of diabetic retinopathy in retinal fundus photographs. Jama. 2016;304:649–56.
9.
go back to reference Kwon J, Jeon K-H, Kim HM, Kim MJ, Lim S, Kim K-H, et al. Deep-learning-based out-of-hospital cardiac arrest prognostic system to predict clinical outcomes. Resuscitation. 2019;139:84–91.CrossRef Kwon J, Jeon K-H, Kim HM, Kim MJ, Lim S, Kim K-H, et al. Deep-learning-based out-of-hospital cardiac arrest prognostic system to predict clinical outcomes. Resuscitation. 2019;139:84–91.CrossRef
10.
go back to reference Breiman L. Statistical Modeling : The Two Cultures. 2011;16:199–215. Breiman L. Statistical Modeling : The Two Cultures. 2011;16:199–215.
12.
go back to reference Schalkoff RJ. Pattern recognition - statistical, structural and neural approaches; 1992. Schalkoff RJ. Pattern recognition - statistical, structural and neural approaches; 1992.
13.
go back to reference Ioffe S, Szegedy C. Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift; 2015. Ioffe S, Szegedy C. Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift; 2015.
14.
go back to reference Abadi M, Barham P, Chen J, Chen Z, Davis A, Dean J, et al. TensorFlow: A System for Large-Scale Machine Learning TensorFlow: A system for large-scale machine learning. In: 12th USENIX Symp Oper Syst Des Implement (OSDI ‘16); 2016. p. 265–84. Abadi M, Barham P, Chen J, Chen Z, Davis A, Dean J, et al. TensorFlow: A System for Large-Scale Machine Learning TensorFlow: A system for large-scale machine learning. In: 12th USENIX Symp Oper Syst Des Implement (OSDI ‘16); 2016. p. 265–84.
15.
go back to reference Gilboy N, Tanabe P, Travers D. Rosenau AM. Emergency Severity Index (ESI): A Triage Tool for Emergency Department Care, Version 4: Implementation Handbook 2012 Edition; 2012. Gilboy N, Tanabe P, Travers D. Rosenau AM. Emergency Severity Index (ESI): A Triage Tool for Emergency Department Care, Version 4: Implementation Handbook 2012 Edition; 2012.
16.
go back to reference Mistry B, Stewart De Ramirez S, Kelen G, PSK S, Balhara KS, Levin S, et al. Accuracy and Reliability of Emergency Department Triage Using the Emergency Severity Index: An International Multicenter Assessment. Ann Emerg Med. 2018;71:581–7 e3.CrossRef Mistry B, Stewart De Ramirez S, Kelen G, PSK S, Balhara KS, Levin S, et al. Accuracy and Reliability of Emergency Department Triage Using the Emergency Severity Index: An International Multicenter Assessment. Ann Emerg Med. 2018;71:581–7 e3.CrossRef
17.
go back to reference Kwon H, Kim YJ, Jo YH, Lee JH, Lee JH, Kim J, et al. The Korean triage and acuity scale: associations with admission, disposition, mortality and length of stay in the emergency department. Int J Qual Heal Care. 2018. Kwon H, Kim YJ, Jo YH, Lee JH, Lee JH, Kim J, et al. The Korean triage and acuity scale: associations with admission, disposition, mortality and length of stay in the emergency department. Int J Qual Heal Care. 2018.
18.
go back to reference Lee B, Kim DK, Park JD, Kwak YH. Clinical considerations when applying vital signs in pediatric Korean triage and acuity scale. J Korean Med Sci. 2017;32:1702.CrossRef Lee B, Kim DK, Park JD, Kwak YH. Clinical considerations when applying vital signs in pediatric Korean triage and acuity scale. J Korean Med Sci. 2017;32:1702.CrossRef
19.
go back to reference Subbe CP, Kruger M, Rutherford P, Gemmel L. Validation of a modified early warning score in medical admissions. QJM. 2001;94:521–6.CrossRef Subbe CP, Kruger M, Rutherford P, Gemmel L. Validation of a modified early warning score in medical admissions. QJM. 2001;94:521–6.CrossRef
20.
go back to reference Smith GB, Prytherch DR, Meredith P, Schmidt PE, Featherstone PI. The ability of the National Early Warning Score (NEWS) to discriminate patients at risk of early cardiac arrest, unanticipated intensive care unit admission, and death. Resuscitation. 2013;84:465–70.CrossRef Smith GB, Prytherch DR, Meredith P, Schmidt PE, Featherstone PI. The ability of the National Early Warning Score (NEWS) to discriminate patients at risk of early cardiac arrest, unanticipated intensive care unit admission, and death. Resuscitation. 2013;84:465–70.CrossRef
21.
go back to reference Williams TA, Tohira H, Finn J, Perkins GD, Ho KM. The ability of early warning scores (EWS) to detect critical illness in the prehospital setting: a systematic review. Resuscitation. 2016;102:35–43.CrossRef Williams TA, Tohira H, Finn J, Perkins GD, Ho KM. The ability of early warning scores (EWS) to detect critical illness in the prehospital setting: a systematic review. Resuscitation. 2016;102:35–43.CrossRef
22.
go back to reference Jouffroy R, Saade A, Ellouze S, Carpentier A, Michaloux M, Carli P, et al. Prehospital triage of septic patients at the SAMU regulation: comparison of qSOFA, MRST, MEWS and PRESEP scores. Am J Emerg Med. 2018;36:820–4.CrossRef Jouffroy R, Saade A, Ellouze S, Carpentier A, Michaloux M, Carli P, et al. Prehospital triage of septic patients at the SAMU regulation: comparison of qSOFA, MRST, MEWS and PRESEP scores. Am J Emerg Med. 2018;36:820–4.CrossRef
23.
go back to reference Carpenter J, Bithell J. Bootstrap confidence intervals: when, which, what? A practical guide for medical statisticians. Stat Med. 2000;19:1141–64.CrossRef Carpenter J, Bithell J. Bootstrap confidence intervals: when, which, what? A practical guide for medical statisticians. Stat Med. 2000;19:1141–64.CrossRef
24.
go back to reference Dietterich TG. Ensemble methods in machine learning; 2007. Dietterich TG. Ensemble methods in machine learning; 2007.
26.
go back to reference Buschhorn HM, Strout TD, Sholl JM, Baumann MR. Emergency medical services triage using the emergency severity index: is it reliable and valid? J Emerg Nurs. 2013;39:e55–63.CrossRef Buschhorn HM, Strout TD, Sholl JM, Baumann MR. Emergency medical services triage using the emergency severity index: is it reliable and valid? J Emerg Nurs. 2013;39:e55–63.CrossRef
27.
go back to reference Leeies M, Ffrench C, Strome T, Weldon E, Bullard M, Grierson R. Prehospital application of the Canadian triage and acuity scale by emergency medical services. CJEM. 2017;19:26–31.CrossRef Leeies M, Ffrench C, Strome T, Weldon E, Bullard M, Grierson R. Prehospital application of the Canadian triage and acuity scale by emergency medical services. CJEM. 2017;19:26–31.CrossRef
28.
go back to reference Sun G, Shook TL, Kay GL. Inappropriate Use of Bivariable Analysis to Screen Risk Factors for Use in Multivariable Analysis. J Clin Epidemiol. 1996;49(8):907-16.CrossRef Sun G, Shook TL, Kay GL. Inappropriate Use of Bivariable Analysis to Screen Risk Factors for Use in Multivariable Analysis. J Clin Epidemiol. 1996;49(8):907-16.CrossRef
29.
go back to reference Bagley SC, White H, Golomb BA. Logistic regression in the medical literature: standards for use and reporting, with particular attention to one medical domain. J Clin Epidemiol. 2001;54:979–85.CrossRef Bagley SC, White H, Golomb BA. Logistic regression in the medical literature: standards for use and reporting, with particular attention to one medical domain. J Clin Epidemiol. 2001;54:979–85.CrossRef
30.
31.
go back to reference Kwon J, Lee Y, Lee Y, Lee S, Park J. An algorithm based on deep learning for predicting in-hospital cardiac arrest. J Am Heart Assoc. 2018;7:e008678.PubMedPubMedCentral Kwon J, Lee Y, Lee Y, Lee S, Park J. An algorithm based on deep learning for predicting in-hospital cardiac arrest. J Am Heart Assoc. 2018;7:e008678.PubMedPubMedCentral
32.
go back to reference Wolpert DH, Macready WG. No free lunch theorems for optimization. IEEE Trans Evol Comput. 1997;1:67–82.CrossRef Wolpert DH, Macready WG. No free lunch theorems for optimization. IEEE Trans Evol Comput. 1997;1:67–82.CrossRef
33.
go back to reference Christ M, Grossmann F, Winter D, Bingisser R, Platz E. Modern triage in the emergency department. Dtsch Arztebl Int. 2010;107:892–8.PubMedPubMedCentral Christ M, Grossmann F, Winter D, Bingisser R, Platz E. Modern triage in the emergency department. Dtsch Arztebl Int. 2010;107:892–8.PubMedPubMedCentral
34.
go back to reference Dugas AF, Kirsch TD, Toerper M, Korley F, Yenokyan G, France D, et al. An electronic emergency triage system to improve patient distribution by critical outcomes. J Emerg Med. 2016;50:910–8.CrossRef Dugas AF, Kirsch TD, Toerper M, Korley F, Yenokyan G, France D, et al. An electronic emergency triage system to improve patient distribution by critical outcomes. J Emerg Med. 2016;50:910–8.CrossRef
36.
go back to reference Chen X, Duan Y, Houthooft R, Schulman J, Sutskever I, Abbeel P. InfoGAN: Interpretable Representation Learning by Information Maximizing Generative Adversarial Nets; 2016. Chen X, Duan Y, Houthooft R, Schulman J, Sutskever I, Abbeel P. InfoGAN: Interpretable Representation Learning by Information Maximizing Generative Adversarial Nets; 2016.
37.
go back to reference Fong RC, Vedaldi A. Interpretable Explanations of Black Boxes by Meaningful Perturbation. In: Proc IEEE Int Conf Comput Vis 2017; 2017. p. 3449–57. Fong RC, Vedaldi A. Interpretable Explanations of Black Boxes by Meaningful Perturbation. In: Proc IEEE Int Conf Comput Vis 2017; 2017. p. 3449–57.
Metadata
Title
Artificial intelligence algorithm to predict the need for critical care in prehospital emergency medical services
Authors
Da-Young Kang
Kyung-Jae Cho
Oyeon Kwon
Joon-myoung Kwon
Ki-Hyun Jeon
Hyunho Park
Yeha Lee
Jinsik Park
Byung-Hee Oh
Publication date
01-12-2020
Publisher
BioMed Central
DOI
https://doi.org/10.1186/s13049-020-0713-4

Other articles of this Issue 1/2020

Scandinavian Journal of Trauma, Resuscitation and Emergency Medicine 1/2020 Go to the issue