Top

BMC Emergency Medicine

Published in:

01-12-2021 | Care | Research article

Development and validation of a scoring system for mortality prediction and application of standardized W statistics to assess the performance of emergency departments

Authors: Jinwoo Jeong, Sung Woo Lee, Won Young Kim, Kap Su Han, Su Jin Kim, Hyungoo Kang

Published in: BMC Emergency Medicine | Issue 1/2021

Abstract

Background

In-hospital mortality and short-term mortality are indicators that are commonly used to evaluate the outcome of emergency department (ED) treatment. Although several scoring systems and machine learning-based approaches have been suggested to grade the severity of the condition of ED patients, methods for comparing severity-adjusted mortality in general ED patients between different systems have yet to be developed. The aim of the present study was to develop a scoring system to predict mortality in ED patients using data collected at the initial evaluation and to validate the usefulness of the scoring system for comparing severity-adjusted mortality between institutions with different severity distributions.

Methods

The study was based on the registry of the National Emergency Department Information System, which is maintained by the National Emergency Medical Center of the Republic of Korea. Data from 2016 were used to construct the prediction model, and data from 2017 were used for validation. Logistic regression was used to build the mortality prediction model. Receiver operating characteristic curves were used to evaluate the performance of the prediction model. We calculated the standardized W statistic and its 95% confidence intervals using the newly developed mortality prediction model.

Results

The area under the receiver operating characteristic curve of the developed scoring system for the prediction of mortality was 0.883 (95% confidence interval [CI]: 0.882–0.884). The Ws score calculated from the 2016 dataset was 0.000 (95% CI: − 0.021 – 0.021). The Ws score calculated from the 2017 dataset was 0.049 (95% CI: 0.030–0.069).

Conclusions

The scoring system developed in the present study utilizing the parameters gathered in initial ED evaluations has acceptable performance for the prediction of in-hospital mortality. Standardized W statistics based on this scoring system can be used to compare the performance of an ED with the reference data or with the performance of other institutions.

Pines JM, Fee C, Fermann GJ, Ferroggiaro AA, Irvin CB, Mazer M, et al. The role of the Society for Academic Emergency Medicine in the development of guidelines and performance measures. Acad Emerg Med. 2010;17(11):e130–40. https://doi.org/10.1111/j.1553-2712.2010.00914.x.CrossRefPubMed

Donabedian A. Evaluating the quality of medical care. 1966. Milbank Q. 2005;83(4):691–729. https://doi.org/10.1111/j.1468-0009.2005.00397.x.CrossRefPubMedPubMedCentral

Mataloni F, Colais P, Galassi C, Davoli M, Fusco D. Patients who leave emergency department without being seen or during treatment in the Lazio region (Central Italy): determinants and short term outcomes. PLoS One. 2018;13(12):e0208914. https://doi.org/10.1371/journal.pone.0208914.CrossRefPubMedPubMedCentral

Jones P, Schimanski K. The four hour target to reduce emergency department 'waiting time': a systematic review of clinical outcomes. Emerg Med Australas. 2010;22(5):391–8. https://doi.org/10.1111/j.1742-6723.2010.01330.x.CrossRefPubMed

Zimmerman JE, Kramer AA, McNair DS, Malila FM. Acute physiology and chronic health evaluation (APACHE) IV: hospital mortality assessment for today's critically ill patients. Crit Care Med. 2006;34(5):1297–310. https://doi.org/10.1097/01.CCM.0000215112.84523.F0.CrossRefPubMed

Boyd CR, Tolson MA, Copes WS. Evaluating trauma care: the TRISS method. Trauma score and the injury severity score. J Trauma. 1987;27(4):370–8. https://doi.org/10.1097/00005373-198704000-00005.CrossRefPubMed

de Munter L, Polinder S, Lansink KW, Cnossen MC, Steyerberg EW, de Jongh MA. Mortality prediction models in the general trauma population: a systematic review. Injury. 2017;48(2):221–9. https://doi.org/10.1016/j.injury.2016.12.009.CrossRefPubMed

Jones P, Wells S, Harper A, Le Fevre J, Stewart J, Curtis E, et al. Impact of a national time target for ED length of stay on patient outcomes. N Z Med J. 2017;130(1455):15–34.PubMed

Singer AJ, Thode HC Jr, Viccellio P, Pines JM. The association between length of emergency department boarding and mortality. Acad Emerg Med. 2011;18(12):1324–9. https://doi.org/10.1111/j.1553-2712.2011.01236.x.CrossRefPubMed

10.

Kim OH, Roh YI, Kim HI, Cha YS, Cha KC, Kim H, et al. Reduced mortality in severely injured patients using hospital-based helicopter emergency medical Services in Interhospital Transport. J Korean Med Sci. 2017;32(7):1187–94. https://doi.org/10.3346/jkms.2017.32.7.1187.CrossRefPubMedPubMedCentral

11.

Jung K, Huh Y, Lee JC, Kim Y, Moon J, Youn SH, et al. Reduced mortality by physician-staffed HEMS dispatch for adult blunt trauma patients in Korea. J Korean Med Sci. 2016;31(10):1656–61. https://doi.org/10.3346/jkms.2016.31.10.1656.CrossRefPubMedPubMedCentral

12.

Kim J, Heo Y, Lee JC, Baek S, Kim Y, Moon J, et al. Effective transport for trauma patients under current circumstances in Korea: a single institution analysis of treatment outcomes for trauma patients transported via the domestic 119 service. J Korean Med Sci. 2015;30(3):336–42. https://doi.org/10.3346/jkms.2015.30.3.336.CrossRefPubMedPubMedCentral

13.

Hollis S, Yates DW, Woodford M, Foster P. Standardized comparison of performance indicators in trauma: a new approach to case-mix variation. J Trauma. 1995;38(5):763–6. https://doi.org/10.1097/00005373-199505000-00015.CrossRefPubMed

14.

Champion HR, Copes WS, Sacco WJ, Lawnick MM, Keast SL, Bain LW Jr, et al. The major trauma outcome study: establishing national norms for trauma care. J Trauma. 1990;30(11):1356–65. https://doi.org/10.1097/00005373-199011000-00008.CrossRefPubMed

15.

Yu JY, Jeong GY, Jeong OS, Chang DK, Cha WC. Machine learning and initial nursing assessment-based triage system for emergency department. Healthc Inform Res. 2020;26(1):13–9. https://doi.org/10.4258/hir.2020.26.1.13.CrossRefPubMedPubMedCentral

16.

Eckart A, Hauser SI, Kutz A, Haubitz S, Hausfater P, Amin D, et al. Combination of the National Early Warning Score (NEWS) and inflammatory biomarkers for early risk stratification in emergency department patients: results of a multinational, observational study. BMJ Open. 2019;9(1):e024636. https://doi.org/10.1136/bmjopen-2018-024636.CrossRefPubMedPubMedCentral

17.

Ryu J-H, Min M-K, Lee D-S, Yeom S-R, Lee S-H, Wang I-J, et al. Changes in Relative Importance of the 5-Level Triage System, Korean Triage and Acuity Scale, for the Disposition of Emergency Patients Induced by Forced Reduction in Its Level Number: a Multi-Center Registry-based Retrospective Cohort Study. J Korean Med Sci. 2019;34(14):e114. https://doi.org/10.3346/jkms.2019.34.e114.

18.

Lee CA, Cho JP, Choi SC, Kim HH, Park JO. Patients who leave the emergency department against medical advice. Clin Exp Emerg Med. 2016;3(2):88–94. https://doi.org/10.15441/ceem.15.015.CrossRefPubMedPubMedCentral

19.

Park S, Moon S, Pai H, Kim B. Appropriate duration of peripherally inserted central catheter maintenance to prevent central line-associated bloodstream infection. PLoS One. 2020;15(6):e0234966. https://doi.org/10.1371/journal.pone.0234966.CrossRefPubMedPubMedCentral

20.

Lee R, Choi S-M, Jo SJ, Lee J, Cho S-Y, Kim S-H, et al. Clinical characteristics and antimicrobial susceptibility trends in Citrobacter bacteremia: an 11-year single-center experience. Infect Chemother. 2019;51(1):1–9. https://doi.org/10.3947/ic.2019.51.1.1.CrossRefPubMedPubMedCentral

21.

Rochon J, Gondan M, Kieser M. To test or not to test: preliminary assessment of normality when comparing two independent samples. BMC Med Res Methodol. 2012;12(1):81. https://doi.org/10.1186/1471-2288-12-81.CrossRefPubMedPubMedCentral

22.

Fagerland MW. T-tests, non-parametric tests, and large studies—a paradox of statistical practice? BMC Med Res Methodol. 2012;12(1):1–7.CrossRef

23.

Tohira H, Jacobs I, Mountain D, Gibson N, Yeo A. Systematic review of predictive performance of injury severity scoring tools. Scandinavian J Trauma Resuscitation Emerg Med. 2012;20:63.CrossRef

24.

Wheeler MM. APACHE: an evaluation. Critical Care Nurs Quarter. 2009;32(1):46–8. https://doi.org/10.1097/01.CNQ.0000343134.12071.a5.CrossRef

25.

Zimmerman JE, Kramer AA. Outcome prediction in critical care: the acute physiology and chronic health evaluation models. Curr Opin Crit Care. 2008;14(5):491–7. https://doi.org/10.1097/MCC.0b013e32830864c0.CrossRefPubMed

26.

Le Gall J-R, Lemeshow S, Saulnier F. A new simplified acute physiology score (SAPS II) based on a European/north American multicenter study. JAMA. 1993;270(24):2957–63. https://doi.org/10.1001/jama.1993.03510240069035.CrossRefPubMed

27.

Knaus WA, Draper EA, Wagner DP, Zimmerman JE. APACHE II: a severity of disease classification system. Crit Care Med. 1985;13(10):818–29. https://doi.org/10.1097/00003246-198510000-00009.CrossRefPubMed

28.

Raita Y, Goto T, Faridi MK, Brown DFM, Camargo CA Jr, Hasegawa K. Emergency department triage prediction of clinical outcomes using machine learning models. Crit Care. 2019;23(1):64. https://doi.org/10.1186/s13054-019-2351-7.CrossRefPubMedPubMedCentral

29.

Olsson T, Terent A, Lind L. Rapid emergency medicine score: a new prognostic tool for in-hospital mortality in nonsurgical emergency department patients. J Intern Med. 2004;255(5):579–87. https://doi.org/10.1111/j.1365-2796.2004.01321.x.CrossRefPubMed

30.

Prytherch DR, Smith GB, Schmidt PE, Featherstone PI. ViEWS—towards a national early warning score for detecting adult inpatient deterioration. Resuscitation. 2010;81(8):932–7. https://doi.org/10.1016/j.resuscitation.2010.04.014.CrossRefPubMed

31.

Kwon JM, Lee Y, Lee Y, Lee S, Park J. An Algorithm Based on Deep Learning for Predicting In-Hospital Cardiac Arrest. J Am Heart Assoc. 2018;7(13):e008678. https://doi.org/10.1161/JAHA.118.008678.

Title: Development and validation of a scoring system for mortality prediction and application of standardized W statistics to assess the performance of emergency departments
Authors: Jinwoo Jeong
Sung Woo Lee
Won Young Kim
Kap Su Han
Su Jin Kim
Hyungoo Kang
Publication date: 01-12-2021
Publisher: BioMed Central
Keyword: Care
Published in: BMC Emergency Medicine / Issue 1/2021
Electronic ISSN: 1471-227X
DOI: https://doi.org/10.1186/s12873-021-00466-8

Keynote webinar | Spotlight on medication adherence

Springer Medicine

Development and validation of a scoring system for mortality prediction and application of standardized W statistics to assess the performance of emergency departments

Abstract

Background

Methods

Results

Conclusions

Keynote webinar | Spotlight on medication adherence

Springer Medicine

Abstract

Background

Methods

Results

Conclusions

Please log in to get access to this content

Other articles of this Issue 1/2021

24-hr observation unit is safe location for rapid glucose control in uncomplicated severe hyperglycaemia

Lessons learned from reviewing a hospital’s disaster response to the hydrofluoric acid leak in Gumi city in 2012

Qualitative development and content validation of the “SPART” model; a focused ethnography study of observable diagnostic and therapeutic activities in the emergency medical services care process

Predicting venous thromboembolism in hospitalized trauma patients: a combination of the Caprini score and data-driven machine learning model

Family satisfaction in the intensive care unit, a cross-sectional study from Norway

Bomb blast: imaging findings, treatment and clinical course of extremity traumas