Skip to main content
Key Specialties
Cardiology Diabetology Endocrinology Gastroenterology Geriatrics and Gerontology Gynecology and Obstetrics Hematology Infectious Disease Internal Medicine Nephrology Neurology Oncology Pediatrics Respiratory Medicine Rheumatology Surgery
Congress Coverage
2024 ACC Congress 2023 ESMO Congress 2023 EASD Congress 2023 ERS Congress 2023 ESC Congress 2023 EHA Congress 2023 ASCO Annual Meeting
Clinical Collections
Advances in Alzheimer’s

Keynote webinar | Spotlight on sleep in brain health

LIVE: Thursday 2nd May 2024, 18:00-19:30 (CEST)

Quality sleep is essential for health. But what happens to our brains when sleep patterns are disturbed? Join our experts to explore the interplay between sleep disruption and neurological diseases, and the questions that you need to be asking your patients to help you prevent the harmful effects of sleep deprivation.
ADVANCED SEARCH
Log in
Top
Critical Care
Published in: Critical Care 1/2018

Open Access 01-12-2018 | Research

Multicenter derivation and validation of an early warning score for acute respiratory failure or death in the hospital

Authors: Mikhail A Dziadzko, Paul J Novotny, Jeff Sloan, Ognjen Gajic, Vitaly Herasevich, Parsa Mirhaji, Yiyuan Wu, Michelle Ng Gong

Published in: Critical Care | Issue 1/2018

Login to get access

Abstract

Background

Acute respiratory failure occurs frequently in hospitalized patients and often starts before ICU admission. A risk stratification tool to predict mortality and risk for mechanical ventilation (MV) may allow for earlier evaluation and intervention. We developed and validated an automated electronic health record (EHR)-based model—Accurate Prediction of Prolonged Ventilation (APPROVE)—to identify patients at risk of death or respiratory failure requiring >= 48 h of MV.

Methods

This was an observational study of adults admitted to four hospitals in 2013 or a fifth hospital in 2017. Clinical data were extracted from the EHRs. The 2013 patients were randomly split 50:50 into a derivation/validation cohort. The qualifying event was death or intubation leading to MV >= 48 h. Random forest method was used in model derivation. APPROVE was calculated retrospectively whenever data were available in 2013, and prospectively every 4 h after hospital admission in 2017. The Modified Early Warning Score (MEWS) and National Early Warning Score (NEWS) were calculated at the same times as APPROVE. Clinicians were not alerted except for APPROVE in 2017cohort.

Results

There were 68,775 admissions in 2013 and 2258 in 2017. APPROVE had an area under the receiver operator curve of 0.87 (95% CI 0.85–0.88) in 2013 and 0.90 (95% CI 0.84–0.95) in 2017, which is significantly better than the MEWS and NEWS in 2013 but similar to the MEWS and NEWS in 2017. At a threshold of > 0.25, APPROVE had similar sensitivity and positive predictive value (PPV) (sensitivity 63% and PPV 21% in 2013 vs 64% and 16%, respectively, in 2017). Compared to APPROVE in 2013, at a threshold to achieve comparable PPV (19% at MEWS > 4 and 22% at NEWS > 6), the MEWS and NEWS had lower sensitivity (16% for MEWS and NEWS). Similarly in 2017, at a comparable sensitivity threshold (64% for APPROVE > 0.25 and 67% for MEWS and NEWS > 4), more patients who triggered an alert developed the event with APPROVE (PPV 16%) while achieving a lower false positive rate (FPR 5%) compared to the MEWS (PPV 7%, FPR 14%) and NEWS (PPV 4%, FPR 25%).

Conclusions

An automated EHR model to identify patients at high risk of MV or death was validated retrospectively and prospectively, and was determined to be feasible for real-time risk identification.

Trial registration

ClinicalTrials.gov, NCT02488174. Registered on 18 March 2015.
Appendix
Available only for authorised users
Literature
1.
Stefan MS, Shieh MS, Pekow PS, Rothberg MB, Steingrub JS, Lagu T, Lindenauer PK. Epidemiology and outcomes of acute respiratory failure in the United States, 2001 to 2009: a national survey. J Hosp Med. 2013;8(2):76–82.CrossRef
2.
Wunsch H, Guerra C, Barnato AE, Angus DC, Li G, Linde-Zwirble WT. Three-year outcomes for Medicare beneficiaries who survive intensive care. JAMA. 2010;303(9):849–56.CrossRef
3.
Barrett ML, Smith MW, Elixhauser A, Honigman LS, Pines JM. Utilization of Intensive Care Services, 2011. HCUP Statistical Brief #185. Rockville: Agency for Healthcare Research and Quality; 2014. p. 14.
4.
Wilcox ME, Herridge MS. Long-term outcomes in patients surviving acute respiratory distress syndrome. Semin Respir Crit Care Med. 2010;31(1):55–65.CrossRef
5.
Schein RM, Hazday N, Pena M, Ruben BH, Sprung CL. Clinical antecedents to in-hospital cardiopulmonary arrest. Chest. 1990;98(6):1388–92.CrossRef
6.
Chen J, Bellomo R, Flabouris A, Hillman K, Assareh H, Ou L. Delayed emergency team calls and associated hospital mortality: a multicenter study. Crit Care Med. 2015;43(10):2059–65.CrossRef
7.
Thornlow DK, Oddone E, Anderson R. Cascade iatrogenesis: a case-control study to detect postoperative respiratory failure in hospitalized older adults. Res Gerontol Nurs. 2014;7(2):66–77.CrossRef
8.
Barwise A, Thongprayoon C, Gajic O, Jensen J, Herasevich V, Pickering BW. Delayed rapid response team activation is associated with increased hospital mortality, morbidity, and length of stay in a tertiary care institution. Crit Care Med. 2016;44(1):54–63.CrossRef
9.
Morgan RJM, Williams F, MM W. An early warning scoring system for detecting developing critical illness. Clin Intensive Care. 1997;8:100.
10.
Smith ME, Chiovaro JC, O'Neil M, Kansagara D, Quinones AR, Freeman M, Motu'apuaka ML, Slatore CG. Early warning system scores for clinical deterioration in hospitalized patients: a systematic review. Ann Am Thorac Soc. 2014;11(9):1454–65.CrossRef
11.
Nates JL, Nunnally M, Kleinpell R, Blosser S, Goldner J, Birriel B, Fowler CS, Byrum D, Miles WS, Bailey H, et al. ICU admission, discharge, and triage guidelines: a framework to enhance clinical operations, development of institutional policies, and further research. Crit Care Med. 2016;44(8):1553–602.CrossRef
12.
Admon AJ, Seymour CW, Gershengorn HB, Wunsch H, Cooke CR. Hospital-level variation in ICU admission and critical care procedures for patients hospitalized for pulmonary embolism. Chest. 2014;146(6):1452–61.CrossRef
13.
Gershengorn HB, Iwashyna TJ, Cooke CR, Scales DC, Kahn JM, Wunsch H. Variation in use of intensive care for adults with diabetic ketoacidosis. Crit Care Med. 2012;40(7):2009–15.CrossRef
14.
Safavi KC, Dharmarajan K, Kim N, Strait KM, Li SX, Chen SI, Lagu T, Krumholz HM. Variation exists in rates of admission to intensive care units for heart failure patients across hospitals in the United States. Circulation. 2013;127(8):923–9.CrossRef
15.
Admon AJ, Wunsch H, Iwashyna TJ, Cooke CR. Hospital contributions to variability in the use of ICUs among elderly medicare recipients. Crit Care Med. 2017;45(1):75–84.CrossRef
16.
Chen LM, Render M, Sales A, Kennedy EH, Wiitala W, Hofer TP. Intensive care unit admitting patterns in the Veterans Affairs health care system. Arch Intern Med. 2012;172(16):1220–6.CrossRef
17.
Chen R, Strait KM, Dharmarajan K, Li SX, Ranasinghe I, Martin J, Fazel R, Masoudi FA, Cooke CR, Nallamothu BK, et al. Hospital variation in admission to intensive care units for patients with acute myocardial infarction. Am Heart J. 2015;170(6):1161–9.CrossRef
18.
Wunsch H, Linde-Zwirble WT, Angus DC, Hartman ME, Milbrandt EB, Kahn JM. The epidemiology of mechanical ventilation use in the United States. Crit Care Med. 2010;38(10):1947–53.CrossRef
19.
Churpek MM, Yuen TC, Winslow C, Robicsek AA, Meltzer DO, Gibbons RD, Edelson DP. Multicenter development and validation of a risk stratification tool for ward patients. Am J Respir Crit Care Med. 2014;190(6):649–55.CrossRef
20.
Churpek MM, Yuen TC, Winslow C, Meltzer DO, Kattan MW, Edelson DP. Multicenter comparison of machine learning methods and conventional regression for predicting clinical deterioration on the wards. Crit Care Med. 2016;44(2):368–74.CrossRef
21.
Escobar GJ, LaGuardia JC, Turk BJ, Ragins A, Kipnis P, Draper D. Early detection of impending physiologic deterioration among patients who are not in intensive care: development of predictive models using data from an automated electronic medical record. J Hosp Med. 2012;7(5):388–95.CrossRef
22.
Churpek MM, Snyder A, Han X, Sokol S, Pettit N, Howell MD, Edelson DP. Quick sepsis-related organ failure assessment, systemic inflammatory response syndrome, and early warning scores for detecting clinical deterioration in infected patients outside the intensive care unit. Am J Respir Crit Care Med. 2017;195(7):906–11.CrossRef
23.
Yu S, Leung S, Heo M, Soto GJ, Shah RT, Gunda S, Gong MN. Comparison of risk prediction scoring systems for ward patients: a retrospective nested case-control study. Crit Care. 2014;18(3):R132.CrossRef
24.
Goulden R, Hoyle MC, Monis J, Railton D, Riley V, Martin P, Martina R, Nsutebu E. qSOFA, SIRS and NEWS for predicting inhospital mortality and ICU admission in emergency admissions treated as sepsis. Emerg Med J. 2018;35(6):345–9.CrossRef
25.
Brabrand M, Hallas P, Hansen SN, Jensen KM, Madsen JLB, Posth S. Using scores to identify patients at risk of short term mortality at arrival to the acute medical unit: a validation study of six existing scores. Eur J Intern Med. 2017;45:32–6.CrossRef
26.
Escobar GJ, Gardner MN, Greene JD, Draper D, Kipnis P. Risk-adjusting hospital mortality using a comprehensive electronic record in an integrated health care delivery system. Med Care. 2013;51(5):446–53.CrossRef
27.
Kang MA, Churpek MM, Zadravecz FJ, Adhikari R, Twu NM, Edelson DP. Real-time risk prediction on the wards: a feasibility study. Crit Care Med. 2016;44(8):1468–73.CrossRef
28.
Finlay GD, Rothman MJ, Smith RA. Measuring the modified early warning score and the Rothman index: advantages of utilizing the electronic medical record in an early warning system. J Hosp Med. 2014;9(2):116–9.CrossRef
29.
Needham DM, Yang T, Dinglas VD, Mendez-Tellez PA, Shanholtz C, Sevransky JE, Brower RG, Pronovost PJ, Colantuoni E. Timing of low tidal volume ventilation and intensive care unit mortality in acute respiratory distress syndrome. A prospective cohort study. Am J Respir Crit Care Med. 2015;191(2):177–85.CrossRef
30.
Shehabi Y, Bellomo R, Reade MC, Bailey M, Bass F, Howe B, McArthur C, Seppelt IM, Webb S, Weisbrodt L, et al. Early intensive care sedation predicts long-term mortality in ventilated critically ill patients. Am J Respir Crit Care Med. 2012;186(8):724–31.CrossRef
31.
Strom T, Martinussen T, Toft P. A protocol of no sedation for critically ill patients receiving mechanical ventilation: a randomised trial. Lancet. 2010;375(9713):475–80.CrossRef
32.
Jaderling G, Bell M, Martling CR, Ekbom A, Konrad D. Limitations of medical treatment among patients attended by the rapid response team. Acta Anaesthesiol Scand. 2013;57(10):1268–74.CrossRef
33.
Cassel CK, Guest JA. Choosing wisely: helping physicians and patients make smart decisions about their care. JAMA. 2012;307(17):1801–2.CrossRef
34.
Quenot JP, Binquet C, Kara F, Martinet O, Ganster F, Navellou JC, Castelain V, Barraud D, Cousson J, Louis G, et al. The epidemiology of septic shock in French intensive care units: the prospective multicenter cohort EPISS study. Crit Care. 2013;17(2):R65.CrossRef
35.
Rhee C, Dantes R, Epstein L, Murphy DJ, Seymour CW, Iwashyna TJ, Kadri SS, Angus DC, Danner RL, Fiore AE, et al. Incidence and trends of sepsis in US hospitals using clinical vs claims data, 2009-2014. JAMA. 2017;318(13):1241–9.CrossRef
36.
Shah AD, Bartlett JW, Carpenter J, Nicholas O, Hemingway H. Comparison of random forest and parametric imputation models for imputing missing data using MICE: a CALIBER study. Am J Epidemiol. 2014;179(6):764–74.CrossRef
Metadata
Title
Multicenter derivation and validation of an early warning score for acute respiratory failure or death in the hospital
Authors
Mikhail A Dziadzko
Paul J Novotny
Jeff Sloan
Ognjen Gajic
Vitaly Herasevich
Parsa Mirhaji
Yiyuan Wu
Michelle Ng Gong
Publication date
01-12-2018
Publisher
BioMed Central
Published in
Critical Care / Issue 1/2018
Electronic ISSN: 1364-8535
DOI
https://doi.org/10.1186/s13054-018-2194-7

Other articles of this Issue 1/2018

Critical Care 1/2018 Go to the issue

Research

Effects of hyperoxia on vascular tone in animal models: systematic review and meta-analysis

Research

Is goal-directed fluid therapy based on dynamic variables alone sufficient to improve clinical outcomes among patients undergoing surgery? A meta-analysis

Letter

Trustworthy or flawed clinical prediction rule?

Research

Effect of decompressive laparotomy on organ function in patients with abdominal compartment syndrome: a systematic review and meta-analysis

Research

Identification of subclasses of sepsis that showed different clinical outcomes and responses to amount of fluid resuscitation: a latent profile analysis

Editorial

Is access to intensive care equitable?