Skip to main content
Top
Published in: Critical Care 1/2019

Open Access 01-12-2019 | Care | Research

A deep learning model for real-time mortality prediction in critically ill children

Authors: Soo Yeon Kim, Saehoon Kim, Joongbum Cho, Young Suh Kim, In Suk Sol, Youngchul Sung, Inhyeok Cho, Minseop Park, Haerin Jang, Yoon Hee Kim, Kyung Won Kim, Myung Hyun Sohn

Published in: Critical Care | Issue 1/2019

Login to get access

Abstract

Background

The rapid development in big data analytics and the data-rich environment of intensive care units together provide unprecedented opportunities for medical breakthroughs in the field of critical care. We developed and validated a machine learning-based model, the Pediatric Risk of Mortality Prediction Tool (PROMPT), for real-time prediction of all-cause mortality in pediatric intensive care units.

Methods

Utilizing two separate retrospective observational cohorts, we conducted model development and validation using a machine learning algorithm with a convolutional neural network. The development cohort comprised 1445 pediatric patients with 1977 medical encounters admitted to intensive care units from January 2011 to December 2017 at Severance Hospital (Seoul, Korea). The validation cohort included 278 patients with 364 medical encounters admitted to the pediatric intensive care unit from January 2016 to November 2017 at Samsung Medical Center.

Results

Using seven vital signs, along with patient age and body weight on intensive care unit admission, PROMPT achieved an area under the receiver operating characteristic curve in the range of 0.89–0.97 for mortality prediction 6 to 60 h prior to death. Our results demonstrated that PROMPT provided high sensitivity with specificity and outperformed the conventional severity scoring system, the Pediatric Index of Mortality, in predictive ability. Model performance was indistinguishable between the development and validation cohorts.

Conclusions

PROMPT is a deep model-based, data-driven early warning score tool that can predict mortality in critically ill children and may be useful for the timely identification of deteriorating patients.
Appendix
Available only for authorised users
Literature
1.
go back to reference Straney L, Clements A, Parslow RC, Pearson G, Shann F, Alexander J, Slater A. Paediatric index of mortality 3: an updated model for predicting mortality in pediatric intensive care*. Pediatr Crit Care Med. 2013;14:673–81.CrossRef Straney L, Clements A, Parslow RC, Pearson G, Shann F, Alexander J, Slater A. Paediatric index of mortality 3: an updated model for predicting mortality in pediatric intensive care*. Pediatr Crit Care Med. 2013;14:673–81.CrossRef
2.
go back to reference Pollack MM, Holubkov R, Funai T, Dean JM, Berger JT, Wessel DL, Meert K, Berg RA, Newth CJ, Harrison RE, Carcillo J, Dalton H, Shanley T, Jenkins TL, Tamburro R. The pediatric risk of mortality score: update 2015. Pediatr Crit Care Med. 2016;17:2–9.CrossRef Pollack MM, Holubkov R, Funai T, Dean JM, Berger JT, Wessel DL, Meert K, Berg RA, Newth CJ, Harrison RE, Carcillo J, Dalton H, Shanley T, Jenkins TL, Tamburro R. The pediatric risk of mortality score: update 2015. Pediatr Crit Care Med. 2016;17:2–9.CrossRef
3.
go back to reference Skrobik Y, Kavanagh BP. Scoring systems for the critically ill: use, misuse and abuse. Can J Anaesth. 2006;53:432–6.CrossRef Skrobik Y, Kavanagh BP. Scoring systems for the critically ill: use, misuse and abuse. Can J Anaesth. 2006;53:432–6.CrossRef
4.
go back to reference Vincent JL, Opal SM, Marshall JC. Ten reasons why we should NOT use severity scores as entry criteria for clinical trials or in our treatment decisions. Crit Care Med. 2010;38:283–7.CrossRef Vincent JL, Opal SM, Marshall JC. Ten reasons why we should NOT use severity scores as entry criteria for clinical trials or in our treatment decisions. Crit Care Med. 2010;38:283–7.CrossRef
5.
go back to reference Berlot G, Pangher A, Petrucci L, Bussani R, Lucangelo U. Anticipating events of in-hospital cardiac arrest. Eur J Emerg Med. 2004;11:24–8.CrossRef Berlot G, Pangher A, Petrucci L, Bussani R, Lucangelo U. Anticipating events of in-hospital cardiac arrest. Eur J Emerg Med. 2004;11:24–8.CrossRef
6.
go back to reference Moss TJ, Lake DE, Calland JF, Enfield KB, Delos JB, Fairchild KD, Moorman JR. Signatures of subacute potentially catastrophic illness in the ICU: model development and validation. Crit Care Med. 2016;44:1639–48.CrossRef Moss TJ, Lake DE, Calland JF, Enfield KB, Delos JB, Fairchild KD, Moorman JR. Signatures of subacute potentially catastrophic illness in the ICU: model development and validation. Crit Care Med. 2016;44:1639–48.CrossRef
7.
go back to reference Monaghan A. Detecting and managing deterioration in children. Paediatr Nurs. 2005;17:32–5.CrossRef Monaghan A. Detecting and managing deterioration in children. Paediatr Nurs. 2005;17:32–5.CrossRef
8.
go back to reference Alam N, Hobbelink EL, van Tienhoven AJ, van de Ven PM, Jansma EP, Nanayakkara PW. The impact of the use of the Early Warning Score (EWS) on patient outcomes: a systematic review. Resuscitation. 2014;85:587–94.CrossRef Alam N, Hobbelink EL, van Tienhoven AJ, van de Ven PM, Jansma EP, Nanayakkara PW. The impact of the use of the Early Warning Score (EWS) on patient outcomes: a systematic review. Resuscitation. 2014;85:587–94.CrossRef
9.
go back to reference Parshuram CS, Hutchison J, Middaugh K. Development and initial validation of the Bedside Paediatric Early Warning System score. Crit Care. 2009;13:R135.CrossRef Parshuram CS, Hutchison J, Middaugh K. Development and initial validation of the Bedside Paediatric Early Warning System score. Crit Care. 2009;13:R135.CrossRef
10.
go back to reference Skaletzky SM, Raszynski A, Totapally BR. Validation of a modified pediatric early warning system score: a retrospective case-control study. Clin Pediatr (Phila). 2012;51:431–5.CrossRef Skaletzky SM, Raszynski A, Totapally BR. Validation of a modified pediatric early warning system score: a retrospective case-control study. Clin Pediatr (Phila). 2012;51:431–5.CrossRef
11.
go back to reference Churpek MM, Yuen TC, Winslow C, Robicsek AA, Meltzer DO, Gibbons RD, Edelson DP. Multicenter development and validation of a risk stratification tool for ward patients. Am J Respir Crit Care Med. 2014;190:649–55.CrossRef Churpek MM, Yuen TC, Winslow C, Robicsek AA, Meltzer DO, Gibbons RD, Edelson DP. Multicenter development and validation of a risk stratification tool for ward patients. Am J Respir Crit Care Med. 2014;190:649–55.CrossRef
12.
go back to reference Egdell P, Finlay L, Pedley DK. The PAWS score: validation of an early warning scoring system for the initial assessment of children in the emergency department. Emerg Med J. 2008;25:745–9.CrossRef Egdell P, Finlay L, Pedley DK. The PAWS score: validation of an early warning scoring system for the initial assessment of children in the emergency department. Emerg Med J. 2008;25:745–9.CrossRef
13.
go back to reference Royston P, Moons KG, Altman DG, Vergouwe Y. Prognosis and prognostic research: developing a prognostic model. Bmj. 2009;338:b604.CrossRef Royston P, Moons KG, Altman DG, Vergouwe Y. Prognosis and prognostic research: developing a prognostic model. Bmj. 2009;338:b604.CrossRef
14.
go back to reference Baronov D, McManus M, Butler E, Chung D, Almodovar MC. Next generation patient monitor powered by in-silico physiology. Conf Proc IEEE Eng Med Biol Soc. 2015;2015:4447–53.PubMedPubMedCentral Baronov D, McManus M, Butler E, Chung D, Almodovar MC. Next generation patient monitor powered by in-silico physiology. Conf Proc IEEE Eng Med Biol Soc. 2015;2015:4447–53.PubMedPubMedCentral
15.
go back to reference Rusin CG, Acosta SI, Shekerdemian LS, Vu EL, Bavare AC, Myers RB, Patterson LW, Brady KM, Penny DJ. Prediction of imminent, severe deterioration of children with parallel circulations using real-time processing of physiologic data. J Thorac Cardiovasc Surg. 2016;152:171–7.CrossRef Rusin CG, Acosta SI, Shekerdemian LS, Vu EL, Bavare AC, Myers RB, Patterson LW, Brady KM, Penny DJ. Prediction of imminent, severe deterioration of children with parallel circulations using real-time processing of physiologic data. J Thorac Cardiovasc Surg. 2016;152:171–7.CrossRef
16.
go back to reference Yun C, Hui Y. Heterogeneous postsurgical data analytics for predictive modeling of mortality risks in intensive care units. Conf Proc IEEE Eng Med Biol Soc. 2014;2014:4310–4. Yun C, Hui Y. Heterogeneous postsurgical data analytics for predictive modeling of mortality risks in intensive care units. Conf Proc IEEE Eng Med Biol Soc. 2014;2014:4310–4.
18.
go back to reference Johnson AEW, Mark RG. Real-time mortality prediction in the intensive care unit. AMIA Annu Symp Proc. 2017;2017:994–1003.PubMed Johnson AEW, Mark RG. Real-time mortality prediction in the intensive care unit. AMIA Annu Symp Proc. 2017;2017:994–1003.PubMed
19.
go back to reference Bailly S, Meyfroidt G, Timsit JF. What’s new in ICU in 2050: big data and machine learning. Intensive Care Med. 2018;44:1524–7.CrossRef Bailly S, Meyfroidt G, Timsit JF. What’s new in ICU in 2050: big data and machine learning. Intensive Care Med. 2018;44:1524–7.CrossRef
20.
go back to reference Wong HR, Dalton HJ. The PICU perspective on monitoring hemodynamics and oxygen transport. Pediatr Crit Care Med. 2011;12:S66–8.CrossRef Wong HR, Dalton HJ. The PICU perspective on monitoring hemodynamics and oxygen transport. Pediatr Crit Care Med. 2011;12:S66–8.CrossRef
21.
go back to reference Olive MK, Owens GE. Current monitoring and innovative predictive modeling to improve care in the pediatric cardiac intensive care unit. Transl Pediatr. 2018;7:120–8.CrossRef Olive MK, Owens GE. Current monitoring and innovative predictive modeling to improve care in the pediatric cardiac intensive care unit. Transl Pediatr. 2018;7:120–8.CrossRef
22.
go back to reference Kennedy CE, Turley JP. Time series analysis as input for clinical predictive modeling: modeling cardiac arrest in a pediatric ICU. Theor Biol Med Model. 2011;8:40.CrossRef Kennedy CE, Turley JP. Time series analysis as input for clinical predictive modeling: modeling cardiac arrest in a pediatric ICU. Theor Biol Med Model. 2011;8:40.CrossRef
23.
go back to reference Che Z, Purushotham S, Khemani R, Liu Y. Interpretable deep models for ICU outcome prediction. AMIA Annu Symp Proc. 2016;2016:371–80.PubMed Che Z, Purushotham S, Khemani R, Liu Y. Interpretable deep models for ICU outcome prediction. AMIA Annu Symp Proc. 2016;2016:371–80.PubMed
24.
go back to reference Rawat W, Wang Z. Deep convolutional neural networks for image classification: a comprehensive review. Neural Comput. 2017;29:2352–449.CrossRef Rawat W, Wang Z. Deep convolutional neural networks for image classification: a comprehensive review. Neural Comput. 2017;29:2352–449.CrossRef
25.
go back to reference Anwar SM, Majid M, Qayyum A, Awais M, Alnowami M, Khan MK. Medical image analysis using convolutional neural networks: a review. J Med Syst. 2018;42:226.CrossRef Anwar SM, Majid M, Qayyum A, Awais M, Alnowami M, Khan MK. Medical image analysis using convolutional neural networks: a review. J Med Syst. 2018;42:226.CrossRef
26.
go back to reference Kam HJ, Kim HY. Learning representations for the early detection of sepsis with deep neural networks. Comput Biol Med. 2017;89:248–55.CrossRef Kam HJ, Kim HY. Learning representations for the early detection of sepsis with deep neural networks. Comput Biol Med. 2017;89:248–55.CrossRef
27.
go back to reference Kamaleswaran R, Akbilgic O, Hallman MA, West AN, Davis RL, Shah SH. Applying artificial intelligence to identify physiomarkers predicting severe sepsis in the PICU. Pediatr Crit Care Med. 2018;19:e495–503.CrossRef Kamaleswaran R, Akbilgic O, Hallman MA, West AN, Davis RL, Shah SH. Applying artificial intelligence to identify physiomarkers predicting severe sepsis in the PICU. Pediatr Crit Care Med. 2018;19:e495–503.CrossRef
28.
go back to reference Savalia S, Emamian V. Cardiac arrhythmia classification by multi-layer perceptron and convolution neural networks. Bioengineering. 2018;5:1–12.CrossRef Savalia S, Emamian V. Cardiac arrhythmia classification by multi-layer perceptron and convolution neural networks. Bioengineering. 2018;5:1–12.CrossRef
29.
go back to reference Chen T, Guestrin C. XGBoost: a scalable tree boosting system. Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. San Francisco: ACM; 2016. p. 785–94. Chen T, Guestrin C. XGBoost: a scalable tree boosting system. Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. San Francisco: ACM; 2016. p. 785–94.
30.
go back to reference Hochreiter S. #252, Schmidhuber r. long short-term memory. Neural Comput. 1997;9:1735–80.CrossRef Hochreiter S. #252, Schmidhuber r. long short-term memory. Neural Comput. 1997;9:1735–80.CrossRef
31.
go back to reference Davis J, Goadrich M. The relationship between Precision-Recall and ROC curves. Proceedings of the 23rd international conference on Machine learning. Pittsburgh: ACM; 2006. p. 233–40. Davis J, Goadrich M. The relationship between Precision-Recall and ROC curves. Proceedings of the 23rd international conference on Machine learning. Pittsburgh: ACM; 2006. p. 233–40.
32.
go back to reference Keegan MT, Gajic O, Afessa B. Severity of illness scoring systems in the intensive care unit. Crit Care Med. 2011;39:163–9.CrossRef Keegan MT, Gajic O, Afessa B. Severity of illness scoring systems in the intensive care unit. Crit Care Med. 2011;39:163–9.CrossRef
33.
go back to reference Murphy-Filkins R, Teres D, Lemeshow S, Hosmer DW. Effect of changing patient mix on the performance of an intensive care unit severity-of-illness model: how to distinguish a general from a specialty intensive care unit. Crit Care Med. 1996;24:1968–73.CrossRef Murphy-Filkins R, Teres D, Lemeshow S, Hosmer DW. Effect of changing patient mix on the performance of an intensive care unit severity-of-illness model: how to distinguish a general from a specialty intensive care unit. Crit Care Med. 1996;24:1968–73.CrossRef
34.
go back to reference Booth FV, Short M, Shorr AF, Arkins N, Bates B, Qualy RL, Levy H. Application of a population-based severity scoring system to individual patients results in frequent misclassification. Crit Care. 2005;9:R522–9.CrossRef Booth FV, Short M, Shorr AF, Arkins N, Bates B, Qualy RL, Levy H. Application of a population-based severity scoring system to individual patients results in frequent misclassification. Crit Care. 2005;9:R522–9.CrossRef
35.
go back to reference Bailey TC, Chen Y, Mao Y, Lu C, Hackmann G, Micek ST, Heard KM, Faulkner KM, Kollef MH. A trial of a real-time alert for clinical deterioration in patients hospitalized on general medical wards. J Hosp Med. 2013;8:236–42.CrossRef Bailey TC, Chen Y, Mao Y, Lu C, Hackmann G, Micek ST, Heard KM, Faulkner KM, Kollef MH. A trial of a real-time alert for clinical deterioration in patients hospitalized on general medical wards. J Hosp Med. 2013;8:236–42.CrossRef
36.
go back to reference Henry KE, Hager DN, Pronovost PJ, Saria S. A targeted real-time early warning score (TREWScore) for septic shock. Sci Transl Med. 2015;7:299ra122.CrossRef Henry KE, Hager DN, Pronovost PJ, Saria S. A targeted real-time early warning score (TREWScore) for septic shock. Sci Transl Med. 2015;7:299ra122.CrossRef
37.
go back to reference Hillman KM, Bristow PJ, Chey T, Daffurn K, Jacques T, Norman SL, Bishop GF, Simmons G. Duration of life-threatening antecedents prior to intensive care admission. Intensive Care Med. 2002;28:1629–34.CrossRef Hillman KM, Bristow PJ, Chey T, Daffurn K, Jacques T, Norman SL, Bishop GF, Simmons G. Duration of life-threatening antecedents prior to intensive care admission. Intensive Care Med. 2002;28:1629–34.CrossRef
38.
go back to reference Smith ME, Chiovaro JC, O'Neil M, Kansagara D, Quinones AR, Freeman M, Motu'apuaka ML, Slatore CG. Early warning system scores for clinical deterioration in hospitalized patients: a systematic review. Ann Am Thorac Soc. 2014;11:1454–65.CrossRef Smith ME, Chiovaro JC, O'Neil M, Kansagara D, Quinones AR, Freeman M, Motu'apuaka ML, Slatore CG. Early warning system scores for clinical deterioration in hospitalized patients: a systematic review. Ann Am Thorac Soc. 2014;11:1454–65.CrossRef
39.
go back to reference Cater DT, Tori AJ, Moser EAS, Rowan CM. Modification and assessment of the bedside pediatric early warning score in the pediatric allogeneic hematopoietic cell transplant population. Pediatr Crit Care Med. 2018;19:483–8.CrossRef Cater DT, Tori AJ, Moser EAS, Rowan CM. Modification and assessment of the bedside pediatric early warning score in the pediatric allogeneic hematopoietic cell transplant population. Pediatr Crit Care Med. 2018;19:483–8.CrossRef
40.
go back to reference Akre M, Finkelstein M, Erickson M, Liu M, Vanderbilt L, Billman G. Sensitivity of the pediatric early warning score to identify patient deterioration. Pediatrics. 2010;125:e763–9.CrossRef Akre M, Finkelstein M, Erickson M, Liu M, Vanderbilt L, Billman G. Sensitivity of the pediatric early warning score to identify patient deterioration. Pediatrics. 2010;125:e763–9.CrossRef
41.
go back to reference Parshuram CS, Dryden-Palmer K, Farrell C, Gottesman R, Gray M, Hutchison JS, Helfaer M, Hunt EA, Joffe AR, Lacroix J, Moga MA, Nadkarni V, Ninis N, Parkin PC, Wensley D, Willan AR, Tomlinson GA. Effect of a pediatric early warning system on all-cause mortality in hospitalized pediatric patients: the EPOCH randomized clinical trial. JAMA. 2018;319:1002–12.CrossRef Parshuram CS, Dryden-Palmer K, Farrell C, Gottesman R, Gray M, Hutchison JS, Helfaer M, Hunt EA, Joffe AR, Lacroix J, Moga MA, Nadkarni V, Ninis N, Parkin PC, Wensley D, Willan AR, Tomlinson GA. Effect of a pediatric early warning system on all-cause mortality in hospitalized pediatric patients: the EPOCH randomized clinical trial. JAMA. 2018;319:1002–12.CrossRef
42.
go back to reference Gold DL, Mihalov LK, Cohen DM. Evaluating the Pediatric Early Warning Score (PEWS) system for admitted patients in the pediatric emergency department. Acad Emerg Med. 2014;21:1249–56.CrossRef Gold DL, Mihalov LK, Cohen DM. Evaluating the Pediatric Early Warning Score (PEWS) system for admitted patients in the pediatric emergency department. Acad Emerg Med. 2014;21:1249–56.CrossRef
43.
go back to reference Williams TA, Tohira H, Finn J, Perkins GD, Ho KM. The ability of early warning scores (EWS) to detect critical illness in the prehospital setting: a systematic review. Resuscitation. 2016;102:35–43.CrossRef Williams TA, Tohira H, Finn J, Perkins GD, Ho KM. The ability of early warning scores (EWS) to detect critical illness in the prehospital setting: a systematic review. Resuscitation. 2016;102:35–43.CrossRef
44.
go back to reference Vincent JL, Moreno R. Clinical review: scoring systems in the critically ill. Crit Care. 2010;14:207.CrossRef Vincent JL, Moreno R. Clinical review: scoring systems in the critically ill. Crit Care. 2010;14:207.CrossRef
45.
go back to reference Liu V. Keeping score of severity scores: taking the next step. Crit Care Med. 2016;44:639–40.CrossRef Liu V. Keeping score of severity scores: taking the next step. Crit Care Med. 2016;44:639–40.CrossRef
46.
go back to reference Williams JB, Ghosh D, Wetzel RC. Applying machine learning to pediatric critical care data. Pediatr Crit Care Med. 2018;19:599–608.CrossRef Williams JB, Ghosh D, Wetzel RC. Applying machine learning to pediatric critical care data. Pediatr Crit Care Med. 2018;19:599–608.CrossRef
47.
go back to reference Delahanty RJ, Kaufman D, Jones SS. Development and evaluation of an automated machine learning algorithm for in-hospital mortality risk adjustment among critical care patients. Crit Care Med. 2018;46:e481–e8.CrossRef Delahanty RJ, Kaufman D, Jones SS. Development and evaluation of an automated machine learning algorithm for in-hospital mortality risk adjustment among critical care patients. Crit Care Med. 2018;46:e481–e8.CrossRef
48.
go back to reference Nemati S, Holder A, Razmi F, Stanley MD, Clifford GD, Buchman TG. An interpretable machine learning model for accurate prediction of Sepsis in the ICU. Crit Care Med. 2018;46:547–53.CrossRef Nemati S, Holder A, Razmi F, Stanley MD, Clifford GD, Buchman TG. An interpretable machine learning model for accurate prediction of Sepsis in the ICU. Crit Care Med. 2018;46:547–53.CrossRef
49.
go back to reference Bengio Y, Courville A, Vincent P. Representation learning: a review and new perspectives. IEEE Trans Pattern Anal Mach Intell. 2013;35:1798–828.CrossRef Bengio Y, Courville A, Vincent P. Representation learning: a review and new perspectives. IEEE Trans Pattern Anal Mach Intell. 2013;35:1798–828.CrossRef
50.
go back to reference Capan M, Hoover S, Miller KE, Pal C, Glasgow JM, Jackson EV, Arnold RC. Data-driven approach to Early Warning Score-based alert management. BMJ Open Qual. 2018;7:e000088.CrossRef Capan M, Hoover S, Miller KE, Pal C, Glasgow JM, Jackson EV, Arnold RC. Data-driven approach to Early Warning Score-based alert management. BMJ Open Qual. 2018;7:e000088.CrossRef
Metadata
Title
A deep learning model for real-time mortality prediction in critically ill children
Authors
Soo Yeon Kim
Saehoon Kim
Joongbum Cho
Young Suh Kim
In Suk Sol
Youngchul Sung
Inhyeok Cho
Minseop Park
Haerin Jang
Yoon Hee Kim
Kyung Won Kim
Myung Hyun Sohn
Publication date
01-12-2019
Publisher
BioMed Central
Keyword
Care
Published in
Critical Care / Issue 1/2019
Electronic ISSN: 1364-8535
DOI
https://doi.org/10.1186/s13054-019-2561-z

Other articles of this Issue 1/2019

Critical Care 1/2019 Go to the issue