Top

Published in:

Open Access 01-03-2020 | Septicemia | Systematic Review

Machine learning for the prediction of sepsis: a systematic review and meta-analysis of diagnostic test accuracy

Authors: Lucas M. Fleuren, Thomas L. T. Klausch, Charlotte L. Zwager, Linda J. Schoonmade, Tingjie Guo, Luca F. Roggeveen, Eleonora L. Swart, Armand R. J. Girbes, Patrick Thoral, Ari Ercole, Mark Hoogendoorn, Paul W. G. Elbers

Published in: Intensive Care Medicine | Issue 3/2020

Abstract

Purpose

Early clinical recognition of sepsis can be challenging. With the advancement of machine learning, promising real-time models to predict sepsis have emerged. We assessed their performance by carrying out a systematic review and meta-analysis.

Methods

A systematic search was performed in PubMed, Embase.com and Scopus. Studies targeting sepsis, severe sepsis or septic shock in any hospital setting were eligible for inclusion. The index test was any supervised machine learning model for real-time prediction of these conditions. Quality of evidence was assessed using the Grading of Recommendations Assessment, Development and Evaluation (GRADE) methodology, with a tailored Quality Assessment of Diagnostic Accuracy Studies (QUADAS-2) checklist to evaluate risk of bias. Models with a reported area under the curve of the receiver operating characteristic (AUROC) metric were meta-analyzed to identify strongest contributors to model performance.

Results

After screening, a total of 28 papers were eligible for synthesis, from which 130 models were extracted. The majority of papers were developed in the intensive care unit (ICU, n = 15; 54%), followed by hospital wards (n = 7; 25%), the emergency department (ED, n = 4; 14%) and all of these settings (n = 2; 7%). For the prediction of sepsis, diagnostic test accuracy assessed by the AUROC ranged from 0.68–0.99 in the ICU, to 0.96–0.98 in-hospital and 0.87 to 0.97 in the ED. Varying sepsis definitions limit pooling of the performance across studies. Only three papers clinically implemented models with mixed results. In the multivariate analysis, temperature, lab values, and model type contributed most to model performance.

Conclusion

This systematic review and meta-analysis show that on retrospective data, individual machine learning models can accurately predict sepsis onset ahead of time. Although they present alternatives to traditional scoring systems, between-study heterogeneity limits the assessment of pooled results. Systematic reporting and clinical implementation studies are needed to bridge the gap between bytes and bedside.

Available only for authorised users

Fleischmann C, Scherag A, Adhikari NKJ et al (2016) Assessment of global incidence and mortality of hospital-treated sepsis. current estimates and limitations. Am J Respir Crit Care Med 193:259–272. https://doi.org/10.1164/rccm.201504-0781OC CrossRefPubMed

Rhee C, Dantes R, Epstein L et al (2017) Incidence and trends of sepsis in US hospitals using clinical vs claims data, 2009–2014. JAMA 318:1241. https://doi.org/10.1001/jama.2017.13836 CrossRefPubMedPubMedCentral

Álvaro-Meca A, Jiménez-Sousa MA, Micheloud D et al (2018) Epidemiological trends of sepsis in the twenty-first century (2000–2013): an analysis of incidence, mortality, and associated costs in Spain. Popul Health Metr 16:4. https://doi.org/10.1186/s12963-018-0160-x CrossRefPubMedPubMedCentral

Seymour CW, Gesten F, Prescott HC et al (2017) Time to treatment and mortality during mandated emergency care for sepsis. N Engl J Med 376:2235–2244. https://doi.org/10.1056/NEJMoa1703058 CrossRefPubMedPubMedCentral

Liu VX, Fielding-Singh V, Greene JD et al (2017) The timing of early antibiotics and hospital mortality in sepsis. Am J Respir Crit Care Med 196:856–863. https://doi.org/10.1164/rccm.201609-1848OC CrossRefPubMedPubMedCentral

Rhodes A, Evans LE, Alhazzani W et al (2017) Surviving sepsis campaign: international guidelines for management of sepsis and septic shock: 2016. Intensive Care Med 43:304–377. https://doi.org/10.1007/s00134-017-4683-6 CrossRefPubMed

Ferrer R, Martin-Loeches I, Phillips G et al (2014) Empiric antibiotic treatment reduces mortality in severe sepsis and septic shock from the first hour. Crit Care Med 42:1749–1755. https://doi.org/10.1097/CCM.0000000000000330 CrossRefPubMed

Kumar A, Roberts D, Wood KE et al (2006) Duration of hypotension before initiation of effective antimicrobial therapy is the critical determinant of survival in human septic shock. Crit Care Med 34:1589–1596. https://doi.org/10.1097/01.CCM.0000217961.75225.E9 CrossRefPubMed

Vincent J-L (2016) The clinical challenge of sepsis identification and monitoring. PLoS Med 13:e1002022. https://doi.org/10.1371/journal.pmed.1002022 CrossRefPubMedPubMedCentral

10.

Talisa VB, Yende S, Seymour CW, Angus DC (2018) Arguing for adaptive clinical trials in sepsis. Front Immunol 9:1502. https://doi.org/10.3389/fimmu.2018.01502 CrossRefPubMedPubMedCentral

11.

de Grooth H-J, Postema J, Loer SA et al (2018) Unexplained mortality differences between septic shock trials: a systematic analysis of population characteristics and control-group mortality rates. Intensive Care Med 44:311–322. https://doi.org/10.1007/s00134-018-5134-8 CrossRefPubMedPubMedCentral

12.

Beam AL, Kohane IS (2018) Big data and machine learning in health care. JAMA 319:1317. https://doi.org/10.1001/jama.2017.18391 CrossRefPubMed

13.

Yu K-H, Beam AL, Kohane IS (2018) Artificial intelligence in healthcare. Nat Biomed Eng 2:719–731. https://doi.org/10.1038/s41551-018-0305-z CrossRefPubMed

14.

Khoshnevisan F, Ivy J, Capan M, et al (2018) Recent temporal pattern mining for septic shock early prediction. In: 2018 IEEE international conference on healthcare informatics (ICHI). IEEE, pp 229–240

15.

Nachimuthu SK, Haug PJ (2012) Early detection of sepsis in the emergency department using dynamic Bayesian networks. AMIA Annu Symp Proc AMIA Symp 2012:653–662PubMed

16.

Thottakkara P, Ozrazgat-Baslanti T, Hupf BB et al (2016) Application of machine learning techniques to high-dimensional clinical data to forecast postoperative complications. PLoS One 11:e0155705. https://doi.org/10.1371/journal.pone.0155705 CrossRefPubMedPubMedCentral

17.

Bihorac A, Ozrazgat-Baslanti T, Ebadi A et al (2019) My surgery risk. Ann Surg 269:652–662. https://doi.org/10.1097/SLA.0000000000002706 CrossRefPubMed

18.

McInnes MDF, Moher D, Thombs BD et al (2018) Preferred reporting items for a systematic review and meta-analysis of diagnostic test accuracy studies the PRISMA-DTA statement. JAMA J Am Med Assoc 319:388–396. https://doi.org/10.1001/jama.2017.19163 CrossRef

19.

Singer M, Deutschman CS, Seymour CW et al (2016) The third international consensus definitions for sepsis and septic shock (sepsis-3). JAMA 315:801. https://doi.org/10.1001/jama.2016.0287 CrossRefPubMedPubMedCentral

20.

Supervised learning—scikit-learn 0.21.2 documentation (2019) https://scikit-learn.org/stable/supervised_learning.html. Accessed 8 Jul 2019

21.

Schünemann H, Brożek J, Guyatt G OA (2013) GRADE handbook for grading quality of evidence and strength of recommendations

22.

Whiting PF, Rutjes AWS, Westwood ME et al (2011) QUADAS-2: a revised tool for the quality assessment of diagnostic accuracy studies. Ann Intern Med 155:529. https://doi.org/10.7326/0003-4819-155-8-201110180-00009 CrossRefPubMed

23.

Critical Appraisal Tools|Joanna Briggs Institute (2019) https://joannabriggs.org/critical_appraisal_tools. Accessed 8 Jul 2019

24.

Kwong MT, Colopy GW, Weber AM et al (2019) The efficacy and effectiveness of machine learning for weaning in mechanically ventilated patients at the intensive care unit: a systematic review. Bio-Design Manuf 2:31–40. https://doi.org/10.1007/s42242-018-0030-1 CrossRef

25.

Mao Q, Jay M, Hoffman JL et al (2018) Multicentre validation of a sepsis prediction algorithm using only vital sign data in the emergency department, general ward and ICU. BMJ Open 8:1–11. https://doi.org/10.1136/bmjopen-2017-017833 CrossRef

26.

Core Team R (2019) R: a language and environment for statistical computing. R Foundation for Statistical Computing, Vienna

27.

Barton C, Chettipally U, Zhou Y et al (2019) Evaluation of a machine learning algorithm for up to 48-hour advance prediction of sepsis using six vital signs. Comput Biol Med 109:79–84. https://doi.org/10.1016/j.compbiomed.2019.04.027 CrossRefPubMedPubMedCentral

28.

Brown SM, Jones J, Kuttler KG et al (2016) Prospective evaluation of an automated method to identify patients with severe sepsis or septic shock in the emergency department. BMC Emerg Med 16:1–7. https://doi.org/10.1186/s12873-016-0095-0 CrossRef

29.

Thiel SW, Rosini JM, Shannon W et al (2010) Early prediction of septic shock in hospitalized patients. J Hosp Med 5:19–25. https://doi.org/10.1002/jhm.530 CrossRefPubMed

30.

Giannini HM, Ginestra JC, Chivers C et al (2019) A machine learning algorithm to predict severe sepsis and septic shock. Crit Care Med. https://doi.org/10.1097/ccm.0000000000003891 CrossRefPubMedPubMedCentral

31.

McCoy A, Das R (2017) Reducing patient mortality, length of stay and readmissions through machine learning-based sepsis prediction in the emergency department, intensive care unit and hospital floor units. BMJ open Qual 6:e000158. https://doi.org/10.1136/bmjoq-2017-000158 CrossRefPubMedPubMedCentral

32.

Shimabukuro DW, Barton CW, Feldman MD et al (2017) Effect of a machine learning-based severe sepsis prediction algorithm on patient survival and hospital length of stay: a randomised clinical trial. BMJ Open Respir Res 4:e000234. https://doi.org/10.1136/bmjresp-2017-000234 CrossRefPubMedPubMedCentral

33.

Calvert J, Desautels T, Chettipally U et al (2016) High-performance detection and early prediction of septic shock for alcohol-use disorder patients. Ann Med Surg 8:50–55. https://doi.org/10.1016/j.amsu.2016.04.023 CrossRef

34.

Seymour CW, Liu VX, Iwashyna TJ et al (2016) Assessment of clinical criteria for sepsis. JAMA 315:762. https://doi.org/10.1001/jama.2016.0288 CrossRefPubMedPubMedCentral

35.

Horng S, Sontag DA, Halpern Y et al (2017) Creating an automated trigger for sepsis clinical decision support at emergency department triage using machine learning. PLoS One 12:e0174708. https://doi.org/10.1371/journal.pone.0174708 CrossRefPubMedPubMedCentral

36.

Moss TJ, Lake DE, Calland JF et al (2016) Signatures of subacute potentially catastrophic illness in the ICU: model development and validation. Crit Care Med 44:1639–1648. https://doi.org/10.1097/CCM.0000000000001738 CrossRefPubMedPubMedCentral

37.

Møller MH, Alhazzani W, Shankar-Hari M (2019) Focus on sepsis. Intensive Care Med 45:1459–1461. https://doi.org/10.1007/s00134-019-05680-4 CrossRefPubMed

38.

Makam AN, Nguyen OK, Auerbach AD (2015) Diagnostic accuracy and effectiveness of automated electronic sepsis alert systems: a systematic review. J Hosp Med 10:396–402. https://doi.org/10.1002/jhm.2347 CrossRefPubMed

39.

Alsolamy S, Al Salamah M, Al Thagafi M et al (2014) Diagnostic accuracy of a screening electronic alert tool for severe sepsis and septic shock in the emergency department. BMC Med Inform Decis Mak 14:105. https://doi.org/10.1186/s12911-014-0105-7 CrossRefPubMedPubMedCentral

40.

Serafim R, Gomes JA, Salluh J, Póvoa P (2018) A comparison of the quick-SOFA and systemic inflammatory response syndrome criteria for the diagnosis of sepsis and prediction of mortality: a systematic review and meta-analysis. Chest 153:646–655. https://doi.org/10.1016/J.CHEST.2017.12.015 CrossRefPubMed

41.

Hiensch R, Poeran J, Saunders-Hao P et al (2017) Impact of an electronic sepsis initiative on antibiotic use and health care facility–onset clostridium difficile infection rates. Am J Infect Control 45:1091–1100. https://doi.org/10.1016/j.ajic.2017.04.005 CrossRefPubMed

42.

Parlato M, Philippart F, Rouquette A et al (2018) Circulating biomarkers may be unable to detect infection at the early phase of sepsis in ICU patients: the CAPTAIN prospective multicenter cohort study. Intensive Care Med 44:1061–1070. https://doi.org/10.1007/s00134-018-5228-3 CrossRefPubMed

43.

Shankar-Hari M, Datta D, Wilson J, et al (2018) Early PREdiction of sepsis using leukocyte surface biomarkers: the ExPRES-sepsis cohort study. Intensive Care Med 44:1836–1848. https://doi.org/10.1007/s00134-018-5389-0 CrossRefPubMed

44.

Fleischmann-Struzek C, Thomas-Rüddel DO, Schettler A et al (2018) Comparing the validity of different ICD coding abstraction strategies for sepsis case identification in German claims data. PLoS One 13:e0198847. https://doi.org/10.1371/journal.pone.0198847 CrossRefPubMedPubMedCentral

45.

Bouza C, Lopez-Cuadrado T, Amate-Blanco JM (2016) Use of explicit ICD9-CM codes to identify adult severe sepsis: impacts on epidemiological estimates. Crit Care 20:313. https://doi.org/10.1186/s13054-016-1497-9 CrossRefPubMedPubMedCentral

46.

Jones M (2012) NEWSDIG: the national early warning score development and implementation group. Clin Med 12:501–503. https://doi.org/10.7861/clinmedicine.12-6-501 CrossRef

47.

Brown SM, Jones J, Kuttler KG et al (2016) Prospective evaluation of an automated method to identify patients with severe sepsis or septic shock in the emergency department. BMC Emerg Med 16:31. https://doi.org/10.1186/s12873-016-0095-0 CrossRefPubMedPubMedCentral

48.

Nguyen SQ, Mwakalindile E, Booth JS et al (2014) Automated electronic medical record sepsis detection in the emergency department. PeerJ 2:e343. https://doi.org/10.7717/peerj.343 CrossRefPubMedPubMedCentral

49.

Nelson JL, Smith BL, Jared JD, Younger JG (2011) Prospective trial of real-time electronic surveillance to expedite early care of severe sepsis. Ann Emerg Med 57:500–504. https://doi.org/10.1016/j.annemergmed.2010.12.008 CrossRefPubMed

50.

Hooper MH, Weavind L, Wheeler AP et al (2012) Randomized trial of automated, electronic monitoring to facilitate early detection of sepsis in the intensive care unit. Crit Care Med 40:2096–2101. https://doi.org/10.1097/CCM.0b013e318250a887 CrossRefPubMedPubMedCentral

51.

Giannini HM, Ginestra JC, Chivers C et al (2019) A machine learning algorithm to predict severe sepsis and septic shock: development, implementation, and impact on clinical practice. Crit Care Med. https://doi.org/10.1097/CCM.0000000000003891 CrossRefPubMedPubMedCentral

52.

Calvert JS, Price DA, Chettipally UK et al (2016) A computational approach to early sepsis detection. Comput Biol Med 74:69–73. https://doi.org/10.1016/j.compbiomed.2016.05.003 CrossRefPubMed

53.

Kam HJ, Kim HY (2017) Learning representations for the early detection of sepsis with deep neural networks. Comput Biol Med 89:248–255. https://doi.org/10.1016/j.compbiomed.2017.08.015 CrossRefPubMed

54.

Desautels T, Calvert J, Hoffman J et al (2016) Prediction of sepsis in the intensive care unit with minimal electronic health record data: a machine learning approach. JMIR Med Inf 4:e28. https://doi.org/10.2196/medinform.5909 CrossRef

55.

Nemati S, Holder A, Razmi F et al (2018) An interpretable machine learning model for accurate prediction of sepsis in the ICU. Crit Care Med 46:547–553. https://doi.org/10.1097/CCM.0000000000002936 CrossRefPubMedPubMedCentral

56.

Henry KE, Hager DN, Pronovost PJ, Saria S (2015) A targeted real-time early warning score (TREWScore) for septic shock. Sci Transl Med 7:299ra122. https://doi.org/10.1126/scitranslmed.aab3719 CrossRefPubMed

57.

Wang RZ, Sun CH, Schroeder PH et al (2018) Predictive models of sepsis in adult ICU patients. In: 2018 IEEE international conference on healthcare informatics (ICHI), IEEE, pp 390–391. https://doi.org/10.1109/ICHI.2018.00068

58.

Guillén J, Liu J, Furr M et al (2015) Predictive models for severe sepsis in adult ICU patients. In: 2015 systems and information engineering design symposium, IEEE, pp 182–187. https://doi.org/10.1109/SIEDS.2015.7116970

59.

Scherpf M, Gräßer F, Malberg H, Zaunseder S (2019) Predicting sepsis with a recurrent neural network using the MIMIC III database. Comput Biol Med 113:103395. https://doi.org/10.1016/j.compbiomed.2019.103395 CrossRefPubMed

60.

He Haibo, Garcia EA (2009) Learning from imbalanced data. IEEE Trans Knowl Data Eng 21:1263–1284. https://doi.org/10.1109/TKDE.2008.239 CrossRef

61.

Liu X, Faes L, Kale AU et al (2019) A comparison of deep learning performance against health-care professionals in detecting diseases from medical imaging: a systematic review and meta-analysis. Lancet Digit Heal 1:e271–e297. https://doi.org/10.1016/s2589-7500(19)30123-2 CrossRef

62.

Haug P, Ferraro J (2016) Using a semi-automated modeling environment to construct a Bayesian, sepsis diagnostic system. BCB '16. https://doi.org/10.1145/2975167.2985841 CrossRef

63.

Delahanty RJ, Alvarez J, Flynn LM et al (2018) Development and evaluation of a machine learning model for the early identification of patients at risk for sepsis. Ann Emerg Med. https://doi.org/10.1016/j.annemergmed.2018.11.036 CrossRef

64.

Khojandi A, Tansakul V, Li X et al (2018) Prediction of sepsis and in-hospital mortality using electronic health records. Methods Inf Med 57:185–193. https://doi.org/10.3414/ME18-01-0014 CrossRefPubMed

65.

Futoma J, Hariharan S, Heller K (2017) Learning to detect sepsis with a multitask Gaussian process RNN classifier. In: Proceedings of the 34th international conference on machine learning

66.

Lin C, Zhang Y, Ivy J et al (2018) Early diagnosis and prediction of sepsis shock by combining static and dynamic information using convolutional-LSTM. In: 2018 IEEE international conference on healthcare informatics (ICHI), IEEE, pp 219–228. https://doi.org/10.1109/ICHI.2018.00032

67.

Shashikumar SP, Stanley MD, Sadiq I et al (2017) Early sepsis detection in critical care patients using multiscale blood pressure and heart rate dynamics. J Electrocardiol 50:739–743. https://doi.org/10.1016/j.jelectrocard.2017.08.013 CrossRefPubMedPubMedCentral

68.

Shashikumar SP, Li Q, Clifford GD, Nemati S (2017) Multiscale network representation of physiological time series for early prediction of sepsis. Physiol Meas 38:2235–2248. https://doi.org/10.1088/1361-6579/aa9772 CrossRefPubMedPubMedCentral

69.

Van Wyk F, Khojandi A, Mohammed A et al (2019) A minimal set of physiomarkers in continuous high frequency data streams predict adult sepsis onset earlier. Int J Med Inform 122:55–62. https://doi.org/10.1016/j.ijmedinf.2018.12.002 CrossRef

70.

Van Wyk F, Khojandi A, Kamaleswaran R (2018) Improving prediction performance using hierarchical analysis of real-time data : a sepsis case study. IEEE J Biomed Health Inf 2018:1–9. https://doi.org/10.1109/JBHI.2019.2894570 CrossRef

71.

Mao Q, Jay M, Hoffman JL et al (2018) Multicentre validation of a sepsis prediction algorithm using only vital sign data in the emergency department, general ward and ICU. BMJ Open 8:e017833. https://doi.org/10.1136/bmjopen-2017-017833 CrossRefPubMedPubMedCentral

Title: Machine learning for the prediction of sepsis: a systematic review and meta-analysis of diagnostic test accuracy
Authors: Lucas M. Fleuren
Thomas L. T. Klausch
Charlotte L. Zwager
Linda J. Schoonmade
Tingjie Guo
Luca F. Roggeveen
Eleonora L. Swart
Armand R. J. Girbes
Patrick Thoral
Ari Ercole
Mark Hoogendoorn
Paul W. G. Elbers
Publication date: 01-03-2020
Publisher: Springer Berlin Heidelberg
Keywords: Septicemia
Septicemia
Septic Shock
Published in: Intensive Care Medicine / Issue 3/2020
Print ISSN: 0342-4642
Electronic ISSN: 1432-1238
DOI: https://doi.org/10.1007/s00134-019-05872-y

At a glance: The STEP trials

Springer Medicine

Machine learning for the prediction of sepsis: a systematic review and meta-analysis of diagnostic test accuracy

Abstract

Purpose

Methods

Results

Conclusion

At a glance: The STEP trials

Springer Medicine

Abstract

Purpose

Methods

Results

Conclusion

Please log in to get access to this content

Other articles of this Issue 3/2020

Fake news on euthanasia and organ donation

Inter-country variability over time in the mortality of mechanically ventilated patients

The persistent potential of extracorporeal therapies in liver failure

Perioperative maintenance fluid therapy in patients undergoing thoracic surgery: more risks than benefits?

The Janus faces of bicarbonate therapy in the ICU: con

Cheyne–Stokes breathing pattern and neurally adjusted ventilatory assist in a neuro-critical patient