Skip to main content
Top
Published in: Scandinavian Journal of Trauma, Resuscitation and Emergency Medicine 1/2024

Open Access 01-12-2024 | Artificial Intelligence | Study protocol

Machine learning for risk stratification in the emergency department (MARS-ED) study protocol for a randomized controlled pilot trial on the implementation of a prediction model based on machine learning technology predicting 31-day mortality in the emergency department

Authors: Paul M.E.L. van Dam, William P.T.M. van Doorn, Floor van Gils, Lotte Sevenich, Lars Lambriks, Steven J.R. Meex, Jochen W.L. Cals, Patricia M. Stassen

Published in: Scandinavian Journal of Trauma, Resuscitation and Emergency Medicine | Issue 1/2024

Login to get access

Abstract

Background

Many prediction models have been developed to help identify emergency department (ED) patients at high risk of poor outcome. However, these models often underperform in clinical practice and their actual clinical impact has hardly ever been evaluated. We aim to perform a clinical trial to investigate the clinical impact of a prediction model based on machine learning (ML) technology.

Methods

The study is a prospective, randomized, open-label, non-inferiority pilot clinical trial. We will investigate the clinical impact of a prediction model based on ML technology, the RISKINDEX, which has been developed to predict the risk of 31-day mortality based on the results of laboratory tests and demographic characteristics. In previous studies, the RISKINDEX was shown to outperform internal medicine specialists and to have high discriminatory performance. Adults patients (18 years or older) will be recruited in the ED. All participants will be randomly assigned to the control group or the intervention group in a 1:1 ratio. Participants in the control group will receive care as usual in which the study team asks the attending physicians questions about their clinical intuition. Participants in the intervention group will also receive care as usual, but in addition to asking the clinical impression questions, the study team presents the RISKINDEX to the attending physician in order to assess the extent to which clinical treatment is influenced by the results.

Discussion

This pilot clinical trial investigates the clinical impact and implementation of an ML based prediction model in the ED. By assessing the clinical impact and prognostic accuracy of the RISKINDEX, this study aims to contribute valuable insights to optimize patient care and inform future research in the field of ML based clinical prediction models.

Trial registration

ClinicalTrials.gov NCT05497830. Machine Learning for Risk Stratification in the Emergency Department (MARS-ED). Registered on August 11, 2022. URL: https://​clinicaltrials.​gov/​study/​NCT05497830.
Appendix
Available only for authorised users
Literature
1.
go back to reference van der Linden N, van der Linden MC, Richards JR, Derlet RW, Grootendorst DC, van den Brand CL. Effects of emergency department crowding on the delivery of timely care in an inner-city hospital in the Netherlands. Eur J Emerg Med. 2016;23(5):337–43.CrossRefPubMed van der Linden N, van der Linden MC, Richards JR, Derlet RW, Grootendorst DC, van den Brand CL. Effects of emergency department crowding on the delivery of timely care in an inner-city hospital in the Netherlands. Eur J Emerg Med. 2016;23(5):337–43.CrossRefPubMed
2.
go back to reference Ter Avest E, Onnes BT, van der Vaart T, Land MJ. Hurry up, it’s quiet in the emergency department. Neth J Med. 2018;76(1):32–5.PubMed Ter Avest E, Onnes BT, van der Vaart T, Land MJ. Hurry up, it’s quiet in the emergency department. Neth J Med. 2018;76(1):32–5.PubMed
3.
go back to reference Guttmann A, Schull MJ, Vermeulen MJ, Stukel TA. Association between waiting times and short term mortality and hospital admission after departure from emergency department: population based cohort study from Ontario, Canada. BMJ. 2011;342:d2983.CrossRefPubMedPubMedCentral Guttmann A, Schull MJ, Vermeulen MJ, Stukel TA. Association between waiting times and short term mortality and hospital admission after departure from emergency department: population based cohort study from Ontario, Canada. BMJ. 2011;342:d2983.CrossRefPubMedPubMedCentral
4.
go back to reference Liew D, Liew D, Kennedy MP. Emergency department length of stay independently predicts excess inpatient length of stay. Med J Aust. 2003;179(10):524–6.CrossRefPubMed Liew D, Liew D, Kennedy MP. Emergency department length of stay independently predicts excess inpatient length of stay. Med J Aust. 2003;179(10):524–6.CrossRefPubMed
5.
go back to reference Sun BC, Hsia RY, Weiss RE, Zingmond D, Liang LJ, Han W, et al. Effect of emergency department crowding on outcomes of admitted patients. Ann Emerg Med. 2013;61(6):605–11. e6.CrossRefPubMed Sun BC, Hsia RY, Weiss RE, Zingmond D, Liang LJ, Han W, et al. Effect of emergency department crowding on outcomes of admitted patients. Ann Emerg Med. 2013;61(6):605–11. e6.CrossRefPubMed
6.
go back to reference Zelis N, Mauritz AN, Kuijpers LIJ, Buijs J, de Leeuw PW, Stassen PM. Short-term mortality in older medical emergency patients can be predicted using clinical intuition: a prospective study. PLoS ONE. 2019;14(1):e0208741.CrossRefPubMedPubMedCentral Zelis N, Mauritz AN, Kuijpers LIJ, Buijs J, de Leeuw PW, Stassen PM. Short-term mortality in older medical emergency patients can be predicted using clinical intuition: a prospective study. PLoS ONE. 2019;14(1):e0208741.CrossRefPubMedPubMedCentral
7.
go back to reference van Geffen M, van der Waaij KM, Stassen PM. Number, nature & impact of incoming telephone calls on residents and their work during evening shifts. Acute Med. 2022;21(1):5–11.CrossRefPubMed van Geffen M, van der Waaij KM, Stassen PM. Number, nature & impact of incoming telephone calls on residents and their work during evening shifts. Acute Med. 2022;21(1):5–11.CrossRefPubMed
8.
go back to reference Laxmisan A, Hakimzada F, Sayan OR, Green RA, Zhang J, Patel VL. The multitasking clinician: decision-making and cognitive demand during and after team handoffs in emergency care. Int J Med Inform. 2007;76(11–12):801–11.CrossRefPubMed Laxmisan A, Hakimzada F, Sayan OR, Green RA, Zhang J, Patel VL. The multitasking clinician: decision-making and cognitive demand during and after team handoffs in emergency care. Int J Med Inform. 2007;76(11–12):801–11.CrossRefPubMed
9.
go back to reference Chisholm CD, Collison EK, Nelson DR, Cordell WH. Emergency department workplace interruptions: are emergency physicians interrupt-driven and multitasking? Acad Emerg Med. 2000;7(11):1239–43.CrossRefPubMed Chisholm CD, Collison EK, Nelson DR, Cordell WH. Emergency department workplace interruptions: are emergency physicians interrupt-driven and multitasking? Acad Emerg Med. 2000;7(11):1239–43.CrossRefPubMed
10.
go back to reference Subbe CP, Kruger M, Rutherford P, Gemmel L. Validation of a modified early warning score in medical admissions. QJM. 2001;94(10):521–6.CrossRefPubMed Subbe CP, Kruger M, Rutherford P, Gemmel L. Validation of a modified early warning score in medical admissions. QJM. 2001;94(10):521–6.CrossRefPubMed
11.
go back to reference Vincent JL, Moreno R, Takala J, Willatts S, De Mendonca A, Bruining H, et al. The SOFA (Sepsis-related Organ failure Assessment) score to describe organ dysfunction/failure. On behalf of the Working Group on Sepsis-related problems of the European Society of Intensive Care Medicine. Intensive Care Med. 1996;22(7):707–10.CrossRefPubMed Vincent JL, Moreno R, Takala J, Willatts S, De Mendonca A, Bruining H, et al. The SOFA (Sepsis-related Organ failure Assessment) score to describe organ dysfunction/failure. On behalf of the Working Group on Sepsis-related problems of the European Society of Intensive Care Medicine. Intensive Care Med. 1996;22(7):707–10.CrossRefPubMed
12.
go back to reference Zelis N, Buijs J, de Leeuw PW, van Kuijk SMJ, Stassen PM. A new simplified model for predicting 30-day mortality in older medical emergency department patients: the rise up score. Eur J Intern Med. 2020;77:36–43.CrossRefPubMed Zelis N, Buijs J, de Leeuw PW, van Kuijk SMJ, Stassen PM. A new simplified model for predicting 30-day mortality in older medical emergency department patients: the rise up score. Eur J Intern Med. 2020;77:36–43.CrossRefPubMed
13.
go back to reference Challen K, Goodacre SW. Predictive scoring in non-trauma emergency patients: a scoping review. Emerg Med J. 2011;28(10):827–37.CrossRefPubMed Challen K, Goodacre SW. Predictive scoring in non-trauma emergency patients: a scoping review. Emerg Med J. 2011;28(10):827–37.CrossRefPubMed
14.
go back to reference Taylor RA, Pare JR, Venkatesh AK, Mowafi H, Melnick ER, Fleischman W, Hall MK. Prediction of In-hospital mortality in Emergency Department patients with Sepsis: a local Big Data-Driven, Machine Learning Approach. Acad Emerg Med. 2016;23(3):269–78.CrossRefPubMedPubMedCentral Taylor RA, Pare JR, Venkatesh AK, Mowafi H, Melnick ER, Fleischman W, Hall MK. Prediction of In-hospital mortality in Emergency Department patients with Sepsis: a local Big Data-Driven, Machine Learning Approach. Acad Emerg Med. 2016;23(3):269–78.CrossRefPubMedPubMedCentral
15.
go back to reference Shafaf N, Malek H. Applications of machine learning approaches in Emergency Medicine; a review article. Arch Acad Emerg Med. 2019;7(1):34.PubMedPubMedCentral Shafaf N, Malek H. Applications of machine learning approaches in Emergency Medicine; a review article. Arch Acad Emerg Med. 2019;7(1):34.PubMedPubMedCentral
16.
go back to reference van Doorn W, Helmich F, van Dam PMEL, Jacobs LHJ, Stassen PM, Bekers O, Meex SJR. Transparent Machine Learning models for Rapid Risk Stratification in the Emergency Department: A multi-center evaluation. JALM 2023. van Doorn W, Helmich F, van Dam PMEL, Jacobs LHJ, Stassen PM, Bekers O, Meex SJR. Transparent Machine Learning models for Rapid Risk Stratification in the Emergency Department: A multi-center evaluation. JALM 2023.
17.
go back to reference Cruz Rivera S, Liu X, Chan AW, Denniston AK, Calvert MJ, Spirit AI, et al. Guidelines for clinical trial protocols for interventions involving artificial intelligence: the SPIRIT-AI extension. Nat Med. 2020;26(9):1351–63.CrossRefPubMedPubMedCentral Cruz Rivera S, Liu X, Chan AW, Denniston AK, Calvert MJ, Spirit AI, et al. Guidelines for clinical trial protocols for interventions involving artificial intelligence: the SPIRIT-AI extension. Nat Med. 2020;26(9):1351–63.CrossRefPubMedPubMedCentral
18.
19.
go back to reference Jansen TCKE, Druml C, Menon DK, Wiedermann CJ, Bakker J. Deferred consent in emergency intensive care research: what if the patient dies early? Use the data or not? Intensive Care Med. 2007;33(5):894–900.CrossRefPubMedPubMedCentral Jansen TCKE, Druml C, Menon DK, Wiedermann CJ, Bakker J. Deferred consent in emergency intensive care research: what if the patient dies early? Use the data or not? Intensive Care Med. 2007;33(5):894–900.CrossRefPubMedPubMedCentral
20.
go back to reference Downar JGR, Pinto R, Englesakis M, Adhikari NKJ. The surprise question for predicting death in seriously ill patients: a systematic review and meta-analysis. CMAJ. 2017;189(13):E484–93.CrossRefPubMedPubMedCentral Downar JGR, Pinto R, Englesakis M, Adhikari NKJ. The surprise question for predicting death in seriously ill patients: a systematic review and meta-analysis. CMAJ. 2017;189(13):E484–93.CrossRefPubMedPubMedCentral
21.
go back to reference Ouchi KST, Haydar S, Baker O, Wang W, Bernacki R, et al. Association of Emergency clinicians’ Assessment of Mortality Risk with actual 1-Month Mortality among older adults admitted to the hospital. JAMA Netw Open. 2019;2(9):e1911139–e.CrossRefPubMedPubMedCentral Ouchi KST, Haydar S, Baker O, Wang W, Bernacki R, et al. Association of Emergency clinicians’ Assessment of Mortality Risk with actual 1-Month Mortality among older adults admitted to the hospital. JAMA Netw Open. 2019;2(9):e1911139–e.CrossRefPubMedPubMedCentral
22.
go back to reference Barais M, vdWM, Groell N, Dany A, Montier T, van Royen P et al. Gut feelings Questionnaire in daily practice: a feasibility study using a mixed-methods approach in three European countries. BMJ open. 2018;8(11). Barais M, vdWM, Groell N, Dany A, Montier T, van Royen P et al. Gut feelings Questionnaire in daily practice: a feasibility study using a mixed-methods approach in three European countries. BMJ open. 2018;8(11).
23.
go back to reference Organization WH. ICD-10, the ICD-10 Classification of Mental and Behavioural Disorders: Diagnostic Criteria for Research. 1993. Organization WH. ICD-10, the ICD-10 Classification of Mental and Behavioural Disorders: Diagnostic Criteria for Research. 1993.
24.
go back to reference Regulation (EU). 2016/679 on the protection of natural persons with regard to the processing of personal data and on the free movement of such data, and repealing Directive 95/46/EC (General Data Protection Regulation), (2016). Regulation (EU). 2016/679 on the protection of natural persons with regard to the processing of personal data and on the free movement of such data, and repealing Directive 95/46/EC (General Data Protection Regulation), (2016).
25.
go back to reference General Data Protection Regulation and General Data Protection Regulation Implementation Act (Dutch: Algemene Verordering Gegevensbescherming en Uitvoeringswet Algemene Verordening Gegevensbescherming). (2018). General Data Protection Regulation and General Data Protection Regulation Implementation Act (Dutch: Algemene Verordering Gegevensbescherming en Uitvoeringswet Algemene Verordening Gegevensbescherming). (2018).
26.
go back to reference Klug MBY, Bechler S, Resheff YS, Tron T, Ironi A, et al. A gradient boosting machine learning model for Predicting Early Mortality in the Emergency Department Triage: devising a nine-point triage score. J Gen Intern Med. 2019;35(1):220–7.CrossRefPubMedPubMedCentral Klug MBY, Bechler S, Resheff YS, Tron T, Ironi A, et al. A gradient boosting machine learning model for Predicting Early Mortality in the Emergency Department Triage: devising a nine-point triage score. J Gen Intern Med. 2019;35(1):220–7.CrossRefPubMedPubMedCentral
27.
go back to reference Perng JKI, Kung C, Hung S, Lai Y, Su C. Mortality prediction of septic patients in the emergency department based on machine learning. J Clin Med. 2019;8 (11). Perng JKI, Kung C, Hung S, Lai Y, Su C. Mortality prediction of septic patients in the emergency department based on machine learning. J Clin Med. 2019;8 (11).
28.
go back to reference van Doorn WPTMSP, Borggreve HF, Schalkwijk MJ, Stoffers J, Bekers O et al. A comparison of machine learning models versus clinical evaluation for mortality prediction in patients with sepsis. PLoS ONE. 2021;16(1). van Doorn WPTMSP, Borggreve HF, Schalkwijk MJ, Stoffers J, Bekers O et al. A comparison of machine learning models versus clinical evaluation for mortality prediction in patients with sepsis. PLoS ONE. 2021;16(1).
29.
go back to reference van Dam PMEL, van Doorn W, Meex SJR, Stassen PM. Machine learning for risk stratification in patients with COVID-19 in the emergency department. van Dam PMEL, van Doorn W, Meex SJR, Stassen PM. Machine learning for risk stratification in patients with COVID-19 in the emergency department.
Metadata
Title
Machine learning for risk stratification in the emergency department (MARS-ED) study protocol for a randomized controlled pilot trial on the implementation of a prediction model based on machine learning technology predicting 31-day mortality in the emergency department
Authors
Paul M.E.L. van Dam
William P.T.M. van Doorn
Floor van Gils
Lotte Sevenich
Lars Lambriks
Steven J.R. Meex
Jochen W.L. Cals
Patricia M. Stassen
Publication date
01-12-2024
Publisher
BioMed Central
DOI
https://doi.org/10.1186/s13049-024-01177-2

Other articles of this Issue 1/2024

Scandinavian Journal of Trauma, Resuscitation and Emergency Medicine 1/2024 Go to the issue