Evidence-based Clinical Decision Support Systems for the prediction and detection of three disease states in critical care: A systematic literature review

Search strategy

A systematic literature review was carried out to identify evidence-based study designs, methods and outcome measures that have been used to determine the clinical effectiveness of CDS systems in the detection and prediction of three populations representing the variety and majority of morbid conditions in a critical care setting: Shock (hemodynamic (in-)stability), respiratory distress/failure and infection/sepsis. The search strategy combined ‘intervention terms’ and ‘disease terms’ to identify primary research evaluating the diagnostic performance of CDS systems and other machine learning algorithms in three different populations of any age, sex, and race. Systematic literature reviews were also included for locating further relevant primary research. The search was conducted in MEDLINE (PubMed), ClinicalTrials.gov and Cochrane Database of Systematic Reviews (CDSR); and limited to studies published or registered between January 1, 2013 and November 8, 2018 and reported in English. Publication dates were limited to focus results on the most recent developments in this fast-evolving research domain. Another method to ensure up-to-date results was to include conference abstracts from 2017 onwards regardless of whether or not they were followed up with a detailed publication. Ongoing studies identified in the clinical trials register were also kept in the review. Study protocols identified from bibliographic databases were, however, excluded assuming that final study results would be available and identified elsewhere. The strategy employed in PubMed is provided as Extended data, Table 1–Table 3^25–27.

Studies conducted in US, Canada, UK, Germany or France with more than 10 subjects per arm were included. These countries were selected because they are known to be active in CDS development. The inclusion and exclusion criteria for selecting abstracts and subsequent full-text publications were based on the population, interventions, comparators, outcomes, and study design (PICOS). These criteria are listed in Table 1.

Study selection and data extraction

Study selection and data extraction was carried out by a single reviewer (MKK or SP). In cases of uncertainty, a second, or even third reviewer, was consulted. Data extraction was performed using a standard data extraction form (DEF). Key data from each additional eligible study were extracted by recording data from original reports into the DEF. The DEF included information on study design, inclusion/exclusion criteria, sample size and characteristics, interventions, outcome measures (measures of predictability like: sensitivity, specificity, negative predictive value (NPV), positive predictive value (PPV), likelihood ratio, accuracy (percentage of correctly identified cases in relation to the whole sample), odds ratio (OR), hazard ratio (HR), median, receiver operating characteristic (ROC) area under the curve (AUC); and length of hospitalization among others).

Studies identified from the ClinicalTrials.gov registry that did not report results were also included in the extraction to give some indication of the outcomes being collected.

Study quality appraisal

This research was not aimed at summarizing study results and assessing the relative effectiveness of CDS systems. Therefore, an appraisal of study quality was not deemed necessary.

Shock (hemodynamic (in-)stability)

The search yielded 1588 hits. Screening the titles and abstracts led to 1502 being excluded. The full texts of the remaining 86 titles were obtained and assessed against the PICOS criteria. Studies were excluded due to irrelevant study design (n=22), population (n=1), intervention (n=5), and outcomes (n=38). A total of 20 studies were finally included in this systematic literature review. This included 5 trials identified from ClinicalTrials.gov. The study selection process is depicted in Figure 1.

Figure 1. Study selection – Shock.

Pop. = Population.

Study characteristics. Of the 15 published studies, five were conducted by research groups outside the USA^28–32. Ten studies were conducted in the US^19,33–41, Thirteen studies were retrospective^{19,28–33,35,37–41} and only two were prospective^34,36. Nine studies were single-center^{28,30,31,33,37–41} and six studies were multi-center^{19,29,32,34–36}. Five studies were time-series^{28,30–32,40} and nine were case-series^{19,29,33–35,37–39,41}.

Across all studies, three had sample sizes ≤100^29,30,36; three had sample sizes of 101–1000^28,31,32; four studies had sample sizes of 1001–10,000^{19,33,34,37,42}; and another five studies, four retrospective single-center studies and one multi-center, had sample sizes larger than 10,000^35,38–41. The three largest studies included patients admitted to various wards of a specified hospital. The majority of the studies did not restrict their sample to a specific in-patient hospital setting. Five studies reported on patients in the ICU^{19,28,32,40,41} and one study reported on patients admitted to the surgical ward³³.

The characteristics of the published studies are summarized in Table 2.

CDS systems. Machine learning algorithms were developed to detect or predict septic shock^{28,33,35,40,41}, various heart arrhythmias^29,30,34, heart failure^37–39, hemodynamic instability and hypovolemia^19,36, myocardial infarction³¹, as well as hypotension³².

All studies, except one, trained a single algorithm. Ebrahimzadeh et al. 2018³⁰ trained and compared support vector machine (SVM), instance-based and neural network models to predict paroxysmal atrial fibrillation. SVMs were the most frequently used algorithms, followed by least absolute shrinkage and selection operator (LASSO) regularization. In one study, the SVM was trained using sequential minimal optimization³⁷.

Machine learning models were trained and validated in 14 studies and subsequently tested in an independent dataset in 3 studies^19,35,37. In one study an algorithm trained to classify arrythmias was not validated but compared to physician`s manual classifications³⁴.

An overview of the investigated machine learning algorithms is presented in Table 3.

Outcome measures. Three of the 15 papers measured a single outcome of model performance. In two studies the preferred measure was accuracy^28,34; whereas in another study this was the ROC AUC. This study was large and based their algorithm on EHRs³³. Across all studies, accuracy was reported in about half of the instances and the ROC AUC was one of the most frequently reported outcomes.

Sensitivity and specificity were reported together in 10 studies. Blecker et al. 2016³⁸ reported sensitivity together with PPV. Sensitivity and specificity were not measured in the study by Sideris et al. 2016³⁷, instead model accuracy and the ROC AUC were preferred. This study was concerned with developing an alternative `comorbidity` framework based on disease and symptom diagnostic codes to cluster individuals at low to high risk of developing chronic heart failure.

PPVs were reported in six studies and accompanied with negative predictive values in two studies. These studies developed and validated machine-learning algorithms for the early detection of less investigated health conditions, these being hemodynamic instability in children¹⁹ and acute decompensated heart failure³⁹. The highest number of outcome measures, including likelihood ratios, was observed in Calvert et al. 2016⁴⁰ who investigated an under-represented population of patients with Alcohol Use Disorder.

The outcomes measured are summarized in Table 4.

Ongoing studies. Five studies are currently ongoing, one in Germany⁴³ and the others in the USA^44–47. Two studies are prospective case series^44,47, two studies are prospective cohort studies^43,45 and one is a RCT⁴⁶. Two of the studies are concerned with developing prediction models, and the others are concerned with implementing machine learning algorithms into clinical practice as early warning systems.

The details of these trials are summarized in Table 5.

Respiratory distress/failure

The search yielded 1279 hits. Screening the titles and abstracts lead to 1142 being excluded. The full texts of the remaining 137 titles were obtained and assessed against the PICOS criteria. Studies were excluded due to irrelevant study design (n=42), population (n=6); intervention (n=18) and outcomes (n=47), and conference proceeding from before 2017 (n=2). A total of 22 studies were finally included in this systematic literature review. None of the trials retrieved from ClinicalTrials.gov were included. The study selection process is depicted in Figure 2.

Figure 2. Study selection - Respiratory distress-failure.

Pop. = Population.

Study characteristics. Of the included studies, 17 were conducted in the US^33,48–63. Five studies were conducted outside the US; two in Canada^64,65 by the same research group, two in France^66,67 and one in the UK⁶⁸. In total, 17 studies were retrospective^{33,48–50,52–55,58–66} and five were prospective^{51,56,57,67,68}. Of these studies, 12 were single-center^{33,48,49,51,52,54,55,58,59,64–66} and 10 studies were multi-center^{50,53,56,57,60–63,67,68}. Five studies were time-series^{48,52,55,56,64}, 14 studies were case-series^{33,49,51,53,54,57–62,65,66,68}, one was case-control⁵⁰ and one was case/time series study⁶³.

The smallest sample of 100 patients came from two single-center retrospective studies^48,66. Ten studies had sample sizes of 101–1000^{33,49–53,57,63,67,68}; seven studies had sample sizes of 1001–10,000^{54,55,59,60,62,64,65}; and three had sample sizes larger than 10,000^56,58,61. The largest study included more than 50,000 patients admitted to the ED of two centers over a 3-year period⁶¹. Several published studies did not report their in-patient setting. When reported, some evaluated data from different wards^{56,59,64,65,68}, and some included patients admitted only to the ED^53,54,61,63, the ICU^48,60,67 and the surgical ward^33,51,55.

The characteristics of all published studies are given in Table 6.

CDS systems. About half of the studies developed machine-learning algorithms, whereas the other half focused on natural language processing (NLP) algorithms. One study differed from the rest by developing a computer-aided detection (CAD) system to measure the axial diameter of the right and left pulmonary ventricles, aiding in the diagnosis of pulmonary embolisms⁴⁹. Many learning algorithms were concerned with detecting pulmonary embolisms and deep vein thrombosis^{53,54,58,59,64–67} as well as pneumonia^{33,48,57,60–63}. Three studies developed machine-learning algorithms to detect COPD^50,56,69. One study developed a machine learning algorithm to detect acute respiratory distress syndrome⁵²; while other studies developed machine learning algorithms to detect respiratory distress or failure following a pressure support ventilation trial⁶⁷, cardiovascular surgery⁵⁵ and pediatric tonsillectomy⁵¹.

The classifiers used in the NLP-based studies were various. However, some commonalities emerged between the studies developing machine-learning algorithms. Multiple studies applied SVM, logistic regression, random forests, K- nearest neighbor (kNN), gradient boosting and neural network models. Various classifiers were explored in 5 studies.

Machine learning and NLP-based algorithms were trained and validated in 20 studies and subsequently tested in an independent dataset in 6 studies^{52,56,60–62,67}. The CAD system mentioned above and an electronic pulmonary embolism severity index were trained and compared to a reference dataset classified by physicians^49,53.

An overview of the developed learning algorithms is provided in Table 7.

One study, Reamoroon et al. 2018⁵², used a novel sampling technique to accommodate for inter-dependency in longitudinal data. Model accuracy and ROC AUC with this method was <5% better than random sampling and 4–11% better than no sampling.

Outcome measures. The majority of the studies reported multiple outcome measures of model performance. The most frequently reported outcome measure was sensitivity, followed by specificity and ROC AUC. Likelihood ratios, on the other hand, were only reported in one study: Silva et al. 2017⁶⁷ reported eight outcome measures of their novel machine learning model to predict post extubation distress. The outcomes measured across all studies are summarized in Table 8.

Many of the studies that developed NLP-based algorithms reported negative and positive predictive values, as well as sensitivity and specificity. In contrast, the ROC AUC was the most frequently reported outcome measure of machine learning algorithm performance. It was also the single preferred outcome in three studies^33,50,55. About half of the studies additionally reported sensitivity, specificity, and accuracy. One study reported specificity with sensitivity set at 90% and 95% to ensure that few disease positive cases were missed⁵². The single study that developed a CAD system measured the ROC AUC and model accuracy⁴⁹.

Infection or sepsis

The search yielded 2659 hits. Screening the titles and abstracts lead to 2562 being excluded. The full texts of the remaining 97 titles were obtained and assessed against the PICOS criteria. Studies were excluded due to irrelevant study design (n=41), population (n=4); intervention (n=6) and outcomes (n=14). A total of 31 studies were finally included in this systematic literature review. Four of these were ongoing trials. The study selection process is depicted in Figure 3.

Figure 3. Study selection - infection or sepsis.

Pop. = Population.

Study characteristics. Of the included studies, 24 were conducted in the US. Three studies were conducted outside the US; one in France; one in the Netherlands and one in the UK. In total, 21 studies were retrospective^{33,35,70–88} and six were prospective^89–94. There were 21 single-center studies^{33,70–75,77–83,86–88,90–92,94} and six multi-center studies^{35,76,84,85,89,93}. Seven studies were time series^{71,78,82,84–86,92}, 18 studies were case series^{33,35,70,72–76,80,81,83,87–91,93,94}, one was a case-control⁷⁷ and one was a matched-controlled study⁷⁹.

The smallest studies included patients with leukemia⁸⁹ and combat casualty patients⁹⁰. Four studies had a sample size below 1000^70,72,73,79, three had a sample size between 1001–10,000^33,71,87 and 12 had a sample size larger than 10,000^{35,74,77–78,80–82,84–87,88}. Eight studies had samples even larger than 50,000^{35,74,77,78,82,84,85,88}. Large samples were achieved by less restrictive inclusion criteria where all patients admitted to specific ward(s) or hospital(s) over a given time were defined.

Majority of the published studies evaluated data from different wards; several studies included patients admitted only to the ICU^{70,72,81,84–86,93} and surgical ward^{73,76,78,87,91,92}, less often the General ward³³ and Emergency Department⁷⁴. Of these, 23 studies included data collected at their own hospital; and four utilized previously collated databases^76,81,84,86.

The characteristics of all published studies are given in Table 9.

CDS systems. The machine learning algorithms evaluated in the studies were developed to predict a range of diseases. These included sepsis^{33,35,72,78,81,85,93,94}, acute kidney injury^{70,78–80,82,84,91}, surgical site infections^{33,73,76,87,92}, central line-associated bloodstream infections^77,86, Clostridium difficile^83,88, pulmonary aspergillosis⁸⁹, bacteremia⁹⁰, fibrosis⁷¹, urine tract infection^33,74 and infections in general⁷⁵.

Almost half of the studies compared different machine learning algorithms, while the others focused only on Bayesian algorithms^73,92, decision tree algorithms⁸⁴, ensemble algorithms^{35,71,82,83,90,93}, regression algorithms^33,78,85, regularization algorithms^81,88 and rule learning⁷⁰. The most frequently applied model was random forest (15 studies) followed by logistic regression (10 studies), support vector machines (5 studies), naïve Bayes (5 studies) and gradient tree boosting (5 studies).

One study compared three different sampling methods for handling class imbalance; under-sampling the majority class (RANDu), over-sampling the minority class (RANDo) and synthetic minority over-sampling (SMOTE). This was a very large study including more than 500,000 patients to predict the onset of infections⁷⁵. The authors found that SMOTE outperformed the other techniques and improved model sensitivity. Two other very large studies used the RANDu method⁸⁰ and mini-batch stochastic gradient descent with backpropagation⁸⁵. No other studies were concerned with imbalance in disease positive and negative classification.

Machine learning models were trained and validated in 26 studies and subsequently tested in an independent dataset in four studies^35,72,75,77.

The machine learning algorithms used are illustrated in Table 10.

Outcome measures. The most frequently reported outcome measure was the ROC AUC. Three studies did not report this measure: Ahmed et al. 2015⁷⁰ developed an algorithm based on decision rules; Legrand et al. 2013⁹¹ was primarily interested in identifying risk factors of AKI after cardiac surgery; and Scicluna et al. 2017⁹³ was primarily concerned with identifying genetic biomarkers of sepsis.

Sensitivity and specificity were reported together in 14 studies^{35,70–72,74,75,78,81–84,87,90,92}. When specificity was not reported, sensitivity was reported together with PPV; and when sensitivity was not reported, this was due to sensitivity being set at a fixed value to report other diagnostic performance measures. In relation to the prior observation, more studies reported PPV than NPV. Four studies reporting likelihood ratios reported both negative and positive likelihood ratios^70,74,81,84.

An overview of measured outcomes is illustrated in Table 11.