Skip to main content
Top
Published in: BMC Medical Informatics and Decision Making 1/2019

Open Access 01-12-2019 | Computed Tomography | Research article

Use of natural language processing to improve predictive models for imaging utilization in children presenting to the emergency department

Authors: Xingyu Zhang, M. Fernanda Bellolio, Pau Medrano-Gracia, Konrad Werys, Sheng Yang, Prashant Mahajan

Published in: BMC Medical Informatics and Decision Making | Issue 1/2019

Login to get access

Abstract

Objective

To examine the association between the medical imaging utilization and information related to patients’ socioeconomic, demographic and clinical factors during the patients’ ED visits; and to develop predictive models using these associated factors including natural language elements to predict the medical imaging utilization at pediatric ED.

Methods

Pediatric patients’ data from the 2012–2016 United States National Hospital Ambulatory Medical Care Survey was included to build the models to predict the use of imaging in children presenting to the ED. Multivariable logistic regression models were built with structured variables such as temperature, heart rate, age, and unstructured variables such as reason for visit, free text nursing notes and combined data available at triage. NLP techniques were used to extract information from the unstructured data.

Results

Of the 27,665 pediatric ED visits included in the study, 8394 (30.3%) received medical imaging in the ED, including 6922 (25.0%) who had an X-ray and 1367 (4.9%) who had a computed tomography (CT) scan. In the predictive model including only structured variables, the c-statistic was 0.71 (95% CI: 0.70–0.71) for any imaging use, 0.69 (95% CI: 0.68–0.70) for X-ray, and 0.77 (95% CI: 0.76–0.78) for CT. Models including only unstructured information had c-statistics of 0.81 (95% CI: 0.81–0.82) for any imaging use, 0.82 (95% CI: 0.82–0.83) for X-ray, and 0.85 (95% CI: 0.83–0.86) for CT scans. When both structured variables and free text variables were included, the c-statistics reached 0.82 (95% CI: 0.82–0.83) for any imaging use, 0.83 (95% CI: 0.83–0.84) for X-ray, and 0.87 (95% CI: 0.86–0.88) for CT.

Conclusions

Both CT and X-rays are commonly used in the pediatric ED with one third of the visits receiving at least one. Patients’ socioeconomic, demographic and clinical factors presented at ED triage period were associated with the medical imaging utilization. Predictive models combining structured and unstructured variables available at triage performed better than models using structured or unstructured variables alone, suggesting the potential for use of NLP in determining resource utilization.
Appendix
Available only for authorised users
Literature
1.
go back to reference Horwitz LI, Green J, Bradley EH. US emergency department performance on wait time and length of visit. Ann Emerg Med. 2010;55(2):133–41.PubMedCrossRef Horwitz LI, Green J, Bradley EH. US emergency department performance on wait time and length of visit. Ann Emerg Med. 2010;55(2):133–41.PubMedCrossRef
2.
go back to reference Kovacs G, Croskerry P. Clinical decision making: an emergency medicine perspective. Acad Emerg Med. 1999;6(9):947–52.PubMedCrossRef Kovacs G, Croskerry P. Clinical decision making: an emergency medicine perspective. Acad Emerg Med. 1999;6(9):947–52.PubMedCrossRef
3.
4.
go back to reference Baumann MR, Strout TD. Evaluation of the emergency severity index (version 3) triage algorithm in pediatric patients. Acad Emerg Med. 2005;12(3):219–24.PubMedCrossRef Baumann MR, Strout TD. Evaluation of the emergency severity index (version 3) triage algorithm in pediatric patients. Acad Emerg Med. 2005;12(3):219–24.PubMedCrossRef
6.
go back to reference Maldonado T, Avner JR. Triage of the pediatric patient in the emergency department: are we all in agreement? Pediatrics. 2004;114(2):356–60.PubMedCrossRef Maldonado T, Avner JR. Triage of the pediatric patient in the emergency department: are we all in agreement? Pediatrics. 2004;114(2):356–60.PubMedCrossRef
7.
go back to reference Lee EK, Atallah HY, Wright MD, Post ET, Thomas Iv C, Wu DT, Haley LL Jr. Transforming hospital emergency department workflow and patient care. Interfaces. 2015;45(1):58–82.CrossRef Lee EK, Atallah HY, Wright MD, Post ET, Thomas Iv C, Wu DT, Haley LL Jr. Transforming hospital emergency department workflow and patient care. Interfaces. 2015;45(1):58–82.CrossRef
8.
go back to reference Kanzaria HK, Probst MA, Ponce NA, Hsia RY. The association between advanced diagnostic imaging and ED length of stay. Am J Emerg Med. 2014;32(10):1253–8.PubMedCrossRefPubMedCentral Kanzaria HK, Probst MA, Ponce NA, Hsia RY. The association between advanced diagnostic imaging and ED length of stay. Am J Emerg Med. 2014;32(10):1253–8.PubMedCrossRefPubMedCentral
9.
go back to reference Yoon P, Steiner I, Reinhardt G. Analysis of factors influencing length of stay in the emergency department. CJEM. 2003;5(3):155–61.PubMedCrossRef Yoon P, Steiner I, Reinhardt G. Analysis of factors influencing length of stay in the emergency department. CJEM. 2003;5(3):155–61.PubMedCrossRef
10.
go back to reference Macias CG, Sahouria JJ. The appropriate use of CT: quality improvement and clinical decision-making in pediatric emergency medicine. Pediatr Radiol. 2011;41(2):498.PubMedCrossRef Macias CG, Sahouria JJ. The appropriate use of CT: quality improvement and clinical decision-making in pediatric emergency medicine. Pediatr Radiol. 2011;41(2):498.PubMedCrossRef
11.
go back to reference Ben-Assuli O, Leshno M, Shabtai I. Using electronic medical record systems for admission decisions in emergency departments: examining the crowdedness effect. J Med Syst. 2012;36(6):3795–803.PubMedCrossRef Ben-Assuli O, Leshno M, Shabtai I. Using electronic medical record systems for admission decisions in emergency departments: examining the crowdedness effect. J Med Syst. 2012;36(6):3795–803.PubMedCrossRef
12.
go back to reference Zhang X, Kim J, Patzer RE, Pitts SR, Patzer A, Schrager JD. Prediction of emergency department hospital admission based on natural language processing and neural networks. Methods Inf Med. 2017;56(05):377–89.PubMedCrossRef Zhang X, Kim J, Patzer RE, Pitts SR, Patzer A, Schrager JD. Prediction of emergency department hospital admission based on natural language processing and neural networks. Methods Inf Med. 2017;56(05):377–89.PubMedCrossRef
13.
go back to reference Goto T, Camargo CA, Faridi MK, Freishtat RJ, Hasegawa K. Machine learning–based prediction of clinical outcomes for children during emergency department triage. JAMA Netw Open. 2019;2(1):e186937.PubMedPubMedCentralCrossRef Goto T, Camargo CA, Faridi MK, Freishtat RJ, Hasegawa K. Machine learning–based prediction of clinical outcomes for children during emergency department triage. JAMA Netw Open. 2019;2(1):e186937.PubMedPubMedCentralCrossRef
14.
go back to reference Demner-Fushman D, Chapman WW, McDonald CJ. What can natural language processing do for clinical decision support? J Biomed Inform. 2009;42(5):760–72.PubMedPubMedCentralCrossRef Demner-Fushman D, Chapman WW, McDonald CJ. What can natural language processing do for clinical decision support? J Biomed Inform. 2009;42(5):760–72.PubMedPubMedCentralCrossRef
15.
go back to reference Claster W, Shanmuganathan S, Ghotbi N. Text Mining of Medical Records for Radiodiagnostic decision-making. JCP. 2008;3(1):1–6. Claster W, Shanmuganathan S, Ghotbi N. Text Mining of Medical Records for Radiodiagnostic decision-making. JCP. 2008;3(1):1–6.
16.
go back to reference McCallum A. Information extraction: distilling structured data from unstructured text. Queue. 2005;3(9):48–57.CrossRef McCallum A. Information extraction: distilling structured data from unstructured text. Queue. 2005;3(9):48–57.CrossRef
17.
go back to reference Luo W, Phung D, Tran T, Gupta S, Rana S, Karmakar C, Shilton A, Yearwood J, Dimitrova N, Ho TB. Guidelines for developing and reporting machine learning predictive models in biomedical research: a multidisciplinary view. J Med Internet Res. 2016;18(12):e323.PubMedPubMedCentralCrossRef Luo W, Phung D, Tran T, Gupta S, Rana S, Karmakar C, Shilton A, Yearwood J, Dimitrova N, Ho TB. Guidelines for developing and reporting machine learning predictive models in biomedical research: a multidisciplinary view. J Med Internet Res. 2016;18(12):e323.PubMedPubMedCentralCrossRef
18.
go back to reference McCaig LF, Burt CW. Understanding and interpreting the National Hospital Ambulatory Medical Care Survey: key questions and answers. Ann Emerg Med. 2012;60(6):716–21.PubMedCrossRef McCaig LF, Burt CW. Understanding and interpreting the National Hospital Ambulatory Medical Care Survey: key questions and answers. Ann Emerg Med. 2012;60(6):716–21.PubMedCrossRef
19.
go back to reference McAdams-Demarco MA, Grams ME, Hall EC, Coresh J, Segev DL. Early hospital readmission after kidney transplantation: patient and center-level associations. Am J Transplant Off J Am Soc Transplant Am Soc Transplant Surg. 2012;12(12):3283–8.CrossRef McAdams-Demarco MA, Grams ME, Hall EC, Coresh J, Segev DL. Early hospital readmission after kidney transplantation: patient and center-level associations. Am J Transplant Off J Am Soc Transplant Am Soc Transplant Surg. 2012;12(12):3283–8.CrossRef
20.
go back to reference Harhay M, Lin E, Pai A, Harhay MO, Huverserian A, Mussell A, Abt P, Levine M, Bloom R, Shea JA, et al. Early rehospitalization after kidney transplantation: assessing preventability and prognosis. Am J Transplant Off J Am Soc Transplant Am Soc Transplant Surg. 2013;13(12):3164–72.CrossRef Harhay M, Lin E, Pai A, Harhay MO, Huverserian A, Mussell A, Abt P, Levine M, Bloom R, Shea JA, et al. Early rehospitalization after kidney transplantation: assessing preventability and prognosis. Am J Transplant Off J Am Soc Transplant Am Soc Transplant Surg. 2013;13(12):3164–72.CrossRef
21.
go back to reference Schneider D, Appleton L, McLemore T. A reason for visit classification for ambulatory care. Vital Health Stat 2. 1979;(78):1–63 i-vi. Schneider D, Appleton L, McLemore T. A reason for visit classification for ambulatory care. Vital Health Stat 2. 1979;(78):1–63 i-vi.
22.
go back to reference Zhang Y, Jin R, Zhou Z-H. Understanding bag-of-words model: a statistical framework. Int J Mach Learn Cybern. 2010;1(1–4):43–52.CrossRef Zhang Y, Jin R, Zhou Z-H. Understanding bag-of-words model: a statistical framework. Int J Mach Learn Cybern. 2010;1(1–4):43–52.CrossRef
23.
go back to reference Zhang X, Ambale-Venkatesh B, Bluemke DA, Cowan BR, Finn JP, Kadish AH, Lee DC, Lima JAC, Hundley WG, Suinesiaputra A. Information maximizing component analysis of left ventricular remodeling due to myocardial infarction. J Transl Med. 2015;13(1):343.PubMedPubMedCentralCrossRef Zhang X, Ambale-Venkatesh B, Bluemke DA, Cowan BR, Finn JP, Kadish AH, Lee DC, Lima JAC, Hundley WG, Suinesiaputra A. Information maximizing component analysis of left ventricular remodeling due to myocardial infarction. J Transl Med. 2015;13(1):343.PubMedPubMedCentralCrossRef
24.
go back to reference Froud R, Abel G. Using ROC Curves to Choose Minimally Important Change Thresholds when Sensitivity and Specificity Are Valued Equally: The Forgotten Lesson of Pythagoras. Theoretical Considerations and an Example Application of Change in Health Status. PLoS One. 2014;9(12):e114468.PubMedPubMedCentralCrossRef Froud R, Abel G. Using ROC Curves to Choose Minimally Important Change Thresholds when Sensitivity and Specificity Are Valued Equally: The Forgotten Lesson of Pythagoras. Theoretical Considerations and an Example Application of Change in Health Status. PLoS One. 2014;9(12):e114468.PubMedPubMedCentralCrossRef
25.
go back to reference Natale JE, Joseph JG, Rogers AJ, Tunik M, Monroe D, Kerrey B, Bonsu BK, Cook LJ, Page K, Adelgais K, et al. Relationship of physician-identified patient race and ethnicity to use of computed tomography in pediatric blunt torso trauma. Acad Emerg Med Off J Soc Acad Emerg Med. 2016;23(5):584–90.CrossRef Natale JE, Joseph JG, Rogers AJ, Tunik M, Monroe D, Kerrey B, Bonsu BK, Cook LJ, Page K, Adelgais K, et al. Relationship of physician-identified patient race and ethnicity to use of computed tomography in pediatric blunt torso trauma. Acad Emerg Med Off J Soc Acad Emerg Med. 2016;23(5):584–90.CrossRef
26.
go back to reference Hryhorczuk AL, Mannix RC, Taylor GA. Pediatric abdominal pain: use of imaging in the emergency department in the United States from 1999 to 2007. Radiology. 2012;263(3):778–85.PubMedCrossRef Hryhorczuk AL, Mannix RC, Taylor GA. Pediatric abdominal pain: use of imaging in the emergency department in the United States from 1999 to 2007. Radiology. 2012;263(3):778–85.PubMedCrossRef
27.
go back to reference Payne NR, Puumala SE. Racial disparities in ordering laboratory and radiology tests for pediatric patients in the emergency department. Pediatr Emerg Care. 2013;29(5):598–606.PubMedCrossRef Payne NR, Puumala SE. Racial disparities in ordering laboratory and radiology tests for pediatric patients in the emergency department. Pediatr Emerg Care. 2013;29(5):598–606.PubMedCrossRef
28.
go back to reference Timm NL, Ho ML, Luria JW. Pediatric emergency department overcrowding and impact on patient flow outcomes. Acad Emerg Med. 2008;15(9):832–7.PubMedCrossRef Timm NL, Ho ML, Luria JW. Pediatric emergency department overcrowding and impact on patient flow outcomes. Acad Emerg Med. 2008;15(9):832–7.PubMedCrossRef
29.
go back to reference Michelson KA, Monuteaux MC, Stack AM, Bachur RG. Pediatric emergency department crowding is associated with a lower likelihood of hospital admission. Acad Emerg Med. 2012;19(7):816–20.PubMedCrossRef Michelson KA, Monuteaux MC, Stack AM, Bachur RG. Pediatric emergency department crowding is associated with a lower likelihood of hospital admission. Acad Emerg Med. 2012;19(7):816–20.PubMedCrossRef
30.
go back to reference Schuur JD, Hsia RY, Burstin H, Schull MJ, Pines JM. Quality measurement in the emergency department: past and future. Health Aff. 2013;32(12):2129–38.CrossRef Schuur JD, Hsia RY, Burstin H, Schull MJ, Pines JM. Quality measurement in the emergency department: past and future. Health Aff. 2013;32(12):2129–38.CrossRef
31.
go back to reference Zeng Z, Ma X, Hu Y, Li J, Bryant D. A simulation study to improve quality of care in the emergency department of a community hospital. J Emerg Nurs. 2012;38(4):322–8.PubMedCrossRef Zeng Z, Ma X, Hu Y, Li J, Bryant D. A simulation study to improve quality of care in the emergency department of a community hospital. J Emerg Nurs. 2012;38(4):322–8.PubMedCrossRef
32.
go back to reference Asplin BR, Magid DJ, Rhodes KV, Solberg LI, Lurie N, Camargo CA Jr. A conceptual model of emergency department crowding. Ann Emerg Med. 2003;42(2):173–80.PubMedCrossRef Asplin BR, Magid DJ, Rhodes KV, Solberg LI, Lurie N, Camargo CA Jr. A conceptual model of emergency department crowding. Ann Emerg Med. 2003;42(2):173–80.PubMedCrossRef
33.
go back to reference Janke AT, Overbeek DL, Kocher KE, Levy PD. Exploring the potential of predictive analytics and big data in emergency care. Ann Emerg Med. 2016;67(2):227–36.PubMedCrossRef Janke AT, Overbeek DL, Kocher KE, Levy PD. Exploring the potential of predictive analytics and big data in emergency care. Ann Emerg Med. 2016;67(2):227–36.PubMedCrossRef
34.
go back to reference Zhang X, Kim J, Patzer RE, Pitts SR, Chokshi FH, Schrager JD. Advanced diagnostic imaging utilization during emergency department visits in the United States: a predictive modeling study for emergency department triage. PLoS One. 2019;14(4):e0214905.PubMedPubMedCentralCrossRef Zhang X, Kim J, Patzer RE, Pitts SR, Chokshi FH, Schrager JD. Advanced diagnostic imaging utilization during emergency department visits in the United States: a predictive modeling study for emergency department triage. PLoS One. 2019;14(4):e0214905.PubMedPubMedCentralCrossRef
35.
go back to reference Zhu W, Zeng N, Wang N. Sensitivity, specificity, accuracy, associated confidence interval and ROC analysis with practical SAS implementations. NESUG proceedings: health care and life sciences. 2010;19:67. Zhu W, Zeng N, Wang N. Sensitivity, specificity, accuracy, associated confidence interval and ROC analysis with practical SAS implementations. NESUG proceedings: health care and life sciences. 2010;19:67.
36.
go back to reference Tanabe P, Gimbel R, Yarnold PR, Adams JG. The emergency severity index (version 3) 5-level triage system scores predict ED resource consumption. J Emerg Nurs. 2004;30(1):22–9.PubMedCrossRef Tanabe P, Gimbel R, Yarnold PR, Adams JG. The emergency severity index (version 3) 5-level triage system scores predict ED resource consumption. J Emerg Nurs. 2004;30(1):22–9.PubMedCrossRef
37.
go back to reference McHugh M, Tanabe P, McClelland M, Khare RK. More patients are triaged using the emergency severity index than any other triage acuity system in the United States. Acad Emerg Med Off J Soc Acad Emerg Med. 2012;19(1):106–9.CrossRef McHugh M, Tanabe P, McClelland M, Khare RK. More patients are triaged using the emergency severity index than any other triage acuity system in the United States. Acad Emerg Med Off J Soc Acad Emerg Med. 2012;19(1):106–9.CrossRef
38.
go back to reference Wuerz R. Emergency severity index triage category is associated with six-month survival. ESI triage study group. Acad Emerg Med Off J Soc Acad Emerg Med. 2001;8(1):61–4.CrossRef Wuerz R. Emergency severity index triage category is associated with six-month survival. ESI triage study group. Acad Emerg Med Off J Soc Acad Emerg Med. 2001;8(1):61–4.CrossRef
39.
go back to reference Levin S, Toerper M, Hamrock E, Hinson JS, Barnes S, Gardner H, Dugas A, Linton B, Kirsch T, Kelen G. Machine-learning-based electronic triage more accurately differentiates patients with respect to clinical outcomes compared with the emergency severity index. Ann Emerg Med. 2018;71(5):565–74.PubMedCrossRef Levin S, Toerper M, Hamrock E, Hinson JS, Barnes S, Gardner H, Dugas A, Linton B, Kirsch T, Kelen G. Machine-learning-based electronic triage more accurately differentiates patients with respect to clinical outcomes compared with the emergency severity index. Ann Emerg Med. 2018;71(5):565–74.PubMedCrossRef
40.
go back to reference Michalowski W, Slowinski R, Wilk S, Farion K, Pike J, Rubin S. Design and development of a mobile system for supporting emergency triage. Methods Inf Med. 2005;44(1):14–24.PubMedCrossRef Michalowski W, Slowinski R, Wilk S, Farion K, Pike J, Rubin S. Design and development of a mobile system for supporting emergency triage. Methods Inf Med. 2005;44(1):14–24.PubMedCrossRef
41.
go back to reference Sun BC, Hsia RY, Weiss RE, Zingmond D, Liang LJ, Han W, McCreath H, Asch SM. Effect of emergency department crowding on outcomes of admitted patients. Ann Emerg Med. 2013;61(6):605–11 e606.PubMedCrossRef Sun BC, Hsia RY, Weiss RE, Zingmond D, Liang LJ, Han W, McCreath H, Asch SM. Effect of emergency department crowding on outcomes of admitted patients. Ann Emerg Med. 2013;61(6):605–11 e606.PubMedCrossRef
42.
43.
go back to reference Pencina MJ, D’agostino RB. Evaluating discrimination of risk prediction models: the C statistic. Jama. 2015;314(10):1063–4.PubMedCrossRef Pencina MJ, D’agostino RB. Evaluating discrimination of risk prediction models: the C statistic. Jama. 2015;314(10):1063–4.PubMedCrossRef
44.
go back to reference Worster A, Carpenter C. Incorporation bias in studies of diagnostic tests: how to avoid being biased about bias. CJEM. 2008;10(2):174–5.PubMedCrossRef Worster A, Carpenter C. Incorporation bias in studies of diagnostic tests: how to avoid being biased about bias. CJEM. 2008;10(2):174–5.PubMedCrossRef
Metadata
Title
Use of natural language processing to improve predictive models for imaging utilization in children presenting to the emergency department
Authors
Xingyu Zhang
M. Fernanda Bellolio
Pau Medrano-Gracia
Konrad Werys
Sheng Yang
Prashant Mahajan
Publication date
01-12-2019
Publisher
BioMed Central
Published in
BMC Medical Informatics and Decision Making / Issue 1/2019
Electronic ISSN: 1472-6947
DOI
https://doi.org/10.1186/s12911-019-1006-6

Other articles of this Issue 1/2019

BMC Medical Informatics and Decision Making 1/2019 Go to the issue