Skip to main content
Key Specialties
Cardiology Diabetology Endocrinology Gastroenterology Geriatrics and Gerontology Gynecology and Obstetrics Hematology Infectious Disease Internal Medicine Nephrology Neurology Oncology Pediatrics Respiratory Medicine Rheumatology Surgery
Congress Coverage
2024 ACC Congress 2023 ESMO Congress 2023 EASD Congress 2023 ERS Congress 2023 ESC Congress
Clinical Collections
Advances in Alzheimer’s

At a glance: The STEP trials

A round-up of the STEP phase 3 clinical trials evaluating semaglutide for weight loss in people with overweight or obesity.
ADVANCED SEARCH
Log in
Top
Implementation Science
Published in: Implementation Science 1/2015

Open Access 01-12-2015 | Study protocol

Accuracy of using automated methods for detecting adverse events from electronic health record data: a research protocol

Authors: Christian M Rochefort, David L Buckeridge, Alan J Forster

Published in: Implementation Science | Issue 1/2015

Login to get access

Abstract

Background

Adverse events are associated with significant morbidity, mortality and cost in hospitalized patients. Measuring adverse events is necessary for quality improvement, but current detection methods are inaccurate, untimely and expensive. The advent of electronic health records and the development of automated methods for encoding and classifying electronic narrative data, such as natural language processing, offer an opportunity to identify potentially better methods. The objective of this study is to determine the accuracy of using automated methods for detecting three highly prevalent adverse events: a) hospital-acquired pneumonia, b) catheter-associated bloodstream infections, and c) in-hospital falls.

Methods/design

This validation study will be conducted at two large Canadian academic health centres: the McGill University Health Centre (MUHC) and The Ottawa Hospital (TOH). The study population consists of all medical, surgical and intensive care unit patients admitted to these centres between 2008 and 2014. An automated detection algorithm will be developed and validated for each of the three adverse events using electronic data extracted from multiple clinical databases. A random sample of MUHC patients will be used to develop the automated detection algorithms (cohort 1, development set). The accuracy of these algorithms will be assessed using chart review as the reference standard. Then, receiver operating characteristic curves will be used to identify optimal cut points for each of the data sources. Multivariate logistic regression and the areas under curve (AUC) will be used to identify the optimal combination of data sources that maximize the accuracy of adverse event detection. The most accurate algorithms will then be validated on a second random sample of MUHC patients (cohort 1, validation set), and accuracy will be measured using chart review as the reference standard. The most accurate algorithms validated at the MUHC will then be applied to TOH data (cohort 2), and their accuracy will be assessed using a reference standard assessment of the medical chart.

Discussion

There is a need for more accurate, timely and efficient measures of adverse events in acute care hospitals. This is a critical requirement for evaluating the effectiveness of preventive interventions and for tracking progress in patient safety through time.
Literature
1.
Kohn LT, Corrigan J, Donaldson MS. To Err is Human. Building a Safer Health System. Washington, D.C: Institute of Medicine, National Acadey Press; 2000.
2.
Brennan TA, Leape LL, Laird NM, Hebert L, Localio AR, Lawthers AG, et al. Incidence of adverse events and negligence in hospitalized patients. Results of the Harvard Medical Practice Study I. N Engl J Med. 1991;324(6):370–6.PubMedCrossRef
3.
Leape LL, Brennan TA, Laird N, Lawthers AG, Localio AR, Barnes BA, et al. The nature of adverse events in hospitalized patients. Results of the Harvard Medical Practice Study II. N Engl J Med. 1991;324(6):377–84.PubMedCrossRef
4.
Thomas EJ, Studdert DM, Burstin HR, Orav EJ, Zeena T, Williams EJ, et al. Incidence and types of adverse events and negligent care in Utah and Colorado. Med Care. 2000;38(3):261–71.PubMedCrossRef
5.
Thomas EJ, Studdert DM, Newhouse JP, Zbar BI, Howard KM, Williams EJ, et al. Costs of medical injuries in Utah and Colorado. Inquiry. 1999;36(3):255–64.PubMed
6.
Baker GR, Norton PG, Flintoft V, Blais R, Brown A, Cox J, et al. The Canadian Adverse Events Study: the incidence of adverse events among hospital patients in Canada. CMAJ. 2004;170(11):1678–86.PubMedCentralPubMedCrossRef
7.
Davis P, Lay-Yee R, Briant R, Ali W, Scott A, Schug S. Adverse events in New Zealand public hospitals I: occurrence and impact. N Z Med J. 2002;115(1167):U271.
8.
Davis P, Lay-Yee R, Briant R, Ali W, Scott A, Schug S. Adverse events in New Zealand public hospitals II: preventability and clinical context. N Z Med J. 2003;116(1183):U624.
9.
Wilson RM, Runciman WB, Gibberd RW, Harrison BT, Newby L, Hamilton JD. The Quality in Australian Health Care Study. Med J Aust. 1995;163(9):458–71.PubMed
10.
The Research Priority Setting Working Group. Global Priorities for Research in Patient Safety. Geneva: World Health Organization; 2008.
11.
Aspden P, Corrigan JM, Wolcott J, Erickson SM. Patient Safety: Achieving a New Standard for Care. Washington, D.C.: National Academy Press; 2004.
12.
Govindan M, Van Citters AD, Nelson EC, Kelly-Cummings J, Suresh G. Automated detection of harm in healthcare with information technology: a systematic review. Qual Saf Health Care. 2010;19(5):e11.
13.
Murff HJ, Patel VL, Hripcsak G, Bates DW. Detecting adverse events for patient safety research: a review of current methodologies. J Biomed Inform. 2003;36(1–2):131–43.PubMedCrossRef
14.
Klompas M, Yokoe DS. Automated surveillance of health care-associated infections. Clin Infect Dis. 2009;48(9):1268–75.PubMedCrossRef
15.
Bates DW, Evans RS, Murff H, Stetson PD, Pizziferri L, Hripcsak G. Detecting adverse events using information technology. J Am Med Inform Assoc. 2003;10(2):115–28.PubMedCentralPubMedCrossRef
16.
Classen DC, Resar R, Griffin F, Federico F, Frankel T, Kimmel N, et al. 'Global trigger tool' shows that adverse events in hospitals may be ten times greater than previously measured. Health Aff (Millwood). 2011;30(4):581–9.CrossRef
17.
Kaafarani HM, Borzecki AM, Itani KM, Loveland S, Mull HJ, Hickson K, et al. Validity of selected patient safety indicators: opportunities and concerns. J Am Coll Surg. 2011;212(6):924–34.PubMedCrossRef
18.
Goto M, Ohl ME, Schweizer ML, Perencevich EN. Accuracy of administrative code data for the surveillance of healthcare-associated infections: a systematic review and meta-analysis. Clin Infect Dis. 2014;58(5):688–96.PubMedCrossRef
19.
Romano PS, Mull HJ, Rivard PE, Zhao S, Henderson WG, Loveland S, et al. Validity of selected AHRQ patient safety indicators based on VA National Surgical Quality Improvement Program data. Health Serv Res. 2009;44(1):182–204.PubMedCentralPubMedCrossRef
20.
Houchens RL, Elixhauser A, Romano PS. How often are potential patient safety events present on admission? Jt Comm J Qual Patient Saf. 2008;34(3):154–63.PubMed
21.
Bahl V, Thompson MA, Kau TY, Hu HM, Campbell Jr DA. Do the AHRQ patient safety indicators flag conditions that are present at the time of hospital admission? Med Care. 2008;46(5):516–22.PubMedCrossRef
22.
Murff HJ, Forster AJ, Peterson JF, Fiskio JM, Heiman HL, Bates DW. Electronically screening discharge summaries for adverse medical events. J Am Med Inform Assoc. 2003;10(4):339–50.PubMedCentralPubMedCrossRef
23.
Forster AJ, Andrade J, van Walraven C. Validation of a discharge summary term search method to detect adverse events. J Am Med Inform Assoc. 2005;12(2):200–6.PubMedCentralPubMedCrossRef
24.
Allen J. Natural Language Understanding. Redwood City, CA: Benjamin/Cummings Publishing Company; 1995.
25.
26.
Chapman WW, Wagner MM, Moore AW, Aryel RM. Natural Language Processing for Biosurveillance. In: Handbook of Biosurveillance. Burlington, MA: Elsevier Academic Press; 2006. p. 255–71.CrossRef
27.
Mitchell TM. Machine Learning. Boston, MA: McGraw-Hill; 1997.
28.
Hastie T, Tibshirani R, Friedman J. The Elements of Statistical Learning. Data mining, Inference, and Prediction. Secondth edition. New York, NY: Springer; 2009.
29.
Murff HJ, FitzHenry F, Matheny ME, Gentry N, Kotter KL, Crimin K, et al. Automated identification of postoperative complications within an electronic medical record using natural language processing. JAMA. 2011;306(8):848–55.PubMedCrossRef
30.
FitzHenry F, Murff HJ, Matheny ME, Gentry N, Fielstein EM, Brown SH, et al. Exploring the frontier of electronic health record surveillance: the case of postoperative complications. Med Care. 2013;51(6):509–16.PubMedCentralPubMedCrossRef
31.
Rochefort CM, Verma AD, Eguale T, Lee TC, Buckeridge DL: A novel method of adverse event detection can accurately identify venous thromboembolism (VTE) from narrative electronic health record data. JAMIA 2014. doi:10.1136/amiajnl-2014-002768.
32.
33.
34.
Chastre J, Fagon JY. Ventilator-associated pneumonia. Am J Respir Crit Care Med. 2002;165(7):867–903.PubMedCrossRef
35.
American Thoracic Society and Infection Disease Society of America. Guidelines for the management of adults with hospital-acquired, ventilator-associated, and healthcare-associated pneumonia. Am J Respir Crit Care Med. 2005;171(4):388–416.CrossRef
36.
Kieninger AN, Lipsett PA. Hospital-acquired pneumonia: pathophysiology, diagnosis, and treatment. Surg Clin North Am. 2009;89(2):439–461. ix.PubMedCrossRef
37.
Joseph NM, Sistla S, Dutta TK, Badhe AS, Parija SC. Ventilator-associated pneumonia: a review. Eur J Intern Med. 2010;21(5):360–8.PubMedCrossRef
38.
Hockenhull JC, Dwan K, Boland A, Smith G, Bagust A, Dundar Y, et al. The clinical effectiveness and cost-effectiveness of central venous catheters treated with anti-infective agents in preventing bloodstream infections: a systematic review and economic evaluation. Health Technol Assess. 2008;12(12):ii–xi. 1.CrossRef
39.
Maki DG, Kluger DM, Crnich CJ. The risk of bloodstream infection in adults with different intravascular devices: a systematic review of 200 published prospective studies. Mayo Clin Proc. 2006;81(9):1159–71.PubMedCrossRef
40.
Salgado RI, Lord SR, Ehrlich F, Janji N, Rahman A. Predictors of falling in elderly hospital patients. Arch Gerontol Geriatr. 2004;38(3):213–9.PubMedCrossRef
41.
Patel PJ, Leeper Jr KV, McGowan Jr JE. Epidemiology and microbiology of hospital-acquired pneumonia. Semin Respir Crit Care Med. 2002;23(5):415–25.PubMedCrossRef
42.
O'Grady NP, Alexander M, Burns LA, Dellinger EP, Garland J, Heard SO, et al. Guidelines for the prevention of intravascular catheter-related infections. Am J Infect Control. 2011;39(4 Suppl 1):S1–S34.PubMedCrossRef
43.
Rotstein C, Evans G, Born A, Grossman R, Light RB, Magder S, et al. Clinical practice guidelines for hospital-acquired pneumonia and ventilator-associated pneumonia in adults. Can J Infect Dis Med Microbiol. 2008;19(1):19–53.PubMedCentralPubMed
44.
Eggimann P, Pittet D. Overview of catheter-related infections with special emphasis on prevention based on educational programs. Clin Microbiol Infect. 2002;8(5):295–309.PubMedCrossRef
45.
Oliver D, Daly F, Martin FC, McMurdo ME. Risk factors and risk assessment tools for falls in hospital in-patients: a systematic review. Age Ageing. 2004;33(2):122–30.PubMedCrossRef
46.
Hill AM, Hoffmann T, Hill K, Oliver D, Beer C, McPhail S, et al. Measuring falls events in acute hospitals-a comparison of three reporting methods to identify missing data in the hospital reporting system. J Am Geriatr Soc. 2010;58(7):1347–52.PubMedCrossRef
47.
48.
49.
Hirschhorn LR, Currier JS, Platt R. Electronic surveillance of antibiotic exposure and coded discharge diagnoses as indicators of postoperative infection and other quality assurance measures. Infect Control Hosp Epidemiol. 1993;14(1):21–8.PubMedCrossRef
50.
Trick WE, Chapman WW, Wisniewski MF, Peterson BJ, Solomon SL, Weinstein RA. Electronic interpretation of chest radiograph reports to detect central venous catheters. Infect Control Hosp Epidemiol. 2003;24(12):950–4.PubMedCrossRef
51.
Emori TG, Edwards JR, Culver DH, Sartor C, Stroud LA, Gaunt EE, et al. Accuracy of reporting nosocomial infections in intensive-care-unit patients to the National Nosocomial Infections Surveillance System: a pilot study. Infect Control Hosp Epidemiol. 1998;19(5):308–16.PubMedCrossRef
52.
Tejerina E, Esteban A, Fernandez-Segoviano P, Frutos-Vivar F, Aramburu J, Ballesteros D, et al. Accuracy of clinical definitions of ventilator-associated pneumonia: comparison with autopsy findings. J Crit Care. 2010;25(1):62–8.PubMedCrossRef
53.
Lin MY, Hota B, Khan YM, Woeltje KF, Borlawsky TB, Doherty JA, et al. Quality of traditional surveillance for public reporting of nosocomial bloodstream infection rates. JAMA. 2010;304(18):2035–41.PubMedCrossRef
54.
Hanley JA, McNeil BJ. A method of comparing the areas under receiver operating characteristic curves derived from the same cases. Radiology. 1983;148(3):839–43.PubMedCrossRef
55.
Charlson ME, Pompei P, Ales KL, MacKenzie CR. A new method of classifying prognostic comorbidity in longitudinal studies: development and validation. J Chronic Dis. 1987;40(5):373–83.PubMedCrossRef
56.
Chuang JH, Friedman C, Hripcsak G. A comparison of the Charlson comorbidities derived from medical language processing and administrative data. Proc AMIA Symp. 2002;2002:160–4.
57.
Pepe M. The Statistical Evaluation of Medical Test Classification and Prediction. New York: Oxford Press; 2004.
58.
Irwig L, Glasziou PP, Berry G, Chock C, Mock P, Simpson JM. Efficient study designs to assess the accuracy of screening tests. Am J Epidemiol. 1994;140(8):759–69.PubMed
59.
60.
Dowell SF, Ho MS. Seasonality of infectious diseases and severe acute respiratory syndrome-what we don’t know can hurt us. Lancet Infect Dis. 2004;4(11):704–8.PubMedCrossRef
61.
Dublin S, Baldwin E, Walker RL, Christensen LM, Haug PJ, Jackson ML, et al. Natural Language Processing to identify pneumonia from radiology reports. Pharmacoepidemiol Drug Saf. 2013;22(8):834–1.PubMedCrossRef
62.
Elkin PL, Froehling D, Wahner-Roedler D, Trusko B, Welsh G, Ma H, et al. NLP-based identification of pneumonia cases from free-text radiological reports. AMIA Annu Symp Proc. 2008;6:172–6.
63.
Trick WE, Zagorski BM, Tokars JI, Vernon MO, Welbel SF, Wisniewski MF, et al. Computer algorithms to detect bloodstream infections. Emerg Infect Dis. 2004;10(9):1612–20.PubMedCentralPubMedCrossRef
64.
Bellini C, Petignat C, Francioli P, Wenger A, Bille J, Klopotov A, et al. Comparison of automated strategies for surveillance of nosocomial bacteremia. Infect Control Hosp Epidemiol. 2007;28(9):1030–5.PubMedCrossRef
65.
Hota B, Lin M, Doherty JA, Borlawsky T, Woeltje K, Stevenson K, et al. Formulation of a model for automating infection surveillance: algorithmic detection of central-line associated bloodstream infection. J Am Med Inform Assoc. 2010;17(1):42–8.PubMedCentralPubMedCrossRef
66.
Toyabe S. Detecting inpatient falls by using natural language processing of electronic medical records. BMC Health Serv Res. 2012;12:448.
Metadata
Title
Accuracy of using automated methods for detecting adverse events from electronic health record data: a research protocol
Authors
Christian M Rochefort
David L Buckeridge
Alan J Forster
Publication date
01-12-2015
Publisher
BioMed Central
Published in
Implementation Science / Issue 1/2015
Electronic ISSN: 1748-5908
DOI
https://doi.org/10.1186/s13012-014-0197-6

Other articles of this Issue 1/2015

Implementation Science 1/2015 Go to the issue

Research

Understanding feedback report uptake: process evaluation findings from a 13-month feedback intervention in long-term care settings

Systematic review

Outcomes for implementation science: an enhanced systematic review of instruments using evidence-based rating criteria

Research

The transcatheter aortic valve implementation (TAVI)—a qualitative approach to the implementation and diffusion of a minimally invasive surgical procedure

Erratum

Erratum to: Expanded Quality Management Using Information Power (EQUIP): protocol for a quasi-experimental study to improve maternal and newborn health in Tanzania and Uganda

Study protocol

Study protocol for “Study of Practices Enabling Implementation and Adaptation in the Safety Net (SPREAD-NET)”: a pragmatic trial comparing implementation strategies

Systematic review

Exploring the function and effectiveness of knowledge brokers as facilitators of knowledge translation in health-related settings: a systematic review and thematic analysis