Skip to main content
Key Specialties
Cardiology Diabetology Endocrinology Gastroenterology Geriatrics and Gerontology Gynecology and Obstetrics Hematology Infectious Disease Internal Medicine Nephrology Neurology Oncology Pediatrics Respiratory Medicine Rheumatology Surgery
Congress Coverage
2024 ACC Congress 2023 ESMO Congress 2023 EASD Congress 2023 ERS Congress 2023 ESC Congress
Clinical Collections
Advances in Alzheimer’s

At a glance: The STEP trials

A round-up of the STEP phase 3 clinical trials evaluating semaglutide for weight loss in people with overweight or obesity.
ADVANCED SEARCH
Log in
Top
BMC Medical Informatics and Decision Making
Published in: BMC Medical Informatics and Decision Making 1/2020

Open Access 01-12-2020 | Research article

Assessment of the impact of EHR heterogeneity for clinical research through a case study of silent brain infarction

Authors: Sunyang Fu, Lester Y. Leung, Anne-Olivia Raulli, David F. Kallmes, Kristin A. Kinsman, Kristoff B. Nelson, Michael S. Clark, Patrick H. Luetmer, Paul R. Kingsbury, David M. Kent, Hongfang Liu

Published in: BMC Medical Informatics and Decision Making | Issue 1/2020

Login to get access

Abstract

Background

The rapid adoption of electronic health records (EHRs) holds great promise for advancing medicine through practice-based knowledge discovery. However, the validity of EHR-based clinical research is questionable due to poor research reproducibility caused by the heterogeneity and complexity of healthcare institutions and EHR systems, the cross-disciplinary nature of the research team, and the lack of standard processes and best practices for conducting EHR-based clinical research.

Method

We developed a data abstraction framework to standardize the process for multi-site EHR-based clinical studies aiming to enhance research reproducibility. The framework was implemented for a multi-site EHR-based research project, the ESPRESSO project, with the goal to identify individuals with silent brain infarctions (SBI) at Tufts Medical Center (TMC) and Mayo Clinic. The heterogeneity of healthcare institutions, EHR systems, documentation, and process variation in case identification was assessed quantitatively and qualitatively.

Result

We discovered a significant variation in the patient populations, neuroimaging reporting, EHR systems, and abstraction processes across the two sites. The prevalence of SBI for patients over age 50 for TMC and Mayo is 7.4 and 12.5% respectively. There is a variation regarding neuroimaging reporting where TMC are lengthy, standardized and descriptive while Mayo’s reports are short and definitive with more textual variations. Furthermore, differences in the EHR system, technology infrastructure, and data collection process were identified.

Conclusion

The implementation of the framework identified the institutional and process variations and the heterogeneity of EHRs across the sites participating in the case study. The experiment demonstrates the necessity to have a standardized process for data abstraction when conducting EHR-based clinical studies.
Appendix
Available only for authorised users
Literature
1.
Friedman CP, Wong AK, Blumenthal D. Achieving a nationwide learning health system. Sci Transl Med. 2010;2(57):57cm29.CrossRefPubMed
2.
Gelijns AC, Gabriel SE. Looking beyond translation--integrating clinical research with medical practice. N Engl J Med. 2012;366(18):1659–61.CrossRefPubMed
3.
Milstein A. Code red and blue--safely limiting health care's GDP footprint. N Engl J Med. 2013;368(1):1–3.CrossRefPubMed
4.
Richesson RL, Horvath MM, Rusincovitch SA. Clinical research informatics and electronic health record data. Yearb Med Inform. 2014;9:215–23.PubMedPubMedCentral
5.
Kaggal VC, Elayavilli RK, Mehrabi S, Pankratz JJ, Sohn S, Wang Y, Li D, Rastegar MM, Murphy SP, Ross JL, et al. Toward a learning health-care system - knowledge delivery at the point of care empowered by big data and NLP. Biomed Inform Insights. 2016;8(Suppl 1):13–22.PubMedPubMedCentral
6.
7.
Wen A, Fu S, Moon S, El Wazir M, Rosenbaum A, Kaggal VC, Liu S, Sohn S, Liu H, JJnDM F. Desiderata for delivering NLP to accelerate healthcare AI advancement and a Mayo Clinic NLP-as-a-service implementation. NPJ Digit Med. 2019;2(1):1–7..CrossRef
8.
Frankovich J, Longhurst CA, Sutherland SM. Evidence-based medicine in the EMR era. N Engl J Med. 2011;365(19):1758–9.CrossRefPubMed
9.
Gearing RE, Mian IA, Barber J, Ickowicz A. A methodology for conducting retrospective chart review research in child and adolescent psychiatry. J Can Acad Child Adolesc Psychiatry. 2006;15(3):126–34.PubMedPubMedCentral
10.
Vassar M, Holzmann M. The retrospective chart review: important methodological considerations. J Educ Eval Health Prof. 2013;10:12.CrossRefPubMed
11.
Xu H, Jiang M, Oetjens M, Bowton EA, Ramirez AH, Jeff JM, Basford MA, Pulley JM, Cowan JD, Wang X. Facilitating pharmacogenetic studies using electronic health records and natural-language processing: a case study of warfarin. J Am Med Inform Assoc. 2011;18(4):387–91.CrossRefPubMedPubMedCentral
12.
Grishman R, Huttunen S, Yangarber R. Information extraction for enhanced access to disease outbreak reports. J Biomed Inform. 2002;35(4):236–46.CrossRefPubMed
13.
South BR, Shen S, Jones M, Garvin J, Samore MH, Chapman WW, Gundlapalli AV. Developing a manually annotated clinical document corpus to identify phenotypic information for inflammatory bowel disease. BMC Bioinformatics. 2009;10(Suppl 9):S12.CrossRefPubMedPubMedCentral
14.
Gilbert EH, Lowenstein SR, Koziol-McLain J, Barta DC, Steiner J. Chart reviews in emergency medicine research: where are the methods? Ann Emerg Med. 1996;27(3):305–8.CrossRefPubMed
15.
Wu ST, Sohn S, Ravikumar K, Wagholikar K, Jonnalagadda SR, Liu H, Juhn YJ. Automated chart review for asthma cohort identification using natural language processing: an exploratory study. Ann Allergy Asthma Immunol. 2013;111(5):364–9.CrossRefPubMed
16.
Dresser MV, Feingold L, Rosenkranz SL, Coltin KL. Clinical quality measurement. Comparing chart review and automated methodologies. Med Care. 1997;35(6):539–52.CrossRefPubMed
17.
Melton GB, Hripcsak G. Automated detection of adverse events using natural language processing of discharge summaries. J Am Med Inform Assoc. 2005;12(4):448–57.CrossRefPubMedPubMedCentral
18.
Carrell DS, Halgrim S, Tran DT, Buist DS, Chubak J, Chapman WW, Savova G. Using natural language processing to improve efficiency of manual chart abstraction in research: the case of breast cancer recurrence. Am J Epidemiol. 2014;179(6):749–58.CrossRefPubMedPubMedCentral
19.
Cohen KB, Xia J, Roeder C, Hunter LE. Reproducibility in natural language processing: a case study of two R libraries for mining PubMed/MEDLINE. LREC Int Conf Lang Resour Eval. 2016;2016(W23):6–12.PubMedPubMedCentral
20.
Branco A. Reliability and meta-reliability of language resources: ready to initiate the integrity debate? In: 12th Workshop on Treebanks and Linguistic Theories: December 13–14, 2013 2013; Sofia, Bulgaria; 2013.
21.
Baker D, Lidster K, Sottomayor A, Amor S. Reproducibility: research-reporting standards fall short. Nature. 2012;492(7427):41.CrossRefPubMed
22.
Johnson KE, Kamineni A, Fuller S, Olmstead D, Wernli KJ. How the provenance of electronic health record data matters for research: a case example using system mapping. EGEMS (Wash DC). 2014;2(1):1058.
23.
Karczewski KJ, Tatonetti NP, Manrai AK, Patel CJ, Titus Brown C, Ioannidis JPA. Methods to ensure the reproducibility of biomedical research. Pac Symp Biocomput. 2017;22:117–9.PubMedPubMedCentral
24.
25.
Zozus MN, Richesson RL, Walden A, Tenenbaum JD, Hammond WE. Research reproducibility in longitudinal multi-center studies using data from electronic health records. AMIA Jt Summits on Transl. 2016;2016:279–85.
26.
Manrai AK, Patel CJ, Gehlenborg N, Tatonetti NP, Ioannidis JP, Kohane IS. Methods to enhance the reproducibility of precision medicine. Pac Symp Biocomput. 2016;21:180–2.PubMedPubMedCentral
27.
Madigan D, Ryan PB, Schuemie M, Stang PE, Overhage JM, Hartzema AG, Suchard MA, DuMouchel W, Berlin JA. Evaluating the impact of database heterogeneity on observational study results. Am J Epidemiol. 2013;178(4):645–51.CrossRefPubMedPubMedCentral
28.
Sohn S, Wang Y, Wi CI, Krusemark EA, Ryu E, Ali MH, Juhn YJ, Liu H. Clinical documentation variations and NLP system portability: a case study in asthma birth cohorts across institutions. J Am Med Inform Assoc. 2018;25(3):353–9.
29.
Kharrazi HH, Wang C, Scharfstein DO. Prospective EHR-based clinical trials: The challenge of missing data. J Gen Intern Med. 2014;29(7):976–8.
30.
Wells BJ, Chagin KM, Nowacki AS, Kattan MWJE. Strategies for handling missing data in electronic health record derived data. EGEMS (Wash DC). 2013;1(3):1035.
31.
Weber GM, Murphy SN, McMurry AJ, Macfadden D, Nigrin DJ, Churchill S, Kohane IS. The shared Health Research information network (SHRINE): a prototype federated query tool for clinical data repositories. J Am Med Inform Assoc. 2009;16(5):624–30.CrossRefPubMedPubMedCentral
32.
Selby JV, Beal AC, Frank L. The Patient-Centered Outcomes Research Institute (PCORI) national priorities for research and initial research agenda. JAMA. 2012;307(15):1583–4.CrossRefPubMed
33.
Consortium PCP, Daugherty SE, Wahba S, Fleurence R. Patient-powered research networks: building capacity for conducting patient-centered clinical outcomes research. J Am Med Inform Assoc. 2014;21(4):583–6.CrossRef
34.
Hripcsak G, Duke JD, Shah NH, Reich CG, Huser V, Schuemie MJ, Suchard MA, Park RW, Wong IC, Rijnbeek PR, et al. Observational health data sciences and informatics (OHDSI): opportunities for observational researchers. Stud Health Technol Inform. 2015;216:574–8.PubMedPubMedCentral
35.
Savova GK, Chapman WW, Zheng J, Crowley RS. Anaphoric relations in the clinical narrative: corpus creation. J Am Med Inform Assoc. 2011;18(4):459–65.CrossRefPubMedPubMedCentral
36.
Albright D, Lanfranchi A, Fredriksen A, Styler WF, Warner C, Hwang JD, Choi JD, Dligach D, Nielsen RD, Martin J, et al. Towards comprehensive syntactic and semantic annotations of the clinical narrative. J Am Med Inform Assoc. 2013;20(5):922–30.CrossRefPubMedPubMedCentral
37.
Scuba W, Tharp M, Mowery D, Tseytlin E, Liu Y, Drews FA, Chapman WW. Knowledge author: facilitating user-driven, domain content development to support clinical information extraction. J Biomed Semantics. 2016;7(1):42.CrossRefPubMedPubMedCentral
38.
Leung LY, Han PK, Lundquist C, Weinstein G, Thaler DE, Kent D. Clinicians’ perspectives on incidentally discovered silent brain infarcts–a qualitative study. PLoS One. 2018;13(3):e0194971.CrossRefPubMedPubMedCentral
39.
Leech G. Corpus annotation schemes. Literary Linguist Comput. 1993;8(4):275–81.CrossRef
40.
Friedman LM, Furberg C, DeMets DL. Fundamentals of clinical trials: springer; 1998.CrossRef
41.
Strasser C. Research data management. National Information Standards Organization; 2015.
42.
Fu S, Leung LY, Wang Y, Raulli A-O, Kallmes DF, Kinsman KA, Nelson KB, Clark MS, Luetmer PH. Kingsbury PRJJmi: Natural Language Processing for the Identification of Silent Brain Infarcts From Neuroimaging Reports. 2019;7(2):e12109.
43.
Vermeer SE, Longstreth WT Jr, Koudstaal PJ. Silent brain infarcts: a systematic review. Lancet Neurol. 2007;6(7):611–9.CrossRefPubMed
44.
45.
Fanning JP, Wesley AJ, Wong AA, Fraser JF. Emerging spectra of silent brain infarction. Stroke. 2014;45(11):3461–71.CrossRefPubMed
46.
Conklin J, Silver FL, Mikulis DJ, Mandell DM. Are acute infarcts the cause of leukoaraiosis? Brain mapping for 16 consecutive weeks. Ann Neurol. 2014;76(6):899–904.CrossRefPubMed
47.
Chen Y, Wang A, Tang J, Wei D, Li P, Chen K, Wang Y, Zhang Z. Association of white matter integrity and cognitive functions in patients with subcortical silent lacunar infarcts. Stroke. 2015;46(4):1123–6.CrossRefPubMed
48.
Aberdeen J, Bayer S, Yeniterzi R, Wellner B, Clark C, Hanauer D, Malin B, Hirschman L. The MITRE identification scrubber toolkit: design, training, and assessment. Int J Med Inform. 2010;79(12):849–59.CrossRefPubMed
49.
Liu H, Bielinski SJ, Sohn S, Murphy S, Wagholikar KB, Jonnalagadda SR, Ravikumar KE, Wu ST, Kullo IJ, Chute CG. An information extraction framework for cohort identification using electronic health records. AMIA Jt Summits Transl Sci Proc. 2013;2013:149–53.PubMedPubMedCentral
50.
Rim K. Mae2: Portable annotation tool for general natural language use. In: 12th Joint ACL-ISO Workshop on Interoperable Semantic Annotation, vol. 2016; 2016.
51.
Cohen J. A coefficient of agreement for nominal scales. Educ Psychol Meas. 1960;20(1):37–46.CrossRef
52.
Sasaki Y. The truth of the F-measure. Teach Tutor Mater. 2007;1(5):1–5.
53.
Holtzblatt KWJ, Wood S. Rapid contextual design: a how-to guide to key techniques for user-centered design: Elsevier; 2004.
54.
Murphy SN, Weber G, Mendis M, Gainer V, Chueh HC, Churchill S, IJJotAMIA K. Serving the enterprise and beyond with informatics for integrating biology and the bedside (i2b2). J Am Med Inform Assoc. 2010;17(2):124–30.CrossRefPubMedPubMedCentral
Metadata
Title
Assessment of the impact of EHR heterogeneity for clinical research through a case study of silent brain infarction
Authors
Sunyang Fu
Lester Y. Leung
Anne-Olivia Raulli
David F. Kallmes
Kristin A. Kinsman
Kristoff B. Nelson
Michael S. Clark
Patrick H. Luetmer
Paul R. Kingsbury
David M. Kent
Hongfang Liu
Publication date
01-12-2020
Publisher
BioMed Central
Published in
BMC Medical Informatics and Decision Making / Issue 1/2020
Electronic ISSN: 1472-6947
DOI
https://doi.org/10.1186/s12911-020-1072-9

Other articles of this Issue 1/2020

BMC Medical Informatics and Decision Making 1/2020 Go to the issue

Research article

Facilitating the design of HL7 domain models through a model-driven solution

Research article

Iranian general populations' and health care providers' preferences for benefits and harms of statin therapy for primary prevention of cardiovascular disease

Software

A comprehensive tool for creating and evaluating privacy-preserving biomedical prediction models

Research article

Bayesian variable selection for high dimensional predictors and self-reported outcomes

Research article

A stacking-based model for predicting 30-day all-cause hospital readmissions of patients with acute myocardial infarction

Research article

Clinicians perceptions of a telemedicine system: a mixed method study of Makassar City, Indonesia