Top

BMC Medical Informatics and Decision Making

Published in:

Open Access 01-12-2010 | Technical advance

ExaCT: automatic extraction of clinical trial characteristics from journal publications

Authors: Svetlana Kiritchenko, Berry de Bruijn, Simona Carini, Joel Martin, Ida Sim

Published in: BMC Medical Informatics and Decision Making | Issue 1/2010

Abstract

Background

Clinical trials are one of the most important sources of evidence for guiding evidence-based practice and the design of new trials. However, most of this information is available only in free text - e.g., in journal publications - which is labour intensive to process for systematic reviews, meta-analyses, and other evidence synthesis studies. This paper presents an automatic information extraction system, called ExaCT, that assists users with locating and extracting key trial characteristics (e.g., eligibility criteria, sample size, drug dosage, primary outcomes) from full-text journal articles reporting on randomized controlled trials (RCTs).

Methods

ExaCT consists of two parts: an information extraction (IE) engine that searches the article for text fragments that best describe the trial characteristics, and a web browser-based user interface that allows human reviewers to assess and modify the suggested selections. The IE engine uses a statistical text classifier to locate those sentences that have the highest probability of describing a trial characteristic. Then, the IE engine's second stage applies simple rules to these sentences to extract text fragments containing the target answer. The same approach is used for all 21 trial characteristics selected for this study.

Results

We evaluated ExaCT using 50 previously unseen articles describing RCTs. The text classifier (first stage) was able to recover 88% of relevant sentences among its top five candidates (top5 recall) with the topmost candidate being relevant in 80% of cases (top1 precision). Precision and recall of the extraction rules (second stage) were 93% and 91%, respectively. Together, the two stages of the extraction engine were able to provide (partially) correct solutions in 992 out of 1050 test tasks (94%), with a majority of these (696) representing fully correct and complete answers.

Conclusions

Our experiments confirmed the applicability and efficacy of ExaCT. Furthermore, they demonstrated that combining a statistical method with 'weak' extraction rules can identify a variety of study characteristics. The system is flexible and can be extended to handle other characteristics and document types (e.g., study protocols).

Available only for authorised users

Guyatt GH, Haynes RB, Jaeschke RZ, Cook DJ, Green L, Naylor CD, Wilson MC, Richardson WS: Users' Guides to the Medical Literature: XXV. Evidence-based medicine: principles for applying the Users' Guides to patient care. Evidence-Based Medicine Working Group. JAMA. 2000, 284 (10): 1290-1296. 10.1001/jama.284.10.1290.CrossRefPubMed

Sim I, Owens DK, Lavori PW, Rennels GD: Electronic trial banks: a complementary method for reporting randomized trials. Med Decis Making. 2000, 20 (4): 440-450. 10.1177/0272989X0002000408.CrossRefPubMed

Sim I, Gorman P, Greenes RA, Haynes RB, Kaplan B, Lehmann H, Tang PC: Clinical decision support systems for the practice of evidence-based medicine. J Am Med Inform Assoc. 2001, 8 (6): 527-534.CrossRefPubMedPubMedCentral

Science Commons. --- Either ISSN or Journal title must be supplied.. [http://sciencecommons.org/]

ClinicalTrials.gov Results. --- Either ISSN or Journal title must be supplied.. [http://clinicaltrials.gov/ct2/info/results]

The Human Study Database. --- Either ISSN or Journal title must be supplied.. [http://hsdbwiki.org]

Sim I, Carini S, Tu S, Wynden R, Pollock BH, Mollah SA, Gabriel D, Hagler HK, Scheuermann RH, Lehmann HP, Wittkowski KM, Nahm M, Bakken S: The Human Studies Database project: federating human studies design data using the Ontology of Clinical Research. Proceedings of the AMIA CRI Summit: 12-13 March 2010; San Francisco, CA. 2010, American Medical Informatics Association

Alex B, Grover C, Haddow B, Kabadjov M, Klein E, Matthews M, Roebuck S, Tobin R, Wang X: Assisted curation: does text mining really help?. Proceedings of the Pacific Symposium on Biocomputing: 4-8 January 2008; Hawaii. Edited by: Altman RB, Dunker AK, Hunter L, Murray T, Klein TE. 2008, Singapore: World Scientific Publishing Co., 556-567. full_text.

Demner-Fushman D, Lin J: Knowledge extraction for clinical question answering: preliminary results. Proceedings of the AAAI Workshop on Question Answering in Restricted Domains: 10 July 2005; Pittsburgh, PA. Edited by: Molla D, Vicedo JL. 2005, AAAI Press, 1-9.

10.

Niu Y, Hirst G: Analysis of semantic classes in medical text for question answering. Proceedings of the ACL Workshop on Question Answering in Restricted Domains: 25 July 2004; Barcelona, Spain. Edited by: Aliod DM, Vicedo JL. 2004, Association for Computational Linguistics, 54-61.

11.

Borlawsky T, Friedman C, Lussier YA: Generating executable knowledge for evidence-based medicine using natural language and semantic processing. Proceedings of the AMIA Annual Symposium: 11-16 November 2006; Washington, DC. 2006, American Medical Informatics Association, 56-60.

12.

Chung GY, Coiera E: A study of structured clinical abstracts and the semantic classification of sentences. Proceedings of the ACL Workshop BioNLP: 29 June 2007; Prague, Czech Republic. Edited by: Cohen KB, Demner-Fushman D, Friedman C, Hirschman L, Pestian J. 2007, Association for Computational Linguistics, 121-128.

13.

Hara K, Matsumoto Y: Extracting Clinical Trial Design Information from MEDLINE Abstracts. New Generation Computing. 2007, 25 (3): 263-275. 10.1007/s00354-007-0017-5.CrossRef

14.

Xu R, Garten Y, Supekar KS, Das AK, Altman RB, Garber AM: Extracting subject demographic information from abstracts of randomized clinical trial reports. Proceedings of the 12th World Congress on Health (Medical) Informatics: 20-24 August 2007; Brisbane, Australia. Edited by: Kuhn KA, Warren JR, Leong TY. 2007, Amsterdam: IOS Press, 550-554.

15.

Chung GY: Towards identifying intervention arms in randomized controlled trials: extracting coordinating constructions. J Biomed Inform. 2009, 42 (5): 790-800. 10.1016/j.jbi.2008.12.011.CrossRefPubMed

16.

Hansen MJ, Rasmussen NO, Chung G: Extracting number of trial participants from abstracts of randomized controlled trials. Proceedings of Tromsø Telemedicine and eHealth Conference: 9-11 June 2008; Tromsø, Norway. 2008

17.

Cohen AM, Hersh WR: A survey of current work in biomedical text mining. Brief Bioinform. 2005, 6 (1): 57-71. 10.1093/bib/6.1.57.CrossRefPubMed

18.

Meystre SM, Savova GK, Kipper-Schuler KC, Hurdle JF: Extracting information from textual documents in the electronic health record: a review of recent research. Methods Inf Med. 2008, 47 (Suppl 1): 128-144.

19.

Chen ES, Hripcsak G, Xu H, Markatou M, Friedman C: Automated acquisition of disease-drug knowledge from biomedical and clinical documents: an initial study. J Am Med Inform Assoc. 2008, 15: 87-98. 10.1197/jamia.M2401.CrossRefPubMedPubMedCentral

20.

McKnight L, Srinivasan P: Categorization of sentence types in medical abstracts. Proceedings of the AMIA Annual Symposium: 8-12 November 2003; Washington, DC. Edited by: Musen M. 2003, American Medical Informatics Association, 440-444.

21.

Lin J, Karakos D, Demner-Fushman D, Khudanpur S: Generative content models for structural analysis of medical abstracts. Proceedings of HLT-NAACL Workshop on Linking Natural Language Processing and Biology: 8 June 2006; New York, NY. Edited by: Verspoor K, Cohen KB, Goertzel B, Mani I. 2006, Association for Computational Linguistics, 65-72. full_text.

22.

Xu R, Supekar K, Huang Y, Das A, Garber A: Combining text classification and hidden Markov modeling techniques for structuring randomized clinical trial abstracts. Proceedings of the AMIA Annual Symposium: 11-16 November 2006; Washington, DC. 2006, American Medical Informatics Association, 824-828.

23.

Ruch P, Geissbühler A, Gobeill J, Lisacek F, Tbahriti I, Veuthey A-L, Aronson AR: Using discourse analysis to improve text categorization in MEDLINE. Proceedings of the 12th World Congress on Health (Medical) Informatics: 20-24 August 2007; Brisbane, Australia. Edited by: Kuhn KA, Warren JR, Leong TY. 2007, Amsterdam: IOS Press, 710-715.

24.

Hirohata K, Okazaki N, Ananiadou S, Ishizuka M: Identifying sections in scientific abstracts using conditional random fields. Proceedings of the International Joint Conference on Natural Language Processing: 7-12 January 2008; Hyderabad, India. 2008, 381-388.

25.

Mullen A, Mizuta Y, Collier N: A baseline feature set for learning rhetorical zones using full articles in the biomedical domain. ACM SIGKDD Explorations. 2005, 7 (1): 52-58. 10.1145/1089815.1089823.CrossRef

26.

Agarwal S, Yu H: Automatically classifying sentences in full-text biomedical articles into Introduction, Methods, Results and Discussion. Bioinformatics. 2009, 25 (23): 3174-3180. 10.1093/bioinformatics/btp548.CrossRefPubMedPubMedCentral

27.

Chung GY: Sentence retrieval for abstracts of randomized controlled trials. BMC Med Inform Decis Mak. 2009, 9: 10-10.1186/1472-6947-9-10.CrossRefPubMedPubMedCentral

28.

Paek H, Kogan Y, Thomas P, Codish S, Krauthammer M: Shallow semantic parsing of randomized controlled trial reports. Proceedings of the AMIA Annual Symposium: 11-16 November 2006; Washington, DC. 2006, American Medical Informatics Association, 604-608.

29.

De Bruijn B, Carini S, Kiritchenko S, Martin J, Sim I: Automated information extraction of key trial design elements from clinical trial publications. Proceedings of the AMIA Annual Symposium: 8-12 November 2008; Washington, DC. 2008, American Medical Informatics Association, 141-145.

30.

Erhardt RA, Schneider R, Blaschke C: Status of text-mining techniques applied to biomedical text. Drug Discov Today. 2006, 11 (7-8): 315-325. 10.1016/j.drudis.2006.02.011.CrossRefPubMed

31.

Patwardhan S, Riloff E: Effective information extraction with semantic affinity patterns and relevant regions. Proceedings of the Joint Conference on Empirical Methods in NLP and Computational Natural Language Learning: 28-30 June 2007; Prague, Czech Republic. Edited by: Eisner J. 2007, Association for Computational Linguistics, 717-727.

32.

Moher D, Schulz KF, Altman DG: The CONSORT statement: revised recommendations for improving the quality of reports of parallel-group randomised trials. Lancet. 2001, 357 (9263): 1191-1194. 10.1016/S0140-6736(00)04337-3.CrossRefPubMed

33.

Altman DG, Schulz KF, Moher D, Egger M, Davidoff F, Elbourne D, Gøtzsche PC, Lang T: The revised CONSORT statement for reporting randomized trials: explanation and elaboration. Ann Intern Med. 2001, 134 (8): 663-694.CrossRefPubMed

34.

Sim I, Olasov B, Carini S: An ontology of randomized controlled trials for evidence-based practice: content specification and evaluation using the competency decomposition method. J Biomed Inform. 2004, 37 (2): 108-119. 10.1016/j.jbi.2004.03.001.CrossRefPubMed

35.

Tbahriti I, Chichester C, Lisacek F, Ruch P: Using argumentation to retrieve articles with similar citations: An inquiry into improving related articles search in the MEDLINE digital library. Int J Med Inform. 2006, 75 (6): 488-495. 10.1016/j.ijmedinf.2005.06.007.CrossRefPubMed

36.

Chakrabarti S, Dom B, Agrawal R, Raghavan P: Using taxonomy, discriminants, and signatures for navigating in text databases. Proceedings of the International Conference on Very Large Data Bases: 25-29 August 1997; Athens, Greece. Edited by: Jarke M, Carey MJ, Dittrich KR, Lochovsky FH, Loucopoulos P, Jeusfeld MA. 1997, San Francisco: Morgan Kaufmann, 446-455.

37.

Van Rijsbergen CJ: Information Retrieval. 1979, London: Buttersworth, 2

38.

Sim I, Carini S, Olasov B, Jeng S: Trial bank publishing: phase I results. Proceedings of the World Congress on Medical Informatics: 7-11 September 2004; San Francisco, CA. Edited by: Fieschi M, Coiera E, Li Y-CJ. 2004, Amsterdam: IOS Press, 1476-1480.

39.

Carini S, Pollock BH, Lehmann HP, Bakken S, Barbour EM, Gabriel D, Hagler HK, Harper CR, Mollah SA, Nahm M, Nguyen HH, Scheuermann RH, Sim I: Development and evaluation of a study design typology for human subjects research. Proceedings of the AMIA Annual Symposium: 14-18 November 2009; San Francisco, CA. 2009, American Medical Informatics Association, 81-85.

The pre-publication history for this paper can be accessed here:http://www.biomedcentral.com/1472-6947/10/56/prepub

Title: ExaCT: automatic extraction of clinical trial characteristics from journal publications
Authors: Svetlana Kiritchenko
Berry de Bruijn
Simona Carini
Joel Martin
Ida Sim
Publication date: 01-12-2010
Publisher: BioMed Central
Published in: BMC Medical Informatics and Decision Making / Issue 1/2010
Electronic ISSN: 1472-6947
DOI: https://doi.org/10.1186/1472-6947-10-56

Keynote webinar | Spotlight on medication adherence

Springer Medicine

ExaCT: automatic extraction of clinical trial characteristics from journal publications

Abstract

Background

Methods

Results

Conclusions

Keynote webinar | Spotlight on medication adherence

Springer Medicine

Abstract

Background

Methods

Results

Conclusions

Please log in to get access to this content

Other articles of this Issue 1/2010

Use of outcomes to evaluate surveillance systems for bioterrorist attacks

Web 2.0 and Internet Social Networking: A New tool for Disaster Management? - Lessons from Taiwan

A predictive model for the early identification of patients at risk for a prolonged intensive care unit length of stay

Use of the internet as a source of health information by Spanish adolescents

From design to implementation - The Joint Asia Diabetes Evaluation (JADE) program: A descriptive report of an electronic web-based diabetes management program

Modeling screening, prevention, and delaying of Alzheimer's disease: an early-stage decision analytic model