Top

BMC Medical Research Methodology

Published in:

Open Access 01-12-2020 | Artificial Intelligence | Research article

An evaluation of DistillerSR’s machine learning-based prioritization tool for title/abstract screening – impact on reviewer-relevant outcomes

Authors: C. Hamel, S. E. Kelly, K. Thavorn, D. B. Rice, G. A. Wells, B. Hutton

Published in: BMC Medical Research Methodology | Issue 1/2020

Abstract

Background

Systematic reviews often require substantial resources, partially due to the large number of records identified during searching. Although artificial intelligence may not be ready to fully replace human reviewers, it may accelerate and reduce the screening burden. Using DistillerSR (May 2020 release), we evaluated the performance of the prioritization simulation tool to determine the reduction in screening burden and time savings.

Methods

Using a true recall @ 95%, response sets from 10 completed systematic reviews were used to evaluate: (i) the reduction of screening burden; (ii) the accuracy of the prioritization algorithm; and (iii) the hours saved when a modified screening approach was implemented. To account for variation in the simulations, and to introduce randomness (through shuffling the references), 10 simulations were run for each review. Means, standard deviations, medians and interquartile ranges (IQR) are presented.

Results

Among the 10 systematic reviews, using true recall @ 95% there was a median reduction in screening burden of 47.1% (IQR: 37.5 to 58.0%). A median of 41.2% (IQR: 33.4 to 46.9%) of the excluded records needed to be screened to achieve true recall @ 95%. The median title/abstract screening hours saved using a modified screening approach at a true recall @ 95% was 29.8 h (IQR: 28.1 to 74.7 h). This was increased to a median of 36 h (IQR: 32.2 to 79.7 h) when considering the time saved not retrieving and screening full texts of the remaining 5% of records not yet identified as included at title/abstract. Among the 100 simulations (10 simulations per review), none of these 5% of records were a final included study in the systematic review. The reduction in screening burden to achieve true recall @ 95% compared to @ 100% resulted in a reduced screening burden median of 40.6% (IQR: 38.3 to 54.2%).

Conclusions

The prioritization tool in DistillerSR can reduce screening burden. A modified or stop screening approach once a true recall @ 95% is achieved appears to be a valid method for rapid reviews, and perhaps systematic reviews. This needs to be further evaluated in prospective reviews using the estimated recall.

Available only for authorised users

About Cochrane reviews 2020. https://www.cochranelibrary.com/about/about-cochrane-reviews. Accessed 16 Jan 2020.

Borah R, Brown A, Capers P, Kaiser K. Analysis of the time and workers needed to conduct systematic reviews of medical interventions using data from the PROSPERO registry. BMJ Open. 2017;7:e012545.CrossRef

Wang Z, Nayfeh T, Tetzlaff J, O’Blenis P, Murad MH. Error rates of human reviewers during abstract screening in systematic reviews. PLoS One 2020;15:e0227742. doi: https://doi.org/10.1371/journal.pone.0227742.

Taljaard M, McGowan J, Grimshaw JM, Brehaut JC, McRae A, Eccles MP, et al. Electronic search strategies to identify reports of cluster randomized trials in MEDLINE: low precision will improve with adherence to reporting standards. BMC Med Res Methodol. 2010;10:15. https://doi.org/10.1186/1471-2288-10-15.CrossRefPubMedPubMedCentral

Waffenschmidt S, Knelangen M, Sieben W, Bühn S, Pieper D. Single screening versus conventional double screening for study selection in systematic reviews: a methodological systematic review. BMC Med Res Methodol. 2019;19:132. https://doi.org/10.1186/s12874-019-0782-0.CrossRefPubMedPubMedCentral

Mateen FJ, Oh J, Tergas AI, Bhayani NH, Kamdar BB. Titles versus titles and abstracts for initial screening of articles for systematic reviews. Clin Epidemiol. 2013;5:89–95. https://doi.org/10.2147/CLEP.S43118.CrossRefPubMedPubMedCentral

Wang Z, Asi N, Elraiyah TA, Abu Dabrh AM, Undavalli C, Glasziou P, et al. Dual computer monitors to increase efficiency of conducting systematic reviews. J Clin Epidemiol. 2014;67:1353–7. https://doi.org/10.1016/j.jclinepi.2014.06.011.CrossRefPubMed

Rathbone J, Albarqouni L, Bakhit M, Beller E, Byambasuren O, Hoffmann T, et al. Expediting citation screening using PICo-based title-only screening for identifying studies in scoping searches and rapid reviews. Syst Rev. 2017;6:233. https://doi.org/10.1186/s13643-017-0629-x.CrossRefPubMedPubMedCentral

Pham MT, Waddell L, Rajić A, Sargeant JM, Papadopoulos A, McEwen SA. Implications of applying methodological shortcuts to expedite systematic reviews: three case studies using systematic reviews from Agri-food public health. Res Synth Methods. 2016;7:433–46. https://doi.org/10.1002/jrsm.1215.CrossRefPubMedPubMedCentral

10.

Edwards P, Clarke M, DiGuiseppi C, Pratap S, Roberts I, Wentz R. Identification of randomized controlled trials in systematic reviews: accuracy and reliability of screening records. Stat Med. 2002;21:1635–40. https://doi.org/10.1002/sim.1190.CrossRefPubMed

11.

Shemilt I, Khan N, Park S, Thomas J. Use of cost-effectiveness analysis to compare the efficiency of study identification methods in systematic reviews. Syst Rev. 2016;5:140. https://doi.org/10.1186/s13643-016-0315-4.CrossRefPubMedPubMedCentral

12.

Gartlehner G, Affengruber L, Titscher V, Noel-Storr A, Dooley G, Ballarini N, et al. Single-reviewer abstract screening missed 13 percent of relevant studies: a crowd-based, randomized controlled trial. J Clin Epidemiol. 2020. https://doi.org/10.1016/j.jclinepi.2020.01.005.

13.

Gartlehner G, Wagner G, Lux L, Affengruber L, Dobrescu A, Kaminski-Hartenthaler A, et al. Assessing the accuracy of machine-assisted abstract screening with DistillerAI: a user study. Syst Rev. 2019;8:277. https://doi.org/10.1186/s13643-019-1221-3.CrossRefPubMedPubMedCentral

14.

Gates A, Guitard S, Pillay J, Elliott SA, Dyson MP, Newton AS, et al. Performance and usability of machine learning for screening in systematic reviews: a comparative evaluation of three tools. Syst Rev. 2019;8:278. https://doi.org/10.1186/s13643-019-1222-2.CrossRefPubMedPubMedCentral

15.

Tsou AY, Treadwell JR, Erinoff E, Schoelles K. Machine learning for screening prioritization in systematic reviews: comparative performance of Abstrackr and EPPI-reviewer. Syst Rev. 2020;9:73.CrossRef

16.

Howard BE, Phillips J, Miller K, Tandon A, Mav D, Shah MR, et al. SWIFT-review: a text-mining workbench for systematic review. Syst Rev. 2016;5:87. https://doi.org/10.1186/s13643-016-0263-z.CrossRefPubMedPubMedCentral

17.

O’Mara-Eves A, Thomas J, McNaught J, Miwa M, Ananiadou S. Using text mining for study identification in systematic reviews: a systematic review of current approaches. Syst Rev. 2015;4:5. https://doi.org/10.1186/2046-4053-4-5.CrossRefPubMedPubMedCentral

18.

Harrison H, Griffin SJ, Kuhn I, Usher-Smith JA. Software tools to support title and abstract screening for systematic reviews in healthcare: an evaluation. BMC Med Res Methodol 2020;20:7. doi: https://doi.org/10.1186/s12874-020-0897-3.

19.

Marshall IJ, Wallace BC. Toward systematic review automation: a practical guide to using machine learning tools in research synthesis. Syst Rev. 2019;8:163. https://doi.org/10.1186/s13643-019-1074-9.CrossRefPubMedPubMedCentral

20.

Gates A, Johnson C, Hartling L. Technology-assisted Title and Abstract Screening for Systematic Reviews: A Retrospective Evaluation of the Abstrackr Machine Learning Tool. Systematic Reviews. 2018;7:45. https://doi.org/10.1186/s13643-018-0707-8.CrossRefPubMedPubMedCentral

21.

Rathbone J, Hoffmann T, Glasziou P. Faster Title and Abstract Screening? Evaluating Abstrackr, a Semi-Automated Online Screening Program for Systematic Reviewers. Syst Rev 2015;4:80. doi: https://doi.org/10.1186/s13643-015-0067-6.

22.

Giummarra MJ, Lau G, Gabbe BJ. Evaluation of text mining to reduce screening workload for injury-focused systematic reviews. Inj Prev. 2020;26:55–60. https://doi.org/10.1136/injuryprev-2019-043247.CrossRefPubMed

23.

Howard BE, Phillips J, Tandon A, Maharana A, Elmore R, Mav D, et al. SWIFT-active screener: accelerated document screening through active learning and integrated recall estimation. Environ Int. 2020;138:105623. https://doi.org/10.1016/j.envint.2020.105623.CrossRefPubMed

24.

Gates A, Gates M, Sebastianski M, Guitard S, Elliott SA, Hartling L. The semi-automation of title and abstract screening: a retrospective exploration of ways to leverage Abstrackr’s relevance predictions in systematic and rapid reviews. BMC Med Res Methodol 2020;20:139. doi: https://doi.org/10.1186/s12874-020-01031-w.

25.

O’Connor AM, Tsafnat G, Thomas J, Glasziou P, Gilbert SB, Hutton B. A question of trust: can we build an evidence base to gain trust in systematic review automation technologies? Syst Rev. 2019;8:143. https://doi.org/10.1186/s13643-019-1062-0.CrossRefPubMedPubMedCentral

26.

Cohen AM, Hersh WR, Peterson K, Yen P-Y. Reducing workload in systematic review preparation using automated citation classification. J Am Med Inform Assoc. 2006;13:206–19. https://doi.org/10.1197/jamia.M1929.CrossRefPubMedPubMedCentral

27.

Ganann R, Ciliska D, Thomas H. Expediting systematic reviews: methods and implications of rapid reviews. Implement Sci. 2010;5:56. https://doi.org/10.1186/1748-5908-5-56.CrossRefPubMedPubMedCentral

28.

Tricco AC, Antony J, Zarin W, Strifler L, Ghassemi M, Ivory J, et al. A scoping review of rapid review methods. BMC Med. 2015;13:224. https://doi.org/10.1186/s12916-015-0465-6.CrossRefPubMedPubMedCentral

29.

Watt A, Cameron A, Sturm L, Lathlean T, Babidge W, Blamey S, et al. Rapid reviews versus full systematic reviews: an inventory of current methods and practice in health technology assessment. Int J Technol Assess Health Care. 2008;24:133–9.CrossRef

30.

Grant MJ, Booth A. A typology of reviews: an analysis of 14 review types and associated methodologies. Health Inf Libr J. 2009;26:91–108. https://doi.org/10.1111/j.1471-1842.2009.00848.x.CrossRef

31.

Elliott JH, Synnot A, Turner T, Simmonds M, Akl EA, McDonald S, et al. Living systematic review: 1. Introduction-the why, what, when, and how. J Clin Epidemiol. 2017;91:23–30. https://doi.org/10.1016/j.jclinepi.2017.08.010.CrossRefPubMed

32.

Thomas J, Noel-Storr A, Marshall I, Wallace B, McDonald S, Mavergames C, et al. Living systematic reviews: 2. Combining human and machine effort. J Clin Epidemiol. 2017;91:31–7. https://doi.org/10.1016/j.jclinepi.2017.08.011.CrossRefPubMed

33.

Munn Z, Stern C, Aromataris E, Lockwood C, Jordan Z. What kind of systematic review should I conduct? A proposed typology and guidance for systematic reviewers in the medical and health sciences. BMC Med Res Methodol. 2018;18:5. https://doi.org/10.1186/s12874-017-0468-4.CrossRefPubMedPubMedCentral

34.

Khangura S, Konnyu K, Cushman R, Grimshaw J, Moher D. Evidence summaries: the evolution of a rapid review approach. Syst Rev. 2012;1:10. https://doi.org/10.1186/2046-4053-1-10.CrossRefPubMedPubMedCentral

35.

Wagner G, Nussbaumer-Streit B, Greimel J, Ciapponi A, Gartlehner G. Trading certainty for speed - how much uncertainty are decisionmakers and guideline developers willing to accept when using rapid reviews: an international survey. BMC Med Res Methodol. 2017;17:121. https://doi.org/10.1186/s12874-017-0406-5.CrossRefPubMedPubMedCentral

36.

Hutton B, Yazdi F, Bordeleau L, Morgan S, Cameron C, Kanji S, et al. Comparison of physical interventions, behavioral interventions, natural health products, and pharmacologics to manage hot flashes in patients with breast or prostate cancer: protocol for a systematic review incorporating network meta-analyses. Syst Rev. 2015;4:114. https://doi.org/10.1186/s13643-015-0099-y.CrossRefPubMedPubMedCentral

37.

Rice DB, Hutton B, Poulin P, Sproule BA, Wolfe D, Moher D, et al. Evaluating comparative effectiveness of psychosocial interventions for persons receiving opioid agonist therapy for opioid use disorder: protocol for a systematic review. BMJ Open. 2018;8:e023902. https://doi.org/10.1136/bmjopen-2018-023902.CrossRefPubMedPubMedCentral

38.

Ahmadzai N, Cheng W, Wolfe D, Bonaparte J, Schramm D, Fitzpatrick E, et al. Pharmacologic and surgical therapies for patients with Meniere’s disease: a protocol for a systematic review and meta-analysis. Syst Rev. 2019;8:341. https://doi.org/10.1186/s13643-019-1195-1.CrossRefPubMedPubMedCentral

39.

Elliott J, Bai Z, Hsieh S-C, Kelly SE, Chen L, Skidmore B, et al. ALK inhibitors for non-small cell lung cancer: a systematic review and network meta-analysis. PLoS One. 2020;15:e0229179. https://doi.org/10.1371/journal.pone.0229179.CrossRefPubMedPubMedCentral

40.

Hersi M, Traversy G, Thombs BD, Beck A, Skidmore B, Groulx S, et al. Effectiveness of stop smoking interventions among adults: protocol for an overview of systematic reviews and an updated systematic review. Syst Rev. 2019;8:28. https://doi.org/10.1186/s13643-018-0928-x.CrossRefPubMedPubMedCentral

41.

Johnston A, Smith C, Zheng C, Aaron SD, Kelly SE, Skidmore B, et al. Influence of prolonged treatment with omalizumab on the development of solid epithelial cancer in patients with atopic asthma and chronic idiopathic urticaria: a systematic review and meta-analysis. Clin Exp Allergy. 2019;49:1291–305. https://doi.org/10.1111/cea.13457.CrossRefPubMed

42.

Hamel C, Lang E, Morissette K, Beck A, Stevens A, Skidmore B, et al. Screening for depression in women during pregnancy or the first year postpartum and in the general adult population: a protocol for two systematic reviews to update a guideline of the Canadian task force on preventive health care. Syst Rev. 2019;8:27. https://doi.org/10.1186/s13643-018-0930-3.CrossRefPubMedPubMedCentral

43.

Hamel C, Stevens A, Singh K, Ansari MT, Myers E, Ziegler P, et al. Do sugar-sweetened beverages cause adverse health outcomes in adults? A systematic review protocol. Syst Rev. 2014;3:108. https://doi.org/10.1186/2046-4053-3-108.CrossRefPubMedPubMedCentral

44.

Stevens A, Hamel C, Singh K, Ansari MT, Myers E, Ziegler P, et al. Do sugar-sweetened beverages cause adverse health outcomes in children? A systematic review protocol. Syst Rev. 2014;3:96. https://doi.org/10.1186/2046-4053-3-96.CrossRefPubMedPubMedCentral

Title: An evaluation of DistillerSR’s machine learning-based prioritization tool for title/abstract screening – impact on reviewer-relevant outcomes
Authors: C. Hamel
S. E. Kelly
K. Thavorn
D. B. Rice
G. A. Wells
B. Hutton
Publication date: 01-12-2020
Publisher: BioMed Central
Keyword: Artificial Intelligence
Published in: BMC Medical Research Methodology / Issue 1/2020
Electronic ISSN: 1471-2288
DOI: https://doi.org/10.1186/s12874-020-01129-1

Keynote webinar | Spotlight on medication adherence

Springer Medicine

An evaluation of DistillerSR’s machine learning-based prioritization tool for title/abstract screening – impact on reviewer-relevant outcomes

Abstract

Background

Methods

Results

Conclusions

Keynote webinar | Spotlight on medication adherence

Springer Medicine

Abstract

Background

Methods

Results

Conclusions

Please log in to get access to this content

Other articles of this Issue 1/2020

Implications of sex offender classification on reporting demographic characteristics, health, and criminal careers: results from an Australian jurisdiction

Power difference in a χ2 test vs generalized linear mixed model in the presence of missing data – a simulation study

Adjusting for comorbidity in incidence-based DALY calculations: an individual-based modeling approach

A scoping review of core outcome sets and their ‘mapping’ onto real-world data using prostate cancer as a case study

Factors associated with patient willingness to participate in anaesthesia clinical trials: a vignette-based cross-sectional study

Telephone versus web panel National Survey for monitoring adoption of preventive behaviors to climate change in populations: a case study of Lyme disease in Québec, Canada