Top

BMC Medical Research Methodology

Published in:

Open Access 01-12-2020 | Research article

The semi-automation of title and abstract screening: a retrospective exploration of ways to leverage Abstrackr’s relevance predictions in systematic and rapid reviews

Authors: Allison Gates, Michelle Gates, Meghan Sebastianski, Samantha Guitard, Sarah A. Elliott, Lisa Hartling

Published in: BMC Medical Research Methodology | Issue 1/2020

Abstract

Background

We investigated the feasibility of using a machine learning tool’s relevance predictions to expedite title and abstract screening.

Methods

We subjected 11 systematic reviews and six rapid reviews to four retrospective screening simulations (automated and semi-automated approaches to single-reviewer and dual independent screening) in Abstrackr, a freely-available machine learning software. We calculated the proportion missed, workload savings, and time savings compared to single-reviewer and dual independent screening by human reviewers. We performed cited reference searches to determine if missed studies would be identified via reference list scanning.

Results

For systematic reviews, the semi-automated, dual independent screening approach provided the best balance of time savings (median (range) 20 (3–82) hours) and reliability (median (range) proportion missed records, 1 (0–14)%). The cited references search identified 59% (n = 10/17) of the records missed. For the rapid reviews, the fully and semi-automated approaches saved time (median (range) 9 (2–18) hours and 3 (1–10) hours, respectively), but less so than for the systematic reviews. The median (range) proportion missed records for both approaches was 6 (0–22)%.

Conclusion

Using Abstrackr to assist one of two reviewers in systematic reviews saves time with little risk of missing relevant records. Many missed records would be identified via other means.

Available only for authorised users

Borah R, Brown AW, Capers PL, Kaiser KA. Analysis of the time and workers needed to conduct systematic reviews of medical interventions using data from the PROSPERO registry. BMJ Open. 2017;7(2):e012545. https://doi.org/10.1136/bmjopen-2016-012545.CrossRefPubMedPubMedCentral

Lefebvre CME, Glanville J. Chapter 6: Searching for studies. In: JPT H, Green S, editors. Cochrane Handbook for Systematic Reviews of Interventions. London: Cochrane; 2011. Available at: https://training.cochrane.org/handbook.

Marshall C, Sutton A. Systematic review toolbox. 2020. Available at: http://systematicreviewtools.com/index.php. Accessed 20 April 2020.

O’Mara-Eves A, Thomas J, McNaught J, Miwa M, Ananiadou S. Using text mining for study identification in systematic reviews: a systematic review of current approaches. Syst Rev. 2015;4(1):5. https://doi.org/10.1186/2046-4053-4-5.CrossRefPubMedPubMedCentral

Thomas J, McNaught J, Ananiadou S. Applications of text mining within systematic reviews. Res Synth Methods. 2011;2(1):1–14. https://doi.org/10.1002/jrsm.27.CrossRefPubMed

O’Connor AM, Tsafnat G, Thomas J, Glasziou P, Gilbert SB, Hutton B. A question of trust: can we build an evidence base to gain trust in systematic review automation technologies? Syst Rev. 2019;8(1):143. https://doi.org/10.1186/s13643-019-1062-0.CrossRefPubMedPubMedCentral

Wallace BC, Small K, Brodley CE, Lau J, Trikalinos TA. Deploying an interactive machine learning system in an evidence-based practice center: abstrackr. Proceedings of the 2^nd ACM SIGHIT International Health Informatics Symposium. New York: Association for Computing Machinery; 2012. p. 819–24. https://doi.org/10.1145/2110363.2110464.CrossRef

Gates A, Johnson C, Hartling L. Technology-assisted title and abstract screening for systematic reviews: a retrospective evaluation of the abstrackr machine learning tool. Syst Rev. 2018;7(1):45. https://doi.org/10.1186/s13643-018-0707-8.CrossRefPubMedPubMedCentral

Giummarra MJ, Lau G, Gabbe BJ. Evaluation of text mining to reduce screening workload for injury-focused systematic reviews. Inj Prev. 2019. https://doi.org/10.1136/injuryprev-2019-043247.

10.

Rathbone J, Hoffmann T, Glasziou P. Faster title and abstract screening? Evaluating abstrackr, a semi-automated online screening program for systematic reviewers. Syst Rev. 2015;4(1):80. https://doi.org/10.1186/s13643-015-0067-6.CrossRefPubMedPubMedCentral

11.

Gates A, Guitard S, Pillay J, Elliott SA, Dyson MP, Newton AS, Hartling L. Performance and uability of machine learning for screening in systematic reviews: a comparative evaluation of three tools. Syst Rev. 2019;8:278. https://doi.org/10.1186/s13643-019-1222-2.CrossRefPubMedPubMedCentral

12.

Waffenschmidt S, Knelangen M, Sieben W, Bühn S, Pieper D. Single screening versus conventional double screening for study selection in systematic reviews: a methodological systematic review. BMC Med Res Methodol. 2019;19:132. https://doi.org/10.1186/s12874-019-07820.CrossRefPubMedPubMedCentral

13.

Evidence Partners. DistillerAI FAQs. 2019. Available at: https://www.evidencepartners.com/distillerai-faqs/. Accessed 11 October 2019.

14.

Shea BJ, Reeves BC, Wells G, Thuku M, Hamel C, Moran J, et al. AMSTAR 2: a critical appraisal tool for systematic reviews that include randomised or non-randomised studies of healthcare interventions, or both. BMJ. 2017;358:j4008. https://doi.org/10.1136/bmj.j4008.CrossRefPubMedPubMedCentral

15.

Wallace BC, Trikalinos TA, Lau J, Brodley C, Schmid CH. Semi-automated screening of biomedical citations for systematic reviews. BMC Bioinformatics. 2010;11(1):55. https://doi.org/10.1186/1471-2105-11-55.CrossRefPubMedPubMedCentral

16.

Tricco AC, Antony J, Zarin W, Strifler L, Ghassemi M, Ivory J, et al. A scoping review of rapid review methods. BMC Med. 2015;13(1):224. https://doi.org/10.1186/s12916-015-0465-6.CrossRefPubMedPubMedCentral

17.

Hartling L, Guise J-M, Hempel S, Featherstone R, Mitchell MD, Motu’apuaka ML, et al. Fit for purpose: perspectives on rapid reviews from end-user interviews. Syst Rev. 2017;6(1):32. https://doi.org/10.1186/s13643-017-0425-7.CrossRefPubMedPubMedCentral

Title: The semi-automation of title and abstract screening: a retrospective exploration of ways to leverage Abstrackr’s relevance predictions in systematic and rapid reviews
Authors: Allison Gates
Michelle Gates
Meghan Sebastianski
Samantha Guitard
Sarah A. Elliott
Lisa Hartling
Publication date: 01-12-2020
Publisher: BioMed Central
Published in: BMC Medical Research Methodology / Issue 1/2020
Electronic ISSN: 1471-2288
DOI: https://doi.org/10.1186/s12874-020-01031-w

Keynote webinar | Spotlight on medication adherence

Springer Medicine

The semi-automation of title and abstract screening: a retrospective exploration of ways to leverage Abstrackr’s relevance predictions in systematic and rapid reviews

Abstract

Background

Methods

Results

Conclusion

Keynote webinar | Spotlight on medication adherence

Springer Medicine

Abstract

Background

Methods

Results

Conclusion

Please log in to get access to this content

Other articles of this Issue 1/2020

Searching for Programme theories for a realist evaluation: a case study comparing an academic database search and a simple Google search

Re-engaging an inactive cohort of young adults: evaluating recruitment for the Kidskin Young Adult Myopia Study

An evaluation of inverse probability weighting using the propensity score for baseline covariate adjustment in smaller population randomised controlled trials with a continuous outcome

Bayesian splines versus fractional polynomials in network meta-analysis

Development of an integrative coding framework for evaluating context within implementation science

Modelling seizure rates rather than time to an event within clinical trials of antiepileptic drugs