Top

Published in:

Open Access 01-12-2019 | Antipsychotics | Research

Performance and usability of machine learning for screening in systematic reviews: a comparative evaluation of three tools

Authors: Allison Gates, Samantha Guitard, Jennifer Pillay, Sarah A. Elliott, Michele P. Dyson, Amanda S. Newton, Lisa Hartling

Published in: Systematic Reviews | Issue 1/2019

Abstract

Background

We explored the performance of three machine learning tools designed to facilitate title and abstract screening in systematic reviews (SRs) when used to (a) eliminate irrelevant records (automated simulation) and (b) complement the work of a single reviewer (semi-automated simulation). We evaluated user experiences for each tool.

Methods

We subjected three SRs to two retrospective screening simulations. In each tool (Abstrackr, DistillerSR, RobotAnalyst), we screened a 200-record training set and downloaded the predicted relevance of the remaining records. We calculated the proportion missed and workload and time savings compared to dual independent screening. To test user experiences, eight research staff tried each tool and completed a survey.

Results

Using Abstrackr, DistillerSR, and RobotAnalyst, respectively, the median (range) proportion missed was 5 (0 to 28) percent, 97 (96 to 100) percent, and 70 (23 to 100) percent for the automated simulation and 1 (0 to 2) percent, 2 (0 to 7) percent, and 2 (0 to 4) percent for the semi-automated simulation. The median (range) workload savings was 90 (82 to 93) percent, 99 (98 to 99) percent, and 85 (85 to 88) percent for the automated simulation and 40 (32 to 43) percent, 49 (48 to 49) percent, and 35 (34 to 38) percent for the semi-automated simulation. The median (range) time savings was 154 (91 to 183), 185 (95 to 201), and 157 (86 to 172) hours for the automated simulation and 61 (42 to 82), 92 (46 to 100), and 64 (37 to 71) hours for the semi-automated simulation. Abstrackr identified 33–90% of records missed by a single reviewer. RobotAnalyst performed less well and DistillerSR provided no relative advantage. User experiences depended on user friendliness, qualities of the user interface, features and functions, trustworthiness, ease and speed of obtaining predictions, and practicality of the export file(s).

Conclusions

The workload savings afforded in the automated simulation came with increased risk of missing relevant records. Supplementing a single reviewer’s decisions with relevance predictions (semi-automated simulation) sometimes reduced the proportion missed, but performance varied by tool and SR. Designing tools based on reviewers’ self-identified preferences may improve their compatibility with present workflows.

Systematic review registration

Not applicable.

Available only for authorised users

Borah R, Brown AW, Capers PL, Kaiser KA. Analysis of the time and workers needed to conduct systematic reviews of medical interventions using data from the PROSPERO registry. BMJ Open. 2017;7:e012545. https://doi.org/10.1136/bmjopen-2016-012545.CrossRefPubMedPubMedCentral

Thomas J, McNaught J, Ananiadou S. Applications of text mining within systematic reviews. Res Synth Methods. 2011;2:1–14. https://doi.org/10.1002/jrsm.27.

Tsafnat G, Glasziou P, Choong MK, Dunn A, Galgani F, Coiera E. Systematic review automation technologies. Syst Rev. 2014;3:74. https://doi.org/10.1186/2046-4053-3-74.CrossRefPubMedPubMedCentral

Beller E, Clark J, Tsafnat G, Adams C, Diehl H, Lund H, et al. Making progress with the automation of systematic reviews: principles of the International Collaboration for the Automation of Systematic Reviews (ICASR). Syst Rev. 2018;7:77. https://doi.org/10.1186/s13643-018-0740-7.CrossRefPubMedPubMedCentral

Marshall IJ, Wallace BC. Toward systematic review automation: a practical guide to using ML tools in research synthesis. Syst Rev. 2019;8:163. https://doi.org/10.1186/s13643-019-1074-9.CrossRefPubMedPubMedCentral

O’Mara-Eves A, Thomas J, McNaught J, Miwa M, Ananiadou S. Using text mining for study identification in systematic reviews: a systematic review of current approaches. Syst Rev. 2015;4:5. https://doi.org/10.1186/2046-4053-4-5.CrossRefPubMedPubMedCentral

O'Connor AM, Tsafnat G, Gilbert SB, Thayer KA, Wolfe MS. Moving toward the automation of the systematic review process: a summary of discussions at the second meeting of International Collaboration for the Automation of Systematic Reviews (ICASR). Syst Rev. 2018;7:3. https://doi.org/10.1186/s13643-017-0667-4.CrossRefPubMedPubMedCentral

Thomas J. Diffusion of innovation in systematic review methodology: why is study selection not yet assisted by automation? OA Evidence-Based Med. 2013;1:12.CrossRef

O’Connor AM, Tsafnat G, Thomas J, Glasziou P, Gilbert SB, Hutton B. A question of trust: can we build an evidence base to gain trust in systematic review automation technologies? Syst Rev. 2019;8:143. https://doi.org/10.1186/s13643-019-1062-0.CrossRefPubMedPubMedCentral

10.

van Altena AJ, Spijker R, Olabarriaga SD. Usage of automation tools in systematic reviews. Res Synth Methods. 2019;10:72–82. https://doi.org/10.1002/jrsm.1335.CrossRefPubMed

11.

Paynter R, Bañez LL, Berliner E, Erinoff E, Lege-Matsuura J, Potter S, et al. EPC methods: an exploration of the use of text-mining software in systematic reviews. Report no.: 16-EHC023-EF. Rockville: Agency for Healthcare Research and Quality; 2016.

12.

O'Connor AM, Tsafnat G, Gilbert SB, Thayer KA, Shemilt I, Thomas J, et al. Still moving toward automation of the systematic review process: a summary of discussions at the third meeting of the International Collaboration for Automation of Systematic Reviews (ICASR). Syst Rev. 2019;8:57. https://doi.org/10.1186/s13643-019-0975-y.CrossRefPubMedPubMedCentral

13.

Marshall C. Systematic review toolbox. 2019. http://systematicreviewtools.com/index.php. Accessed 5 Apr 2019.

14.

Wallace BC, Small K, Brodley CE, Lau J, Trikalinos TA. Deploying An Interactive ML System In An Evidence-Based Practice Center: Abstrackr. Proceedings of the 2nd ACM SIGHIT International Health Informatics Symposium; 2012 Jan 28-30; New York, New York: Association for Computing Machinery; 2012. doi:https://doi.org/10.1145/2110363.2110464.

15.

The National Centre for Text Mining. RobotAnalyst. 2019. http://www.nactem.ac.uk/robotanalyst/. Accessed 5 Apr 2019.

16.

Evidence Partners. Publications. 2019. https://www.evidencepartners.com/about/publications/. Accessed 5 Apr 2019.

17.

Przybyła P, Brockmeier AJ, Kontonatsios G, Le Pogam MA, McNaught J, von Elm E, et al. Prioritising references for systematic reviews with RobotAnalyst: a user study. Res Synth Methods. 2018;9:470–88. https://doi.org/10.1002/jrsm.1311.CrossRefPubMedPubMedCentral

18.

Gates A, Johnson C, Hartling L. Technology-assisted title and abstract screening for systematic reviews: a retrospective evaluation of the Abstrackr machine learning tool. Syst Rev. 2018;7:45. https://doi.org/10.1186/s13643-0.18-0707-8.

19.

Rathbone J, Hoffmann T, Glasziou P. Faster title and abstract screening? Evaluating Abstrackr, a semi-automated online screening program for systematic reviewers. Syst Rev. 2015;4:80 doi:26073974.CrossRef

20.

Pillay J, Boylan K, Carrey N, Newton A, Vandermeer B, Nuspl M, et al. First-and second-generation antipsychotics in children and young adults: systematic review update. Report no.: 17-EHC001-EF. Rockville: Agency for Healthcare Research and Quality; 2017.CrossRef

21.

Pillay J, Freeman EE, Hodge W, MacGregor T, Featherstone R, Vandermeer B, et al. Screening for impaired visual acuity and vision-related functional limitations in adults 65 years and older in primary health care: systematic review. Edmonton: Evidence Review Synthesis Centre; 2017. http://canadiantaskforce.ca/ctfphc-guidelines/overview/.

22.

Evidence Partners. DistillerAI FAQs. 2019. https://www.evidencepartners.com/resources/distillerai-faqs/. Accessed 5 Apr 2019.

23.

Waffenschmidt S, Knelangen M, Sieben W, Bühn S, Pieper D. Single screening versus conventional double screening for study selection in systematic reviews: a methodological systematic review. BMC Med Res Methodol. 2019;19:132. https://doi.org/10.1186/s12874-019-0782-0.CrossRefPubMedPubMedCentral

24.

Harris PA, Taylor R, Thielke R, Payne J, Gonzalez N, Conde JG. Research electronic data capture (REDCap)—a metadata-driven methodology and workflow process for providing translational research informatics support. J Biomed Inform. 2009;42:377–81. https://doi.org/10.1016/j.jbi.2008.08.010.CrossRef

25.

Brooke J. SUS-a quick and dirty usability scale. Usability Eval Industry. 1996;189:4–7.

26.

Wallace BC, Trikalinos TA, Lau J, Brodley C, Schmid CH. Semi-automated screening of biomedical citations for systematic reviews. BMC Bioinformatics. 2010;11:55. https://doi.org/10.1186/1471-2105-11-55.CrossRefPubMedPubMedCentral

27.

Bangor A, Kortum PT, Miller JT. An empirical evaluation of the system usability scale. Int J Hum Comput Interact. 2008;24:574–94. https://doi.org/10.1080/10447310802205776.CrossRef

28.

Vaismoradi M, Turunen H, Bondas T. Content analysis and thematic analysis: implications for conducting a qualitative descriptive study. Nurs Health Sci. 2013;15:398–405. https://doi.org/10.1111/nhs.12048.CrossRefPubMed

29.

Wallace BC, Small K, Brodley CE, Lau J, Trikalinos TA. Modeling annotation time to reduce workload in comparative effectiveness reviews. Proceedings of the 1st ACM International Health Informatics Symposium; 2010 Nov 11–12. New York, New York: Association for Computing Machinery; 2010. doi:1−/1145/1882992.1882999.

30.

Chen Y. Developing stopping rules for a ML system in citation screening [dissertation]. Providence: Brown University; 2019.

31.

Olorisade BK, Brereton P, Andras P. Reproducibility of studies on text mining for citation screening in systematic reviews: evaluation and checklist. J Biomed Inform. 2017;73:1–13. https://doi.org/10.1016/j.jbi.2017.07.010.CrossRefPubMed

32.

Olorisade BK, Quincey E, Brereton P, Andras P. A critical analysis of studies that address the use of text mining for citation screening in systematic reviews. Proceedings of the 20th International Conference on Evaluation and Assessment in Software Engineering; 2016 Jun 1–3. Limerick: Association for Computing Machinery; 2016. https://doi.org/10.1145/2915970.2915982.CrossRef

Title: Performance and usability of machine learning for screening in systematic reviews: a comparative evaluation of three tools
Authors: Allison Gates
Samantha Guitard
Jennifer Pillay
Sarah A. Elliott
Michele P. Dyson
Amanda S. Newton
Lisa Hartling
Publication date: 01-12-2019
Publisher: BioMed Central
Keywords: Antipsychotics
Antipsychotics
Bronchiolitis
Published in: Systematic Reviews / Issue 1/2019
Electronic ISSN: 2046-4053
DOI: https://doi.org/10.1186/s13643-019-1222-2

At a glance: The ONWARDS insulin icodec trials

Springer Medicine

Performance and usability of machine learning for screening in systematic reviews: a comparative evaluation of three tools

Abstract

Background

Methods

Results

Conclusions

Systematic review registration

At a glance: The ONWARDS insulin icodec trials

Springer Medicine

Abstract

Background

Methods

Results

Conclusions

Systematic review registration

Please log in to get access to this content

Other articles of this Issue 1/2019

Predictors of aging out of heavy episodic drinking in emerging adults: a systematic review protocol

Does routine surveillance imaging after completing treatment for childhood solid tumours cause more harm than good? A systematic review and meta-analysis protocol

Sonographic reference values of median nerve cross-sectional area: a protocol for a systematic review and meta-analysis

Correction to: Information needs in people with diabetes mellitus: a systematic review

Computer-based cognitive interventions for mild cognitive impairment and dementia in older adults: protocol for a systematic review of published studies and meta-analysis

Toward systematic review automation: a practical guide to using machine learning tools in research synthesis