Top

Systematic Reviews

Published in:

Open Access 01-12-2021 | Methodology

Research Screener: a machine learning tool to semi-automate abstract screening for systematic reviews

Authors: Kevin E. K. Chai, Robin L. J. Lines, Daniel F. Gucciardi, Leo Ng

Published in: Systematic Reviews | Issue 1/2021

Abstract

Background

Systematic reviews and meta-analyses provide the highest level of evidence to help inform policy and practice, yet their rigorous nature is associated with significant time and economic demands. The screening of titles and abstracts is the most time consuming part of the review process with analysts required review thousands of articles manually, taking on average 33 days. New technologies aimed at streamlining the screening process have provided initial promising findings, yet there are limitations with current approaches and barriers to the widespread use of these tools. In this paper, we introduce and report initial evidence on the utility of Research Screener, a semi-automated machine learning tool to facilitate abstract screening.

Methods

Three sets of analyses (simulation, interactive and sensitivity) were conducted to provide evidence of the utility of the tool through both simulated and real-world examples.

Results

Research Screener delivered a workload saving of between 60 and 96% across nine systematic reviews and two scoping reviews. Findings from the real-world interactive analysis demonstrated a time saving of 12.53 days compared to the manual screening, which equates to a financial saving of USD 2444. Conservatively, our results suggest that analysts who scan 50% of the total pool of articles identified via a systematic search are highly likely to have identified 100% of eligible papers.

Conclusions

In light of these findings, Research Screener is able to reduce the burden for researchers wishing to conduct a comprehensive systematic review without reducing the scientific rigour for which they strive to achieve.

Available only for authorised users

Allen IE, Olkin I. Estimating time to conduct a meta-analysis from number of citations retrieved. JAMA. 1999;282(7):634–5. https://doi.org/10.1001/jama.282.7.634.CrossRefPubMed

Athanasiadou R, Bankston A, Carlisle M, Niziolek CA, McDowell GS. Assessing the landscape of US postdoctoral salaries. Stud Grad Postdoctoral Educ. 2018;19(2):213–42. https://doi.org/10.1108/SGPE-D-17-00048.CrossRef

Bannach-Brown A, Przybyła P, Thomas J, Rice AS, Ananiadou S, Liao J, Macleod MR. Machine learning algorithms for systematic review: reducing workload in a preclinical review of animal studies and reducing human screening error. Syst Rev. 2019;8(1):1–2. https://doi.org/10.1186/s13643-019-0942-7.CrossRef

Bastian H, Glasziou P, Chalmers I. Seventy-five trials and eleven systematic reviews a day: how will we ever keep up? Plos Med. 2010;7(9):e1000326. https://doi.org/10.1371/journal.pmed.1000326.CrossRefPubMedPubMedCentral

Borah R, Brown AW, Capers PL, Kaiser KA. Analysis of the time and workers needed to conduct systematic reviews of medical interventions using data from the PROSPERO registry. BMJ Open. 2017;7(2):e012545. https://doi.org/10.1136/bmjopen-2016-012545.CrossRefPubMedPubMedCentral

Bramer WM, Giustini D, de Jonge GB, Holland L, Bekhuis T. De-duplication of database search results for systematic reviews in EndNote. J Med Libr Assoc. 2016;104(3):240–3. https://doi.org/10.3163/1536-5050.104.3.014.CrossRefPubMedPubMedCentral

Bramer WM, Milic J, Mast F. Reviewing retrieved references for inclusion in systematic reviews using EndNote. J Med Libr Assoc. 2017;105:84–7. https://doi.org/10.5195/jmla.2017.111.CrossRefPubMedPubMedCentral

Burton E, Farrier K, Lewin G, Petrich M, Boyle E, Hill KD. Are interventions effective in improving the ability of older adults to rise from the floor independently? A mixed method systematic review. Disabil Rehabil. 2020;42(6):743–53. https://doi.org/10.1080/09638288.2018.1508509.CrossRefPubMed

Cavalheri V, Granger C. Preoperative exercise training for patients with non-small cell lung cancer. Cochrane Database Syst Rev. 2017;6. https://doi.org/10.1002/14651858.CD012020.pub2.

10.

Cavalheri V, Burtin C, Formico VR, Nonoyama ML, Jenkins S, Spruit MA, Hill K. Exercise training undertaken by people within 12 months of lung resection for non-small cell lung cancer. Cochrane Database Syst Rev. 2019;6. https://doi.org/10.1002/14651858.CD009955.pub3.

11.

Cohen AM, Hersh WR, Peterson K, Yen PY. Reducing workload in systematic review preparation using automated citation classification. J Am Med Informatics Assoc. 2006;13:206–19. https://doi.org/10.1197/jamia.M1929.CrossRef

12.

Devlin, J., Chang, M.W., Lee, K., Toutanova, K. BERT: Pre-training Of deep bidirectional transformers for language understanding. In Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies. 2019;1:4171–86.

13.

Gates A, Guitard S, Pillay J, Elliott SA, Dyson MP, Newton AS, Hartling L. Performance and usability of machine learning for screening in systematic reviews: a comparative evaluation of three tools. Syst Rev. 2019;8(1):278. https://doi.org/10.1186/s13643-019-1222-2.CrossRefPubMedPubMedCentral

14.

Gates A, Johnson C, Hartling L. Technology-assisted title and abstract screening for systematic reviews: a retrospective evaluation of the Abstrackr machine learning tool. Syst Rev. 2018;7:45. https://doi.org/10.1186/s13643-018-0707-8.CrossRefPubMedPubMedCentral

15.

Haddaway NR, Westgate MJ. Predicting the time needed for environmental systematic reviews and systematic maps. Conserv Biol. 2019;33(2):434–43. https://doi.org/10.1111/cobi.13231.CrossRefPubMed

16.

Harrison H, Griffin SJ, Kuhn I, Usher-Smith JA. Software tools to support title and abstract screening for systematic reviews in healthcare: an evaluation. BMC Med Res Methodol. 2020;20:7. https://doi.org/10.1186/s12874-020-0897-3.CrossRefPubMedPubMedCentral

17.

Kohl C, McIntosh EJ, Unger S, Haddaway NR, Kecke S, Schiemann J, Wilhelm R. Online tools supporting the conduct and reporting of systematic reviews and systematic maps: a case study on CADIMA and review of existing tools. Environ Evid. 2018;7(1):8. https://doi.org/10.1186/s13750-018-0115-5.CrossRef

18.

Lau J. Systematic review automation thematic series. Syst Rev. 2019;8(1):70. https://doi.org/10.1186/s13643-019-0974-z.CrossRefPubMedPubMedCentral

19.

Le Q, Mikolov T. Distributed representations of sentences and documents. In: International Conference on Machine Learning; 2014. p. 1188–96.

20.

Lorenzetti DL, Ghali WA. Reference management software for systematic reviews and meta-analyses: an exploration of usage and usability. BMC Med Res Methodol. 2013;13(1):141. http://www.biomedcentral.com/1471-2288/13/141. https://doi.org/10.1186/1471-2288-13-141.CrossRefPubMedPubMedCentral

21.

Marshall C, Brereton P. Systematic review toolbox: a catalogue of tools to support systematic reviews. In: Proceedings of the 19th International Conference on Evaluation and Assessment in Software Engineering; 2015. p. 1–6. https://doi.org/10.1145/2745802.2745824.CrossRef

22.

Michelson M, Reuter K. The significant cost of systematic reviews and meta-analyses: a call for greater involvement of machine learning to assess the promise of clinical trials. Contemp Clin Trials Commun. 2019;16:100443. https://doi.org/10.1016/j.conctc.2019.100443.CrossRefPubMedPubMedCentral

23.

Mikolov, T., Sutskever, I., Chen, K., Corrado, G. & Dean, J. Distributed representations of words and phrases and their compositionality. Advances in Neural Information Processing Systems. 2013;26:3111–9.

24.

Miwa M, Thomas J, O’Mara-Eves A, Ananiadou S. Reducing systematic review workload through certainty-based screening. J Biomed Informatics. 2014;51:242–53. https://doi.org/10.1016/j.jbi.2014.06.005.CrossRef

25.

Moher D, Shamseer L, Clarke M, Ghersi D, Liberati A, Petticrew M, Shekelle P, Stewart LA. Preferred reporting items for systematic review and meta-analysis protocols (PRISMA-P) 2015 statement. Syst Rev. 2015;4(1):1. https://doi.org/10.1186/2046-4053-4-1.CrossRefPubMedPubMedCentral

26.

Munn Z, Aromataris E, Tufanaru C, Stern C, Porritt K, Farrow J, Lockwood C, Stephenson M, Moola S, Lizarondo L, McArthur A. The development of software to support multiple systematic review types: the Joanna Briggs Institute System for the Unified Management, Assessment and Review of Information (JBI SUMARI). Int J Evid Based Healthcare. 2019;17(1):36–43. https://doi.org/10.1097/XEB.0000000000000152.CrossRef

27.

O’Connor AM, Tsafnat G, Gilbert SB, Thayer KA, Wolfe MS. Moving toward the automation of the systematic review process: a summary of discussions at the second meeting of International Collaboration for the Automation of Systematic Reviews (ICASR). Syst Rev. 2018;7:3.CrossRef

28.

O’Connor AM, Tsafnat G, Thomas J, Glasziou P, Gilbert SB, Hutton B. A question of trust: can we build an evidence base to gain trust in systematic review automation technologies? Systematic Reviews. 2019;8(1):143. https://doi.org/10.1186/s13643-019-1062-0.CrossRefPubMedPubMedCentral

29.

Olofsson H, Brolund A, Hellberg C, Silverstein R, Stenström K, Österberg M, Dagerhamn J. Can abstract screening workload be reduced using text mining? User experiences of the tool Rayyan. Res Synth Methods. 2017;8(3):275–80. https://doi.org/10.1002/jrsm.1237.

30.

O’Mara-Eves A, Thomas J, McNaught J, Miwa M, Ananiadou S. Using text mining for study identification in systematic reviews: a systematic review of current approaches. Syst Rev. 2015;4:5. http://www.systematicreviewsjournal.com/content/4/1/5. https://doi.org/10.1186/2046-4053-4-5.CrossRefPubMedPubMedCentral

31.

Ouzzani M, Hammady H, Fedorowicz Z, Elmagarmid A. Rayyan—a web and mobile app for systematic reviews. Systematic Reviews. 2016;5(1):210. https://doi.org/10.1186/s13643-016-0384-4.CrossRefPubMedPubMedCentral

32.

Page MJ, Shamseer L, Tricco AC. Registration of systematic reviews in PROSPERO: 30,000 records and counting. Systematic Reviews. 2018;7(1):32. https://doi.org/10.1186/s13643-018-0699-4.CrossRefPubMedPubMedCentral

33.

Pennington J, Socher R, Manning CD. Glove: global vectors for word representation. In: Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP); 2014. p. 1532–43.CrossRef

34.

Polanin JR, Pigott TD, Espelage DL, Grotpeter JK. Best practice guidelines for abstract screening large-evidence systematic reviews and meta-analyses. Rese Synth Methods. 2019;10(3):330–42. https://doi.org/10.1002/jrsm.1354.CrossRef

35.

Przybyła P, Brockmeier AJ, Kontonatsios G, Le Pogam MA, McNaught J, von Elm E, Nolan K, Ananiadou S. Prioritising references for systematic reviews with RobotAnalyst: a user study. Res Synth Methods. 2018;9(3):470–88. https://doi.org/10.1002/jrsm.1311.CrossRefPubMedPubMedCentral

36.

Rathbone J, Hoffmann T, Glasziou P. Faster title and abstract screening? Evaluating Abstrackr, a semi-automated online screening program for systematic reviewers. Syst Rev. 2015;4:80. https://doi.org/10.1186/s13643-015-0067-6.CrossRefPubMedPubMedCentral

37.

Saraceni N, Kent P, Ng L, Campbell A, Straker L, O'Sullivan P. To flex or not to flex? is there a relationship between lumbar spine flexion during lifting and low back pain? A systematic review with meta-analysis. J Orthop Sports Phys Ther. 2020;50(3):121–30. https://www.jospt.org/doi/10.2519/jospt.2020.9218.CrossRef

38.

Shemilt I, Khan N, Park S, Thomas J. Use of cost-effectiveness analysis to compare the efficiency of study identification methods in systematic reviews. Res Synth Methods. 2016;5(1):31–49. https://doi.org/10.1002/jrsm.1093.CrossRef

39.

Tsafnat G, Glasziou P, Choong MK, Dunn A, Galgani F, Coiera E. Systematic review automation technologies. Systematic Reviews. 2014;3:74. http://www.systematicreviewsjournal.com/content/3/1/74. https://doi.org/10.1186/2046-4053-3-74.CrossRefPubMedPubMedCentral

40.

Tsafnat G, Glasziou P, Karystianis G, Coiera E. Automated screening of research studies for systematic reviews using study characteristics. Syst Rev. 2018;7(1):64. https://doi.org/10.1186/s13643-018-0724-7.CrossRefPubMedPubMedCentral

41.

Vaswani A, Shazeer N, Parmar N, Uszkoreit J, Jones L, Gomez AN, Kaiser Ł, Polosukhin I. Attention is all you need. Adv Neural Inf Process Syst. 2017;30:5998–6008. Accessed 8 Jul 2020.

42.

Wallace BC, Small K, Brodley CE, Lau J, Schmid CH, Bertram L, Lill CM, Cohen JT, Trikalinos TA. Toward modernizing the systematic review pipeline in genetics: efficient updating via data mining. Genet Med. 2012a;14(7):663–9. https://doi.org/10.1038/gim.2012.7.CrossRefPubMedPubMedCentral

43.

Wallace BC, Trikalinos TA, Lau J, Brodley C, Schmid CH. Semi-automated screening of biomedical citations for systematic reviews. BMC Bioinformatics. 2010;11(1):55. https://doi.org/10.1186/1471-2105-11-55.CrossRefPubMedPubMedCentral

44.

Wallace BC, Small K, Brodley CE, Lau J, Trikalinos TA. Deploying an interactive machine learning system in an evidence-based practice center: abstrackr. In: Proceedings of the 2nd ACM SIGHIT International Health Informatics Symposium; 2012b. p. 819–24.CrossRef

45.

Zabatiero J, Ng L, Clayton R, Middlemiss S, Kang K, Harrold M, Cavalheri V. Effectiveness of interventions aiming at reducing sedentary behaviour in a non-surgical population with overweight or obesity: a systematic review and meta-analysis. Obes Res Clin Pract. 2019;13(2):115–28. https://doi.org/10.1016/j.orcp.2018.10.004.CrossRefPubMed

46.

Gray C, Crawford G, Maycock B, Lobo R. Socioecological Factors Influencing Sexual Health Experiences and Health Outcomes of Migrant Asian Women Living in ‘Western’ High-Income Countries: A Systematic Review. International Journal of Environmental Research and Public Health. 2021;18(5):2469. https://doi.org/10.3390/ijerph18052469

47.

Robin L. J. Lines, Simon Pietsch, Monique Crane, Nikos Ntoumanis, Philip Temby, Sally Graham, Daniel F. Gucciardi, (2021) The effectiveness of team reflexivity interventions: A systematic review and meta-analysis of randomized controlled trials.. Sport, Exercise, and Performance Psychology

48.

Iain J. Marshall, Byron C. Wallace, (2019) Toward systematic review automation: a practical guide to using machine learning tools in research synthesis. Systematic Reviews 8 (1)

49.

Rens van de Schoot, Jonathan de Bruin, Raoul Schram, Parisa Zahedi, Jan de Boer, Felix Weijdema, Bianca Kramer, Martijn Huijts, Maarten Hoogerwerf, Gerbrich Ferdinands, Albert Harkema, Joukje Willemsen, Yongchao Ma, Qixiang Fang, Sybren Hindriks, Lars Tummers, Daniel L. Oberski,. An open source machine learning framework for efficient and transparent systematic reviews. Nature Machine Intelligence 3. 2021;(2):125–33

Title: Research Screener: a machine learning tool to semi-automate abstract screening for systematic reviews
Authors: Kevin E. K. Chai
Robin L. J. Lines
Daniel F. Gucciardi
Leo Ng
Publication date: 01-12-2021
Publisher: BioMed Central
Published in: Systematic Reviews / Issue 1/2021
Electronic ISSN: 2046-4053
DOI: https://doi.org/10.1186/s13643-021-01635-3

At a glance: The STEP trials

Springer Medicine

Research Screener: a machine learning tool to semi-automate abstract screening for systematic reviews

Abstract

Background

Methods

Results

Conclusions

At a glance: The STEP trials

Springer Medicine

Abstract

Background

Methods

Results

Conclusions

Please log in to get access to this content

Other articles of this Issue 1/2021

McDonald versus Shirodkar cerclage technique in women requiring a prophylactic cerclage: a systematic review and meta-analysis protocol

Communication strategies in the prevention of type 2 diabetes and gestational diabetes in vulnerable groups: a scoping review

Tricyclic antidepressants versus ‘active placebo’, placebo or no intervention for adults with major depressive disorder: a protocol for a systematic review with meta-analysis and Trial Sequential Analysis

Mesenchymal stromal cells as a therapeutic intervention for COVID-19: a living systematic review and meta-analysis protocol

Team functioning and implementation of innovations in healthcare and human service settings: a systematic review protocol

Comparison of COVID-19 outcomes in organ transplant recipients (OTr) and non-transplant patients: a study protocol for rapid review