Top

Published in:

Open Access 01-12-2021 | Research

LOCATE: a prospective evaluation of the value of Leveraging Ongoing Citation Acquisition Techniques for living Evidence syntheses

Authors: Michelle Gates, Sarah A. Elliott, Allison Gates, Meghan Sebastianski, Jennifer Pillay, Liza Bialy, Lisa Hartling

Published in: Systematic Reviews | Issue 1/2021

Abstract

Background

Living systematic reviews (LSRs) can expedite evidence synthesis by incorporating new evidence in real time. However, the methods needed to identify new studies in a timely manner are not well established.

Objectives

To explore the value of complementary search approaches in terms of search performance, impact on results and conclusions, screening workload, and feasibility compared to the reference standard.

Methods

We developed three complementary search approaches for a systematic review on treatments for bronchiolitis: Automated Full Search, PubMed Similar Articles, and Scopus Citing References. These were automated to retrieve results monthly; pairs of reviewers screened the records and commented on feasibility. After 1 year, we conducted a full update search (reference standard). For each complementary approach, we compared search performance (proportion missed, number needed to read [NNR]) and reviewer workload (number of records screened, time required) to the reference standard. We investigated the impact of the new trials on the effect estimate and certainty of evidence for the primary outcomes. We summarized comments about feasibility.

Results

Via the reference standard, reviewers screened 505 titles/abstracts, 24 full texts, and identified four new trials (NNR 127; 12.4 h). Of the complementary approaches, only the Automated Full Search located all four trials; these were located 6 to 12 months sooner than via the reference standard but did not alter the results nor certainty in the evidence. The Automated Full Search was the most resource-intensive approach (816 records screened; NNR 204; 17.1 h). The PubMed Similar Articles and Scopus Citing References approaches located far fewer records (452 and 244, respectively), thereby requiring less screening time (9.4 and 5.2 h); however, each approach located only one of the four new trials. Reviewers found it feasible and convenient to conduct monthly screening for searches of this yield (median 15–65 records/month).

Conclusions

The Automated Full Search was the most resource-intensive approach, but also the only to locate all of the newly published trials. Although the monthly screening time for the PubMed Similar Articles and Scopus Citing Articles was far less, most relevant records were missed. These approaches were feasible to integrate into reviewer work processes.

Systematic review registration

Open Science Framework. https://doi.org/10.17605/OSF.IO/6M28H.

Available only for authorised users

Garritty C, Stevens A, Hamel C, Golfam M, Hutton B, Wolfe D. Knowledge synthesis in evidence-based medicine. Semin Nucl Med. 2019;49(2):136–44. https://doi.org/10.1053/j.semnuclmed.2018.11.006.

Egger M, Smith GD, O'Rourke K. Introduction: rationale, potentials, and promise of systematic reviews. Systematic reviews in health care: meta-analysis in context. London: 2nd, Wiley; 2008. p. 1–19.

Borah R, Brown AW, Capers PL, Kaiser KA. Analysis of the time and workers needed to conduct systematic reviews of medical interventions using data from the PROSPERO registry. BMJ Open. 2017;7(2):e012545. https://doi.org/10.1136/bmjopen-2016-012545.

Tricco AC, Brehaut J, Chen MH, Moher D. Following 411 Cochrane protocols to completion: a retrospective cohort study. PLoS One. 2008;3(11):e3684. https://doi.org/10.1371/journal.pone.0003684.CrossRefPubMedPubMedCentral

Bragge P, Clavisi O, Turner T, Tavender E, Collie A, Gruen RL. The global evidence mapping initiative: scoping research in broad topic areas. BMC Med Res Methodol. 2011;11(1):92. https://doi.org/10.1186/1471-2288-11-92.CrossRefPubMedPubMedCentral

Créquit P, Trinquart L, Yavchitz A, Ravaud P. Wasted research when systematic reviews fail to provide a complete and up-to-date evidence synthesis: the example of lung cancer. BMC Med. 2016;14(1):8. https://doi.org/10.1186/s12916-016-0555-0.CrossRefPubMedPubMedCentral

Shojania K, Sampson M, Ansari M, Ji J, Doucette S, Moher D. How quickly do systematic reviews go out of date? A survival analysis. Ann Intern Med. 2007;147(4):224–33. https://doi.org/10.7326/0003-4819-147-4-200708210-00179.CrossRefPubMed

Bero L, Busuttil G, Farquhar C, Pérez Koehlmoos, Moher D, Nylenna M, et al. Measuring the performance of The Cochrane Library [editorial]. Cochrane Syst Rev. 2012;12:ED00048.

Tsertsvadze A, Chen Y-F, Moher D, Sutcliffe P, McCarthy N. How to conduct systematic reviews more expeditiously? Syst Rev. 2015;4(1):160. https://doi.org/10.1186/s13643-015-0147-7.CrossRefPubMedPubMedCentral

10.

Elliott JH, Turner T, Clavisi O, Thomas J, Higgins JPT, Mavergames C, et al. Living systematic reviews: an emerging opportunity to narrow the evidence-practice gap. PLoS Med. 2014;11(2):e1001603-e.

11.

Elliott JH, Synnot A, Turner T, Simmonds M, Akl EA, McDonald S, et al. Living systematic review: 1. Introduction—the why, what, when, and how. J Clin Epidemiol. 2017;91:23–30. https://doi.org/10.1016/j.jclinepi.2017.08.010.CrossRefPubMed

12.

Brooker J, Synnot A, McDonald S, Elliott J, Turner T, Hodder R, et al. Guidance for the production and publication of Cochrane living systematic reviews: Cochrane Reviews in living mode (Version 1.0). Cochrane: London; 2019.

13.

Synnot A, Turner T, Elliott J. Cochrane living systematic reviews: interim guidance for pilots (version 0.3). Cochrane: London; 2017.

14.

Thomas J, Noel-Storr A, Marshall I, Wallace B, McDonald S, Mavergames C, et al. Living systematic reviews: 2. Combining human and machine effort. J Clin Epidemiol. 2017;91:31–7. https://doi.org/10.1016/j.jclinepi.2017.08.011.CrossRefPubMed

15.

Rice M, Ali MU, Fitzpatrick-Lewis D, Kenny M, Raina P, Sherifali D. Testing the effectiveness of simplified search strategies for updating systematic reviews. J Clin Epidemiol. 2017;88:148–53. https://doi.org/10.1016/j.jclinepi.2017.06.005.CrossRefPubMed

16.

Bialy L, Foisy M, Smith M, Fernandes RM. The Cochrane Library and the treatment of bronchiolitis in children: an overview of reviews. Evid-Based Child Health. 2011;6(1):258–75. https://doi.org/10.1002/ebch.673.CrossRef

17.

Kirolos A, Manti S, Blacow R, Tse G, Wilson T, Lister M, et al. A systematic review of clinical practice guidelines for the diagnosis and management of bronchiolitis. J Infect Dis. 2020;222(Supplement 7):S672–S9.CrossRefPubMed

18.

Korppi M, Mecklin M, Heikkilä P. Review shows substantial variations in the use of medication for infant bronchiolitis between and within countries. Acta Paediatr. 2019;108(6):1016–22. https://doi.org/10.1111/apa.14713.CrossRefPubMed

19.

McGowan J, Sampson M, Salzwedel DM, Cogo E, Foerster V, Lefebvre C. PRESS peer review of electronic search strategies: 2015 guideline statement. J Clin Epidemiol. 2016;75:40–6. https://doi.org/10.1016/j.jclinepi.2016.01.021.CrossRefPubMed

20.

Lin J, Wilbur WJ. PubMed related articles: a probabilistic topic-based model for content similarity. BMC Bioinformatics. 2007;8(1):423. https://doi.org/10.1186/1471-2105-8-423.CrossRefPubMedPubMedCentral

21.

Cooper C, Varley-Campbell J, Booth A, Britten N, Garside R. Systematic review identifies six metrics and one method for assessing literature search effectiveness but no consensus on appropriate use. J Clin Epidemiol. 2018;99:53–63. https://doi.org/10.1016/j.jclinepi.2018.02.025.CrossRefPubMed

22.

Higgins JPT, Altman DG, Gøtzsche PC, Jüni P, Moher D, Oxman AD, et al. The Cochrane Collaboration’s tool for assessing risk of bias in randomised trials. BMJ. 2011;343(oct18 2):d5928. https://doi.org/10.1136/bmj.d5928.CrossRefPubMedPubMedCentral

23.

DerSimonian R, Laird N. Meta-analysis in clinical trials. Control Clin Trials. 1986;7(3):177–88. https://doi.org/10.1016/0197-2456(86)90046-2.CrossRefPubMed

24.

Guyatt GH, Oxman AD, Vist GE, Kunz R, Falck-Ytter Y, Alonso-Coello P, et al. GRADE: an emerging consensus on rating quality of evidence and strength of recommendations. BMJ. 2008;336(7650):924–6. https://doi.org/10.1136/bmj.39489.470347.AD.CrossRefPubMedPubMedCentral

25.

Chen DY, Zee ED, Gildengorin G, Fong EW. A pilot study of heated and humidified low flow oxygen therapy: an assessment in infants with mild and moderate bronchiolitis (HHOT AIR study). Pediatr Pulmonol. 2019;54(5):620–7. https://doi.org/10.1002/ppul.24267.CrossRefPubMed

26.

Ergul AB, Caliskan E, Samsa H, Gokcek I, Kaya A, Zararsiz GE, et al. Using a high-flow nasal cannula provides superior results to OxyMask delivery in moderate to severe bronchiolitis: a randomized controlled study. Eur J Pediatr. 2018;177(8):1299–307. https://doi.org/10.1007/s00431-018-3191-1.CrossRefPubMed

27.

Morikawa Y, Miura M, Furuhata MY, Morino S, Omori T, Otsuka M, et al. Nebulized hypertonic saline in infants hospitalized with moderately severe bronchiolitis due to RSV infection: a multicenter randomized controlled trial. Pediatr Pulmonol. 2018;53(3):358–65. https://doi.org/10.1002/ppul.23945.CrossRefPubMed

28.

Seliem W, Sultan AM. Does heliox administered by low-flow nasal cannula improve respiratory distress in infants with respiratory syncytial virus acute bronchiolitis? A randomised controlled trial. An Pediatr (Barc, English Edition). 2019;90(1):3–9.

29.

Simmonds M, Salanti G, McKenzie J, Elliott J, Agoritsas T, Hilton J, et al. Living systematic reviews: 3. Statistical methods for updating meta-analyses. J Clin Epidemiol. 2017;91:38–46. https://doi.org/10.1016/j.jclinepi.2017.08.008.CrossRefPubMed

30.

O’Connor AM, Tsafnat G, Thomas J, Glasziou P, Gilbert SB, Hutton B. A question of trust: can we build an evidence base to gain trust in systematic review automation technologies? Syst Rev. 2019;8(1):143. https://doi.org/10.1186/s13643-019-1062-0.CrossRefPubMedPubMedCentral

31.

Millard T, Synnot A, Elliott J, Green S, McDonald S, Turner T. Feasibility and acceptability of living systematic reviews: results from a mixed-methods evaluation. Syst Rev. 2019;8(1):325. https://doi.org/10.1186/s13643-019-1248-5.CrossRefPubMedPubMedCentral

Title: LOCATE: a prospective evaluation of the value of Leveraging Ongoing Citation Acquisition Techniques for living Evidence syntheses
Authors: Michelle Gates
Sarah A. Elliott
Allison Gates
Meghan Sebastianski
Jennifer Pillay
Liza Bialy
Lisa Hartling
Publication date: 01-12-2021
Publisher: BioMed Central
Published in: Systematic Reviews / Issue 1/2021
Electronic ISSN: 2046-4053
DOI: https://doi.org/10.1186/s13643-021-01665-x

At a glance: The STEP trials

Springer Medicine

LOCATE: a prospective evaluation of the value of Leveraging Ongoing Citation Acquisition Techniques for living Evidence syntheses

Abstract

Background

Objectives

Methods

Results

Conclusions

Systematic review registration

At a glance: The STEP trials

Springer Medicine

Abstract

Background

Objectives

Methods

Results

Conclusions

Systematic review registration

Please log in to get access to this content

Other articles of this Issue 1/2021

AGREEing on clinical practice guidelines for idiopathic steroid-sensitive nephrotic syndrome in children

Molecular mechanisms of Zika virus teratogenesis from animal studies: a systematic review protocol

Incorporating and evaluating citizen engagement in health research: a scoping review protocol

Dealing with predatory journal articles captured in systematic reviews

Iterative guided machine learning-assisted systematic literature reviews: a diabetes case study

Red blood cell transfusion in animal models of acute brain injuries: a systematic review protocol