Top

BMC Medical Research Methodology

Published in:

Open Access 01-12-2016 | Research article

The contribution of databases to the results of systematic reviews: a cross-sectional study

Authors: Lisa Hartling, Robin Featherstone, Megan Nuspl, Kassi Shave, Donna M. Dryden, Ben Vandermeer

Published in: BMC Medical Research Methodology | Issue 1/2016

Abstract

Background

One of the best sources for high quality information about healthcare interventions is a systematic review. A well-conducted systematic review includes a comprehensive literature search. There is limited empiric evidence to guide the extent of searching, in particular the number of electronic databases that should be searched. We conducted a cross-sectional quantitative analysis to examine the potential impact of selective database searching on results of meta-analyses.

Methods

Our sample included systematic reviews (SRs) with at least one meta-analysis from three Cochrane Review Groups: Acute Respiratory Infections (ARI), Infectious Diseases (ID), Developmental Psychosocial and Learning Problems (DPLP) (n = 129). Outcomes included: 1) proportion of relevant studies indexed in each of 10 databases; and 2) changes in results and statistical significance of primary meta-analysis for studies identified in Medline only and in Medline plus each of the other databases.

Results

Due to variation across topics, we present results by group (ARI n = 57, ID n = 38, DPLP n = 34). For ARI, identification of relevant studies was highest for Medline (85 %) and Embase (80 %). Restricting meta-analyses to trials that appeared in Medline + Embase yielded fewest changes in statistical significance: 53/55 meta-analyses showed no change. Point estimates changed in 12 cases; in 7 the change was less than 20 %. For ID, yield was highest for Medline (92 %), Embase (81 %), and BIOSIS (67 %). Restricting meta-analyses to trials that appeared in Medline + BIOSIS yielded fewest changes with 1 meta-analysis changing in statistical significance. Point estimates changed in 8 of 31 meta-analyses; change less than 20 % in all cases. For DPLP, identification of relevant studies was highest for Medline (75 %) and Embase (62 %). Restricting meta-analyses to trials that appeared in Medline + PsycINFO resulted in only one change in significance. Point estimates changed for 13 of 33 meta-analyses; less than 20 % in 9 cases.

Conclusions

Majority of relevant studies can be found within a limited number of databases. Results of meta-analyses based on the majority of studies did not differ in most cases. There were very few cases of changes in statistical significance. Effect estimates changed in a minority of meta-analyses but in most the change was small. Results did not change in a systematic manner (i.e., regularly over- or underestimating treatment effects), suggesting that selective searching may not introduce bias in terms of effect estimates.

Available only for authorised users

Lefebvre C, Manheimer E, Glanville J. Chapter 6: Searching for studies. In: Higgins JP, Green S, editors. Cochrane Handbook for Systematic Reviews of Interventions Version 5.1.0 [Internet]. The Cochrane Collaboration; 2011 [cited 2016 Jun 7]. Available from: http://handbook.cochrane.org/.

Chandler J, Churchill R, Higgins J, Lasserson T, Tovey D. Methodological Expectations of Cochrane Intervention Reviews (MECIR): Methodological standards for the conduct of new Cochrane Intervention Reviews [Internet]. Cochrane Collaboration; 2013 [cited 2016 Jun 7]. Available from: http://editorial-unit.cochrane.org/mecir.

Institute of Medicine. Finding what works in health care: standards for systematic reviews [Internet]. Washington, DC: The National Academies Press; 2011 [cited 2016 Jun 7]. Available from: http://www.nationalacademies.org/hmd/Reports/2011/Finding-What-Works-in-Health-Care-Standards-for-Systematic-Reviews.aspx.

McDonald S, Taylor L, Adams C. Searching the right database. A comparison of four databases for psychiatry journals. Health Libr Rev. 1999;16(3):151–6. Available from Medline: http://www.ncbi.nlm.nih.gov/pubmed/10620849.CrossRefPubMed

Suarez-Almazor ME, Belseck E, Homik J, Dorgan M, Ramos-Remus C. Identifying clinical trials in the medical literature with electronic databases: MEDLINE alone is not enough. Control Clin Trials. 2000;21(5):476–87. Available from Medline: http://www.ncbi.nlm.nih.gov/pubmed/11018564.CrossRefPubMed

Royle P, Milne R. Literature searching for randomized controlled trials used in Cochrane reviews: rapid versus exhaustive searches. Int J Technol Assess Health Care. 2003;19(4):591–603. Available from Medline: http://www.ncbi.nlm.nih.gov/pubmed/15095765.CrossRefPubMed

Royle P, Waugh N. Literature searching for clinical and cost-effectiveness studies used in health technology assessment reports carried out for the National Institute for Clinical Excellence appraisal system. Health Technol Assess. 2003;7(34):iii. ix-x, 1–51. Available from Medline: http://www.ncbi.nlm.nih.gov/pubmed/14609481.CrossRef

Savoie I, Helmer D, Green CJ, Kazanjian A. Beyond Medline: reducing bias through extended systematic review search. Int J Technol Assess Health Care. 2003;19(1):168–78. Available from Medline: http://www.ncbi.nlm.nih.gov/pubmed/12701949.CrossRefPubMed

Stevinson C, Lawlor DA. Searching multiple databases for systematic reviews: added value or diminishing returns? Complement Ther Med. 2004;12(4):228–32. Available from Medline: http://www.ncbi.nlm.nih.gov/pubmed/15649836.CrossRefPubMed

10.

Betran AP, Say L, Gulmezoglu AM, Allen T, Hampson L. Effectiveness of different databases in identifying studies for systematic reviews: experience from the WHO systematic review of maternal morbidity and mortality. BMC Med Res Methodol. 2005;5(1):6. Available from Medline: http://www.ncbi.nlm.nih.gov/pubmed/15679886.CrossRefPubMedPubMedCentral

11.

Lemeshow AR, Blum RE, Berlin JA, Stoto MA, Colditz GA. Searching one or two databases was insufficient for meta-analysis of observational studies. J Clin Epidemiol. 2005;58(9):867–73. Available from Medline: http://www.ncbi.nlm.nih.gov/pubmed/16085190.CrossRefPubMed

12.

Ogilvie D, Hamilton V, Egan M, Petticrew M. Systematic reviews of health effects of social interventions: 1. Finding the evidence: how far should you go? J Epidemiol Community Health. 2005;59(9):804–8. Available from Medline: http://www.ncbi.nlm.nih.gov/pubmed/16100321/.CrossRefPubMedPubMedCentral

13.

Royle P, Bain L, Waugh N. Systematic reviews of epidemiology in diabetes: finding the evidence. BMC Med Res Methodol. 2005;5:2. Available from Medline: http://www.ncbi.nlm.nih.gov/pubmed/15638944/.CrossRefPubMedPubMedCentral

14.

Whiting P, Westwood M, Burke M, Sterne J, Glanville J. Systematic reviews of test accuracy should search a range of databases to identify primary studies. J Clin Epidemiol. 2008;61(4):357–64. Available from Medline: http://www.ncbi.nlm.nih.gov/pubmed/18313560.CrossRefPubMed

15.

Slobogean GP, Verma A, Giustini D, Slobogean BL, Mulpuri K. MEDLINE, EMBASE, and Cochrane index most primary studies but not abstracts included in orthopedic meta-analyses. J Clin Epidemiol. 2009;62(12):1261–7. Available from Medline: http://www.ncbi.nlm.nih.gov/pubmed/19364634.CrossRefPubMed

16.

Golder S, Loke YK. The contribution of different information sources for adverse effects data. Int J Technol Assess Health Care. 2012;28(2):133–7. Available from Medline: http://www.ncbi.nlm.nih.gov/pubmed/22559754.CrossRefPubMed

17.

Beyer FR, Wright K. Can we prioritise which databases to search? A case study using a systematic review of frozen shoulder management. Health Info Libr J. 2013;30(1):49–58. Available from Medline: http://www.ncbi.nlm.nih.gov/pubmed/23413793.CrossRefPubMed

18.

Bramer WM, Giustini D, Kramer BM, Anderson P. The comparative recall of Google Scholar versus PubMed in identical searches for biomedical systematic reviews: a review of searches used in systematic reviews. Syst Rev. 2013;2:115. Available from Medline: http://www.ncbi.nlm.nih.gov/pubmed/24360284.CrossRefPubMedPubMedCentral

19.

Kwon Y, Powelson SE, Wong H, Ghali WA, Conly JM. An assessment of the efficacy of searching in biomedical databases beyond MEDLINE in identifying studies for a systematic review on ward closures as an infection control intervention to control outbreaks. Syst Rev. 2014;3:135. Available from Medline: http://www.ncbi.nlm.nih.gov/pubmed/25387523.CrossRefPubMedPubMedCentral

20.

Lorenzetti DL, Topfer LA, Dennett L, Clement F. Value of databases other than medline for rapid health technology assessments. Int J Technol Assess Health Care. 2014;30(2):173–8. Available from Medline: http://www.ncbi.nlm.nih.gov/pubmed/24774535.CrossRefPubMed

21.

Westphal A, Kriston L, Holzel LP, Harter M, von Wolff A. Efficiency and contribution of strategies for finding randomized controlled trials: a case study from a systematic review on therapeutic interventions of chronic depression. Public Health Res. 2014;3(2):177. Available from Medline: http://www.ncbi.nlm.nih.gov/pubmed/25343133.

22.

Gargon E, Williamson PR, Clarke M. Collating the knowledge base for core outcome set development: developing and appraising the search strategy for a systematic review. BMC Med Res Methodol. 2015;15(1):26. Available from Medline: http://www.ncbi.nlm.nih.gov/pubmed/25888523.CrossRefPubMedPubMedCentral

23.

Sampson M, Barrowman NJ, Moher D, Klassen TP, Pham B, Platt R, et al. Should meta-analysts search Embase in addition to Medline? J Clin Epidemiol. 2003;56(10):943–55. Available from Medline: http://www.ncbi.nlm.nih.gov/pubmed/14568625.CrossRefPubMed

24.

Royle PL, Bain L, Waugh NR. Sources of evidence for systematic reviews of interventions in diabetes. Diabet Med. 2005;22(10):1386–93. Available from Medline: http://www.ncbi.nlm.nih.gov/pubmed/16176201.CrossRefPubMed

25.

Halladay CW, Trikalinos TA, Schmid IT, Schmid CH, Dahabreh IJ. Using data sources beyond PubMed has a modest impact on the results of systematic reviews of therapeutic interventions. J Clin Epidemio. 2015;68(9):1076–84. Available from Medline: http://www.ncbi.nlm.nih.gov/pubmed/26279401.CrossRef

26.

Moher D, Tetzlaff J, Tricco AC, Sampson M, Altman DG. Epidemiology and reporting characteristics of systematic reviews. PLoS Med. 2007;4(3):e78. Available from Medline: http://www.ncbi.nlm.nih.gov/pubmed/17388659.CrossRefPubMedPubMedCentral

27.

Page MJ, Shamseer L, Altman DG, Tetzlaff J, Sampson M, Tricco AC, et al. Epidemiology and reporting characteristics of systematic reviews of biomedical research: a cross-sectional study. PLoS Med. 2016;13(5):e1002028. Available from Medline: http://www.ncbi.nlm.nih.gov/pubmed/27218655.CrossRefPubMedPubMedCentral

28.

Bow S, Klassen J, Chisholm A, Tjosvold L, Thomson D, Klassen TP, et al. A descriptive analysis of child-relevant systematic reviews in the Cochrane database of systematic reviews. BMC Pediatr. 2010;10(1):1–8. Available from Medline: http://www.ncbi.nlm.nih.gov/pubmed/20487565.CrossRef

29.

Mathie RT, Frye J, Fisher P. Homeopathic Oscillococcinum® for preventing and treating influenza and influenza-like illness. Cochrane Database Syst Rev. 2015;28(1):CD001957. doi:10.1002/14651858.CD001957.pub6.

30.

Jiang L, Deng L, Wu T. Chinese medicinal herbs for influenza. Cochrane Database Syst Rev. 2013;28(3):CD004559. doi:10.1002/14651858.CD004559.pub4.

31.

Livingstone N, Macdonald G, Carr N. Restorative justice conferencing for reducing recidivism in young offenders (aged 7 to 21). Cochrane Database Syst Rev. 2013;28(2):CD008898. doi:10.1002/14651858.CD008898.pub2.

32.

Geretsegger M, Elefant C, Mössler KA, Gold C. Music therapy for people with autism spectrum disorder. Cochrane Database Syst Rev. 2014;(6):CD004381. doi:10.1002/14651858.CD004381.pub3.

33.

Lucas PJ, McIntosh K, Petticrew M, Roberts HM, Shiell A. Financial benefits for child health and well-being in low income or socially disadvantaged families in developed world countries. Cochrane Database Syst Rev. 2008;16(2):CD006358. doi:10.1002/14651858.CD006358.pub2.

34.

Straus SE, Tetroe JM, Graham ID. Knowledge translation is the use of knowledge in health care decision making. J Clin Epidemiol. 2011;64(1):6–10. Available from Medline: http://www.ncbi.nlm.nih.gov/pubmed/19926445.CrossRefPubMed

35.

Ganann R, Ciliska D, Thomas H. Expediting systematic reviews: methods and implications of rapid reviews. Implement Sci. 2010;5:56. Available from Medline: http://www.ncbi.nlm.nih.gov/pubmed/20642853.CrossRefPubMedPubMedCentral

36.

Khangura S, Polisena J, Clifford TJ, Farrah K, Kamel C. Rapid review: an emerging approach to evidence synthesis in health technology assessment. Int J Technol Assess Health Care. 2014;30(1):20–7. Available from Medline: http://www.ncbi.nlm.nih.gov/pubmed/24451157.CrossRefPubMed

37.

Hartling L, Guise J-M, Kato E, Anderson J, Belinson S, Berliner E, et al. A taxonomy of rapid reviews links report types and methods to specific decision-making contexts. J Clin Epidemiol. 2015;68(12):1451–62. Available from Medline: http://www.ncbi.nlm.nih.gov/pubmed/26278023.CrossRefPubMed

38.

Tricco AC, Antony J, Zarin W, Strifler L, Ghassemi M, Ivory J, et al. A scoping review of rapid review methods. BMC Med. 2015;13:224. Available from Medline: http://www.ncbi.nlm.nih.gov/pubmed/26377409.CrossRefPubMedPubMedCentral

39.

Hartling L, Guise JM, Hempel S, Featherstone R, Mitchell MD, Motu’apuaka ML, et al. EPC methods: AHRQ End-user perspectives of rapid reviews. Rockville: Agency for Healthcare Research and Quality (US); 2016. Contract No.: 16-EHC014-EF. Available from NCBI Bookshelf: http://www.ncbi.nlm.nih.gov/books/NBK362003/.

40.

Fortier KJ, Kiss N, Tongbram V. What is the optimal search engine for results from embase and medline: ovid or embase.com? Value Health. 2013;1(6):A1–298.

Title: The contribution of databases to the results of systematic reviews: a cross-sectional study
Authors: Lisa Hartling
Robin Featherstone
Megan Nuspl
Kassi Shave
Donna M. Dryden
Ben Vandermeer
Publication date: 01-12-2016
Publisher: BioMed Central
Published in: BMC Medical Research Methodology / Issue 1/2016
Electronic ISSN: 1471-2288
DOI: https://doi.org/10.1186/s12874-016-0232-1

ACC 2024 Congress

Springer Medicine

The contribution of databases to the results of systematic reviews: a cross-sectional study

Abstract

Background

Methods

Results

Conclusions

ACC 2024 Congress

Springer Medicine

Abstract

Background

Methods

Results

Conclusions

Please log in to get access to this content

Other articles of this Issue 1/2016

The use of relative incidence ratios in self-controlled case series studies: an overview

Dose-response meta-analysis of differences in means

National weighting of data from the Behavioral Risk Factor Surveillance System (BRFSS)

Searching for the optimal measuring frequency in longitudinal studies -- an example utilizing short message service (SMS) to collect repeated measures among patients with low back pain

Recommendations for the analysis of individually randomised controlled trials with clustering in one arm – a case of continuous outcomes

Meta-analyses including non-randomized studies of therapeutic interventions: a methodological review