Top

Systematic Reviews

Published in:

Open Access 01-12-2019 | Research

The strong focus on positive results in abstracts may cause bias in systematic reviews: a case study on abstract reporting bias

Authors: Bram Duyx, Gerard M. H. Swaen, Miriam J. E. Urlings, Lex M. Bouter, Maurice P. Zeegers

Published in: Systematic Reviews | Issue 1/2019

Abstract

Background

Research articles tend to focus on positive findings in their abstract, especially if multiple outcomes have been studied. At the same time, search queries in databases are generally limited to the abstract, title and keywords fields of an article. Negative findings are therefore less likely to be detected by systematic searches and to appear in systematic reviews. We aim to assess the occurrence of this ‘abstract reporting bias’ and quantify its impact in the literature on the association between diesel exhaust exposure (DEE) and bladder cancer.

Methods

We set up a broad search query related to DEE and cancer in general. Full-texts of the articles identified in the search output were manually scanned. Articles were included if they reported, anywhere in the full-text, the association between DEE and bladder cancer. We assume that the use of a broad search query and manual full-text scanning allowed us to catch all the relevant articles, including those in which bladder cancer was not mentioned in the abstract, title or keywords.

Results

We identified 28 articles. Only 12 of these (43%) had mentioned bladder in their abstract, title or keywords. A meta-analysis based on these 12 detectable articles yielded a pooled risk estimate of 1.10 (95% confidence interval [CI] 0.97–1.25), whereas the meta-analysis based on all 28 articles yielded a pooled estimate of 1.03 (95% CI 0.96–1.11).

Conclusions

This case study on abstract reporting bias shows that (a) more than half of all relevant articles were missed by a conventional search query and (b) this led to an overestimation of the pooled effect. Detection of articles will be improved if all studied exposure and outcome variables are reported in the keywords. The restriction on the maximum number of keywords should be lifted.

Available only for authorised users

Harbour R, Miller J. A new system for grading recommendations in evidence based guidelines. BMJ. 2001;323(7308):334–6.CrossRef

Evans D. Hierarchy of evidence: a framework for ranking evidence evaluating healthcare interventions. J Clin Nurs. 2003;12(1):77–84.CrossRef

Duyx B, Urlings MJE, Swaen GMH, Bouter LM, Zeegers MP. Scientific citations favor positive results: a systematic review and meta-analysis. J Clin Epidemiol. 2017;88:92–101.CrossRef

Ioannidis JPA. Why most published research findings are false. PLoS Med. 2005;2(8):e124.CrossRef

Sterne JAC, Egger M, Smith GD. Investigating and dealing with publication and other biases in meta-analysis. BMJ. 2001;323(7304):101–5.CrossRef

Song F, Parekh S, Hooper L, Loke YK, Ryder J, Sutton AJ, et al. Dissemination and publication of research findings: an updated review of related biases. Health Technol Assess. 2010;14(8):1–193.CrossRef

Kavvoura FK, Liberopoulos G, Ioannidis JPA. Selection in reported epidemiological risks: an empirical assessment. PLoS Med. 2007;4(3):e79.CrossRef

Web of Science All Databases Help: Searching the topic field: clarivate analytics; 2018 [updated 09/10/2018. Available from: https://images.webofknowledge.com/WOKRS531OR13/help/WOK/hs_topic.html].

PubMed Help: Search Field Descriptions and Tags: U.S. National Library of Medicine; 2018 [updated October 22, 2018. Available from: https://www.ncbi.nlm.nih.gov/books/NBK3827/#pubmedhelp.Search_Field_Descriptions_and].

10.

Systematic error in systematic reviews: a study protocol on abstract reporting bias. DataverseNL Dataverse. 2018. [Available from: http://hdl.handle.net/10411/5798UX. Accessed 9 July 2019].

11.

Dean AG, Sullivan KM, Soe MM. OpenEpi: Open Source Epidemiologic Statistics for Public Health: OpenEpi; [updated 6 April 2013 Available from: http://www.openepi.com/SMR/SMR.htm].

12.

KeyWords Plus generation, creation, and changes: clarivate analytics; 2018 [updated 27 Jun 2018. Available from: https://support.clarivate.com/ScientificandAcademicResearch/s/article/KeyWords-Plus-generation-creation-and-changes?language=en_US].

13.

MEDLINE Indexing Online Training Course Rockville Pike, Bethesda, USA: U.S. National Library of Medicine; 2015 [updated 15 April 2015. Available from: https://www.nlm.nih.gov/bsd/indexing/training/USE_010.html].

14.

Boffetta P, Silverman DT. A meta-analysis of bladder cancer and diesel exhaust exposure. Epidemiology. 2001;12(1):125–30.CrossRef

15.

Chavalarias D, Ioannidis JPA. Science mapping analysis characterizes 235 biases in biomedical research. J Clin Epidemiol. 2010;63(11):1205–15.CrossRef

16.

Swaen GMH, Langendam M, Weyler J, Burger H, Siesling S, Atsma WJ, et al. Responsible Epidemiologic Research Practice: a guideline developed by a working group of the Netherlands Epidemiological Society. J Clin Epidemiol. 2018;100:111–9.CrossRef

17.

Vandenbroucke JP, von Elm E, Altman DG, Gøtzsche PC, Mulrow CD, Pocock SJ, et al. Strengthening the Reporting of Observational Studies in Epidemiology (STROBE): Explanation and Elaboration. PLoS Med. 2007;4(10):e297.CrossRef

18.

Diesel and gasoline engine exhausts and some nitroarenes. IARC monographs on the evaluation of carcinogenic risks to humans. 2013;105:1–704 [Available from: http://monographs.iarc.fr/ENG/Monographs/vol105/index.php].

19.

Google Scholar: Advanced Search [Available from: https://scholar.google.com/#d=gs_asd&p=&u=. Accessed 9 July 2019].

20.

Robinson KA, Dickersin K. Development of a highly sensitive search strategy for the retrieval of reports of controlled trials using PubMed. Int J Epidemiol. 2002;31(1):150–3.CrossRef

Title: The strong focus on positive results in abstracts may cause bias in systematic reviews: a case study on abstract reporting bias
Authors: Bram Duyx
Gerard M. H. Swaen
Miriam J. E. Urlings
Lex M. Bouter
Maurice P. Zeegers
Publication date: 01-12-2019
Publisher: BioMed Central
Published in: Systematic Reviews / Issue 1/2019
Electronic ISSN: 2046-4053
DOI: https://doi.org/10.1186/s13643-019-1082-9

At a glance: The STEP trials

Springer Medicine

The strong focus on positive results in abstracts may cause bias in systematic reviews: a case study on abstract reporting bias

Abstract

Background

Methods

Results

Conclusions

At a glance: The STEP trials

Springer Medicine

Abstract

Background

Methods

Results

Conclusions

Please log in to get access to this content

Other articles of this Issue 1/2019

Living alone and positive mental health: a systematic review

Reporting quality of Cochrane systematic reviews with Chinese herbal medicines

Effectiveness and safety of inpatient versus extended venous thromboembolism (VTE) prophylaxis with heparin following major pelvic surgery for malignancy: protocol for a systematic review

Pragmatic methods for reviewing exceptionally large bodies of evidence: systematic mapping review and overview of systematic reviews using lung cancer survival as an exemplar

Acceptance and commitment therapy for chronic pain: protocol of a systematic review and individual participant data meta-analysis

Mapping the evidence on interventions to raise awareness on lung cancer in resource poor settings: a scoping review protocol