Top

BMC Medical Research Methodology

Published in:

Open Access 01-12-2017 | Research article

Grey literature in systematic reviews: a cross-sectional study of the contribution of non-English reports, unpublished studies and dissertations to the results of meta-analyses in child-relevant reviews

Authors: Lisa Hartling, Robin Featherstone, Megan Nuspl, Kassi Shave, Donna M. Dryden, Ben Vandermeer

Published in: BMC Medical Research Methodology | Issue 1/2017

Abstract

Background

Systematic reviews (SRs) are an important source of information about healthcare interventions. A key component of a well-conducted SR is a comprehensive literature search. There is limited evidence on the contribution of non-English reports, unpublished studies, and dissertations and their impact on results of meta-analyses.

Methods

Our sample included SRs from three Cochrane Review Groups: Acute Respiratory Infections (ARI), Infectious Diseases (ID), Developmental Psychosocial and Learning Problems (DPLP) (n = 129). Outcomes included: 1) proportion of reviews that searched for and included each study type; 2) proportion of relevant studies represented by each study type; and 3) impact on results and conclusions of the primary meta-analysis for each study type.

Results

Most SRs searched for non-English studies; however, these were included in only 12% of reviews and represented less than 5% of included studies. There was a change in results in only four reviews (total sample = 129); in two cases the change did not have an impact on the statistical or clinical significance of results. Most SRs searched for unpublished studies but the majority did not include these (only 6%) and they represented 2% of included studies. In most cases the impact of including unpublished studies was small; a substantial impact was observed in one case that relied solely on unpublished data. Few reviews in ARI (9%) and ID (3%) searched for dissertations compared to 65% in DPLP. Overall, dissertations were included in only nine SRs and represented less than 2% of included studies. In the majority of cases the change in results was negligible or small; in the case where a large change was noted, the estimate was more conservative without dissertations.

Conclusions

The majority of SRs searched for non-English and unpublished studies; however, these represented a small proportion of included studies and rarely impacted the results and conclusions of the review. Inclusion of these study types may have an impact in situations where there are few relevant studies, or where there are questionable vested interests in the published literature. We found substantial variation in whether SRs searched for dissertations; in most reviews that included dissertations, these had little impact on results.

Available only for authorised users

Lavis JN, Davies HT, Gruen RL, Walshe K, Farquhar CM. Working within and beyond the Cochrane Collaboration to make systematic reviews more useful to healthcare managers and policy makers. Healthc Policy. 2006;1(2):21–33. [cited 2017 Jan 6] Available from Medline: https://www.ncbi.nlm.nih.gov/pubmed/19305650/.PubMedPubMedCentral

Lefebvre C, Manheimer E, Glanville J. Chapter 6: Searching for studies. In: Higgins JP, Green S, editors. Cochrane Handbook for Systematic Reviews of Interventions Version 5.1.0. The Cochrane Collaboration. 2011. [cited 2016 Jun 7]. Available from: http://handbook.cochrane.org/.

Sterne JA, Egger M, Moher D. Chapter 10: Addressing reporting biases. In: Higgins JP, Green S, editors. Cochrane Handbook for Systematic Reviews of Interventions Version 5.1.0. The Cochrane Collaboration. 2011. [cited 2017 Jan 3]. Available from: http://handbook.cochrane.org/.

Higgins JPT, Lasserson T, Chandler J, Tovey D, Churchill R. Methodological Expectations of Cochrane Intervention Reviews (MECIR): standards for the conduct and reporting of new Cochrane Intervention Reviews, reporting of protocols and the planning, conduct and reporting of updates. Cochrane; 2016 [cited 2017 Jan 3]. Available from: http://methods.cochrane.org/sites/default/files/public/uploads/mecir_printed_booklet_final.pdf.

Institute of Medicine. Finding what works in health care: standards for systematic reviews. Washington, DC: The National Academies Press; 2011. [cited 2016 Jun 7]. Available from: http://www.nationalacademies.org/hmd/Reports/2011/Finding-What-Works-in-Health-Care-Standards-for-Systematic-Reviews.aspx.

Balshem H, Stevens A, Ansari M, Norris S, Kansagara D, Shamliyan T, Chou R, Chung M, Moher D, Dickersin K. Finding grey literature evidence and assessing for outcome and analysis reporting biases when comparing medical interventions: AHRQ and the Effective Health Care Program. Methods Guide for Comparative Effectiveness Reviews. Rockville: Agency for Healthcare Research and Quality (US); 2013. [cited 2017 Jan 3]; Contract Nos.: 290-2007-10057-I and 290-2007-10059-I. Available from Medline: https://www.ncbi.nlm.nih.gov/pubmed/24404628.

Lefebvre C, Manheimer E, Glanville J. Chapter 6.2.3: Unpublished and ongoing studies. In: Higgins JP, Green S, editors. Cochrane Handbook for Systematic Reviews of Interventions Version 5.1.0. The Cochrane Collaboration. 2011. [cited 2017 Jan 4]. Available from: http://handbook.cochrane.org/.

Egger M, Juni P, Bartlett C, Holenstein F, Sterne J. How important are comprehensive literature searches and the assessment of trial quality in systematic reviews?: Empirical study. Health Technol Asses. 2003;7(1). [cited 2017 Jan 4] Available from Medline: https://www.ncbi.nlm.nih.gov/pubmed/12583822.

Jefferson T, Jones MA, Doshi P, Del Mar CB, Heneghan CJ, Hama R, Thompson MJ. Neuraminidase inhibitors for preventing and treating influenza in healthy adults and children. Cochrane Database of Syst Rev. 2012 [cited 2017 Jan 3];Jan 18. Available from Medline: https://www.ncbi.nlm.nih.gov/pubmed/22258996.

10.

Hart B, Lundh A, Bero L. Effect of reporting bias on meta-analyses of drug trials: reanalysis of meta-analyses. BMJ. 2012;344:d7202. [cited 2017 Jan 3] Available from Medline: https://www.ncbi.nlm.nih.gov/pubmed/22214754.CrossRefPubMed

11.

Turner EH, Knoepflmacher D, Shapley L. Publication bias in antipsychotic trials: an analysis of efficacy comparing the published literature to the US Food and Drug Administration database. PLoS Med. 2012;9(3):e1001189. [cited 2017 Jan 4] Available from Medline: https://www.ncbi.nlm.nih.gov/pubmed/22448149.CrossRefPubMedPubMedCentral

12.

Golder S, Loke YK, Wright K, Norman G. Reporting of adverse events in published and unpublished studies of health care interventions: a systematic review. PLoS Med. 2016;13(9):e1002127. [cited 2017 Jan 4] Available from Medline: https://www.ncbi.nlm.nih.gov/pubmed/27649528.CrossRefPubMedPubMedCentral

13.

Whittington CJ, Kendall T, Fonagy P, Cottrell D, Cotgrove A, Boddington E. Selective serotonin reuptake inhibitors in childhood depression: systematic review of published versus unpublished data. Lancet. 2004;363(9418):1341–5. [cited 2017 Jan 3] Available from Medline: https://www.ncbi.nlm.nih.gov/pubmed/15110490.CrossRefPubMed

14.

Lefebvre C, Manheimer E, Glanville J. Chapter 6.2.1.7 Dissertations and theses databases. In: Higgins JP, Green S, editors. Cochrane Handbook for Systematic Reviews of Interventions Version 5.1.0. The Cochrane Collaboration. 2011. [cited 2017 Jan 4]. Available from: http://handbook.cochrane.org/.

15.

Moher D, Pham B, Lawson ML, Klassen TP. The inclusion of reports of randomised trials published in languages other than English in systematic reviews. Health Technol Assess. 2003;7(41):1–90. [cited 2017 Jan 4] Available from Medline: https://www.ncbi.nlm.nih.gov/pubmed/14670218.CrossRefPubMed

16.

Egger M, Zellweger-Zähner T, Schneider M, Junker C, Lengeler C, Antes G. Language bias in randomised controlled trials published in English and German. Lancet. 1997;350(9074):326–9. Available from Medline: https://www.ncbi.nlm.nih.gov/pubmed/9251637.CrossRefPubMed

17.

Heres S, Wagenpfeil S, Hamann J, Kissling W, Leucht S. Language bias in neuroscience—is the Tower of Babel located in Germany? Eur Psychiatry. 2004;19(4):230–2. [cited 2017 Jan 3] Available from Medline: https://www.ncbi.nlm.nih.gov/pubmed/15196606.CrossRefPubMed

18.

Vickers A, Goyal N, Harland R, Rees R. Do certain countries produce only positive results? A systematic review of controlled trials. Control Clin Trials. 1998;19(2):159–66. [cited 2017 Jan 3] Available from Medline: https://www.ncbi.nlm.nih.gov/pubmed/9551280.CrossRefPubMed

19.

Pham B, Klassen TP, Lawson ML, Moher D. Language of publication restrictions in systematic reviews gave different results depending on whether the intervention was conventional or complementary. J Clin Epidemiol. 2005;58(8):769–76. [cited 2017 Jan 3] Available from Medline: https://www.ncbi.nlm.nih.gov/pubmed/16086467.CrossRefPubMed

20.

Royle PL, Bain L, Waugh NR. Sources of evidence for systematic reviews of interventions in diabetes. Diabet Med. 2005;22(10):1386–93. [cited 2016 Jun 7] Available from Medline: http://www.ncbi.nlm.nih.gov/pubmed/16176201.CrossRefPubMed

21.

Royle P, Waugh N. Literature searching for clinical and cost-effectiveness studies used in health technology assessment reports carried out for the National Institute for Clinical Excellence appraisal system. Health Technol Assess. 2003;7(34):iii, ix-x, 1–51. [cited 2016 Jun 7] Available from Medline: http://www.ncbi.nlm.nih.gov/pubmed/14609481

22.

Hartling L, Guise JM, Hempel S, Featherstone R, Mitchell MD, Motu’apuaka ML, et al. EPC Methods: AHRQ End-User Perspectives of Rapid Reviews. Rockville: Agency for Healthcare Research and Quality (US); 2016. [cited 2016 Jun 7]; Contract No.: 16-EHC014-EF. Available from NCBI Bookshelf: http://www.ncbi.nlm.nih.gov/books/NBK362003/.

23.

Hartling L, Featherstone R, Nuspl M, Shave K, Dryden DM, Vandermeer B. The contribution of databases to the results of systematic reviews: a cross-sectional study. BMC Med Res Methodol. 2016;16(1):127. [cited 2017 Jan 4] Available from Medline: https://www.ncbi.nlm.nih.gov/pubmed/27670136.CrossRefPubMedPubMedCentral

24.

Jones M, Jefferson T, Doshi P, Del Mar C, Heneghan C, Onakpoya I. Commentary on Cochrane review of neuraminidase inhibitors for preventing and treating influenza in healthy adults and children. Clin Microbiol Infect. 2015;21(3):217–21. [cited 2017 Jan 3] Available from Medline: https://www.ncbi.nlm.nih.gov/pubmed/25658541.CrossRefPubMed

25.

Higgins JPT, Altman DG, Sterne JAC. Chapter 8: Assessing risk of bias in included studies. In: Higgins JPT, Green S, editors. Cochrane Handbook for Systematic Reviews of Interventions Version 5.1.0. The Cochrane Collaboration. 2011. [cited 2016 Jun 7]. Available from: http://handbook.cochrane.org/.

26.

Wikipedia contributors. Thesis. Wikipedia, The Free Encyclopedia. 2017, [cited 2017 Jan 3] Available from: https://en.wikipedia.org/w/index.php?title=Thesis&oldid=758058152.

27.

Ganann R, Ciliska D, Thomas H. Expediting systematic reviews: methods and implications of rapid reviews. Implement Sci. 2010;5:56. [cited 2016 Jun 7] Available from Medline: http://www.ncbi.nlm.nih.gov/pubmed/20642853.CrossRefPubMedPubMedCentral

28.

Khangura S, Polisena J, Clifford TJ, Farrah K, Kamel C. Rapid review: an emerging approach to evidence synthesis in health technology assessment. Int J Technol Assess Health Care. 2014;30(1):20–7. [cited 2016 Jun 7] Available from Medline: http://www.ncbi.nlm.nih.gov/pubmed/24451157.CrossRefPubMed

29.

Hartling L, Guise J-M, Kato E, Anderson J, Belinson S, Berliner E, et al. A taxonomy of rapid reviews links report types and methods to specific decision-making contexts. J Clin Epidemiol. 2015;68(12):1451–62. [cited 2016 Jun 7] Available from Medline: http://www.ncbi.nlm.nih.gov/pubmed/26278023.CrossRefPubMed

30.

Tricco AC, Antony J, Zarin W, Strifler L, Ghassemi M, Ivory J, et al. A scoping review of rapid review methods. BMC Med. 2015;13:224. [cited 2016 Jun 7] Available from Medline: http://www.ncbi.nlm.nih.gov/pubmed/26377409.CrossRefPubMedPubMedCentral

Title: Grey literature in systematic reviews: a cross-sectional study of the contribution of non-English reports, unpublished studies and dissertations to the results of meta-analyses in child-relevant reviews
Authors: Lisa Hartling
Robin Featherstone
Megan Nuspl
Kassi Shave
Donna M. Dryden
Ben Vandermeer
Publication date: 01-12-2017
Publisher: BioMed Central
Published in: BMC Medical Research Methodology / Issue 1/2017
Electronic ISSN: 1471-2288
DOI: https://doi.org/10.1186/s12874-017-0347-z

At a glance: The STEP trials

Springer Medicine

Grey literature in systematic reviews: a cross-sectional study of the contribution of non-English reports, unpublished studies and dissertations to the results of meta-analyses in child-relevant reviews

Abstract

Background

Methods

Results

Conclusions

At a glance: The STEP trials

Springer Medicine

Abstract

Background

Methods

Results

Conclusions

Please log in to get access to this content

Other articles of this Issue 1/2017

Why statistical inference from clinical trials is likely to generate false and irreproducible results

Towards evidence-based computational statistics: lessons from clinical research on the role and design of real-data benchmark studies

Bayesian hierarchical piecewise regression models: a tool to detect trajectory divergence between groups in long-term observational studies

Estimating the size of the homeless adolescent population across seven cities in Cambodia

Efficient confidence limits for adaptive one-arm two-stage clinical trials with binary endpoints

Erratum to: Evaluation of biases present in the cohort multiple randomised controlled trial design: a simulation study