Skip to main content
Key Specialties
Cardiology Diabetology Endocrinology Gastroenterology Geriatrics and Gerontology Gynecology and Obstetrics Hematology Infectious Disease Internal Medicine Nephrology Neurology Oncology Pediatrics Respiratory Medicine Rheumatology Surgery
Congress Coverage
2024 ACC Congress 2023 ESMO Congress 2023 EASD Congress 2023 ERS Congress 2023 ESC Congress
Clinical Collections
Advances in Alzheimer’s

Keynote webinar | Spotlight on medication adherence

LIVE: Thursday 27th June 2024, 18:00-19:30 (CEST)
Join our expert panel to discover why you need to understand the drivers of non-adherence in your patients, and how you can optimize medication adherence in your clinics to drastically improve patient outcomes.
ADVANCED SEARCH
Log in
Top
BMC Medical Research Methodology
Published in: BMC Medical Research Methodology 1/2009

Open Access 01-12-2009 | Research article

Extent of publication bias in different categories of research cohorts: a meta-analysis of empirical studies

Authors: Fujian Song, Sheetal Parekh-Bhurke, Lee Hooper, Yoon K Loke, Jon J Ryder, Alex J Sutton, Caroline B Hing, Ian Harvey

Published in: BMC Medical Research Methodology | Issue 1/2009

Login to get access

Abstract

Background

The validity of research synthesis is threatened if published studies comprise a biased selection of all studies that have been conducted. We conducted a meta-analysis to ascertain the strength and consistency of the association between study results and formal publication.

Methods

The Cochrane Methodology Register Database, MEDLINE and other electronic bibliographic databases were searched (to May 2009) to identify empirical studies that tracked a cohort of studies and reported the odds of formal publication by study results. Reference lists of retrieved articles were also examined for relevant studies. Odds ratios were used to measure the association between formal publication and significant or positive results. Included studies were separated into subgroups according to starting time of follow-up, and results from individual cohort studies within the subgroups were quantitatively pooled.

Results

We identified 12 cohort studies that followed up research from inception, four that included trials submitted to a regulatory authority, 28 that assessed the fate of studies presented as conference abstracts, and four cohort studies that followed manuscripts submitted to journals. The pooled odds ratio of publication of studies with positive results, compared to those without positive results (publication bias) was 2.78 (95% CI: 2.10 to 3.69) in cohorts that followed from inception, 5.00 (95% CI: 2.01 to 12.45) in trials submitted to regulatory authority, 1.70 (95% CI: 1.44 to 2.02) in abstract cohorts, and 1.06 (95% CI: 0.80 to 1.39) in cohorts of manuscripts.

Conclusion

Dissemination of research findings is likely to be a biased process. Publication bias appears to occur early, mainly before the presentation of findings at conferences or submission of manuscripts to journals.
Appendix
Available only for authorised users
Literature
1.
Begg CB, Berlin JA: Publication bias; a problem in interpreting medical data. J R Statist Soc A. 1988, 151: 445-463. 10.2307/2982993.CrossRef
2.
Rosenthal R: The "file drawer problem" and tolerance for null results. Psychol Bull. 1979, 86 (3): 638-641. 10.1037/0033-2909.86.3.638.CrossRef
3.
Begg CB, Berlin JA: Publication bias and dissemination of clinical research. J Natl Cancer Inst. 1989, 81: 107-115. 10.1093/jnci/81.2.107.CrossRefPubMed
4.
Dickersin K: The existence of publication bias and risk factors for its occurrence. JAMA. 1990, 263 (10): 1385-1389. 10.1001/jama.263.10.1385.CrossRefPubMed
5.
Sterling T: Publication decisions and their possible effects on inferences drawn from tests of significance - or vice versa. Am Stat Assoc J. 1959, 54: 30-34. 10.2307/2282137.
6.
Sterling TD, Rosenbaum WL, Weinkam JJ: Publication decisions revisited - the effect of the outcome of statistical tests on the decision to publish and vice-versa. American Statistician. 1995, 49 (1): 108-112. 10.2307/2684823.
7.
Sohn D: Publications bias and the evaluation of psychotherapy efficacy In reviews of the research literature. Clin Psychol Rev. 1996, 16 (2): 147-156. 10.1016/0272-7358(96)00005-0.CrossRef
8.
Song F, Eastwood AJ, Gilbody S, Duley L, Sutton AJ: Publication and related biases. Health Technol Assess. 2000, 4 (10): 1-115. 10.3310/hta4100.
9.
Egger M, Juni P, Bartlett C, Holenstein F, Sterne J: How important are comprehensive literature searches and the assessment of trial quality in systematic reviews? Empirical study. Health Technol Assess. 2003, 7: 1-76.PubMed
10.
Moher D, Pham B, Lawson ML, Klassen TP: The inclusion of reports of randomised trials published in languages other than English in systematic reviews. Health Technol Assess. 2003, 7 (41): 1-90.CrossRefPubMed
11.
Chan AW, Altman DG: Identifying outcome reporting bias in randomised trials on PubMed: review of publications and survey of authors. BMJ. 2005, 330 (7494): 753-10.1136/bmj.38356.424606.8F.CrossRefPubMedPubMedCentral
12.
Chan AW, Hrobjartsson A, Haahr MT, Gotzsche PC, Altman DG: Empirical evidence for selective reporting of outcomes in randomized trials: comparison of protocols to published articles. JAMA. 2004, 291 (20): 2457-2465. 10.1001/jama.291.20.2457.CrossRefPubMed
13.
Chan AW, Krleza-Jeri K, Schmid I, Altman DG: Outcome reporting bias in randomized trials funded by the Canadian Institutes of Health Research [see comment]. CMAJ. 2004, 171 (7): 735-740.CrossRefPubMedPubMedCentral
14.
Scherer RW, Langenberg P, von Elm E: Full publication of results initially presented in abstracts. Cochrane Database Syst Rev. 2007, MR000005-2
15.
Song F, Parekh S, Hooper L, Loke Y, Ryder JJ, Sutton AJ, Hing C, Shing C, Pang C, Harvey I: Dissemination and publication of research findings: an updated review of related biases. Health Technol Assess. 2010,
16.
Dickersin K: How important is publication bias? A synthesis of available data. Aids Educ Prev. 1997, 9 (1 SA): 15-21.PubMed
17.
Hopewell S, Loudon K, Clarke MJ, Oxman AD, Dickersin K: Publication bias in clinical trials due to statistical significance or direction of trial results. Cochrane Database Syst Rev. 2009, MR000006-1
18.
Dwan K, Altman DG, Arnaiz JA, Bloom J, Chan AW, Cronin E, Decullier E, Easterbrook PJ, Von Elm E, Gamble C, et al: Systematic review of the empirical evidence of study publication bias and outcome reporting bias. PLoS ONE. 2008, 3 (8): e3081-10.1371/journal.pone.0003081.CrossRefPubMedPubMedCentral
19.
20.
Rucker G, Schwarzer G, Carpenter JR, Schumacher M: Undue reliance on I2 in assessing heterogeneity may mislead. BMC Med Res Methodol. 2008, 8: 79-10.1186/1471-2288-8-79.CrossRefPubMedPubMedCentral
21.
Peters JL, Sutton AJ, Jones DR, Abrams KR, Rushton L: Comparison of two methods to detect publication bias in meta-analysis. JAMA. 2006, 295 (6): 676-680. 10.1001/jama.295.6.676.CrossRefPubMed
22.
23.
Cronin E, Sheldon T: Factors influencing the publication of health research. Int J Technol Assess Health Care. 2004, 20 (3): 351-355. 10.1017/S0266462304001175.CrossRefPubMed
24.
Decullier E, Lheritier V, Chapuis F: Fate of biomedical research protocols and publication bias in France: retrospective cohort study [see comment]. BMJ. 2005, 331 (7507): 19-10.1136/bmj.38488.385995.8F.CrossRefPubMedPubMedCentral
25.
Decullier E, Chapuis F: Impact of funding on biomedical research: a retrospective cohort study. BMC Public Health. 2006, 6: 165-10.1186/1471-2458-6-165.CrossRefPubMedPubMedCentral
26.
Dickersin K, Min YI, Meinert CL: Factors influencing publication of research results. Follow up of applications submitted to two institutional review boards. JAMA. 1992, 267 (3): 374-378. 10.1001/jama.267.3.374.CrossRefPubMed
27.
Dickersin K, Min YI: NIH clinical trials and publication bias. Online J Curr Clin Trials. 1993, Doc No 50 (3):
28.
Easterbrook PJ, Berlin JA, Gopalan R, Matthews DR: Publication bias in clinical research. Lancet. 1991, 337 (8746): 867-872. 10.1016/0140-6736(91)90201-Y.CrossRefPubMed
29.
Ioannidis J: Effect of the statistical significance of results on the time to completion and publication of randomized efficacy trials. JAMA. 1998, 279 (4): 281-286. 10.1001/jama.279.4.281.CrossRefPubMed
30.
Misakian AL, Bero LA: Publication bias and research on passive smoking: comparison of published and unpublished studies. JAMA. 1998, 280 (3): 250-253. 10.1001/jama.280.3.250.CrossRefPubMed
31.
32.
Wormald R, Bloom J, Evans J, Oldfled K: Publication bias in eye trials [abstract]. Second International Conference Scientific Basis of Health Science and 5th Annual Cochrane Colloquium. 1997, Amsterdam, The Netherlands: Cochrane Collaboration
33.
Zimpel T, Windeler J: [Publications of dissertations on unconventional medical therapy and diagnosis procedures--a contribution to "publication bias"]. Forschende Komplementarmedizin und Klassische Naturheilkunde. 2000, 7 (2): 71-74. 10.1159/000021324.CrossRefPubMed
34.
Lee K, Bacchetti P, Sim I: Publication of clinical trials supporting successful new drug applications: a literature analysis. PLoS Med. 2008, 5 (9): e191-10.1371/journal.pmed.0050191.CrossRefPubMedPubMedCentral
35.
Melander H, Ahlqvist-Rastad J, Meijer G, Beermann B: Evidence b(i)ased medicine--selective reporting from studies sponsored by pharmaceutical industry: review of studies in new drug applications. BMJ. 2003, 326 (7400): 1171-1173. 10.1136/bmj.326.7400.1171.CrossRefPubMedPubMedCentral
36.
Rising K, Bacchetti P, Bero L: Reporting bias in drug trials submitted to the Food and Drug Administration: review of publication and presentation. PLoS Med. 2008, 5 (11): e217-10.1371/journal.pmed.0050217.CrossRefPubMedPubMedCentral
37.
Turner EH, Matthews AM, Linardatos E, Tell RA, Rosenthal R: Selective publication of antidepressant trials and its influence on apparent efficacy. N Engl J Med. 2008, 358 (3): 252-260. 10.1056/NEJMsa065779.CrossRefPubMed
38.
Akbari-Kamrani M, Shakiba B, Parsian S: Transition from congress abstract to full publication for clinical trials presented at laser meetings. Lasers Med Sci. 2008, 23 (3): 295-299. 10.1007/s10103-007-0484-4.CrossRefPubMed
39.
Brazzelli M, Lewis SC, Deeks JJ, Sandercock PAG: No evidence of bias in the process of publication of diagnostic accuracy studies in stroke submitted as abstracts. J Clin Epidemiol. 2009, 62 (4): 425-430. 10.1016/j.jclinepi.2008.06.018.CrossRefPubMed
40.
Callaham ML, Wears RL, Weber EJ, Barton C, Young G: Positive-outcome bias and other limitations in the outcome of research abstracts submitted to a scientific meeting [erratum appears in JAMA 1998 Oct 14;280(14):1232]. JAMA. 1998, 280 (3): 254-257. 10.1001/jama.280.3.254.CrossRefPubMed
41.
Castillo J, Garcia Guasch R, Cifuentes I: Fate of abstracts from the Paris 1995 European Society of Anaesthesiologists meeting. Eur J Anaesthesiol. 2002, 19 (12): 888-893.CrossRefPubMed
42.
Cheng K, Preston C, Ashby D, O'Hea U, Smyth RL: Time to publication as full reports of abstracts of randomized controlled trials in cystic fibrosis. Pediatr Pulmonol. 1998, 26 (2): 101-105. 10.1002/(SICI)1099-0496(199808)26:2<101::AID-PPUL5>3.0.CO;2-P.CrossRefPubMed
43.
DeBellefeuille C, Morrison CA, Tannock IF: The fate of abstracts submitted to a cancer meeting: factors which influence presentation and subsequent publication. Ann Oncol. 1992, 3: 187-191.
44.
Delamere F, Williams H: To what extent are conference abstracts reporting randomised controlled trials of skin diseases published subsequently? [abstract]. 13th Cochrane Colloquium; 2005 Oct 22 26; Melbourne, Australia. 2005, 101-
45.
Eloubeidi MA, Wade SB, Provenzale D: Factors associated with acceptance and full publication of GI endoscopic research originally published in abstract form. Gastrointest Endosc. 2001, 53 (3): 275-282.CrossRefPubMed
46.
Evers JL: Publication bias in reproductive research. Hum Reprod. 2000, 15 (10): 2063-2066. 10.1093/humrep/15.10.2063.CrossRefPubMed
47.
Glick N, MacDonald I, Knoll G, Brabant A, Gourishankar S: Factors associated with publication following presentation at a transplantation meeting. Am J Transplant. 2006, 6 (3): 552-556. 10.1111/j.1600-6143.2005.01203.x.CrossRefPubMed
48.
Ha TH, Yoon DY, Goo DH, Chang SK, Seo YL, Yun EJ, Moon JH, Lee YJ, Lim KJ, Choi CS: Publication rates for abstracts presented by korean investigators at major radiology meetings. Korean J Radiol. 2008, 9 (4): 303-311. 10.3348/kjr.2008.9.4.303.CrossRefPubMedPubMedCentral
49.
Halpern SH, Palmer C, Angle P, Tarshis J: Published abstracts in obsteterical anesthesia: full publication rates and data reliability. Anesthesiology. 2001, 94 (1A): A69-
50.
Harris IA, Mourad M, Kadir A, Solomon MJ, Young JM: Publication bias in abstracts presented to the annual meeting of the American Academy of Orthopaedic Surgeons. J Orthop Surg (Hong Kong). 2007, 15 (1): 62-66.
51.
Harris IA, Mourad MS, Kadir A, Solomon MJ, Young JM: Publication bias in papers presented to the Australian Orthopaedic Association Annual Scientific Meeting. ANZ J Surg. 2006, 76 (6): 427-431. 10.1111/j.1445-2197.2006.03747.x.CrossRefPubMed
52.
Hashkes PJ, Uziel Y: The publication rate of abstracts from the 4th Park City Pediatric Rheumatology meeting in peer-reviewed journals: what factors influenced publication?. J Rheumatol. 2003, 30 (3): 597-602.PubMed
53.
Kiroff GK: Publication bias in presentations to the Annual Scientific Congress. ANZ J Surg. 2001, 71 (3): 167-171. 10.1046/j.1440-1622.2001.02058.x.CrossRefPubMed
54.
Klassen TPWNRKSKHLCWRMD: Abstracts of randomized controlled trials presented at the society for pediatric research meeting: an example of publication bias [comment]. Arch Pediatr Adolesc Med. 2002, 156 (5): 474-479.CrossRefPubMed
55.
Krzyzanowska MK, Pintilie M, Tannock IF: Factors associated with failure to publish large randomized trials presented at an oncology meeting. JAMA. 2003, 290 (4): 495-501. 10.1001/jama.290.4.495.CrossRefPubMed
56.
Landry VL: The publication outcome for the papers presented at the 1990 ABA conference. J Burn Care Rehabil. 1996, 17 (1): 23A-26A. 10.1097/00004630-199601000-00002.CrossRefPubMed
57.
Peng PH, Wasserman JM, Rosenfeld RM: Factors influencing publication of abstracts presented at the AAO-HNS Annual Meeting. Otolaryngol Head Neck Surg. 2006, 135 (2): 197-203. 10.1016/j.otohns.2006.04.006.CrossRefPubMed
58.
Petticrew M, Gilbody S, Song F: Lost information? The fate of papers presented at the 40th society for Social Medicine Conference. J Epidemiol Community Health. 1999, 53 (7): 442-443. 10.1136/jech.53.7.442.CrossRefPubMedPubMedCentral
59.
Sanossian N, Ohanian AG, Saver JL, Kim LI, Ovbiagele B: Frequency and determinants of nonpublication of research in the stroke literature. Stroke. 2006, 37 (10): 2588-2592. 10.1161/01.STR.0000240509.05587.a2.CrossRefPubMed
60.
Scherer RW, Dickersin K, Langenberg P: Full publication of results initially presented in abstracts. A meta-analysis. JAMA. 1994, 272: 158-162. 10.1001/jama.272.2.158.CrossRefPubMed
61.
Smith WA, Cancel QV, Tseng TY, Sultan S, Vieweg J, Dahm P: Factors associated with the full publication of studies presented in abstract form at the annual meeting of the American Urological Association. J Urol. 2007, 177 (3): 1084-1088. 10.1016/j.juro.2006.10.029. discussion 1088-1089.CrossRefPubMed
62.
Timmer A, Hilsden RJ, Cole J, Hailey D, Sutherland LR: Publication bias in gastroenterological research - a retrospective cohort study based on abstracts submitted to a scientific meeting. BMC Med Res Methodol. 2002, 2 (1): 7-10.1186/1471-2288-2-7.CrossRefPubMedPubMedCentral
63.
Vecchi S, Belleudi V, Amato L, Davoli M: Full publication of abstracts submitted at a major international conference on drug addiction: a longitudinal study [abstract]. 14th Cochrane Colloquium; 2006 October 23 26; Dublin, Ireland. 2006
64.
Zamakhshary M, Abuznadah W, Zacny J, Giacomantonio M: Research publication in pediatric surgery: a cross-sectional study of papers presented at the Canadian Association of Pediatric Surgeons and the American Pediatric Surgery Association. J Pediatr Surg. 2006, 41 (7): 1298-1301. 10.1016/j.jpedsurg.2006.03.042.CrossRefPubMed
65.
Zaretsky Y, Imrie K: The fate of phase III trial abstracts presented at the American Society of Hematology. Blood. 2002, 100 (11): 185a-
66.
Lee KP, Boyd EA, Holroyd-Leduc JM, Bacchetti P, Bero LA: Predictors of publication: characteristics of submitted manuscripts associated with acceptance at major biomedical journals. Med J Aust. 2006, 184 (12): 621-626.PubMed
67.
Lynch JR, Cunningham MRA, Warme WJ, Schaad DC, Wolf FM, Leopold SS: Commercially funded and United States-based research is more likely to be published; good-quality studies with negative outcomes are not. J Bone Joint Surg - Am. 2007, 89 (5): 1010-1018. 10.2106/JBJS.F.01152.CrossRefPubMed
68.
Okike K, Kocher MS, Mehlman CT, Heckman JD, Bhandari M: Publication bias in orthopaedic research: an analysis of scientific factors associated with publication in the Journal of Bone and Joint Surgery (American Volume). J Bone Joint Surg - Am. 2008, 90 (3): 595-601. 10.2106/JBJS.G.00279.CrossRefPubMed
69.
Olson CM, Rennie D, Cook D, Dickersin K, Flanagin A, Hogan JW, Zhu Q, Reiling J, Pace B: Publication bias in editorial decision making. JAMA. 2002, 287 (21): 2825-2828. 10.1001/jama.287.21.2825.CrossRefPubMed
70.
Blumle A, Antes G, Schumacher M, Just H, von Elm E: Clinical research projects at a German medical faculty: follow-up from ethical approval to publication and citation by others. J Med Ethics. 2008, 34 (9): e20-10.1136/jme.2008.024521.CrossRefPubMed
71.
Druss BG, Marcus SC: Tracking publication outcomes of National Institutes of Health grants. Am J Med. 2005, 118 (6): 658-663. 10.1016/j.amjmed.2005.02.015.CrossRefPubMed
72.
Cooper H, Charlton K: Finding the missing science: the fate of studies submitted for review by a human subjects committee. Psychological Methods. 1997, 2 (4): 447-452. 10.1037/1082-989X.2.4.447.CrossRef
73.
Hahn S, Williamson PR, Hutton JL: Investigation of within-study selective reporting in clinical research: follow-up of applications submitted to a local research ethics committee. J Eval Clin Pract. 2002, 8 (3): 353-359. 10.1046/j.1365-2753.2002.00314.x.CrossRefPubMed
74.
Pich J, Carne X, Arnaiz J-A, Gomez B, Trilla A, Rodes J: Role of a research ethics committee in follow-up and publication of results. Lancet. 2003, 361 (9362): 1015-1016. 10.1016/S0140-6736(03)12799-7.CrossRefPubMed
75.
von Elm E, Rollin A, Blumle A, Huwiler K, Witschi M, Egger M: Publication and non-publication of clinical trials: longitudinal study of applications submitted to a research ethics committee. Swiss Med Wkly. 2008, 138 (13-14): 197-203.PubMed
76.
Dickersin K, Ssemanda E, Mansell C, Rennie D: What do the JAMA editors say when they discuss manuscripts that they are considering for publication? Developing a schema for classifying the content of editorial discussion. BMC Med Res Methodol. 2007, 7: 44-10.1186/1471-2288-7-44.CrossRefPubMedPubMedCentral
77.
Calnan M, Smith GD, Sterne JAC: The publication process itself was the major cause of publication bias in genetic epidemiology. J Clin Epidemiol. 2006, 59 (12): 1312-1318. 10.1016/j.jclinepi.2006.05.002.CrossRefPubMed
78.
Laupacis A: Methodological studies of systematic reviews: is there publication bias?. Arch Intern Med. 1997, 157: 357-10.1001/archinte.157.3.357.CrossRefPubMed
79.
Dubben H-H, Beck-Bornholdt H-P: Systematic review of publication bias in studies on publication bias. BMJ. 2005, 331 (7514): 433-434. 10.1136/bmj.38478.497164.F7.CrossRefPubMedPubMedCentral
80.
Song F: Review of publication bias in studies on publication bias: studies on publication bias are probably susceptible to the bias they study. BMJ. 2005, 331 (7517): 637-638. 10.1136/bmj.331.7517.637-c.CrossRefPubMedPubMedCentral
81.
Hopewell S, Clarke M, Stewart L, Tierney J: Time to publication for results of clinical trials. Cochrane Database Syst Rev. 2007, MR000011-2
Metadata
Title
Extent of publication bias in different categories of research cohorts: a meta-analysis of empirical studies
Authors
Fujian Song
Sheetal Parekh-Bhurke
Lee Hooper
Yoon K Loke
Jon J Ryder
Alex J Sutton
Caroline B Hing
Ian Harvey
Publication date
01-12-2009
Publisher
BioMed Central
Published in
BMC Medical Research Methodology / Issue 1/2009
Electronic ISSN: 1471-2288
DOI
https://doi.org/10.1186/1471-2288-9-79

Other articles of this Issue 1/2009

BMC Medical Research Methodology 1/2009 Go to the issue

Research article

Calculation of NNTs in RCTs with time-to-event outcomes: A literature review

Research article

Extending an evidence hierarchy to include topics other than treatment: revising the Australian 'levels of evidence'

Research article

Stratification of the severity of critically ill patients with classification trees

Technical advance

New journal selection for quantitative survey of infectious disease research: application for Asian trend analysis

Research article

Design effect in multicenter studies: gain or loss of power?

Research article

Effectiveness and cost of recruitment strategies for a community-based randomised controlled trial among rainwater drinkers