Top

BMC Medicine

Published in:

Open Access 01-12-2015 | Research article

Comparison of registered and published outcomes in randomized controlled trials: a systematic review

Authors: Christopher W. Jones, Lukas G. Keil, Wesley C. Holland, Melissa C. Caughey, Timothy F. Platts-Mills

Published in: BMC Medicine | Issue 1/2015

Abstract

Background

Clinical trial registries can improve the validity of trial results by facilitating comparisons between prospectively planned and reported outcomes. Previous reports on the frequency of planned and reported outcome inconsistencies have reported widely discrepant results. It is unknown whether these discrepancies are due to differences between the included trials, or to methodological differences between studies. We aimed to systematically review the prevalence and nature of discrepancies between registered and published outcomes among clinical trials.

Methods

We searched MEDLINE via PubMed, EMBASE, and CINAHL, and checked references of included publications to identify studies that compared trial outcomes as documented in a publicly accessible clinical trials registry with published trial outcomes. Two authors independently selected eligible studies and performed data extraction. We present summary data rather than pooled analyses owing to methodological heterogeneity among the included studies.

Results

Twenty-seven studies were eligible for inclusion. The overall risk of bias among included studies was moderate to high. These studies assessed outcome agreement for a median of 65 individual trials (interquartile range [IQR] 25–110). The median proportion of trials with an identified discrepancy between the registered and published primary outcome was 31 %; substantial variability in the prevalence of these primary outcome discrepancies was observed among the included studies (range 0 % (0/66) to 100 % (1/1), IQR 17–45 %). We found less variability within the subset of studies that assessed the agreement between prospectively registered outcomes and published outcomes, among which the median observed discrepancy rate was 41 % (range 30 % (13/43) to 100 % (1/1), IQR 33–48 %). The nature of observed primary outcome discrepancies also varied substantially between included studies. Among the studies providing detailed descriptions of these outcome discrepancies, a median of 13 % of trials introduced a new, unregistered outcome in the published manuscript (IQR 5–16 %).

Conclusions

Discrepancies between registered and published outcomes of clinical trials are common regardless of funding mechanism or the journals in which they are published. Consistent reporting of prospectively defined outcomes and consistent utilization of registry data during the peer review process may improve the validity of clinical trial publications.

Available only for authorised users

Moher D, Hopewell S, Schulz KF, Montori V, Gotzsche PC, Devereaux PJ, et al. CONSORT 2010 explanation and elaboration: updated guidelines for reporting parallel group randomised trials. BMJ. 2010;340:c869. doi:10.1136/bmj.c869.PubMedCentralCrossRefPubMed

Newman DH, Shreves AE. Treatment of acute otitis media in children. N Engl J Med. 2011;364(18):1775. doi:10.1056/NEJMc1102207#SA1. author reply 7–8.CrossRefPubMed

DeAngelis CD, Drazen JM, Frizelle FA, Haug C, Hoey J, Horton R, et al. Clinical trial registration: a statement from the International Committee of Medical Journal Editors. JAMA. 2004;292(11):1363–4. doi:10.1001/jama.292.11.1363.CrossRefPubMed

Food and Drug Administration Act of 2007, Stat. 2007: 110–85.

Regulation (EU) No 536/2014 of the European Parliament and of the Council of 16 April 2014 on clinical trials on medicinal products for human use, and repealing Directive 2001/20/ED, Stat. 536/2014. 2014.

Califf RM, Zarin DA, Kramer JM, Sherman RE, Aberle LH, Tasneem A. Characteristics of clinical trials registered in ClinicalTrials.gov, 2007–2010. JAMA. 2012;307(17):1838–47. doi:10.1001/jama.2012.3424.CrossRefPubMed

Mathieu S, Boutron I, Moher D, Altman DG, Ravaud P. Comparison of registered and published primary outcomes in randomized controlled trials. JAMA. 2009;302(9):977–84. doi:10.1001/jama.2009.1242.CrossRefPubMed

Ross JS, Mulvey GK, Hines EM, Nissen SE, Krumholz HM. Trial publication after registration in ClinicalTrials.Gov: a cross-sectional analysis. PLoS Med. 2009;6(9):e1000144. doi:10.1371/journal.pmed.1000144.PubMedCentralCrossRefPubMed

Smith SM, Wang AT, Pereira A, Chang RD, McKeown A, Greene K, et al. Discrepancies between registered and published primary outcome specifications in analgesic trials: ACTTION systematic review and recommendations. Pain. 2013;154(12):2769–74. doi:10.1016/j.pain.2013.08.011.CrossRefPubMed

10.

Chan AW, Hrobjartsson A, Haahr MT, Gotzsche PC, Altman DG. Empirical evidence for selective reporting of outcomes in randomized trials: comparison of protocols to published articles. JAMA. 2004;291(20):2457–65. doi:10.1001/jama.291.20.2457.CrossRefPubMed

11.

Moher D, Liberati A, Tetzlaff J, Altman DG, PRISMA Group. Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. BMJ. 2009;339:b2535. doi:10.1136/bmj.b2535.PubMedCentralCrossRefPubMed

12.

Higgins J, Green S, (editors). Cochrane Handbook for Systematic Reviews of Interventions Version 5.1.0 [updated March 2011]. London, UK: The Cochrane Collaboration; 2011.

13.

Wells GA, Shea B, O’Connell D, Peterson J, Welch V, Losos M, et al. The Newcastle-Ottawa Scale (NOS) for assessing the quality of nonrandomised studies in meta-analyses. Ottawa Hospital Research Institute. 2014. http://www.ohri.ca/programs/clinical_epidemiology/oxford.asp. Accessed 24 September 2015.

14.

Newcombe RG. Two-sided confidence intervals for the single proportion: comparison of seven methods. Stat Med. 1998;17(8):857–72.CrossRefPubMed

15.

Ewart R, Lausen H, Millian N. Undisclosed changes in outcomes in randomized controlled trials: an observational study. Ann Fam Med. 2009;7(6):542–6. doi:10.1370/afm.1017.PubMedCentralCrossRefPubMed

16.

Anand V, Scales DC, Parshuram CS, Kavanagh BP. Registration and design alterations of clinical trials in critical care: a cross-sectional observational study. Intensive Care Med. 2014;40(5):700–22. doi:10.1007/s00134-014-3250-7.CrossRefPubMed

17.

Bourgeois FT, Murthy S, Mandl KD. Outcome reporting among drug trials registered in ClinicalTrials.gov. Ann Intern Med. 2010;153(3):158–66. doi:10.7326/0003-4819-153-3-201008030-00006.PubMedCentralCrossRefPubMed

18.

Chahal J, Tomescu SS, Ravi B, Bach Jr BR, Ogilvie-Harris D, Mohamed NN, et al. Publication of sports medicine-related randomized controlled trials registered in ClinicalTrials.gov. Am J Sports Med. 2012;40(9):1970–7. doi:10.1177/0363546512448363.CrossRefPubMed

19.

Gandhi R, Jan M, Smith HN, Mahomed NN, Bhandari M. Comparison of published orthopaedic trauma trials following registration in Clinicaltrials.gov. BMC Musculoskelet Disord. 2011;12:278. doi:10.1186/1471-2474-12-278.PubMedCentralCrossRefPubMed

20.

Hannink G, Gooszen HG, Rovers MM. Comparison of registered and published primary outcomes in randomized clinical trials of surgical interventions. Ann Surg. 2013;257(5):818–23. doi:10.1097/SLA.0b013e3182864fa3.CrossRefPubMed

21.

Hartung DM, Zarin DA, Guise JM, McDonagh M, Paynter R, Helfand M. Reporting discrepancies between the ClinicalTrials.gov results database and peer-reviewed publications. Ann Intern Med. 2014;160(7):477–83. doi:10.7326/M13-0480.PubMedCentralCrossRefPubMed

22.

Huic M, Marusic M, Marusic A. Completeness and changes in registered data and reporting bias of randomized controlled trials in ICMJE journals after trial registration policy. PLoS One. 2011;6(9):e25258. doi:10.1371/journal.pone.0025258.PubMedCentralCrossRefPubMed

23.

Hutfless S, Blair R, Berger Z, Wilson L, Bass EB, Lazarev M. Poor reporting of Crohn’s disease trials in ClinicalTrials.gov. Gastroenterology. 2013;144(5):SUPPL. 1(S232).CrossRef

24.

Jones CW, Platts-Mills TF. Quality of registration for clinical trials published in emergency medicine journals. Ann Emerg Med. 2012;60(4):458–64. doi:10.1016/j.annemergmed.2012.02.005.CrossRefPubMed

25.

Khan NA, Lombeida JI, Singh M, Spencer HJ, Torralba KD. Association of industry funding with the outcome and quality of randomized controlled trials of drug therapy for rheumatoid arthritis. Arthritis Rheum. 2012;64(7):2059–67. doi:10.1002/art.34393.PubMedCentralCrossRefPubMed

26.

Killeen S, Sourallous P, Hunter IA, Hartley JE, Grady HL. Registration rates, adequacy of registration, and a comparison of registered and published primary outcomes in randomized controlled trials published in surgery journals. Ann Surg. 2014;259(1):193–6. doi:10.1097/SLA.0b013e318299d00b.CrossRefPubMed

27.

Li XQ, Yang GL, Tao KM, Zhang HQ, Zhou QH, Ling CQ. Comparison of registered and published primary outcomes in randomized controlled trials of gastroenterology and hepatology. Scand J Gastroenterol. 2013;48(12):1474–83. doi:10.3109/00365521.2013.845909.CrossRefPubMed

28.

Liu JP, Han M, Li XX, Mu YJ, Lewith G, Wang YY, et al. Prospective registration, bias risk and outcome-reporting bias in randomised clinical trials of traditional Chinese medicine: an empirical methodological study. BMJ Open. 2013;3(7):pii: e002968. doi:10.1136/bmjopen-2013-002968.

29.

Mathieu S, Giraudeau B, Soubrier M, Ravaud P. Misleading abstract conclusions in randomized controlled trials in rheumatology: comparison of the abstract conclusions and the results section. Joint Bone Spine. 2012;79(3):262–7. doi:10.1016/j.jbspin.2011.05.008.CrossRefPubMed

30.

Milette K, Roseman M, Thombs BD. Transparency of outcome reporting and trial registration of randomized controlled trials in top psychosomatic and behavioral health journals: a systematic review. J Psychosom Res. 2011;70(3):205–17. doi:10.1016/j.jpsychores.2010.09.015.CrossRefPubMed

31.

Nankervis H, Baibergenova A, Williams HC, Thomas KS. Prospective registration and outcome-reporting bias in randomized controlled trials of eczema treatments: a systematic review. J Invest Dermatol. 2012;132(12):2727–34. doi:10.1038/jid.2012.231.PubMedCentralCrossRefPubMed

32.

Pinto RZ, Elkins MR, Moseley AM, Sherrington C, Herbert RD, Maher CG, et al. Many randomized trials of physical therapy interventions are not adequately registered: a survey of 200 published trials. Phys Ther. 2013;93(3):299–309. doi:10.2522/ptj.20120206.CrossRefPubMed

33.

Rosenthal R, Dwan K. Comparison of randomized controlled trial registry entries and content of reports in surgery journals. Ann Surg. 2013;257(6):1007–15. doi:10.1097/SLA.0b013e318283cf7f.CrossRefPubMed

34.

Smith HN, Bhandari M, Mahomed NN, Jan M, Gandhi R. Comparison of arthroplasty trial publications after registration in ClinicalTrials.gov. J Arthroplasty. 2012;27(7):1283–8. doi:10.1016/j.arth.2011.11.005.CrossRefPubMed

35.

Vera-Badillo FE, Shapiro R, Ocana A, Amir E, Tannock IF. Bias in reporting of end points of efficacy and toxicity in randomized, clinical trials for women with breast cancer. Ann Oncol. 2013;24(5):1238–44. doi:10.1093/annonc/mds636.CrossRefPubMed

36.

Walker KF, Stevenson G, Thornton JG. Discrepancies between registration and publication of randomised controlled trials: an observational study. JRSM Open. 2014;5(5):2042533313517688.PubMedCentralCrossRefPubMed

37.

Wildt S, Krag A, Gluud L. Characteristics of randomised trials on diseases in the digestive system registered in ClinicalTrials.gov: a retrospective analysis. BMJ Open. 2011;1(2):e000309. doi:10.1136/bmjopen-2011-000309.PubMedCentralCrossRefPubMed

38.

You B, Gan HK, Pond G, Chen EX. Consistency in the analysis and reporting of primary end points in oncology randomized controlled trials from registration to publication: a systematic review. J Clin Oncol. 2012;30(2):210–6. doi:10.1200/JCO.2011.37.0890.CrossRefPubMed

39.

Mathieu S, Chan AW, Ravaud P. Use of trial register information during the peer review process. PLoS One. 2013;8(4):e59910. doi:10.1371/journal.pone.0059910.PubMedCentralCrossRefPubMed

40.

Chauvin A, Ravaud P, Baron G, Barnes C, Boutron I. The most important tasks for peer reviewers evaluating a randomized controlled trial are not congruent with the tasks most often requested by journal editors. BMC Med. 2015;13:158. doi:10.1186/s12916-015-0395-3.PubMedCentralCrossRefPubMed

41.

Zarin DA, Tse T, Williams RJ, Califf RM, Ide NC. The ClinicalTrials.gov results database--update and key issues. N Engl J Med. 2011;364(9):852–60. doi:10.1056/NEJMsa1012065.PubMedCentralCrossRefPubMed

42.

Dwan K, Altman DG, Cresswell L, Blundell M, Gamble CL, Williamson PR. Comparison of protocols and registry entries to published reports for randomised controlled trials. Cochrane Database Syst Rev. 2011;1:MR000031. doi:10.1002/14651858.MR000031.pub2.PubMed

Title: Comparison of registered and published outcomes in randomized controlled trials: a systematic review
Authors: Christopher W. Jones
Lukas G. Keil
Wesley C. Holland
Melissa C. Caughey
Timothy F. Platts-Mills
Publication date: 01-12-2015
Publisher: BioMed Central
Published in: BMC Medicine / Issue 1/2015
Electronic ISSN: 1741-7015
DOI: https://doi.org/10.1186/s12916-015-0520-3

At a glance: The STEP trials

Springer Medicine

Comparison of registered and published outcomes in randomized controlled trials: a systematic review

Abstract

Background

Methods

Results

Conclusions

At a glance: The STEP trials

Springer Medicine

Abstract

Background

Methods

Results

Conclusions

Please log in to get access to this content

Other articles of this Issue 1/2015

Incorporating genetics into the identification and treatment of Idiopathic Pulmonary Fibrosis

A randomised controlled trial of three or one breathing technique training sessions for breathlessness in people with malignant lung disease

What is the evidence base for diagnosing hypertension and for subsequent blood pressure treatment targets in the prevention of cardiovascular disease?

Extending the elderly- and risk-group programme of vaccination against seasonal influenza in England and Wales: a cost-effectiveness study

The most important tasks for peer reviewers evaluating a randomized controlled trial are not congruent with the tasks most often requested by journal editors

Molecular essence and endocrine responsiveness of estrogen receptor-negative, progesterone receptor-positive, and HER2-negative breast cancer