Skip to main content
Key Specialties
Cardiology Diabetology Endocrinology Gastroenterology Geriatrics and Gerontology Gynecology and Obstetrics Hematology Infectious Disease Internal Medicine Nephrology Neurology Oncology Pediatrics Respiratory Medicine Rheumatology Surgery
Congress Coverage
2024 ACC Congress 2023 ESMO Congress 2023 EASD Congress 2023 ERS Congress 2023 ESC Congress 2023 EHA Congress 2023 ASCO Annual Meeting
Clinical Collections
Advances in Alzheimer’s

At a glance: The ONWARDS insulin icodec trials

A round-up of the ONWARDS phase 3 clinical trials evaluating the novel weekly insulin icodec in people with diabetes.
ADVANCED SEARCH
Log in
Top
BMC Medicine
Published in: BMC Medicine 1/2018

Open Access 01-12-2018 | Research article

Avoidable waste of research related to outcome planning and reporting in clinical trials

Authors: Youri Yordanov, Agnes Dechartres, Ignacio Atal, Viet-Thi Tran, Isabelle Boutron, Perrine Crequit, Philippe Ravaud

Published in: BMC Medicine | Issue 1/2018

Login to get access

Abstract

Background

Inadequate planning, selective reporting, and incomplete reporting of outcomes in randomized controlled trials (RCTs) contribute to the problem of waste of research. We aimed to describe such a waste and to examine to what extent this waste could be avoided.

Methods

This research-on-research study was based on RCTs included in Cochrane reviews with a summary of findings (SoF) table. We considered the outcomes reported in the SoF tables as surrogates for important outcomes for patients and other decision makers. We used a three-step approach. (1) First, in each review, we identified, for each important outcome, RCTs that were excluded from the corresponding meta-analysis. (2) Then, for these RCTs, we systematically searched for registrations and protocols to distinguish between inadequate planning (an important outcome was not reported in registries or protocols), selective reporting (an important outcome was reported in registries or protocols but not in publications), and incomplete reporting (an important outcome was incompletely reported in publications). (3) Finally, we assessed, with the consensus of five experts, the feasibility and cost of measuring the important outcomes that were not planned. We considered inadequately planned or selectively or incompletely reported important outcomes as avoidable waste if the outcome could have been easily measured at no additional cost based on expert evaluation.

Results

Of the 2711 RCTs included in the main comparison of 290 reviews, 2115 (78%) were excluded from at least one meta-analysis of important outcomes. Every trial contributed to 55%, on average, of the meta-analyses of important outcomes. Of the 310 RCTs published in 2010 or later, 156 were registered. Inadequate planning affected 79% of these RCTs, whereas incomplete and selective reporting affected 41% and 15%, respectively. For 63% of RCTs, we found at least one missing important outcome for which the waste was avoidable and for 30%, the waste was avoidable for all important outcomes.

Conclusions

Most of the RCTs included in our sample did not contribute to all the important outcomes in meta-analyses, mostly because of inadequate planning or incomplete reporting. A large part of this waste of research seemed to be avoidable.
Appendix
Available only for authorised users
Literature
1.
Tugwell P, Boers M. OMERACT conference on outcome measures in rheumatoid arthritis clinical trials: introduction. J Rheumatol. 1993;20(3):528–30.PubMed
2.
Frank L, Basch E, Selby JV. Patient-centered outcomes research I: the PCORI perspective on patient-centered outcomes research. Jama. 2014;312(15):1513–4.CrossRefPubMed
3.
4.
Williamson P, Altman D, Blazeby J, Clarke M, Gargon E. Driving up the quality and relevance of research through the use of agreed core outcomes. J Health Serv Res Policy. 2012;17(1):1–2.CrossRefPubMed
5.
Tunis SR, Clarke M, Gorst SL, Gargon E, Blazeby JM, Altman DG, Williamson PR. Improving the relevance and consistency of outcomes in comparative effectiveness research. J Comp Eff Res. 2016;5(2):193-205.
6.
Murad MH, Shah ND, Van Houten HK, Ziegenfuss JY, Deming JR, Beebe TJ, Smith SA, Guyatt GH, Montori VM. Individuals with diabetes preferred that future trials use patient-important outcomes and provide pragmatic inferences. J Clin Epidemiol. 2011;64(7):743–8.CrossRefPubMed
7.
Higgins JPT, Green S (editors). Cochrane Handbook for Systematic Reviews of Interventions Version 5.1.0 [updated March 2011]. The Cochrane Collaboration, 2011. Available from http://​handbook.​cochrane.​org.
8.
9.
Chalmers I, Glasziou P. Avoidable waste in the production and reporting of research evidence. Lancet. 2009;374(9683):86–9.CrossRefPubMed
10.
Al-Shahi Salman R, Beller E, Kagan J, Hemminki E, Phillips RS, Savulescu J, Macleod M, Wisely J, Chalmers I. Increasing value and reducing waste in biomedical research regulation and management. Lancet. 2014;383(9912):176–85.CrossRefPubMed
11.
Chalmers I, Bracken MB, Djulbegovic B, Garattini S, Grant J, Gulmezoglu AM, Howells DW, Ioannidis JP, Oliver S. How to increase value and reduce waste when research priorities are set. Lancet. 2014;383(9912):156–65.CrossRefPubMed
12.
Chan AW, Song F, Vickers A, Jefferson T, Dickersin K, Gotzsche PC, Krumholz HM, Ghersi D, van der Worp HB. Increasing value and reducing waste: addressing inaccessible research. Lancet. 2014;383(9913):257–66.CrossRefPubMedPubMedCentral
13.
Glasziou P, Altman DG, Bossuyt P, Boutron I, Clarke M, Julious S, Michie S, Moher D, Wager E. Reducing waste from incomplete or unusable reports of biomedical research. Lancet. 2014;383(9913):267–76.CrossRefPubMed
14.
Ioannidis JP, Greenland S, Hlatky MA, Khoury MJ, Macleod MR, Moher D, Schulz KF, Tibshirani R. Increasing value and reducing waste in research design, conduct, and analysis. Lancet. 2014;383(9912):166–75.CrossRefPubMedPubMedCentral
15.
Macleod MR, Michie S, Roberts I, Dirnagl U, Chalmers I, Ioannidis JP, Al-Shahi Salman R, Chan AW, Glasziou P. Biomedical research: increasing value, reducing waste. Lancet. 2014;383(9912):101–4.CrossRefPubMed
16.
Moher D, Glasziou P, Chalmers I, Nasser M, Bossuyt PMM, Korevaar DA, Graham ID, Ravaud P, Boutron I. Increasing value and reducing waste in biomedical research: who's listening? Lancet. 2016;387(10027):1573–86.CrossRefPubMed
17.
Yordanov Y, Dechartres A, Porcher R, Boutron I, Altman DG, Ravaud P. Avoidable waste of research related to inadequate methods in clinical trials. BMJ. 2015;350:h809.CrossRefPubMedPubMedCentral
18.
Smith V, Clarke M, Williamson P, Gargon E. Survey of new 2007 and 2011 Cochrane reviews found 37% of prespecified outcomes not reported. J Clin Epidemiol. 2014;68(3):237–45.
19.
Gandhi GY, Murad MH, Fujiyoshi A, Mullan RJ, Flynn DN, Elamin MB, Swiglo BA, Isley WL, Guyatt GH, Montori VM. Patient-important outcomes in registered diabetes trials. JAMA. 2008;299(21):2543–9.CrossRefPubMed
20.
Rahimi K, Malhotra A, Banning AP, Jenkinson C. Outcome selection and role of patient reported outcomes in contemporary cardiovascular trials: systematic review. BMJ. 2010;341:c5707.CrossRefPubMedPubMedCentral
21.
Svensson S, Menkes DB, Lexchin J. Surrogate outcomes in clinical trials: a cautionary tale. JAMA Intern Med. 2013;173(8):611–2.CrossRefPubMed
22.
Naci H, Ioannidis JP. How good is “evidence” from clinical studies of drug effects and why might such evidence fail in the prediction of the clinical utility of drugs? Annu Rev Pharmacol Toxicol. 2015;55:169–89.CrossRefPubMed
23.
Ocana A, Tannock IF. When are “positive” clinical trials in oncology truly positive? J Natl Cancer Inst. 2010;103(1):16–20.CrossRefPubMed
24.
25.
The Cardiac Arrhythmia Suppression Trial (CAST) Investigators. Preliminary report: effect of encainide and flecainide on mortality in a randomized trial of arrhythmia suppression after myocardial infarction. N Engl J Med. 1989;321(6):406–12.CrossRef
26.
Ciani O, Buyse M, Garside R, Pavey T, Stein K, Sterne JA, Taylor RS. Comparison of treatment effect sizes associated with surrogate and final patient relevant outcomes in randomised controlled trials: meta-epidemiological study. BMJ. 2013;346:f457.CrossRefPubMedPubMedCentral
27.
Moynihan R. Surrogates under scrutiny: fallible correlations, fatal consequences. BMJ. 2011;343:d5160.CrossRefPubMed
28.
Walter SD, Sun X, Heels-Ansdell D, Guyatt G. Treatment effects on patient-important outcomes can be small, even with large effects on surrogate markers. J Clin Epidemiol. 2012;65(9):940–5.CrossRefPubMed
29.
Chan AW, Hrobjartsson A, Haahr MT, Gotzsche PC, Altman DG. Empirical evidence for selective reporting of outcomes in randomized trials: comparison of protocols to published articles. JAMA. 2004;291(20):2457–65.CrossRefPubMed
30.
Dwan K, Altman DG, Cresswell L, Blundell M, Gamble CL, Williamson PR. Comparison of protocols and registry entries to published reports for randomised controlled trials. Cochrane Database Syst Rev. 2011;1:MR000031.
31.
Mathieu S, Boutron I, Moher D, Altman DG, Ravaud P. Comparison of registered and published primary outcomes in randomized controlled trials. JAMA. 2009;302(9):977–84.CrossRefPubMed
32.
Williamson PR, Gamble C. Identification and impact of outcome selection bias in meta-analysis. Stat Med. 2005;24(10):1547–61.CrossRefPubMed
33.
Kirkham JJ, Dwan KM, Altman DG, Gamble C, Dodd S, Smyth R, Williamson PR. The impact of outcome reporting bias in randomised controlled trials on a cohort of systematic reviews. BMJ. 2010;340:c365.CrossRefPubMed
34.
Jones CW, Keil LG, Holland WC, Caughey MC, Platts-Mills TF. Comparison of registered and published outcomes in randomized controlled trials: a systematic review. BMC Med. 2015;13:282.CrossRefPubMedPubMedCentral
35.
Chan AW, Krleza-Jeric K, Schmid I, Altman DG. Outcome reporting bias in randomized trials funded by the Canadian Institutes of Health Research. CMAJ. 2004;171(7):735–40.CrossRefPubMedPubMedCentral
36.
Chan A-W, Altman DG. Identifying outcome reporting bias in randomised trials on PubMed: review of publications and survey of authors. BMJ. 2005;330(7494):753.CrossRefPubMedPubMedCentral
37.
Dechartres A, Ravaud P, Atal I, Riveros C, Boutron I. Association between trial registration and treatment effect estimates: a meta-epidemiological study. BMC Med. 2016;14(1):100.CrossRefPubMedPubMedCentral
38.
Mocellin S, Lens MB, Pasquali S, Pilati P, Chiarion Sileni V. Interferon alpha for the adjuvant treatment of cutaneous melanoma. Cochrane Database Syst Rev. 2013;6:CD008955.
39.
Ameur H, Ravaud P, Fayard F, Riveros C, Dechartres A. Systematic reviews of therapeutic interventions frequently consider patient-important outcomes. J Clin Epidemiol. 2017;84:70–77.
40.
Furukawa TA, Watanabe N, Omori IM, Montori VM, Guyatt GH. Association between unreported outcomes and effect size estimates in Cochrane meta-analyses. JAMA. 2007;297(5):468–70.CrossRefPubMed
41.
Wuytack F, Smith V, Clarke M, Williamson P, Gargon E. Towards core outcome set (COS) development: a follow-up descriptive survey of outcomes in Cochrane reviews. Syst Rev. 2015;4(1):73.CrossRefPubMedPubMedCentral
42.
43.
Methodology Committee of the Patient-Centered Outcomes Research I. Methodological standards and patient-centeredness in comparative effectiveness research: the PCORI perspective. JAMA. 2012;307(15):1636–40.CrossRef
44.
Selby JV, Beal AC, Frank L. The Patient-Centered Outcomes Research Institute (PCORI) national priorities for research and initial research agenda. JAMA. 2012;307(15):1583–4.CrossRefPubMed
45.
46.
Gargon E, Williamson PR, Altman DG, Blazeby JM, Tunis S, Clarke M. The COMET initiative database: progress and activities update (2015). Trials. 2017;18(1):54.CrossRefPubMedPubMedCentral
47.
48.
Gargon E, Williamson PR, Altman DG, Blazeby JM, Clarke M. The COMET initiative database: progress and activities from 2011 to 2013. Trials. 2014;15:279.CrossRefPubMedPubMedCentral
49.
Riveros C, Dechartres A, Perrodeau E, Haneef R, Boutron I, Ravaud P. Timing and completeness of trial results posted at ClinicalTrials.gov and published in journals. PLoS Med. 2013;10(12):e1001566.CrossRefPubMedPubMedCentral
50.
Peters JL, Mengersen KL. Meta-analysis of repeated measures study designs. J Eval Clin Pract. 2008;14(5):941–50.CrossRefPubMed
51.
Useem J, Brennan A, LaValley M, Vickery M, Ameli O, Reinen N, Gill CJ. Systematic differences between Cochrane and non-Cochrane meta-analyses on the same topic: a matched pair analysis. PLoS One. 2015;10(12):e0144980.CrossRefPubMedPubMedCentral
Metadata
Title
Avoidable waste of research related to outcome planning and reporting in clinical trials
Authors
Youri Yordanov
Agnes Dechartres
Ignacio Atal
Viet-Thi Tran
Isabelle Boutron
Perrine Crequit
Philippe Ravaud
Publication date
01-12-2018
Publisher
BioMed Central
Published in
BMC Medicine / Issue 1/2018
Electronic ISSN: 1741-7015
DOI
https://doi.org/10.1186/s12916-018-1083-x

Other articles of this Issue 1/2018

BMC Medicine 1/2018 Go to the issue

Research article

Four-year effects of exercise on fatigue and physical activity in patients with cancer

Research article

Built environmental characteristics and diabetes: a systematic review and meta-analysis

Research article

Analysing the role of complexity in explaining the fortunes of technology programmes: empirical application of the NASSS framework

Research article

Efficacy and safety of methylene blue in the treatment of malaria: a systematic review

Research article

Estimating the distribution of morbidity and mortality of childhood diarrhea, measles, and pneumonia by wealth group in low- and middle-income countries

Correspondence

The SMILES trial: an important first step