Skip to main content
Key Specialties
Cardiology Diabetology Endocrinology Gastroenterology Geriatrics and Gerontology Gynecology and Obstetrics Hematology Infectious Disease Internal Medicine Nephrology Neurology Oncology Pediatrics Respiratory Medicine Rheumatology Surgery
Congress Coverage
2024 ACC Congress 2023 ESMO Congress 2023 EASD Congress 2023 ERS Congress 2023 ESC Congress
Clinical Collections
Advances in Alzheimer’s

At a glance: The STEP trials

A round-up of the STEP phase 3 clinical trials evaluating semaglutide for weight loss in people with overweight or obesity.
ADVANCED SEARCH
Log in
Top
Trials
Published in: Trials 1/2018

Open Access 01-12-2018 | Commentary

Caveat emptor: the combined effects of multiplicity and selective reporting

Authors: Tianjing Li, Evan Mayo-Wilson, Nicole Fusco, Hwanhee Hong, Kay Dickersin

Published in: Trials | Issue 1/2018

Login to get access

Abstract

Clinical trials and systematic reviews of clinical trials inform healthcare decisions. There is growing concern, however, about results from clinical trials that cannot be reproduced. Reasons for nonreproducibility include that outcomes are defined in multiple ways, results can be obtained using multiple methods of analysis, and trial findings are reported in multiple sources (“multiplicity”). Multiplicity combined with selective reporting can influence dissemination of trial findings and decision-making. In particular, users of evidence might be misled by exposure to selected sources and overly optimistic representations of intervention effects. In this commentary, drawing from our experience in the Multiple Data Sources in Systematic Reviews (MUDS) study and evidence from previous research, we offer practical recommendations to enhance the reproducibility of clinical trials and systematic reviews.
Literature
1.
Ebrahim S, Sohani ZN, Montoya L, Agarwal A, Thorlund K, Mills EJ, Ioannidis JP. Reanalyses of randomized clinical trial data. JAMA. 2014;312(10):1024–32.CrossRef
2.
Glasziou P, Altman DG, Bossuyt P, Boutron I, Clarke M, Julious S, Michie S, Moher D, Wager E. Reducing waste from incomplete or unusable reports of biomedical research. Lancet. 2014;383(9913):267–76.CrossRef
3.
4.
Dwan K, Gamble C, Williamson PR, Kirkham JJ. Reporting Bias Group. Systematic review of the empirical evidence of study publication bias and outcome reporting bias — an updated review. PLoS One. 2013;8(7):e66844.CrossRef
5.
Institute of Medicine. Sharing clinical trial data: maximizing benefits, minimizing risk. Washington, DC: National Academies Press; 2015.
6.
Nosek BA, Alter G, Banks GC, et al. Scientific standards: promoting an open research culture. Science. 2015;348(6242):1422–5.CrossRef
7.
DeAngelis CD, Drazen JM, Frizelle FA, Haug C, Hoey J, Horton R, Kotzin S, Laine C, Marusic A, Overbeke AJ, Schroeder TV, Sox HC, Van Der Weyden MB. International Committee of Medical Journal Editors. Clinical trial registration: a statement from the International Committee of Medical Journal Editors. JAMA. 2004;292(11):1363–4.CrossRef
8.
Zarin DA, Tse T, Williams RJ, Rajakannan T. Update on trial registration 11 years after the ICMJE policy was established. N Engl J Med. 2017;376(4):383–91.CrossRef
9.
Food and Drug Administration Amendments Act (FDAAA), 42 USC §801. 2007.
10.
National Institutes of Health. NIH policy on dissemination of NIH-funded clinical trial information. Fed Regist. 2016;81(183):64922.
11.
Clinical trials registration and results information submission: final rule. 42 CFR 11. 2016.
12.
Hudson KL, Lauer MS, Collins FS. Toward a new era of trust and transparency in clinical trials. JAMA. 2016;316(13):1353–4.CrossRef
13.
Zarin DA, Tse T, Williams RJ, Carr S. Trial reporting in ClinicalTrials.gov — the final rule. N Engl J Med. 2016;375(20):1998–2004.CrossRef
14.
15.
Mayo-Wilson E, Hutfless S, Li T, Gresham G, Fusco N, Ehmsen J, Heyward J, Vedula S, Lock D, Haythornthwaite J, Payne JL, Cowley T, Tolbert E, Rosman L, Twose C, Stuart EA, Hong H, Doshi P, Suarez-Cuervo C, Singh S, Dickersin K. Integrating multiple data sources (MUDS) for meta-analysis to improve patient-centered outcomes research. Syst Rev. 2015;4:143.CrossRef
16.
Mayo-Wilson E, Doshi P, Dickersin K. Are manufacturers sharing data as promised? BMJ. 2015;351:h4169.CrossRef
17.
Mayo-Wilson E, Fusco N, Li T, Hong H, Canner J, Dickersin K. MUDS Investigators. Multiple outcomes and analyses in clinical trials create challenges for interpretation and research synthesis. J Clin Epidemiol. 2017;86:39–50.CrossRef
18.
Mayo-Wilson E, Li T, Fusco N, Bertizzolo L, Canner JK, Cowley T, Doshi P, Ehmsen J, Gresham G, Guo N, Haythornthwaite JA, Heyward J, Hong H, Pham D, Payne JL, Rosman L, Stuart EA, Suarez-Cuervo C, Tolbert E, Twose C, Vedula S, Dickersin K. Cherry-picking by trialists and meta-analysts can drive conclusions about intervention efficacy. J Clin Epidemiol. 2017;91:95–110.CrossRef
19.
Mayo-Wilson E, Li T, Fusco N, Dickersin K. MUDS investigators. Practical guidance for using multiple data sources in systematic reviews and meta-analyses (with examples from the MUDS study). Res Synth Methods. 2018;9(1):2–12.CrossRef
20.
Meinert CL. Clinical trials dictionary: terminology and usage recommendations. 2nd ed. Hoboken, NJ: Wiley; 2012.CrossRef
21.
Williamson PR, Altman DG, Blazeby JM, Clarke M, Devane D, Gargon E, Tugwell P. Developing core outcome sets for clinical trials: issues to consider. Trials. 2012;13:132.CrossRef
22.
Saldanha IJ, Dickersin K, Wang X, Li T. Outcomes in Cochrane systematic reviews addressing four common eye conditions: an evaluation of completeness and comparability. PLoS One. 2014;9(10):e109400.CrossRef
23.
Zarin DA, Tse T, Williams RJ, Califf RM, Ide NC. The ClinicalTrials.gov results database — update and key issues. N Engl J Med. 2011;364(9):852–60.CrossRef
24.
Dosenovic S, Jelicic Kadic A, Jeric M, Boric M, Markovic D, Vucic K, Puljak L. Efficacy and safety outcome domains and outcome measures in systematic reviews of neuropathic pain conditions. Clin J Pain. 2018;34(7):674–84.PubMed
25.
Trudeau J, Van Inwegen R, Eaton T, Bhat G, Paillard F, Ng D, Tan K, Katz NP. Assessment of pain and activity using an electronic pain diary and actigraphy device in a randomized, placebo-controlled crossover trial of celecoxib in osteoarthritis of the knee. Pain Pract. 2015;15(3):247–55.CrossRef
26.
Page MJ, McKenzie JE, Kirkham J, Dwan K, Kramer S, Green S, Forbes A. Bias due to selective inclusion and reporting of outcomes and analyses in systematic reviews of randomised trials of healthcare interventions. Cochrane Database Syst Rev. 2014;10:MR000035.
27.
Page MJ, McKenzie JE, Chau M, Green SE, Forbes A. Methods to select results to include in meta-analyses deserve more consideration in systematic reviews. J Clin Epidemiol. 2015;68(11):1282–91.CrossRef
28.
Saldanha IJ, Li T, Yang C, Owczarzak J, Williamson PR, Dickersin K. Clinical trials and systematic reviews addressing similar interventions for the same condition do not consider similar outcomes to be important: a case study in HIV/AIDS. J Clin Epidemiol. 2017;84:85–94.CrossRef
29.
Saldanha IJ, Lindsley K, Do DV, Chuck RS, Meyerle C, Jones L, Coleman AL, Jampel HJ, Dickersin K, Virgili G. Comparison of clinical trials and systematic review outcomes for the 4 most prevalent eye diseases. JAMA Ophthalmol. 2017;135(9):933–40.CrossRef
30.
Chalmers I, Glasziou P. Avoidable waste in the production and reporting of research evidence. Lancet. 2009;374(9683):86–9.CrossRef
31.
32.
33.
Clearfield E, Money S, Saldanha I, Chuck R, Lindsley K. Outcome choice and potential loss of valuable information - an example from a Cochrane Eyes and Vision systematic review. Abstracts of the Global Evidence Summit, Cape Town, South Africa. Cochrane Database Syst Rev. 2017;9(Suppl 1). https://​doi.​org/​10.​1002/​14651858.​CD201702.
34.
Song F, Parekh S, Hooper L, Loke YK, Ryder J, Sutton AJ, Hing C, Kwok CS, Pang C, Harvey I. Dissemination and publication of research findings: an updated review of related biases. Health Technol Assess. 2010;14(8). https://​doi.​org/​10.​3310/​hta14080.
35.
Sterne JA, Sutton AJ, Ioannidis JP, Terrin N, Jones DR, Lau J, Carpenter J, Rücker G, Harbord RM, Schmid CH, Tetzlaff J, Deeks JJ, Peters J, Macaskill P, Schwarzer G, Duval S, Altman DG, Moher D, Higgins JP. Recommendations for examining and interpreting funnel plot asymmetry in meta-analyses of randomised controlled trials. BMJ. 2011;343:d4002.CrossRef
36.
Vedula SS, Li T, Dickersin K. Difference in reporting of analyses in internal company documents versus published trial reports: comparisons in industry-sponsored trials in off-label uses of gabapentin. PLoS Med. 2013;10(1):e1991378.CrossRef
37.
Gamble C, Krishan A, Stocken D, Lewis S, Juszczak E, Doré C, Williamson PR, Altman DG, Montgomery A, Lim P, Berlin J, Senn S, Day S, Barbachano Y, Loder E. Guidelines for the content of statistical analysis plans in clinical trials. JAMA. 2017;318(23):2337–43.CrossRef
38.
39.
Finfer S, Bellomo R. Why publish statistical analysis plans? Crit Care Resusc. 2009;11(1):5–6.PubMed
40.
41.
JPT H, Deeks JJ. Chapter 7: Selecting studies and collecting data. In: JPT H, Green S, editors. Cochrane handbook for systematic reviews of interventions. Version 5.1.0 (updated March 2011): The Cochrane Collaboration; 2011. Available from https://​training.​cochrane.​org/​handbook. Accessed 11 Sept 2018.
42.
Institute of Medicine. Finding what works in health care: standards for systematic reviews. Washington, DC: National Academies Press; 2011.
43.
Tendal B, Nuesch E, Higgins JP, Juni P, Gotzsche PC. Multiplicity of data in trial reports and the reliability of meta-analyses: Empirical study. BMJ. 2011;343:d4829.CrossRef
Metadata
Title
Caveat emptor: the combined effects of multiplicity and selective reporting
Authors
Tianjing Li
Evan Mayo-Wilson
Nicole Fusco
Hwanhee Hong
Kay Dickersin
Publication date
01-12-2018
Publisher
BioMed Central
Published in
Trials / Issue 1/2018
Electronic ISSN: 1745-6215
DOI
https://doi.org/10.1186/s13063-018-2888-9

Other articles of this Issue 1/2018

Trials 1/2018 Go to the issue

Study protocol

Xue-Fu-Zhu-Yu capsule in the treatment of qi stagnation and blood stasis syndrome: a study protocol for a randomised controlled pilot and feasibility trial

Research

‘Recruitment, recruitment, recruitment’ – the need for more focus on retention: a qualitative study of five trials

Study protocol

Clinical- and cost-effectiveness of the STAR care pathway compared to usual care for patients with chronic pain after total knee replacement: study protocol for a UK randomised controlled trial

Study protocol

Quality of life, satisfaction and outcomes after ministernotomy versus full sternotomy isolated aortic valve replacement (QUALITY-AVR): study protocol for a randomised controlled trial

Study protocol

Optimizing treatment protocols for spinal manipulative therapy: study protocol for a randomized trial

Study protocol

Stopping or maintaining oral anticoagulation in patients undergoing photoselective vaporization of the prostate (SOAP) surgery for benign prostate obstruction: study protocol for a multicentre randomized controlled trial