Skip to main content
Top
Published in: Trials 1/2017

Open Access 01-12-2017 | Commentary

Why prudence is needed when interpreting articles reporting clinical trial results in mental health

Authors: Rafael Dal-Ré, Julio Bobes, Pim Cuijpers

Published in: Trials | Issue 1/2017

Login to get access

Abstract

Background

Clinical trial results’ reliability is impacted by reporting bias. This is primarily manifested as publication bias and outcome reporting bias.

Mental health trials’ specific features

Mental health trials are prone to two methodological deficiencies: (1) using small numbers of participants that facilitates false positive findings and exaggerated size effects, and (2) the obligatory use of psychometric scales that require subjective assessments. These two deficiencies contribute to the publication of unreliable results. Considerable reporting bias has been found in safety and efficacy findings in psychotherapy and pharmacotherapy trials. Reporting bias can be carried forward to meta-analyses, a key source for clinical practice guidelines. The final result is the frequent overestimation of treatment effects that could impact patients and clinician-informed decisions.

Mechanisms to prevent outcome reporting bias

Prospective registration of trials and publication of results are the two major methods to reduce reporting bias. Prospective trial registration will allow checking whether they are published (so it will help to prevent publication bias) and, if published, whether those outcomes and analyses that were deemed as appropriate before trial commencement are actually published (hence helping to find out selective reporting of outcomes). Unfortunately, the rate of registered trials in mental health interventions is low and, frequently, of poor quality.

Conclusion

Clinicians should be prudent when interpreting the results of published trials and some meta-analyses – such as those conducted by scientists working for the sponsor company or those that only include published trials. Prescribers, however, should be confident when prescribing drugs following the summary of product characteristics, since regulatory agencies have access to all clinical trial results.
Literature
1.
go back to reference Kirkham JJ, Dwan KM, Altman DG, Gamble C, Dodd S, Smyth R, et al. The impact of outcome reporting bias in randomised controlled trials on a cohort of systematic reviews. BMJ. 2010;340:c365.CrossRefPubMed Kirkham JJ, Dwan KM, Altman DG, Gamble C, Dodd S, Smyth R, et al. The impact of outcome reporting bias in randomised controlled trials on a cohort of systematic reviews. BMJ. 2010;340:c365.CrossRefPubMed
2.
go back to reference Saini P, Loke YK, Gamble C, Altman DG, Williamson PR, Kirkham JJ. Selective reporting bias of harm outcomes within studies: findings from a cohort of systematic reviews. BMJ. 2014;349:g6501.CrossRefPubMedPubMedCentral Saini P, Loke YK, Gamble C, Altman DG, Williamson PR, Kirkham JJ. Selective reporting bias of harm outcomes within studies: findings from a cohort of systematic reviews. BMJ. 2014;349:g6501.CrossRefPubMedPubMedCentral
4.
go back to reference Sterling T. Publication decisions and their possible effects on inferences drawn from tests of significance, or vice versa. J Am Stat Assoc. 1959;285:30–4. Sterling T. Publication decisions and their possible effects on inferences drawn from tests of significance, or vice versa. J Am Stat Assoc. 1959;285:30–4.
5.
go back to reference Masicampo EJ, Lalande DR. A peculiar prevalence of p values just below.05. Q J Exp Psychol (Hove). 2012;65:2271–9.CrossRef Masicampo EJ, Lalande DR. A peculiar prevalence of p values just below.05. Q J Exp Psychol (Hove). 2012;65:2271–9.CrossRef
6.
go back to reference Krawczyk M. The search for significance: a few peculiarities in the distribution of P values in experimental psychology literature. PLoS One. 2015;10:e0127872.CrossRefPubMedPubMedCentral Krawczyk M. The search for significance: a few peculiarities in the distribution of P values in experimental psychology literature. PLoS One. 2015;10:e0127872.CrossRefPubMedPubMedCentral
8.
go back to reference Button KS, Ioannidis JP, Mokrysz C, Nosek BA, Flint J, Robinson ES, et al. Power failure: why small sample size undermines the reliability of neuroscience. Nat Rev Neurosci. 2013;14:365–76.CrossRefPubMed Button KS, Ioannidis JP, Mokrysz C, Nosek BA, Flint J, Robinson ES, et al. Power failure: why small sample size undermines the reliability of neuroscience. Nat Rev Neurosci. 2013;14:365–76.CrossRefPubMed
10.
go back to reference Downing NS, Aminawung JA, Shah ND, Krumholz HM, Ross JS. Clinical trial evidence supporting FDA approval of novel therapeutic agents, 2005-2012. JAMA. 2014;311:368–77.CrossRefPubMedPubMedCentral Downing NS, Aminawung JA, Shah ND, Krumholz HM, Ross JS. Clinical trial evidence supporting FDA approval of novel therapeutic agents, 2005-2012. JAMA. 2014;311:368–77.CrossRefPubMedPubMedCentral
11.
go back to reference Open Science Collaboration. Estimating the reproducibility of psychological sciences. Science. 2015;349(943):aac4716.CrossRef Open Science Collaboration. Estimating the reproducibility of psychological sciences. Science. 2015;349(943):aac4716.CrossRef
12.
13.
go back to reference Cuijpers P, Smit F, Bohlmeijer E, Hollon SD, Andersson G. Efficacy of cognitive-behavioural therapy and other psychological treatments for adult depression: meta-analytic study of publication bias. Br J Psychiatry. 2010;196:173–8.CrossRefPubMed Cuijpers P, Smit F, Bohlmeijer E, Hollon SD, Andersson G. Efficacy of cognitive-behavioural therapy and other psychological treatments for adult depression: meta-analytic study of publication bias. Br J Psychiatry. 2010;196:173–8.CrossRefPubMed
14.
go back to reference Driessen E, Hollon SD, Bockting CL, Cuijpers P, Turner EH. Does publication bias inflate the apparent efficacy of psychological treatment for major depressive disorder? A systematic review and meta-analysis of US National Institutes of Health-Funded Trials. PLoS One. 2015;10:e0137864.CrossRefPubMedPubMedCentral Driessen E, Hollon SD, Bockting CL, Cuijpers P, Turner EH. Does publication bias inflate the apparent efficacy of psychological treatment for major depressive disorder? A systematic review and meta-analysis of US National Institutes of Health-Funded Trials. PLoS One. 2015;10:e0137864.CrossRefPubMedPubMedCentral
15.
go back to reference Turner EH, Matthews AM, Linardatos E, Tell RA, Rosenthal R. Selective publication of antidepressant trials and its influence on apparent efficacy. N Engl J Med. 2008;358:252–60.CrossRefPubMed Turner EH, Matthews AM, Linardatos E, Tell RA, Rosenthal R. Selective publication of antidepressant trials and its influence on apparent efficacy. N Engl J Med. 2008;358:252–60.CrossRefPubMed
16.
go back to reference Roest AM, de Jonge P, Williams CD, de Vries YA, Schoevers RA, Turner EH. Reporting bias in clinical trials investigating the efficacy of second-generation antidepressants in the treatment of anxiety disorders: a report of 2 meta-analyses. JAMA Psychiat. 2015;72:500–10.CrossRef Roest AM, de Jonge P, Williams CD, de Vries YA, Schoevers RA, Turner EH. Reporting bias in clinical trials investigating the efficacy of second-generation antidepressants in the treatment of anxiety disorders: a report of 2 meta-analyses. JAMA Psychiat. 2015;72:500–10.CrossRef
17.
go back to reference Turner EH, Knoepflmacher D, Shapley L. Publication bias in antipsychotic trials: an analysis of efficacy comparing the published literature to the US Food and Drug Administration Database. PLoS Med. 2012;9:e1001189.CrossRefPubMedPubMedCentral Turner EH, Knoepflmacher D, Shapley L. Publication bias in antipsychotic trials: an analysis of efficacy comparing the published literature to the US Food and Drug Administration Database. PLoS Med. 2012;9:e1001189.CrossRefPubMedPubMedCentral
18.
go back to reference Hughes S, Cohen D, Jaggi R. Differences in reporting serious adverse events in industry sponsored clinical trial registries and journal articles on antidepressant and antipsychotic drugs: a cross sectional study. BMJ Open. 2014;4:e005535.CrossRefPubMedPubMedCentral Hughes S, Cohen D, Jaggi R. Differences in reporting serious adverse events in industry sponsored clinical trial registries and journal articles on antidepressant and antipsychotic drugs: a cross sectional study. BMJ Open. 2014;4:e005535.CrossRefPubMedPubMedCentral
19.
go back to reference Vaughan B, Goldstein MH, Alikakos M, Cohen LJ, Serby MJ. Frequency of reporting of adverse events in randomized controlled trials of psychotherapy vs. psychopharmacotherapy. Compr Psychiatry. 2014;55:849–55.CrossRefPubMedPubMedCentral Vaughan B, Goldstein MH, Alikakos M, Cohen LJ, Serby MJ. Frequency of reporting of adverse events in randomized controlled trials of psychotherapy vs. psychopharmacotherapy. Compr Psychiatry. 2014;55:849–55.CrossRefPubMedPubMedCentral
20.
21.
go back to reference Mavridis D, Efthimiou O, Leucht S, Salanti G. Publication bias and small-study effects magnified effectiveness of antipsychotics but their relative ranking remained invariant. J Clin Epidemiol. 2015;69:161–9.CrossRefPubMed Mavridis D, Efthimiou O, Leucht S, Salanti G. Publication bias and small-study effects magnified effectiveness of antipsychotics but their relative ranking remained invariant. J Clin Epidemiol. 2015;69:161–9.CrossRefPubMed
26.
go back to reference Knuppel H, Metz C, Meerpohl JJ, Strech D. How psychiatry journals support the unbiased translation of clinical research. A cross-sectional study of editorial policies. PLoS One. 2013;8:e75995.CrossRefPubMedPubMedCentral Knuppel H, Metz C, Meerpohl JJ, Strech D. How psychiatry journals support the unbiased translation of clinical research. A cross-sectional study of editorial policies. PLoS One. 2013;8:e75995.CrossRefPubMedPubMedCentral
27.
go back to reference Dal-Ré R, Bracken MB, Ioannidis JP. Call to improve transparency of trials of non-regulated interventions. BMJ. 2015;350:h1323.CrossRefPubMed Dal-Ré R, Bracken MB, Ioannidis JP. Call to improve transparency of trials of non-regulated interventions. BMJ. 2015;350:h1323.CrossRefPubMed
30.
go back to reference Scott A, Rucklidge JJ, Mulder RT. Is mandatory prospective trial registration working to prevent publication of unregistered trials and selective outcome reporting? An observational study of five psychiatry journals that mandate prospective clinical trial registration. PLoS One. 2015;10:e0133718.CrossRefPubMedPubMedCentral Scott A, Rucklidge JJ, Mulder RT. Is mandatory prospective trial registration working to prevent publication of unregistered trials and selective outcome reporting? An observational study of five psychiatry journals that mandate prospective clinical trial registration. PLoS One. 2015;10:e0133718.CrossRefPubMedPubMedCentral
31.
go back to reference Shinohara K, Tajika A, Imai H, Takeshima N, Hayasaka Y, Furukawa TA. Protocol registration and selective outcome reporting in recent psychiatry trials: new antidepressants and cognitive behavioral therapies. Acta Psychiatr Scand. 2015;132:489–98.CrossRefPubMed Shinohara K, Tajika A, Imai H, Takeshima N, Hayasaka Y, Furukawa TA. Protocol registration and selective outcome reporting in recent psychiatry trials: new antidepressants and cognitive behavioral therapies. Acta Psychiatr Scand. 2015;132:489–98.CrossRefPubMed
32.
go back to reference Bradley HA, Rucklidge JJ, Mulder RT. A systematic review of trial registration and selective outcome reporting in psychotherapy randomized controlled trials. Acta Psychiatr Scand. 2017;135:66–75.CrossRef Bradley HA, Rucklidge JJ, Mulder RT. A systematic review of trial registration and selective outcome reporting in psychotherapy randomized controlled trials. Acta Psychiatr Scand. 2017;135:66–75.CrossRef
33.
go back to reference Dal-Ré R, Caplan AL. Journal editors’ impasse with outcome reporting bias. Eur J Clin Invest. 2015;45:895–8.CrossRefPubMed Dal-Ré R, Caplan AL. Journal editors’ impasse with outcome reporting bias. Eur J Clin Invest. 2015;45:895–8.CrossRefPubMed
34.
go back to reference Mavridis D, Salanti G. How to assess publication bias: funnel plot, trim-and-fill method and selection models. Evid Based Ment Health. 2014;17:30.CrossRefPubMed Mavridis D, Salanti G. How to assess publication bias: funnel plot, trim-and-fill method and selection models. Evid Based Ment Health. 2014;17:30.CrossRefPubMed
35.
go back to reference Ebrahim S, Bance S, Athale A, Malachowski C, Ioannidis JP. Meta-analyses with industry involvement are massively published and report no caveats for antidepressants. J Clin Epidemiol. 2016;70:155–63.CrossRefPubMed Ebrahim S, Bance S, Athale A, Malachowski C, Ioannidis JP. Meta-analyses with industry involvement are massively published and report no caveats for antidepressants. J Clin Epidemiol. 2016;70:155–63.CrossRefPubMed
36.
go back to reference Hart B, Lundh A, Bero L. Effect of reporting bias on meta-analyses of drug trials: reanalysis of meta-analyses. BMJ. 2012;344:d7202.CrossRefPubMed Hart B, Lundh A, Bero L. Effect of reporting bias on meta-analyses of drug trials: reanalysis of meta-analyses. BMJ. 2012;344:d7202.CrossRefPubMed
37.
go back to reference Vedula SS, Li T, Dickersin K. Differences in reporting of analyses in internal company documents versus published trial reports: comparisons in industry-sponsored trials in off-label uses of gabapentin. PLoS Med. 2013;10:e1001378.CrossRefPubMedPubMedCentral Vedula SS, Li T, Dickersin K. Differences in reporting of analyses in internal company documents versus published trial reports: comparisons in industry-sponsored trials in off-label uses of gabapentin. PLoS Med. 2013;10:e1001378.CrossRefPubMedPubMedCentral
38.
go back to reference van Lent M, IntHout J, Out HJ. Differences between information in registries and articles did not influence publication acceptance. J Clin Epidemiol. 2015;68:1059–67.CrossRefPubMed van Lent M, IntHout J, Out HJ. Differences between information in registries and articles did not influence publication acceptance. J Clin Epidemiol. 2015;68:1059–67.CrossRefPubMed
39.
go back to reference Ioannidis J, Caplan AL, Dal-Ré R. Outcome reporting bias in clinical trials: why monitoring matters. BMJ. 2017;356:j408.CrossRefPubMed Ioannidis J, Caplan AL, Dal-Ré R. Outcome reporting bias in clinical trials: why monitoring matters. BMJ. 2017;356:j408.CrossRefPubMed
Metadata
Title
Why prudence is needed when interpreting articles reporting clinical trial results in mental health
Authors
Rafael Dal-Ré
Julio Bobes
Pim Cuijpers
Publication date
01-12-2017
Publisher
BioMed Central
Published in
Trials / Issue 1/2017
Electronic ISSN: 1745-6215
DOI
https://doi.org/10.1186/s13063-017-1899-2

Other articles of this Issue 1/2017

Trials 1/2017 Go to the issue