Top

Canadian Journal of Anesthesia/Journal canadien d'anesthésie

Published in:

01-08-2021 | Editorials

Increasing the reproducibility of research will reduce the problem of apophenia (and more)

Authors: Philip M. Jones, MD, MSc, Janet Martin, PharmD, MSc (HTA&M)

Published in: Canadian Journal of Anesthesia/Journal canadien d'anesthésie | Issue 8/2021

Excerpt

In their thought-provoking article, Hanson et al.1 eloquently and convincingly describe how humans are naturally predisposed to identifying meaningful patterns where none truly exist and tend to prefer positive over negative results—the tendency known as apophenia. In the research context, when these two elements are combined, this creates the aggregate issue of false positive research findings (hereafter referred to as simply false positives). …

The Canadian Journal of Anesthesia’s Policy Statement on Registration of Randomized Clinical Trials and Systematic Reviews. Available from URL: https://www.springer.com/journal/12630/submission-guidelines (accessed April 2021).

Registration of Observational Studies at ClinicalTrials.gov. Available from URL: https://clinicaltrials.gov/ct2/manage-recs/how-register#Considerations (accessed April 2021).

For example, the Open Science Framework. Available from URL: https://osf.io (accessed April 2021).

The Canadian Journal of Anesthesia’s Policy Statement on Registration of Randomized Clinical Trials and Systematic Reviews. Disponible à l’adresse: https://www.springer.com/journal/12630/submission-guidelines (consulté avril 2021).

Registration of Observational Studies at ClinicalTrials.gov. Disponible à l’adresse: https://clinicaltrials.gov/ct2/manage-recs/howregister#Considerations (consulté avril 2021).

Par exemple, le Open Science Framework. Disponible à l’adresse: https://osf.io/ (consulté avril 2021).

Hanson NA, Lavallee MB, Thiele RH. Apophenia and anesthesia: how we sometimes change our practice prematurely. Can J Anesth 2021; DOI: https://doi.org/10.1007/s12630-021-02005-2.CrossRefPubMed

Ioannidis JP. Why most published research findings are false. PLoS Med 2005; DOI: https://doi.org/10.1371/journal.pmed.0020124.CrossRefPubMedPubMedCentral

Vetter TR, McGwin G Jr, Pittet JF. Replicability, reproducibility, and fragility of research findings-ultimately, caveat emptor. Anesth Analg 2016; 123: 244-8.CrossRef

Spence JR, Stanley DJ. Prediction interval: what to expect when you’re expecting … a replication. PLoS One 2016; DOI: https://doi.org/10.1371/journal.pone.0162874.CrossRefPubMedPubMedCentral

Pawel S, Held L. Probabilistic forecasting of replication studies. PLoS One 2020; DOI: https://doi.org/10.1371/journal.pone.0231416.CrossRefPubMedPubMedCentral

Baker M. 1,500 scientists lift the lid on reproducibility. Nature 2016; 533: 452-4.CrossRef

Open Science Collaboration. Estimating the reproducibility of psychological science. Science 2015; DOI: https://doi.org/10.1126/science.aac4716.CrossRef

Begley CG, Ellis LM. Drug development: raise standards for preclinical cancer research. Nature 2012; 483: 531-3.CrossRef

Prinz F, Schlange T, Asadullah K. Believe it or not: how much can we rely on published data on potential drug targets. Nat Rev Drug Discov 2011; DOI: https://doi.org/10.1038/nrd3439-c1.CrossRefPubMed

10.

Niven DJ, McCormick TJ, Straus SE, et al. Reproducibility of clinical research in critical care: a scoping review. BMC Med 2018; DOI: https://doi.org/10.1186/s12916-018-1018-6.CrossRefPubMedPubMedCentral

11.

Avidan MS, Ioannidis JP, Mashour GA. Independent discussion sections for improving inferential reproducibility in published research. Br J Anaesth 2019; 122: 413-20.CrossRef

12.

Camerer CF, Dreber A, Forsell E, et al. Evaluating replicability of laboratory experiments in economics. Science 2016; 351: 1433-6.CrossRef

13.

Han R, Walton KS, Sholl DS. Does chemical engineering research have a reproducibility problem. Ann Rev 2019; 10: 43-57.

14.

McDermott MB, Wang S, Marinsek N, Ranganath R, Foschini L, Ghassemi M. Reproducibility in machine learning for health research: still a ways to go. Sci Transl Med 2021; DOI: https://doi.org/10.1126/scitranslmed.abb1655.CrossRefPubMed

15.

Dowden H, Munro J. Trends in clinical success rates and therapeutic focus. Nat Rev Drug Discov 2019; 18: 495-6.CrossRef

16.

Broer L, Lill CM, Schuur M, et al. Distinguishing true from false positives in genomic studies: p values. Eur J Epidemiol 2013; 28: 131-8.CrossRef

17.

Hingorani AD, Kuan V, Finan C, et al. Improving the odds of drug development success through human genomics: modelling study. Sci Rep 2019; DOI: https://doi.org/10.1038/s41598-019-54849-w.CrossRefPubMedPubMedCentral

18.

Wong CH, Siah KW, Lo AW. Estimation of clinical trial success rates and related parameters. Biostatistics 2019; 20: 273-86.CrossRef

19.

Djulbegovic B, Kumar A, Glasziou P, Miladinovic B, Chalmers I. Medical research: trial unpredictability yields predictable therapy gains. Nature 2013; 500: 395-6.CrossRef

20.

Colquhoun D. The reproducibility of research and the misinterpretation of p-values. R Soc Open Sci 2017; DOI: https://doi.org/10.1098/rsos.171085.CrossRefPubMedPubMedCentral

21.

Grolleau F, Collins GS, Smarandache A, et al. The fragility and reliability of conclusions of anesthesia and critical care randomized trials with statistically significant findings: a systematic review. Crit Care Med 2019; 47: 456-62.CrossRef

22.

Chow JT, Turkstra TP, Yim E, et al. Sample size calculations for randomized clinical trials published in anesthesiology journals: a comparison of 2010 versus 2016. Can J Anesth 2018; 65: 611-8.CrossRef

23.

Egbewale BE, Lewis M, Sim J. Bias, precision and statistical power of analysis of covariance in the analysis of randomized trials with baseline imbalance: a simulation study. BMC Med Res Methodol 2014; DOI: https://doi.org/10.1186/1471-2288-14-49.CrossRefPubMedPubMedCentral

24.

Kahan BC, Jairath V, Doré CJ, Morris TP. The risks and rewards of covariate adjustment in randomized trials: an assessment of 12 outcomes from 8 studies. Trials 2014; DOI: https://doi.org/10.1186/1745-6215-15-139.CrossRefPubMedPubMedCentral

25.

de Winter JC, Dodou D. A surge of p-values between 0.041 and 0.049 in recent decades (but negative results are increasing rapidly too). PeerJ 2015; https://doi.org/10.7717/peerj.733.

26.

Chavalarias D, Wallach JD, Li AH, Ioannidis JP. Evolution of reporting P values in the biomedical literature, 1990-2015. JAMA 2016; 315: 1141-8.CrossRef

27.

Wang MQ, Yan AF, Katz RV. Researcher requests for inappropriate analysis and reporting: a U.S. survey of consulting biostatisticians. Ann Intern Med 2018; 169: 554-8.

28.

Jones PM, Chow JT, Arango MF, et al. Comparison of registered and reported outcomes in randomized clinical trials published in anesthesiology journals. Anesth Analg 2017; 125: 1292-300.CrossRef

29.

De Oliveira GS, Jr Jung MJ, McCarthy RJ. Discrepancies between randomized controlled trial registry entries and content of corresponding manuscripts reported in anesthesiology journals. Anesth Analg 2015; 121: 1030-3.CrossRef

30.

Turner EH, Matthews AM, Linardatos E, Tell RA, Rosenthal R. Selective publication of antidepressant trials and its influence on apparent efficacy. N Engl J Med 2008; 358: 252-60.CrossRef

31.

De Oliveira GS, Chang R, Kendall MC, et al. Publication bias in the anesthesiology literature. Anesth Analg 2012; 114: 1042-8.CrossRef

32.

McHugh UM, Yentis SM. An analysis of retractions of papers authored by Scott Reuben. Joachim Boldt and Yoshitaka Fujii. Anaesthesia 2019; 74: 17-21.

33.

Fanelli D. How many scientists fabricate and falsify research? A systematic review and meta-analysis of survey data. PLoS One 2009; DOI: https://doi.org/10.1371/journal.pone.0005738.CrossRefPubMedPubMedCentral

34.

Altman DG. The scandal of poor medical research. BMJ 1994; 308: 283-4.CrossRef

35.

Glasziou P, Chalmers I. Research waste is still a scandal—an essay by Paul Glasziou and Iain Chalmers. BMJ 2018; DOI: https://doi.org/10.1136/bmj.k4645.CrossRefPubMed

36.

Unverzagt S, Prondzinsky R, Peinemann F. Single-center trials tend to provide larger treatment effects than multicenter trials: a systematic review. J Clin Epidemiol 2013; 66: 1271-80.CrossRef

37.

Goodman SN. Toward evidence-based medical statistics. 2: the Bayes factor. Ann Intern Med 1999; 130: 1005-13.

38.

Koletsi D, Solmi M, Pandis N, et al. Most recommended medical interventions reach P < 0.005 for their primary outcomes in meta-analyses. Int J Epidemiol 2020; 49: 885-93.

39.

Ioannidis JP. The proposal to lower P value thresholds to .005. JAMA 2018; 319: 1429-30.

40.

Wayant C, Scott J, Vassar M. Evaluation of lowering the P value threshold for statistical significance from .05 to s.005 in previously published randomized clinical trials in major medical journals. JAMA 2018; https://doi.org/10.1001/jama.2018.12288.

Title: Increasing the reproducibility of research will reduce the problem of apophenia (and more)
Authors: Philip M. Jones, MD, MSc
Janet Martin, PharmD, MSc (HTA&M)
Publication date: 01-08-2021
Publisher: Springer International Publishing
Published in: Canadian Journal of Anesthesia/Journal canadien d'anesthésie / Issue 8/2021
Print ISSN: 0832-610X
Electronic ISSN: 1496-8975
DOI: https://doi.org/10.1007/s12630-021-02006-1

At a glance: The ONWARDS insulin icodec trials

Springer Medicine

Increasing the reproducibility of research will reduce the problem of apophenia (and more)

Excerpt

At a glance: The ONWARDS insulin icodec trials

Springer Medicine

Excerpt

Please log in to get access to this content

Other articles of this Issue 8/2021

Transesophageal lung ultrasound in the intensive care unit

COVID-19 infection prevention and control procedures and institutional trust: Perceptions of Canadian intensive care and emergency department nurses

In reply: “Nature exposed to our method of questioning”—resuscitation preferences and complex interventions

Gender and sexuality-based discrimination in anesthesiology within Canada: a cross-sectional survey

Post-thyroidectomy prothrombin time elevation and hematoma in a patient who received sugammadex

Correction to: The impact of unilateral diaphragmatic paralysis on sleepdisordered breathing: a scoping review