Skip to main content
Top
Published in: Reproductive Health 1/2021

Open Access 01-12-2021 | Editorial

There is life beyond the statistical significance

Authors: Agustín Ciapponi, José M. Belizán, Gilda Piaggio, Sanni Yaya

Published in: Reproductive Health | Issue 1/2021

Login to get access

Abstract

This article challenges the “tyranny of P-value” and promote more valuable and applicable interpretations of the results of research on health care delivery. We provide here solid arguments to retire statistical significance as the unique way to interpret results, after presenting the current state of the debate inside the scientific community. Instead, we promote reporting the much more informative confidence intervals and eventually adding exact P-values. We also provide some clues to integrate statistical and clinical significance by referring to minimal important differences and integrating the effect size of an intervention and the certainty of evidence ideally using the GRADE approach. We have argued against interpreting or reporting results as statistically significant or statistically non-significant. We recommend showing important clinical benefits with their confidence intervals in cases of point estimates compatible with results benefits and even important harms. It seems fair to report the point estimate and the more likely values along with a very clear statement of the implications of extremes of the intervals. We recommend drawing conclusions, considering the multiple factors besides P-values such as certainty of the evidence for each outcome, net benefit, economic considerations and values and preferences. We use several examples and figures to illustrate different scenarios and further suggest a wording to standardize the reporting. Several statistical measures have a role in the scientific communication of studies, but it is time to understand that there is life beyond the statistical significance. There is a great opportunity for improvement towards a more complete interpretation and to a more standardized reporting.
Footnotes
1
Given the null hypothesis. p-value = Pr(data | H0).
 
Literature
1.
go back to reference Royston P, Altman DG, Sauerbrei W. Dichotomizing continuous predictors in multiple regression: a bad idea. Stat Med. 2006;25(1):127–41.CrossRef Royston P, Altman DG, Sauerbrei W. Dichotomizing continuous predictors in multiple regression: a bad idea. Stat Med. 2006;25(1):127–41.CrossRef
2.
go back to reference Taylor AB, West SG, Aiken LS. Loss of power in logistic, ordinal logistic, and probit regression when an outcome variable is coarsely categorized. Educ Psychol Meas. 2006;66(2):228–39.CrossRef Taylor AB, West SG, Aiken LS. Loss of power in logistic, ordinal logistic, and probit regression when an outcome variable is coarsely categorized. Educ Psychol Meas. 2006;66(2):228–39.CrossRef
3.
go back to reference Amrhein V, Greenland S, McShane B. Scientists rise up against statistical significance. Nature. 2019;567(7748):305–7.CrossRef Amrhein V, Greenland S, McShane B. Scientists rise up against statistical significance. Nature. 2019;567(7748):305–7.CrossRef
4.
go back to reference Altman DG, Bland JM. Statistics notes: absence of evidence is not evidence of absence. BMJ. 1995;311(7003):485.CrossRef Altman DG, Bland JM. Statistics notes: absence of evidence is not evidence of absence. BMJ. 1995;311(7003):485.CrossRef
5.
go back to reference Wasserstein RL, Lazar NA. The ASA statement on p-values: context, process, and purpose. Am Stat. 2016;70(2):129–33.CrossRef Wasserstein RL, Lazar NA. The ASA statement on p-values: context, process, and purpose. Am Stat. 2016;70(2):129–33.CrossRef
6.
go back to reference Ciapponi A. ¿Significancia clínica o significancia estadística? Evid Act Pract Ambul. 2013;16(4):122–5. Ciapponi A. ¿Significancia clínica o significancia estadística? Evid Act Pract Ambul. 2013;16(4):122–5.
7.
go back to reference Wasserstein RL, Schirm AL, Lazar NA. Moving to a world beyond “p < 0.05.” Am Stat. 2019;73(1):1–19.CrossRef Wasserstein RL, Schirm AL, Lazar NA. Moving to a world beyond “p < 0.05.” Am Stat. 2019;73(1):1–19.CrossRef
8.
go back to reference Gelman A, Stern H. The difference between “significant” and “not significant” is not itself statistically significant. Am Stat. 2006;60(4):328–31.CrossRef Gelman A, Stern H. The difference between “significant” and “not significant” is not itself statistically significant. Am Stat. 2006;60(4):328–31.CrossRef
9.
go back to reference Ioannidis JPA. What have we (not) learnt from millions of scientific papers with p values? Am Stat. 2019;73(sup1):20–5.CrossRef Ioannidis JPA. What have we (not) learnt from millions of scientific papers with p values? Am Stat. 2019;73(sup1):20–5.CrossRef
10.
go back to reference Ciapponi A, Bardach A, Glujovsky D. Potentially misleading reporting comparisons’ results in Cochrane systematic reviews and in major general medical journals. Paper presented at: XXII Cochrane Colloquium. Evidence-Informed Health: Opportunities and Challenges; 21–26 September 2014, 2014; Hyderabad, India. Ciapponi A, Bardach A, Glujovsky D. Potentially misleading reporting comparisons’ results in Cochrane systematic reviews and in major general medical journals. Paper presented at: XXII Cochrane Colloquium. Evidence-Informed Health: Opportunities and Challenges; 21–26 September 2014, 2014; Hyderabad, India.
11.
go back to reference Glujovsky D, Sueldo C, Borghi C, Nicotra P, Andreucci S, Ciapponi A. Misleading reporting in major fertility journals: poor use of confidence intervals and absolute differences. Paper presented at: XXIII Cochrane Colloquium Filtering the information overload for better decisions; 3–7 October 2015, 2015; Vienna, Austria Glujovsky D, Sueldo C, Borghi C, Nicotra P, Andreucci S, Ciapponi A. Misleading reporting in major fertility journals: poor use of confidence intervals and absolute differences. Paper presented at: XXIII Cochrane Colloquium Filtering the information overload for better decisions; 3–7 October 2015, 2015; Vienna, Austria
12.
go back to reference Jakobsen J. The necessity of randomized clinical trials. Br J Med Med Res. 2013;3:1453–68.CrossRef Jakobsen J. The necessity of randomized clinical trials. Br J Med Med Res. 2013;3:1453–68.CrossRef
13.
go back to reference Goodman WM, Spruill SE, Komaroff E. A proposed hybrid effect size plus p-value criterion: empirical evidence supporting its use. Am Stat. 2019;73(sup1):168–85.CrossRef Goodman WM, Spruill SE, Komaroff E. A proposed hybrid effect size plus p-value criterion: empirical evidence supporting its use. Am Stat. 2019;73(sup1):168–85.CrossRef
14.
go back to reference Jakobsen JC, Wetterslev J, Winkel P, Lange T, Gluud C. Thresholds for statistical and clinical significance in systematic reviews with meta-analytic methods. BMC Med Res Methodol. 2014;14:120.CrossRef Jakobsen JC, Wetterslev J, Winkel P, Lange T, Gluud C. Thresholds for statistical and clinical significance in systematic reviews with meta-analytic methods. BMC Med Res Methodol. 2014;14:120.CrossRef
15.
go back to reference Schunemann HJ, Guyatt GH. Commentary–goodbye M(C)ID! Hello MID, where do you come from? Health Serv Res. 2005;40(2):593–7.CrossRef Schunemann HJ, Guyatt GH. Commentary–goodbye M(C)ID! Hello MID, where do you come from? Health Serv Res. 2005;40(2):593–7.CrossRef
16.
go back to reference Wright A, Hannon J, Hegedus EJ, Kavchak AE. Clinimetrics corner: a closer look at the minimal clinically important difference (MCID). J Manual Manip Ther. 2012;20(3):160–6.CrossRef Wright A, Hannon J, Hegedus EJ, Kavchak AE. Clinimetrics corner: a closer look at the minimal clinically important difference (MCID). J Manual Manip Ther. 2012;20(3):160–6.CrossRef
18.
go back to reference Jakobsen JC, Gluud C, Winkel P, Lange T, Wetterslev J. The thresholds for statistical and clinical significance—a five-step procedure for evaluation of intervention effects in randomised clinical trials. BMC Med Res Methodol. 2014;14:34.CrossRef Jakobsen JC, Gluud C, Winkel P, Lange T, Wetterslev J. The thresholds for statistical and clinical significance—a five-step procedure for evaluation of intervention effects in randomised clinical trials. BMC Med Res Methodol. 2014;14:34.CrossRef
19.
go back to reference Revicki D, Hays RD, Cella D, Sloan J. Recommended methods for determining responsiveness and minimally important differences for patient-reported outcomes. J Clin Epidemiol. 2008;61(2):102–9.CrossRef Revicki D, Hays RD, Cella D, Sloan J. Recommended methods for determining responsiveness and minimally important differences for patient-reported outcomes. J Clin Epidemiol. 2008;61(2):102–9.CrossRef
21.
go back to reference Greenland S. Valid p-values behave exactly as they should: some misleading criticisms of p-values and their resolution with s-values. Am Stat. 2019;73(sup1):106–14.CrossRef Greenland S. Valid p-values behave exactly as they should: some misleading criticisms of p-values and their resolution with s-values. Am Stat. 2019;73(sup1):106–14.CrossRef
22.
go back to reference Guyatt G, Oxman AD, Akl EA, et al. GRADE guidelines: 1. Introduction-GRADE evidence profiles and summary of findings tables. J Clin Epidemiol. 2011;64(4):383–94.CrossRef Guyatt G, Oxman AD, Akl EA, et al. GRADE guidelines: 1. Introduction-GRADE evidence profiles and summary of findings tables. J Clin Epidemiol. 2011;64(4):383–94.CrossRef
23.
go back to reference Ciapponi A, Glujovsky D, Comande D, Bardach A. Do Cochrane systematic reviews report results integrating certainty of evidence and effect size? Paper presented at: 25th Cochrane Colloquium. 16–18 September 2018; 16–18 September 2018, 2018; Edinburgh, Scotland. Ciapponi A, Glujovsky D, Comande D, Bardach A. Do Cochrane systematic reviews report results integrating certainty of evidence and effect size? Paper presented at: 25th Cochrane Colloquium. 16–18 September 2018; 16–18 September 2018, 2018; Edinburgh, Scotland.
24.
go back to reference Schünemann H, Vist G, Higgins J, et al. Chapter 15: Interpreting results and drawing conclusions. In: Higgins J, Thomas J, Chandler J, et al., editors. Cochrane handbook for systematic reviews of interventions version 6.0 (updated July 2019): Cochrane; 2019. Schünemann H, Vist G, Higgins J, et al. Chapter 15: Interpreting results and drawing conclusions. In: Higgins J, Thomas J, Chandler J, et al., editors. Cochrane handbook for systematic reviews of interventions version 6.0 (updated July 2019): Cochrane; 2019.
26.
go back to reference Guyatt G, Oxman AD, Kunz R, et al. GRADE guidelines: 6. Rating the quality of evidence-imprecision. J Clin Epidemiol. 2011;64:1283–93.CrossRef Guyatt G, Oxman AD, Kunz R, et al. GRADE guidelines: 6. Rating the quality of evidence-imprecision. J Clin Epidemiol. 2011;64:1283–93.CrossRef
28.
go back to reference Goodman SN. Why is getting rid of p-values so hard? Musings on science and statistics. Am Stat. 2019;73(sup1):26–30.CrossRef Goodman SN. Why is getting rid of p-values so hard? Musings on science and statistics. Am Stat. 2019;73(sup1):26–30.CrossRef
29.
go back to reference Moher D, Hopewell S, Schulz KF, et al. CONSORT 2010 explanation and elaboration: updated guidelines for reporting parallel group randomised trials. BMJ. 2010;340:c869.CrossRef Moher D, Hopewell S, Schulz KF, et al. CONSORT 2010 explanation and elaboration: updated guidelines for reporting parallel group randomised trials. BMJ. 2010;340:c869.CrossRef
Metadata
Title
There is life beyond the statistical significance
Authors
Agustín Ciapponi
José M. Belizán
Gilda Piaggio
Sanni Yaya
Publication date
01-12-2021
Publisher
BioMed Central
Published in
Reproductive Health / Issue 1/2021
Electronic ISSN: 1742-4755
DOI
https://doi.org/10.1186/s12978-021-01131-w

Other articles of this Issue 1/2021

Reproductive Health 1/2021 Go to the issue