Skip to main content
Top
Published in: BMC Medical Research Methodology 1/2014

Open Access 01-12-2014 | Correspondence

The thresholds for statistical and clinical significance – a five-step procedure for evaluation of intervention effects in randomised clinical trials

Authors: Janus Christian Jakobsen, Christian Gluud, Per Winkel, Theis Lange, Jørn Wetterslev

Published in: BMC Medical Research Methodology | Issue 1/2014

Login to get access

Abstract

Background

Thresholds for statistical significance are insufficiently demonstrated by 95% confidence intervals or P-values when assessing results from randomised clinical trials. First, a P-value only shows the probability of getting a result assuming that the null hypothesis is true and does not reflect the probability of getting a result assuming an alternative hypothesis to the null hypothesis is true. Second, a confidence interval or a P-value showing significance may be caused by multiplicity. Third, statistical significance does not necessarily result in clinical significance. Therefore, assessment of intervention effects in randomised clinical trials deserves more rigour in order to become more valid.

Methods

Several methodologies for assessing the statistical and clinical significance of intervention effects in randomised clinical trials were considered. Balancing simplicity and comprehensiveness, a simple five-step procedure was developed.

Results

For a more valid assessment of results from a randomised clinical trial we propose the following five-steps: (1) report the confidence intervals and the exact P-values; (2) report Bayes factor for the primary outcome, being the ratio of the probability that a given trial result is compatible with a ‘null’ effect (corresponding to the P-value) divided by the probability that the trial result is compatible with the intervention effect hypothesised in the sample size calculation; (3) adjust the confidence intervals and the statistical significance threshold if the trial is stopped early or if interim analyses have been conducted; (4) adjust the confidence intervals and the P-values for multiplicity due to number of outcome comparisons; and (5) assess clinical significance of the trial results.

Conclusions

If the proposed five-step procedure is followed, this may increase the validity of assessments of intervention effects in randomised clinical trials.
Appendix
Available only for authorised users
Literature
1.
go back to reference Jakobsen JC, Gluud C: The necessity of randomized clinical trials. Br J Med Res. 2013, 3 (4): 1453-1468.CrossRef Jakobsen JC, Gluud C: The necessity of randomized clinical trials. Br J Med Res. 2013, 3 (4): 1453-1468.CrossRef
3.
go back to reference Fisher R: Statistical methods and scientific induction. J R Stat Soc Ser B. 1955, 17 (1): 69-78. Fisher R: Statistical methods and scientific induction. J R Stat Soc Ser B. 1955, 17 (1): 69-78.
4.
go back to reference Gigerenzer G: Mindless statistics. J Socio Econ. 2004, 33 (5): 587-606. 10.1016/j.socec.2004.09.033.CrossRef Gigerenzer G: Mindless statistics. J Socio Econ. 2004, 33 (5): 587-606. 10.1016/j.socec.2004.09.033.CrossRef
5.
go back to reference Hald A: A history of mathematical statistics from 1750 to 1930. 1998, New York: John Wiley & Sons Hald A: A history of mathematical statistics from 1750 to 1930. 1998, New York: John Wiley & Sons
6.
go back to reference Goodman S: A dirty dozen: twelve p-value misconceptions. Semin Hematol. 2008, 45: 135-140. 10.1053/j.seminhematol.2008.04.003.CrossRefPubMed Goodman S: A dirty dozen: twelve p-value misconceptions. Semin Hematol. 2008, 45: 135-140. 10.1053/j.seminhematol.2008.04.003.CrossRefPubMed
7.
go back to reference Oliveri RS, Gluud C, Wille-Jørgensen PA: Hospital doctors' self-rated skills in and use of evidence-based medicine - a questionnaire survey. J Eval Clin Pract. 2004, 10 (2): 219-226. 10.1111/j.1365-2753.2003.00477.x.CrossRefPubMed Oliveri RS, Gluud C, Wille-Jørgensen PA: Hospital doctors' self-rated skills in and use of evidence-based medicine - a questionnaire survey. J Eval Clin Pract. 2004, 10 (2): 219-226. 10.1111/j.1365-2753.2003.00477.x.CrossRefPubMed
8.
go back to reference Bassler D, Briel M, Montori VM, Lane M, Glasziou P, Zhou Q, Heels-Ansdell D, Walter SD, Guyatt GH, Flynn DN, Elamin MB, Murad MH, Abu Elnour NO, Lampropulos JF, Sood A, Mullan RJ, Erwin PJ, Bankhead CR, Perera R, Ruiz Culebro C, You JJ, Mulla SM, Kaur J, Nerenberg KA, Schunemann H, Cook DJ, Lutz K, Ribic CM, Vale N, Malaga G, Akl EA, et al: Stopping randomized trials early for benefit and estimation of treatment effects: systematic review and meta-regression analysis. JAMA. 2010, 303: 1180-1187. 10.1001/jama.2010.310.CrossRefPubMed Bassler D, Briel M, Montori VM, Lane M, Glasziou P, Zhou Q, Heels-Ansdell D, Walter SD, Guyatt GH, Flynn DN, Elamin MB, Murad MH, Abu Elnour NO, Lampropulos JF, Sood A, Mullan RJ, Erwin PJ, Bankhead CR, Perera R, Ruiz Culebro C, You JJ, Mulla SM, Kaur J, Nerenberg KA, Schunemann H, Cook DJ, Lutz K, Ribic CM, Vale N, Malaga G, Akl EA, et al: Stopping randomized trials early for benefit and estimation of treatment effects: systematic review and meta-regression analysis. JAMA. 2010, 303: 1180-1187. 10.1001/jama.2010.310.CrossRefPubMed
9.
go back to reference Thorlund K, Devereaux PJ, Wetterslev J, Guyatt G, Ioannidis JP, Thabane L, Gluud LL, Als-Nielsen B, Gluud C: Can trial sequential monitoring boundaries reduce spurious inferences from meta-analyses?. Int J Epidemiol. 2009, 38 (1): 276-286. 10.1093/ije/dyn179.CrossRefPubMed Thorlund K, Devereaux PJ, Wetterslev J, Guyatt G, Ioannidis JP, Thabane L, Gluud LL, Als-Nielsen B, Gluud C: Can trial sequential monitoring boundaries reduce spurious inferences from meta-analyses?. Int J Epidemiol. 2009, 38 (1): 276-286. 10.1093/ije/dyn179.CrossRefPubMed
11.
go back to reference Garattini S, Bertele V: Non-inferiority trials are unethical because they disregard patients' interests. Lancet. 2007, 370 (9602): 1875-1877. 10.1016/S0140-6736(07)61604-3.CrossRefPubMed Garattini S, Bertele V: Non-inferiority trials are unethical because they disregard patients' interests. Lancet. 2007, 370 (9602): 1875-1877. 10.1016/S0140-6736(07)61604-3.CrossRefPubMed
12.
go back to reference Sterne JA: Teaching hypothesis tests–time for significant change?. Stat Med. 2002, 21: 985-999. 10.1002/sim.1129.CrossRefPubMed Sterne JA: Teaching hypothesis tests–time for significant change?. Stat Med. 2002, 21: 985-999. 10.1002/sim.1129.CrossRefPubMed
13.
go back to reference Ranstam J: Why the P-value culture is bad and confidence intervals a better alternative. Osteoarthritis Cartilage. 2012, 20: 805-808. 10.1016/j.joca.2012.04.001.CrossRefPubMed Ranstam J: Why the P-value culture is bad and confidence intervals a better alternative. Osteoarthritis Cartilage. 2012, 20: 805-808. 10.1016/j.joca.2012.04.001.CrossRefPubMed
14.
go back to reference Williamson PR, Altman DG, Blazeby JM, Clarke M, Gargon E: The COMET (Core Outcome Measures in Effectiveness Trials) Initiative. Trials. 2011, 12 (Suppl 1): A70-10.1186/1745-6215-12-S1-A70.CrossRefPubMedCentral Williamson PR, Altman DG, Blazeby JM, Clarke M, Gargon E: The COMET (Core Outcome Measures in Effectiveness Trials) Initiative. Trials. 2011, 12 (Suppl 1): A70-10.1186/1745-6215-12-S1-A70.CrossRefPubMedCentral
15.
go back to reference Altman DG, Bland JM: How to obtain the confidence interval from a P value. BMJ. 2011, 343: d2090-10.1136/bmj.d2090.CrossRefPubMed Altman DG, Bland JM: How to obtain the confidence interval from a P value. BMJ. 2011, 343: d2090-10.1136/bmj.d2090.CrossRefPubMed
16.
go back to reference Chow S-C, Shao J, Wang H: Sample Size Calculations in Clinical Research, Second Edition. 2008, Boca Raton, Florida: Chapman and Hall/CRC Chow S-C, Shao J, Wang H: Sample Size Calculations in Clinical Research, Second Edition. 2008, Boca Raton, Florida: Chapman and Hall/CRC
17.
go back to reference Schulz KF, Altman DG, Moher D: CONSORT 2010 statement: updated guidelines for reporting parallel group randomized trials. Ann Int Med. 2010, 152 (11): 726-732. 10.7326/0003-4819-152-11-201006010-00232.CrossRefPubMed Schulz KF, Altman DG, Moher D: CONSORT 2010 statement: updated guidelines for reporting parallel group randomized trials. Ann Int Med. 2010, 152 (11): 726-732. 10.7326/0003-4819-152-11-201006010-00232.CrossRefPubMed
18.
go back to reference Scales DC, Rubenfeld GD: Estimating sample size in critical care clinical trials. J Crit Care. 2005, 20 (1): 6-11. 10.1016/j.jcrc.2005.02.002.CrossRefPubMed Scales DC, Rubenfeld GD: Estimating sample size in critical care clinical trials. J Crit Care. 2005, 20 (1): 6-11. 10.1016/j.jcrc.2005.02.002.CrossRefPubMed
19.
go back to reference Myles DJS, Keith RA, Jonathan P: Bayesian approaches to clinical trials and health-care evaluation (Statistics in Practice). 2004, West Sussex, England: John Wiley & Sons Myles DJS, Keith RA, Jonathan P: Bayesian approaches to clinical trials and health-care evaluation (Statistics in Practice). 2004, West Sussex, England: John Wiley & Sons
20.
go back to reference Roloff V, Higgins JP, Sutton AJ: Planning future studies based on the conditional power of a meta-analysis. Stat Med. 2013, 32 (1): 11-24. 10.1002/sim.5524.CrossRefPubMed Roloff V, Higgins JP, Sutton AJ: Planning future studies based on the conditional power of a meta-analysis. Stat Med. 2013, 32 (1): 11-24. 10.1002/sim.5524.CrossRefPubMed
21.
go back to reference Goodman SN: Introduction to Bayesian methods I: measuring the strength of evidence. Clin Trials. 2005, 2: 282-378. 10.1191/1740774505cn098oa.CrossRefPubMed Goodman SN: Introduction to Bayesian methods I: measuring the strength of evidence. Clin Trials. 2005, 2: 282-378. 10.1191/1740774505cn098oa.CrossRefPubMed
22.
go back to reference Goodman SN: Toward evidence-based medical statistics. 2: The Bayes factor. Ann Int Med. 1999, 130 (12): 1005-1013. 10.7326/0003-4819-130-12-199906150-00019.CrossRefPubMed Goodman SN: Toward evidence-based medical statistics. 2: The Bayes factor. Ann Int Med. 1999, 130 (12): 1005-1013. 10.7326/0003-4819-130-12-199906150-00019.CrossRefPubMed
23.
go back to reference Pogue JM, Yusuf S: Cumulating evidence from randomized trials: utilizing sequential monitoring boundaries for cumulative meta-analysis. Control Clin Trials. 1997, 18 (6): 580-593. 10.1016/S0197-2456(97)00051-2.CrossRefPubMed Pogue JM, Yusuf S: Cumulating evidence from randomized trials: utilizing sequential monitoring boundaries for cumulative meta-analysis. Control Clin Trials. 1997, 18 (6): 580-593. 10.1016/S0197-2456(97)00051-2.CrossRefPubMed
24.
go back to reference Higgins JP, Whitehead A: Borrowing strength from external trials in a meta-analysis. Stat Med. 1996, 15 (24): 2733-2749. 10.1002/(SICI)1097-0258(19961230)15:24<2733::AID-SIM562>3.0.CO;2-0.CrossRefPubMed Higgins JP, Whitehead A: Borrowing strength from external trials in a meta-analysis. Stat Med. 1996, 15 (24): 2733-2749. 10.1002/(SICI)1097-0258(19961230)15:24<2733::AID-SIM562>3.0.CO;2-0.CrossRefPubMed
25.
go back to reference Fayers PM, Cuschieri A, Fielding J, Craven J, Uscinska B, Freedman LS: Sample size calculation for clinical trials: the impact of clinician beliefs. Br J Cancer. 2000, 82 (1): 213-219. 10.1054/bjoc.1999.0902.CrossRefPubMed Fayers PM, Cuschieri A, Fielding J, Craven J, Uscinska B, Freedman LS: Sample size calculation for clinical trials: the impact of clinician beliefs. Br J Cancer. 2000, 82 (1): 213-219. 10.1054/bjoc.1999.0902.CrossRefPubMed
26.
go back to reference Thorlund K, Imberger G, Walsh M, Chu R, Gluud C, Wetterslev J, Guyatt G, Devereaux PJ, Thabane L: The number of patients and events required to limit the risk of overestimation of intervention effects in meta-analysis-a simulation study. PLoS One. 2011, 6: e25491-10.1371/journal.pone.0025491.CrossRefPubMedPubMedCentral Thorlund K, Imberger G, Walsh M, Chu R, Gluud C, Wetterslev J, Guyatt G, Devereaux PJ, Thabane L: The number of patients and events required to limit the risk of overestimation of intervention effects in meta-analysis-a simulation study. PLoS One. 2011, 6: e25491-10.1371/journal.pone.0025491.CrossRefPubMedPubMedCentral
27.
go back to reference Pereira TV, Horwitz RI, Ioannidis JP: Empirical evaluation of very large treatment effects of medical interventions. JAMA. 2012, 308: 1676-1684. 10.1001/jama.2012.13444.CrossRefPubMed Pereira TV, Horwitz RI, Ioannidis JP: Empirical evaluation of very large treatment effects of medical interventions. JAMA. 2012, 308: 1676-1684. 10.1001/jama.2012.13444.CrossRefPubMed
28.
go back to reference Mehta CR, Pocock SJ: Adaptive increase in sample size when interim results are promising: a practical guide with examples. Stat Med. 2011, 30 (28): 3267-3284. 10.1002/sim.4102.CrossRefPubMed Mehta CR, Pocock SJ: Adaptive increase in sample size when interim results are promising: a practical guide with examples. Stat Med. 2011, 30 (28): 3267-3284. 10.1002/sim.4102.CrossRefPubMed
29.
go back to reference Jennison C, Turnbull BW: Efficient group sequential designs when there are several effect sizes under consideration. Stat Med. 2005, 25: 917-932.CrossRef Jennison C, Turnbull BW: Efficient group sequential designs when there are several effect sizes under consideration. Stat Med. 2005, 25: 917-932.CrossRef
30.
go back to reference O'Hagan A, Stevens JW, Campbell MJ: Assurance in clinical trial design. Pharm Stat. 2005, 4 (3): 187-201. 10.1002/pst.175.CrossRef O'Hagan A, Stevens JW, Campbell MJ: Assurance in clinical trial design. Pharm Stat. 2005, 4 (3): 187-201. 10.1002/pst.175.CrossRef
31.
go back to reference Turner RM, Bird SM, Higgins JP: The impact of study size on meta-analyses: examination of underpowered studies in Cochrane reviews. PLoS One. 2013, 8 (3): e59202-10.1371/journal.pone.0059202.CrossRefPubMedPubMedCentral Turner RM, Bird SM, Higgins JP: The impact of study size on meta-analyses: examination of underpowered studies in Cochrane reviews. PLoS One. 2013, 8 (3): e59202-10.1371/journal.pone.0059202.CrossRefPubMedPubMedCentral
32.
go back to reference Sully BG, Julious SA, Nicholl J: A reinvestigation of recruitment to randomised, controlled, multicenter trials: a review of trials funded by two UK funding agencies. Trials. 2013, 14: 166-10.1186/1745-6215-14-166.CrossRefPubMedPubMedCentral Sully BG, Julious SA, Nicholl J: A reinvestigation of recruitment to randomised, controlled, multicenter trials: a review of trials funded by two UK funding agencies. Trials. 2013, 14: 166-10.1186/1745-6215-14-166.CrossRefPubMedPubMedCentral
33.
go back to reference Levin GP, Emerson SC, Emerson SS: Adaptive clinical trial designs with pre-specified rules for modifying the sample size: understanding efficient types of adaptation. Stat Med. 2012, 32 (8): 1259-1275.CrossRefPubMed Levin GP, Emerson SC, Emerson SS: Adaptive clinical trial designs with pre-specified rules for modifying the sample size: understanding efficient types of adaptation. Stat Med. 2012, 32 (8): 1259-1275.CrossRefPubMed
34.
go back to reference DeMets DL, Lan KK: Interim analysis: the alpha spending function approach. Stat Med. 1994, 13 (13–14): 1341-1356.CrossRefPubMed DeMets DL, Lan KK: Interim analysis: the alpha spending function approach. Stat Med. 1994, 13 (13–14): 1341-1356.CrossRefPubMed
35.
go back to reference Bassler D, Montori VM, Briel M, Glasziou P, Walter SD, Ramsay T, Guyatt G: Reflections on meta-analyses involving trials stopped early for benefit: is there a problem and if so, what is it?. Stat Methods Med Res. 2013, 22 (2): 159-168. 10.1177/0962280211432211.CrossRefPubMed Bassler D, Montori VM, Briel M, Glasziou P, Walter SD, Ramsay T, Guyatt G: Reflections on meta-analyses involving trials stopped early for benefit: is there a problem and if so, what is it?. Stat Methods Med Res. 2013, 22 (2): 159-168. 10.1177/0962280211432211.CrossRefPubMed
36.
go back to reference Lindley DV: A statistical paradox. Biometrika. 1957, 44 (1/2): 187-192. 10.2307/2333251.CrossRef Lindley DV: A statistical paradox. Biometrika. 1957, 44 (1/2): 187-192. 10.2307/2333251.CrossRef
37.
go back to reference Guyatt GH, Briel M, Glasziou P, Bassler D, Montori VM: Problems of stopping trials early. BMJ. 2012, 344: e3863-10.1136/bmj.e3863.CrossRefPubMed Guyatt GH, Briel M, Glasziou P, Bassler D, Montori VM: Problems of stopping trials early. BMJ. 2012, 344: e3863-10.1136/bmj.e3863.CrossRefPubMed
38.
go back to reference Wald A: Sequential tests of statistical hypotheses. Ann Math Stat. 1945, 16: 117-186. 10.1214/aoms/1177731118.CrossRef Wald A: Sequential tests of statistical hypotheses. Ann Math Stat. 1945, 16: 117-186. 10.1214/aoms/1177731118.CrossRef
39.
go back to reference Zhang J, Quan H, Ng J, Stepanavage ME: Some statistical methods for multiple endpoints in clinical trials. Control Clin Trials. 1997, 18: 204-221. 10.1016/S0197-2456(96)00129-8.CrossRefPubMed Zhang J, Quan H, Ng J, Stepanavage ME: Some statistical methods for multiple endpoints in clinical trials. Control Clin Trials. 1997, 18: 204-221. 10.1016/S0197-2456(96)00129-8.CrossRefPubMed
40.
go back to reference Imberger G, Vejlby AD, Hansen SB, Møller AM, Wetterslev J: Statistical multiplicity in systematic reviews of anaesthesia interventions: a quantification and comparison between Cochrane and non-Cochrane reviews. PLoS One. 2011, 6: e28422-10.1371/journal.pone.0028422.CrossRefPubMedPubMedCentral Imberger G, Vejlby AD, Hansen SB, Møller AM, Wetterslev J: Statistical multiplicity in systematic reviews of anaesthesia interventions: a quantification and comparison between Cochrane and non-Cochrane reviews. PLoS One. 2011, 6: e28422-10.1371/journal.pone.0028422.CrossRefPubMedPubMedCentral
42.
go back to reference Jennison C, Turnbull BW: Repeated confidence intervals for group sequential clinical trials. Control Clin Trials. 1984, 5 (1): 33-45. 10.1016/0197-2456(84)90148-X.CrossRefPubMed Jennison C, Turnbull BW: Repeated confidence intervals for group sequential clinical trials. Control Clin Trials. 1984, 5 (1): 33-45. 10.1016/0197-2456(84)90148-X.CrossRefPubMed
43.
go back to reference Todd S, Whitehead J, Facey KM: Point and interval estimation following a sequential clinical trial. Biometrika. 1996, 83 (2): 453-461. 10.1093/biomet/83.2.453.CrossRef Todd S, Whitehead J, Facey KM: Point and interval estimation following a sequential clinical trial. Biometrika. 1996, 83 (2): 453-461. 10.1093/biomet/83.2.453.CrossRef
44.
go back to reference Jennison C, Turnbull BW: Group Sequential Methods with Applications to Clinical Trials (Chapman & Hall/CRC Interdisciplinary Statistics). 1999, : Chapman and Hall/CRC Jennison C, Turnbull BW: Group Sequential Methods with Applications to Clinical Trials (Chapman & Hall/CRC Interdisciplinary Statistics). 1999, : Chapman and Hall/CRC
45.
go back to reference Thorlund K, Engstrøm J, Wetterslev J, Brok J, Imberger G, Gluud C: User manual for trial sequential analysis (TSA). 2011, Copenhagen, Denmark: Copenhagen Trial Unit, Centre for Clinical Intervention Research, 1-115. Available from http://www.ctu.dk/tsa Thorlund K, Engstrøm J, Wetterslev J, Brok J, Imberger G, Gluud C: User manual for trial sequential analysis (TSA). 2011, Copenhagen, Denmark: Copenhagen Trial Unit, Centre for Clinical Intervention Research, 1-115. Available from http://​www.​ctu.​dk/​tsa
46.
go back to reference Equator Network: Enhancing the QUAlity and Transparency Of health Research. Available at: http://www.equator-network.org/ 2014 Equator Network: Enhancing the QUAlity and Transparency Of health Research. Available at: http://​www.​equator-network.​org/​ 2014
47.
go back to reference Yang Q, Cui J, Chazaro I, Cupples LA, Demissie S: Power and type I error rate of false discovery rate approaches in genome-wide association studies. BMC Genet. 2005, 6 (Suppl 1): S134-10.1186/1471-2156-6-S1-S134.CrossRefPubMedPubMedCentral Yang Q, Cui J, Chazaro I, Cupples LA, Demissie S: Power and type I error rate of false discovery rate approaches in genome-wide association studies. BMC Genet. 2005, 6 (Suppl 1): S134-10.1186/1471-2156-6-S1-S134.CrossRefPubMedPubMedCentral
48.
go back to reference Bretz F, Hothorn T, Westfall P: Multiple Comparisons Using R. 2010, Boca Raton, Florida: Chapman and Hall/CRCCrossRef Bretz F, Hothorn T, Westfall P: Multiple Comparisons Using R. 2010, Boca Raton, Florida: Chapman and Hall/CRCCrossRef
49.
go back to reference Altman DG, Bland JM: How to obtain the P value from a confidence interval. BMJ. 2011, 343: d2304-10.1136/bmj.d2304.CrossRefPubMed Altman DG, Bland JM: How to obtain the P value from a confidence interval. BMJ. 2011, 343: d2304-10.1136/bmj.d2304.CrossRefPubMed
50.
go back to reference Abdi H: Encyclopedia of Measurement and Statistics. The Bonferonni and Šidák corrections for multiple comparisons. In N.J. Salkind (Ed.) page 103–107. 2007, Thousand Oaks (CA): Sage Abdi H: Encyclopedia of Measurement and Statistics. The Bonferonni and Šidák corrections for multiple comparisons. In N.J. Salkind (Ed.) page 103–107. 2007, Thousand Oaks (CA): Sage
51.
go back to reference Holm S: A simple sequentially rejective multipletestprocedure. Scand J Statist. 1979, 6: 65-70. Holm S: A simple sequentially rejective multipletestprocedure. Scand J Statist. 1979, 6: 65-70.
52.
go back to reference Dmitrienko A, Ajit C, Tamhane AC, Bretz F: Multiple testing problems in pharmaceutical statistics (Chapman & Hall/CRC Biostatistics Series). 2009, Boca Raton, Florida: Chapman and Hall/CRCCrossRef Dmitrienko A, Ajit C, Tamhane AC, Bretz F: Multiple testing problems in pharmaceutical statistics (Chapman & Hall/CRC Biostatistics Series). 2009, Boca Raton, Florida: Chapman and Hall/CRCCrossRef
53.
go back to reference Tu YH, Cheng B, Cheung YK: A note on confidence bounds after fixed-sequence multiple tests. J Stat Plan Inference. 2012, 142 (11): 2993-2998. 10.1016/j.jspi.2012.05.002.CrossRefPubMedPubMedCentral Tu YH, Cheng B, Cheung YK: A note on confidence bounds after fixed-sequence multiple tests. J Stat Plan Inference. 2012, 142 (11): 2993-2998. 10.1016/j.jspi.2012.05.002.CrossRefPubMedPubMedCentral
54.
go back to reference Wiens BL, Dmitrienko A: The fallback procedure for evaluating a single family of hypotheses. J Biopharm Stat. 2005, 15 (6): 929-942. 10.1080/10543400500265660.CrossRefPubMed Wiens BL, Dmitrienko A: The fallback procedure for evaluating a single family of hypotheses. J Biopharm Stat. 2005, 15 (6): 929-942. 10.1080/10543400500265660.CrossRefPubMed
55.
go back to reference Korn EL, Li MC, McShane LM, Simon R: An investigation of two multivariate permutation methods for controlling the false discovery proportion. Stat Med. 2007, 26 (24): 4428-4440. 10.1002/sim.2865.CrossRefPubMed Korn EL, Li MC, McShane LM, Simon R: An investigation of two multivariate permutation methods for controlling the false discovery proportion. Stat Med. 2007, 26 (24): 4428-4440. 10.1002/sim.2865.CrossRefPubMed
56.
go back to reference Westfall PH, Young S: Resampling-Based Multiple Testing: Examples and Methods for p-Value Adjustment (Wiley Series in Probability and Statistics). 1993, New York: Wiley-Interscience Westfall PH, Young S: Resampling-Based Multiple Testing: Examples and Methods for p-Value Adjustment (Wiley Series in Probability and Statistics). 1993, New York: Wiley-Interscience
57.
go back to reference Yu J, Hutson AD, Siddiqui AH, Kedron MA: Group sequential control of overall toxicity incidents in clinical trials - non-Bayesian and Bayesian approaches. Stat Methods Med Res. 2012, Epub ahead of print Yu J, Hutson AD, Siddiqui AH, Kedron MA: Group sequential control of overall toxicity incidents in clinical trials - non-Bayesian and Bayesian approaches. Stat Methods Med Res. 2012, Epub ahead of print
58.
go back to reference Thall PF, Simon RM, Shen Y: Approximate Bayesian evaluation of multiple treatment effects. Biometrics. 2000, 56: 213-219. 10.1111/j.0006-341X.2000.00213.x.CrossRefPubMed Thall PF, Simon RM, Shen Y: Approximate Bayesian evaluation of multiple treatment effects. Biometrics. 2000, 56: 213-219. 10.1111/j.0006-341X.2000.00213.x.CrossRefPubMed
59.
go back to reference Zhang X, Cutter G: Bayesian interim analysis in clinical trials. Contemp Clin Trials. 2008, 29: 751-755. 10.1016/j.cct.2008.05.007.CrossRefPubMed Zhang X, Cutter G: Bayesian interim analysis in clinical trials. Contemp Clin Trials. 2008, 29: 751-755. 10.1016/j.cct.2008.05.007.CrossRefPubMed
60.
go back to reference Jakobsen JC, Lindschou Hansen J, Storebø OJ, Simonsen E, Gluud C: The effects of cognitive therapy versus 'treatment as usual' in patients with major depressive disorder. PLoS One. 2011, 6 (8): e22890-10.1371/journal.pone.0022890.CrossRefPubMedPubMedCentral Jakobsen JC, Lindschou Hansen J, Storebø OJ, Simonsen E, Gluud C: The effects of cognitive therapy versus 'treatment as usual' in patients with major depressive disorder. PLoS One. 2011, 6 (8): e22890-10.1371/journal.pone.0022890.CrossRefPubMedPubMedCentral
61.
go back to reference Knorr U, Vinberg M, Kessing LV, Wetterslev J: Salivary cortisol in depressed patients versus control persons: a systematic review and meta-analysis. Psychoneuroendocrinol. 2010, 35: 1275-1286. 10.1016/j.psyneuen.2010.04.001.CrossRef Knorr U, Vinberg M, Kessing LV, Wetterslev J: Salivary cortisol in depressed patients versus control persons: a systematic review and meta-analysis. Psychoneuroendocrinol. 2010, 35: 1275-1286. 10.1016/j.psyneuen.2010.04.001.CrossRef
62.
go back to reference Downs JR, Clearfield M, Weis S, Whitney E, Shapiro DR, Beere PA, Langendorfer A, Stein EA, Kruyer W, Gotto AM: Primary prevention of acute coronary events with lovastatin in men and women with average cholesterol levels: results of AFCAPS/TexCAPS. Air Force/Texas Coronary Atherosclerosis Prevention Study. JAMA. 1998, 279 (20): 1615-1622. 10.1001/jama.279.20.1615.CrossRefPubMed Downs JR, Clearfield M, Weis S, Whitney E, Shapiro DR, Beere PA, Langendorfer A, Stein EA, Kruyer W, Gotto AM: Primary prevention of acute coronary events with lovastatin in men and women with average cholesterol levels: results of AFCAPS/TexCAPS. Air Force/Texas Coronary Atherosclerosis Prevention Study. JAMA. 1998, 279 (20): 1615-1622. 10.1001/jama.279.20.1615.CrossRefPubMed
63.
go back to reference Stovring H, Harmsen CG, Wisloff T, Jarbol DE, Nexoe J, Nielsen JB, Kristiansen IS: A competing risk approach for the European Heart SCORE model based on cause-specific and all-cause mortality. Eur J Prev Cardiol. 2012, 20 (5): 827-836.CrossRefPubMed Stovring H, Harmsen CG, Wisloff T, Jarbol DE, Nexoe J, Nielsen JB, Kristiansen IS: A competing risk approach for the European Heart SCORE model based on cause-specific and all-cause mortality. Eur J Prev Cardiol. 2012, 20 (5): 827-836.CrossRefPubMed
64.
go back to reference Prasad V, Vandross A: Cardiovascular primary prevention: how high should we set the bar?. Arch Int Med. 2012, 172: 656-659. 10.1001/archinternmed.2012.812.CrossRef Prasad V, Vandross A: Cardiovascular primary prevention: how high should we set the bar?. Arch Int Med. 2012, 172: 656-659. 10.1001/archinternmed.2012.812.CrossRef
65.
go back to reference Guyatt G, Oxman AD, Akl EA, Kunz R, Vist G, Brozek J, Norris S, Falck-Ytter Y, Glasziou P, DeBeer H, Jaeschke R, Rind D, Meerpohl J, Dahm P, Schunemann HJ: GRADE guidelines: 1. Introduction-GRADE evidence profiles and summary of findings Tables. J Clin Epidemiol. 2011, 64 (4): 383-394. 10.1016/j.jclinepi.2010.04.026.CrossRefPubMed Guyatt G, Oxman AD, Akl EA, Kunz R, Vist G, Brozek J, Norris S, Falck-Ytter Y, Glasziou P, DeBeer H, Jaeschke R, Rind D, Meerpohl J, Dahm P, Schunemann HJ: GRADE guidelines: 1. Introduction-GRADE evidence profiles and summary of findings Tables. J Clin Epidemiol. 2011, 64 (4): 383-394. 10.1016/j.jclinepi.2010.04.026.CrossRefPubMed
66.
go back to reference Guyatt G, Oxman AD, Sultan S, Brozek J, Glasziou P, Alonso-Coello P, Atkins D, Kunz R, Montori V, Jaeschke R, Rind D, Dahm P, Akl EA, Meerpohl J, Vist G, Berliner E, Norris S, Falck-Ytter Y, Schunemann HJ: GRADE guidelines: 11. Making an overall rating of confidence in effect estimates for a single outcome and for all outcomes. J Clin Epidemiol. 2013, 66 (2): 151-157. 10.1016/j.jclinepi.2012.01.006.CrossRefPubMed Guyatt G, Oxman AD, Sultan S, Brozek J, Glasziou P, Alonso-Coello P, Atkins D, Kunz R, Montori V, Jaeschke R, Rind D, Dahm P, Akl EA, Meerpohl J, Vist G, Berliner E, Norris S, Falck-Ytter Y, Schunemann HJ: GRADE guidelines: 11. Making an overall rating of confidence in effect estimates for a single outcome and for all outcomes. J Clin Epidemiol. 2013, 66 (2): 151-157. 10.1016/j.jclinepi.2012.01.006.CrossRefPubMed
67.
go back to reference Jüni P, Nartey L, Reichenbach S, Sterchi R, Dieppe PA, Egger M: Risk of cardiovascular events and rofecoxib: cumulative meta-analysis. Lancet. 2004, 364 (9450): 2021-2029. 10.1016/S0140-6736(04)17514-4.CrossRefPubMed Jüni P, Nartey L, Reichenbach S, Sterchi R, Dieppe PA, Egger M: Risk of cardiovascular events and rofecoxib: cumulative meta-analysis. Lancet. 2004, 364 (9450): 2021-2029. 10.1016/S0140-6736(04)17514-4.CrossRefPubMed
69.
go back to reference Johnston BC, Thorlund K, Schunemann HJ, Xie F, Murad MH, Montori VM, Guyatt GH: Improving the interpretation of quality of life evidence in meta-analyses: the application of minimal important difference units. Health Qual Life Outcomes. 2010, 8: 116-10.1186/1477-7525-8-116.CrossRefPubMedPubMedCentral Johnston BC, Thorlund K, Schunemann HJ, Xie F, Murad MH, Montori VM, Guyatt GH: Improving the interpretation of quality of life evidence in meta-analyses: the application of minimal important difference units. Health Qual Life Outcomes. 2010, 8: 116-10.1186/1477-7525-8-116.CrossRefPubMedPubMedCentral
70.
go back to reference Halvorsen PA, Kristiansen IS: Decisions on drug therapies by numbers needed to treat: a randomized trial. Arch Int Med. 2005, 165: 1140-1146. 10.1001/archinte.165.10.1140.CrossRef Halvorsen PA, Kristiansen IS: Decisions on drug therapies by numbers needed to treat: a randomized trial. Arch Int Med. 2005, 165: 1140-1146. 10.1001/archinte.165.10.1140.CrossRef
71.
go back to reference Chalmers I, Milne I, Trohler U, Vandenbroucke J, Morabia A, Tait G, Dukan E: The James Lind Library: explaining and illustrating the evolution of fair tests of medical treatments. J R Coll Physicians Edinb. 2008, 38 (3): 259-264.PubMed Chalmers I, Milne I, Trohler U, Vandenbroucke J, Morabia A, Tait G, Dukan E: The James Lind Library: explaining and illustrating the evolution of fair tests of medical treatments. J R Coll Physicians Edinb. 2008, 38 (3): 259-264.PubMed
72.
go back to reference The Library and Information Services Department, The Royal College of Physicians of Edinburgh: James Lind Library. Available online at: http://www.jameslindlibrary.org/ 2003 The Library and Information Services Department, The Royal College of Physicians of Edinburgh: James Lind Library. Available online at: http://​www.​jameslindlibrary​.​org/​ 2003
74.
go back to reference Garthwaite P, Kadane JB, O'Hagan A: Statistical Methods for Eliciting Probability Distributions. J Am Stat Assoc. 2012, 100 (470): Garthwaite P, Kadane JB, O'Hagan A: Statistical Methods for Eliciting Probability Distributions. J Am Stat Assoc. 2012, 100 (470):
75.
go back to reference Ioannidis J: Contradicted and initially stronger effects in highly cited clinical research. JAMA. 2005, 294 (2): 218-228. 10.1001/jama.294.2.218.CrossRefPubMed Ioannidis J: Contradicted and initially stronger effects in highly cited clinical research. JAMA. 2005, 294 (2): 218-228. 10.1001/jama.294.2.218.CrossRefPubMed
76.
go back to reference Wetterslev J, Thorlund K, Brok J, Gluud C: Trial sequential analysis may establish when firm evidence is reached in cumulative meta-analysis. J Clin Epidemiol. 2008, 61 (1): 64-75. 10.1016/j.jclinepi.2007.03.013.CrossRefPubMed Wetterslev J, Thorlund K, Brok J, Gluud C: Trial sequential analysis may establish when firm evidence is reached in cumulative meta-analysis. J Clin Epidemiol. 2008, 61 (1): 64-75. 10.1016/j.jclinepi.2007.03.013.CrossRefPubMed
77.
go back to reference Higgins JP, Whitehead A, Simmonds M: Sequential methods for random-effects meta-analysis. Stat Med. 2011, 30 (9): 903-921. 10.1002/sim.4088.CrossRefPubMed Higgins JP, Whitehead A, Simmonds M: Sequential methods for random-effects meta-analysis. Stat Med. 2011, 30 (9): 903-921. 10.1002/sim.4088.CrossRefPubMed
78.
go back to reference Keus F, Wetterslev J, Gluud C, van Laarhoven CJ: Evidence at a glance: error matrix approach for overviewing available evidence. BMC Med Res Methodol. 2010, 10: 90-10.1186/1471-2288-10-90.CrossRefPubMedPubMedCentral Keus F, Wetterslev J, Gluud C, van Laarhoven CJ: Evidence at a glance: error matrix approach for overviewing available evidence. BMC Med Res Methodol. 2010, 10: 90-10.1186/1471-2288-10-90.CrossRefPubMedPubMedCentral
80.
go back to reference Higgins JP, Spiegelhalter DJ: Being sceptical about meta-analyses: a Bayesian perspective on magnesium trials in myocardial infarction. Int J Epidemiol. 2002, 31 (1): 96-104. 10.1093/ije/31.1.96.CrossRefPubMed Higgins JP, Spiegelhalter DJ: Being sceptical about meta-analyses: a Bayesian perspective on magnesium trials in myocardial infarction. Int J Epidemiol. 2002, 31 (1): 96-104. 10.1093/ije/31.1.96.CrossRefPubMed
81.
go back to reference Korn EL, Freidlin B: The likelihood as statistical evidence in multiple comparisons in clinical trials: no free lunch. Biom J. 2006, 48 (3): 346-355. 10.1002/bimj.200510216.CrossRefPubMed Korn EL, Freidlin B: The likelihood as statistical evidence in multiple comparisons in clinical trials: no free lunch. Biom J. 2006, 48 (3): 346-355. 10.1002/bimj.200510216.CrossRefPubMed
82.
go back to reference Lunn D, Spiegelhalter D, Thomas A, Best N: The BUGS project: Evolution, critique and future directions. Stat Med. 2009, 28 (25): 3049-3067. 10.1002/sim.3680.CrossRefPubMed Lunn D, Spiegelhalter D, Thomas A, Best N: The BUGS project: Evolution, critique and future directions. Stat Med. 2009, 28 (25): 3049-3067. 10.1002/sim.3680.CrossRefPubMed
83.
go back to reference Gaziano JM, Sesso HD, Christen WG, Bubes V, Smith JP, MacFadyen J, Schvartz M, Manson JE, Glynn RJ, Buring JE: Multivitamins in the prevention of cancer in men: the Physicians' Health Study II. JAMA. 2012, 308 (18): 1871-1880. 10.1001/jama.2012.14641.CrossRefPubMedPubMedCentral Gaziano JM, Sesso HD, Christen WG, Bubes V, Smith JP, MacFadyen J, Schvartz M, Manson JE, Glynn RJ, Buring JE: Multivitamins in the prevention of cancer in men: the Physicians' Health Study II. JAMA. 2012, 308 (18): 1871-1880. 10.1001/jama.2012.14641.CrossRefPubMedPubMedCentral
84.
go back to reference Christen WG, Gaziano JM, Hennekens CH: Design of Physicians' Health Study II–a randomized trial of beta-carotene. Ann Epidemiol. 2000, 10 (2): 125-134. 10.1016/S1047-2797(99)00042-3.CrossRefPubMed Christen WG, Gaziano JM, Hennekens CH: Design of Physicians' Health Study II–a randomized trial of beta-carotene. Ann Epidemiol. 2000, 10 (2): 125-134. 10.1016/S1047-2797(99)00042-3.CrossRefPubMed
85.
go back to reference Bjelakovic G, Nikolova D, Gluud LL, Simonetti RG, Gluud C: Antioxidant supplements for prevention of mortality in healthy participants and patients with various diseases. Cochrane Database Syst Rev. 2012, 3: CD007176 Bjelakovic G, Nikolova D, Gluud LL, Simonetti RG, Gluud C: Antioxidant supplements for prevention of mortality in healthy participants and patients with various diseases. Cochrane Database Syst Rev. 2012, 3: CD007176
86.
go back to reference Bjelakovic G, Nikolova D, Simonetti RG, Gluud C: Antioxidant supplements for preventing gastrointestinal cancers. Cochrane Database of Syst Rev. 3: CD004183- Bjelakovic G, Nikolova D, Simonetti RG, Gluud C: Antioxidant supplements for preventing gastrointestinal cancers. Cochrane Database of Syst Rev. 3: CD004183-
87.
go back to reference Cortés-Jofré M, Rueda JR, Corsini-Muñoz G, Fonseca-Cortés C, Caraballoso M, Bonfill Cosp X: Drugs for preventing lung cancer in healthy people. Cochrane Database of Syst Rev. 10: CD002141- Cortés-Jofré M, Rueda JR, Corsini-Muñoz G, Fonseca-Cortés C, Caraballoso M, Bonfill Cosp X: Drugs for preventing lung cancer in healthy people. Cochrane Database of Syst Rev. 10: CD002141-
88.
go back to reference Shakur H, Roberts I, Bautista R, Caballero J, Coats T, Dewan Y, El-Sayed H, Gogichaishvili T, Gupta S, Herrera J, Hunt B, Iribhogbe P, Izurieta M, Khamis H, Komolafe E, Marrero MA, Mejia-Mantilla J, Miranda J, Morales C, Olaomi O, Olldashi F, Perel P, Peto R, Ramana PV, Ravi RR, Yutthakasemsunt S: Effects of tranexamic acid on death, vascular occlusive events, and blood transfusion in trauma patients with significant haemorrhage (CRASH-2): a randomised, placebo-controlled trial. Lancet. 2010, 376 (9734): 23-32.CrossRefPubMed Shakur H, Roberts I, Bautista R, Caballero J, Coats T, Dewan Y, El-Sayed H, Gogichaishvili T, Gupta S, Herrera J, Hunt B, Iribhogbe P, Izurieta M, Khamis H, Komolafe E, Marrero MA, Mejia-Mantilla J, Miranda J, Morales C, Olaomi O, Olldashi F, Perel P, Peto R, Ramana PV, Ravi RR, Yutthakasemsunt S: Effects of tranexamic acid on death, vascular occlusive events, and blood transfusion in trauma patients with significant haemorrhage (CRASH-2): a randomised, placebo-controlled trial. Lancet. 2010, 376 (9734): 23-32.CrossRefPubMed
89.
go back to reference Perner A, Haase N, Guttormsen AB, Tenhunen J, Klemenzson G, Aneman A, Madsen KR, Moller MH, Elkjaer JM, Poulsen LM, Bendtsen A, Winding R, Steensen M, Berezowicz P, Soe-Jensen P, Bestle M, Strand K, Wiis J, White JO, Thornberg KJ, Quist L, Nielsen J, Andersen LH, Holst LB, Thormar K, Kjaeldgaard AL, Fabritius ML, Mondrup F, Pott FC, Moller T, et al: Hydroxyethyl starch 130/0.42 versus Ringer's acetate in severe sepsis. N Eng J Med. 2012, 367 (2): 124-134. 10.1056/NEJMoa1204242.CrossRef Perner A, Haase N, Guttormsen AB, Tenhunen J, Klemenzson G, Aneman A, Madsen KR, Moller MH, Elkjaer JM, Poulsen LM, Bendtsen A, Winding R, Steensen M, Berezowicz P, Soe-Jensen P, Bestle M, Strand K, Wiis J, White JO, Thornberg KJ, Quist L, Nielsen J, Andersen LH, Holst LB, Thormar K, Kjaeldgaard AL, Fabritius ML, Mondrup F, Pott FC, Moller T, et al: Hydroxyethyl starch 130/0.42 versus Ringer's acetate in severe sepsis. N Eng J Med. 2012, 367 (2): 124-134. 10.1056/NEJMoa1204242.CrossRef
90.
go back to reference Perner A, Haase N, Wetterslev J, Aneman A, Tenhunen J, Guttormsen AB, Klemenzson G, Pott F, Bodker KD, Badstolokken PM, Bendtsen A, Soe-Jensen P, Tousi H, Bestle M, Pawlowicz M, Winding R, Bulow HH, Kancir C, Steensen M, Nielsen J, Fogh B, Madsen KR, Larsen NH, Carlsson M, Wiis J, Petersen JA, Iversen S, Schoidt O, Leivdal S, Berezowicz P, et al: Comparing the effect of hydroxyethyl starch 130/0.4 with balanced crystalloid solution on mortality and kidney failure in patients with severe sepsis (6S--Scandinavian Starch for Severe Sepsis/Septic Shock trial): study protocol, design and rationale for a double-blinded, randomised clinical trial. Trials. 2011, 12 (1): 24-10.1186/1745-6215-12-24.CrossRefPubMedPubMedCentral Perner A, Haase N, Wetterslev J, Aneman A, Tenhunen J, Guttormsen AB, Klemenzson G, Pott F, Bodker KD, Badstolokken PM, Bendtsen A, Soe-Jensen P, Tousi H, Bestle M, Pawlowicz M, Winding R, Bulow HH, Kancir C, Steensen M, Nielsen J, Fogh B, Madsen KR, Larsen NH, Carlsson M, Wiis J, Petersen JA, Iversen S, Schoidt O, Leivdal S, Berezowicz P, et al: Comparing the effect of hydroxyethyl starch 130/0.4 with balanced crystalloid solution on mortality and kidney failure in patients with severe sepsis (6S--Scandinavian Starch for Severe Sepsis/Septic Shock trial): study protocol, design and rationale for a double-blinded, randomised clinical trial. Trials. 2011, 12 (1): 24-10.1186/1745-6215-12-24.CrossRefPubMedPubMedCentral
91.
go back to reference Haase N, Perner A, Hennings LI, Siegemund M, Lauridsen B, Wetterslev M, Wetterslev J: Hydroxyethyl starch 130/0.38-0.45 versus crystalloid or albumin in patients with sepsis: systematic review with meta-analysis and trial sequential analysis. BMJ. 2013, 346: f839-10.1136/bmj.f839.CrossRefPubMedPubMedCentral Haase N, Perner A, Hennings LI, Siegemund M, Lauridsen B, Wetterslev M, Wetterslev J: Hydroxyethyl starch 130/0.38-0.45 versus crystalloid or albumin in patients with sepsis: systematic review with meta-analysis and trial sequential analysis. BMJ. 2013, 346: f839-10.1136/bmj.f839.CrossRefPubMedPubMedCentral
Metadata
Title
The thresholds for statistical and clinical significance – a five-step procedure for evaluation of intervention effects in randomised clinical trials
Authors
Janus Christian Jakobsen
Christian Gluud
Per Winkel
Theis Lange
Jørn Wetterslev
Publication date
01-12-2014
Publisher
BioMed Central
Published in
BMC Medical Research Methodology / Issue 1/2014
Electronic ISSN: 1471-2288
DOI
https://doi.org/10.1186/1471-2288-14-34

Other articles of this Issue 1/2014

BMC Medical Research Methodology 1/2014 Go to the issue