Skip to main content
Key Specialties
Cardiology Diabetology Endocrinology Gastroenterology Geriatrics and Gerontology Gynecology and Obstetrics Hematology Infectious Disease Internal Medicine Nephrology Neurology Oncology Pediatrics Respiratory Medicine Rheumatology Surgery
Congress Coverage
2024 ACC Congress 2023 ESMO Congress 2023 EASD Congress 2023 ERS Congress 2023 ESC Congress
Clinical Collections
Advances in Alzheimer’s

Keynote webinar | Spotlight on medication adherence

LIVE: Thursday 27th June 2024, 18:00-19:30 (CEST)
Join our expert panel to discover why you need to understand the drivers of non-adherence in your patients, and how you can optimize medication adherence in your clinics to drastically improve patient outcomes.
ADVANCED SEARCH
Log in
Top
BMC Medical Research Methodology
Published in: BMC Medical Research Methodology 1/2014

Open Access 01-12-2014 | Correspondence

Thresholds for statistical and clinical significance in systematic reviews with meta-analytic methods

Authors: Janus Christian Jakobsen, Jørn Wetterslev, Per Winkel, Theis Lange, Christian Gluud

Published in: BMC Medical Research Methodology | Issue 1/2014

Login to get access

Abstract

Background

Thresholds for statistical significance when assessing meta-analysis results are being insufficiently demonstrated by traditional 95% confidence intervals and P-values. Assessment of intervention effects in systematic reviews with meta-analysis deserves greater rigour.

Methods

Methodologies for assessing statistical and clinical significance of intervention effects in systematic reviews were considered. Balancing simplicity and comprehensiveness, an operational procedure was developed, based mainly on The Cochrane Collaboration methodology and the Grading of Recommendations Assessment, Development, and Evaluation (GRADE) guidelines.

Results

We propose an eight-step procedure for better validation of meta-analytic results in systematic reviews (1) Obtain the 95% confidence intervals and the P-values from both fixed-effect and random-effects meta-analyses and report the most conservative results as the main results. (2) Explore the reasons behind substantial statistical heterogeneity using subgroup and sensitivity analyses (see step 6). (3) To take account of problems with multiplicity adjust the thresholds for significance according to the number of primary outcomes. (4) Calculate required information sizes (≈ the a priori required number of participants for a meta-analysis to be conclusive) for all outcomes and analyse each outcome with trial sequential analysis. Report whether the trial sequential monitoring boundaries for benefit, harm, or futility are crossed. (5) Calculate Bayes factors for all primary outcomes. (6) Use subgroup analyses and sensitivity analyses to assess the potential impact of bias on the review results. (7) Assess the risk of publication bias. (8) Assess the clinical significance of the statistically significant review results.

Conclusions

If followed, the proposed eight-step procedure will increase the validity of assessments of intervention effects in systematic reviews of randomised clinical trials.
Appendix
Available only for authorised users
Literature
1.
Sun X, Guyatt G: Meta-analysis of randomized trials for health care interventions: one for all?. J Evid Based Med. 2009, 2 (1): 53-56. 10.1111/j.1756-5391.2009.01006.x.CrossRefPubMed
2.
Murad MH, Montori VM, Ioannidis JP, Jaeschke R, Devereaux PJ, Prasad K, Neumann I, Carrasco-Labra A, Agoritsas T, Hatala R, Meade MO, Wyer P, Cook DJ, Guyatt G: How to read a systematic review and meta-analysis and apply the results to patient care: users’ guides to the medical literature. JAMA. 2014, 312 (2): 171-179. 10.1001/jama.2014.5559.CrossRefPubMed
3.
Stegenga J: Is meta-analysis the platinum standard of evidence?. Stud Hist Philos Biol Biomed Sci. 2011, 42 (4): 497-507. 10.1016/j.shpsc.2011.07.003.CrossRefPubMed
4.
Hennekens CH, DeMets D: The need for large-scale randomized evidence without undue emphasis on small trials, meta-analyses, or subgroup analyses. JAMA. 2009, 302 (21): 2361-2362. 10.1001/jama.2009.1756.CrossRefPubMed
5.
Borzak S, Ridker PM: Discordance between meta-analyses and large-scale randomized, controlled trials. Examples from the management of acute myocardial infarction. Ann Intern Med. 1995, 123 (11): 873-877. 10.7326/0003-4819-123-11-199512010-00010.CrossRefPubMed
6.
Inthout J, Ioannidis JP, Borm GF: Obtaining evidence by a single well-powered trial or several modestly powered trials. Stat Methods Med Res. 2012, Epub ahead of print
7.
Borm GF, Lemmers O, Fransen J, Donders R: The evidence provided by a single trial is less reliable than its statistical analysis suggests. J Clin Epidemiol. 2009, 62 (7): 711-715. 10.1016/j.jclinepi.2008.09.013. e711CrossRefPubMed
8.
Deeks JJ, Dinnes J, D’Amico R, Sowden AJ, Sakarovitch C, Song F, Petticrew M, Altman DG: Evaluating non-randomised intervention studies. Health Technol Assess. 2003, 7 (27): iii-x. 1–173CrossRefPubMed
9.
Ioannidis J: Contradicted and initially stronger effects in highly cited clinical research. JAMA. 2005, 294 (2): 218-228. 10.1001/jama.294.2.218.CrossRefPubMed
10.
Ioannidis JP, Haidich AB, Pappa M, Pantazis N, Kokori SI, Tektonidou MG, Contopoulos-Ioannidis DG, Lau J: Comparison of evidence of treatment effects in randomized and nonrandomized studies. JAMA. 2001, 286 (7): 821-830. 10.1001/jama.286.7.821.CrossRefPubMed
11.
Jakobsen JC, Gluud C: The necessity of randomized clinical trials. Br J Med Res. 2013, 3 (4): 1453-1468. 10.9734/BJMMR/2013/3208.CrossRef
12.
Jakobsen JC, Gluud C, Winkel P, Lange T, Wetterslev J: The thresholds for statistical and clinical significance - a five-step procedure for evaluation of intervention effects in randomised clinical trials. BMC Med Res Methodol. 2014, 14 (34): doi:10.1186/1471-2288-14-34
13.
14.
Keus F, Wetterslev J, Gluud C, van Laarhoven CJ: Evidence at a glance: error matrix approach for overviewing available evidence. BMC Med Res Methodol. 2010, 10: 90-10.1186/1471-2288-10-90.CrossRefPubMedPubMedCentral
15.
Clarke M, Brice A, Chalmers I: Accumulating research: a systematic account of how cumulative meta-analyses would have provided knowledge, improved health, reduced harm and saved resources. PLoS One. 2014, 9 (7): e102670-10.1371/journal.pone.0102670.CrossRefPubMedPubMedCentral
16.
Agoritsas T, Guyatt GH: Evidence-based medicine 20 years on: a view from the inside. Can J Neurol Sci. 2013, 40 (4): 448-449.CrossRefPubMed
17.
Guyatt G, Oxman AD, Akl EA, Kunz R, Vist G, Brozek J, Norris S, Falck-Ytter Y, Glasziou P, DeBeer H, Jaeschke R, Rind D, Meerpohl J, Dahm P, Schunemann HJ: GRADE guidelines: 1. Introduction-GRADE evidence profiles and summary of findings tables. J Clin Epidemiol. 2011, 64 (4): 383-394. 10.1016/j.jclinepi.2010.04.026.CrossRefPubMed
18.
Guyatt G, Oxman AD, Sultan S, Brozek J, Glasziou P, Alonso-Coello P, Atkins D, Kunz R, Montori V, Jaeschke R, Rind D, Dahm P, Akl EA, Meerpohl J, Vist G, Berliner E, Norris S, Falck-Ytter Y, Schunemann HJ: GRADE guidelines: 11. Making an overall rating of confidence in effect estimates for a single outcome and for all outcomes. J Clin Epidemiol. 2013, 66 (2): 151-157. 10.1016/j.jclinepi.2012.01.006.CrossRefPubMed
19.
Guyatt GH, Thorlund K, Oxman AD, Walter SD, Patrick D, Furukawa TA, Johnston BC, Karanicolas P, Akl EA, Vist G, Kunz R, Brozek J, Kupper LL, Martin SL, Meerpohl JJ, Alonso-Coello P, Christiansen R, Schunemann HJ: GRADE guidelines: 13. Preparing summary of findings tables and evidence profiles-continuous outcomes. J Clin Epidemiol. 2013, 66 (2): 173-183. 10.1016/j.jclinepi.2012.08.001.CrossRefPubMed
20.
Bjelakovic G, Gluud Lise L, Nikolova D, Whitfield K, Wetterslev J, Simonetti Rosa G, Bjelakovic M, Gluud C: Vitamin D supplementation for prevention of mortality in adults. Cochrane Database Syst Rev. 2014, 1: CD007470-PubMed
21.
DerSimonian R, Laird N: Meta-analysis in clinical trials. Control Clin Trials. 1986, 7 (3): 177-188. 10.1016/0197-2456(86)90046-2.CrossRefPubMed
22.
IntHout J, Ioannidis JP, Borm GF: The Hartung-Knapp-Sidik-Jonkman method for random effects meta-analysis is straightforward and considerably outperforms the standard DerSimonian-Laird method. BMC Med Res Methodol. 2014, 14: 25-10.1186/1471-2288-14-25.CrossRefPubMedPubMedCentral
23.
Higgins J, Thompson S, Deeks J, Altman D: Statistical heterogeneity in systematic reviews of clinical trials: a critical appraisal of guidelines and practice. J Health Serv Res Policy. 2002, 7 (1): 51-61. 10.1258/1355819021927674.CrossRefPubMed
24.
Higgins JP, Spiegelhalter DJ: Being sceptical about meta-analyses: a Bayesian perspective on magnesium trials in myocardial infarction. Int J Epidemiol. 2002, 31 (1): 96-104. 10.1093/ije/31.1.96.CrossRefPubMed
25.
Brok J, Thorlund K, Gluud C, Wetterslev J: Trial sequential analysis reveals insufficient information size and potentially false positive results in many meta-analysis. J Clin Epidemiol. 2008, 61: 763-769. 10.1016/j.jclinepi.2007.10.007.CrossRefPubMed
26.
Wetterslev J, Thorlund K, Brok J, Gluud C: Trial sequential analysis may establish when firm evidence is reached in cumulative meta-analysis. J Clin Epidemiol. 2008, 61 (1): 64-75. 10.1016/j.jclinepi.2007.03.013.CrossRefPubMed
27.
Rücker G, Schwarzer G, Carpenter JR, Schumacher M: Undue reliance on I(2) in assessing heterogeneity may mislead. BMC Med Res Methodol. 2008, 8: 79-10.1186/1471-2288-8-79.CrossRefPubMedPubMedCentral
28.
Berlin JA, Crowe BJ, Whalen E, Xia HA, Koro CE, Kuebler J: Meta-analysis of clinical trial safety data in a drug development program: answers to frequently asked questions. Clin Trials. 2013, 10 (1): 20-31. 10.1177/1740774512465495.CrossRefPubMed
29.
Bender R, Bunce C, Clarke M, Gates S, Lange S, Pace NL, Thorlund K: Attention should be given to multiplicity issues in systematic reviews. J Clin Epidemiol. 2008, 61 (9): 857-865. 10.1016/j.jclinepi.2008.03.004.CrossRefPubMed
30.
The European Agency for the Evaluation of Medical Products: Points to consider on multiplicity issues in clinical trials. Eval Med Human Use. 2002, CPMP/EWP/908/99
31.
Imberger G, Vejlby AD, Hansen SB, Møller AM, Wetterslev J: Statistical multiplicity in systematic reviews of anaesthesia interventions: a quantification and comparison between Cochrane and non-Cochrane reviews. PLoS One. 2011, 6: e28422-10.1371/journal.pone.0028422.CrossRefPubMedPubMedCentral
32.
Schulz KF, Altman DG, Moher D: CONSORT 2010 statement: updated guidelines for reporting parallel group randomized trials. Ann Int Med. 2010, 152 (11): 726-732. 10.7326/0003-4819-152-11-201006010-00232.CrossRefPubMed
33.
34.
Abdi H: The Bonferonni and Šidák corrections for multiple comparisons. Encyclopedia of Measurement and Statistics Thousand Oaks (CA): Sage. Edited by: Salkind NJ. 2007, 103-107.
35.
Williamson PR, Altman DG, Blazeby JM, Clarke M, Gargon E: The COMET (Core Outcome Measures in Effectiveness Trials) Initiative. Trials. 2011, 12 (Suppl 1): A70-10.1186/1745-6215-12-S1-A70.CrossRefPubMedCentral
36.
Altman DG, Bland JM: How to obtain the P value from a confidence interval. BMJ. 2011, 343: d2304-10.1136/bmj.d2304.CrossRefPubMed
37.
Mehrotra DV, Heyse JF: Use of the false discovery rate for evaluating clinical safety data. Stat Methods Med Res. 2004, 13 (3): 227-238.PubMed
38.
Berry SM, Berry DA: Accounting for multiplicities in assessing drug safety: a three-level hierarchical mixture model. Biometrics. 2004, 60 (2): 418-426. 10.1111/j.0006-341X.2004.00186.x.CrossRefPubMed
39.
Xia HA, Ma H, Carlin BP: Bayesian hierarchical modeling for detecting safety signals in clinical trials. J Biopharm Stat. 2011, 21 (5): 1006-1029. 10.1080/10543406.2010.520181.CrossRefPubMed
40.
Higgins JP, Thompson SG, Spiegelhalter DJ: A re-evaluation of random-effects meta-analysis. J R Stat Soc Ser A Stat Soc. 2009, 172 (1): 137-159. 10.1111/j.1467-985X.2008.00552.x.CrossRefPubMedPubMedCentral
41.
Wei Y, Higgins JP: Bayesian multivariate meta-analysis with multiple outcomes. Stat Med. 2013, 32 (17): 2911-34. 10.1002/sim.5745.CrossRefPubMed
42.
Harmonised Tripartite Guideline ICH: Statistical principles for clinical trials. International Conference on Harmonisation E9 Expert Working Group. Stat Med. 1999, 18 (15): 1905-1942.
43.
Turner RM, Bird SM, Higgins JP: The impact of study size on meta-analyses: examination of underpowered studies in Cochrane reviews. PLoS One. 2013, 8 (3): e59202-10.1371/journal.pone.0059202.CrossRefPubMedPubMedCentral
44.
Pereira TV, Horwitz RI, Ioannidis JP: Empirical evaluation of very large treatment effects of medical interventions. JAMA. 2012, 308: 1676-1684. 10.1001/jama.2012.13444.CrossRefPubMed
45.
Thorlund K, Devereaux PJ, Wetterslev J, Guyatt G, Ioannidis JP, Thabane L, Gluud LL, Als-Nielsen B, Gluud C: Can trial sequential monitoring boundaries reduce spurious inferences from meta-analyses?. Int J Epidemiol. 2009, 38 (1): 276-286. 10.1093/ije/dyn179.CrossRefPubMed
46.
Thorlund K, Imberger G, Walsh M, Chu R, Gluud C, Wetterslev J, Guyatt G, Devereaux PJ, Thabane L: The number of patients and events required to limit the risk of overestimation of intervention effects in meta-analysis – a simulation study. PLoS One. 2011, 6: e25491-10.1371/journal.pone.0025491.CrossRefPubMedPubMedCentral
47.
Thorlund K, Engstrøm J, Wetterslev J, Brok J, Imberger G, Gluud C: User manual for trial sequential analysis (TSA). Copenhagen Trial Unit, Centre for Clinical Intervention Research. 2011, Copenhagen Denmark, 1-115. Available from http://​www.​ctu.​dk/​tsa
48.
Chow SC: Sample size calculations for clinical trials. WIREs Comp Stat. 2011, 3: 414-427. 10.1002/wics.155.CrossRef
49.
Pogue JM, Yusuf S: Cumulating evidence from randomized trials: utilizing sequential monitoring boundaries for cumulative meta-analysis. Control Clin Trials. 1997, 18 (6): 580-593. 10.1016/S0197-2456(97)00051-2.CrossRefPubMed
50.
Spiegelhalter DJ, Abrams KR, Myles JP: Bayesian approaches to clinical trials and health-care evaluation (Statistics in Practice). 2004, West Sussex: John Wiley & Sons Ltd
51.
Roloff V, Higgins JP, Sutton AJ: Planning future studies based on the conditional power of a meta-analysis. Stat Med. 2013, 32 (1): 11-24. 10.1002/sim.5524.CrossRefPubMed
52.
Wetterslev J, Thorlund K, Brok J, Gluud C: Estimating required information size by quantifying diversity in random-effects model meta-analyses. BMC Med Res Methodol. 2009, 9: 86-10.1186/1471-2288-9-86.CrossRefPubMedPubMedCentral
53.
Higgins JP, Whitehead A, Simmonds M: Sequential methods for random-effects meta-analysis. Stat Med. 2011, 30 (9): 903-921. 10.1002/sim.4088.CrossRefPubMed
54.
55.
Goodman SN: Introduction to Bayesian methods I: measuring the strength of evidence. Clin Trials. 2005, 2: 282-378. 10.1191/1740774505cn098oa.CrossRefPubMed
56.
Goodman SN: Toward evidence-based medical statistics. 2: The Bayes factor. Ann Intern Med. 1999, 130 (12): 1005-1013. 10.7326/0003-4819-130-12-199906150-00019.CrossRefPubMed
57.
Korn EL, Freidlin B: The likelihood as statistical evidence in multiple comparisons in clinical trials: no free lunch. Biom J. 2006, 48 (3): 346-355. 10.1002/bimj.200510216.CrossRefPubMed
58.
Savovic J, Jones HE, Altman DG, Harris RJ, Juni P, Pildal J, Als-Nielsen B, Balk EM, Gluud C, Gluud LL, Ioannidis JP, Schulz KF, Beynon R, Welton NJ, Wood L, Moher D, Deeks JJ, Sterne JA: Influence of reported study design characteristics on intervention effect estimates from randomized, controlled trials. Ann Intern Med. 2012, 157 (6): 429-438. 10.7326/0003-4819-157-6-201209180-00537.CrossRefPubMed
59.
Schulz KF, Chalmers I, Hayes RJ, Altman DG: Empirical evidence of bias. Dimensions of methodological quality associated with estimates of treatment effects in controlled trials. JAMA. 1995, 273 (5): 408-412. 10.1001/jama.1995.03520290060030.CrossRefPubMed
60.
Moher D, Cook DJ, Eastwood S, Olkin I, Rennie D, Stroup DF: Improving the quality of reports of meta-analyses of randomised controlled trials: the QUOROM statement. Quality of Reporting of Meta-analyses. Lancet. 1999, 354 (9193): 1896-1900. 10.1016/S0140-6736(99)04149-5.CrossRefPubMed
61.
Kjaergaard L, Villumsen J, Gluud C: Reported methodologic quality and discrepancies between large and small randomized trials in meta-analysis. Ann Int Med. 2001, 135 (11): 982-989. 10.7326/0003-4819-135-11-200112040-00010.CrossRef
62.
Gluud LL, Thorlund K, Gluud C, Woods L, Harris R, Sterne JA: Correction: reported methodologic quality and discrepancies between large and small randomized trials in meta-analyses. Ann Int Med. 2008, 149 (3): 219-CrossRefPubMed
63.
Wood L, Egger M, Gluud LL, Schulz KF, Juni P, Altman DG, Gluud C, Martin RM, Wood AJ, Sterne JA: Empirical evidence of bias in treatment effect estimates in controlled trials with different interventions and outcomes: meta-epidemiological study. BMJ. 2008, 336 (7644): 601-605. 10.1136/bmj.39465.451748.AD.CrossRefPubMedPubMedCentral
64.
Lundh A, Sismondo S, Lexchin J, Busuioc OA, Bero L: Industry sponsorship and research outcome. Cochrane Database Syst Rev. 2012, 12: MR000033-PubMed
65.
Sterne JAC, White IA, Carlin JB, Spratt M, Royston P, Kenward MG, Wood AM, Carpenter JR: Multiple imputation for missing data in epidemiological and clinical research: potential and pitfalls. BMJ. 2009, 338 (b2393):
66.
Little RJ, D’Agostino R, Cohen ML, Dickersin K, Emerson SS, Farrar JT, Frangakis C, Hogan JW, Molenberghs G, Murphy SA, Neaton JD, Rotnitzky A, Scharfstein D, Shih WJ, Siegel JP, Stern H: The prevention and treatment of missing data in clinical trials. N Engl J Med. 2012, 367 (14): 1355-1360. 10.1056/NEJMsr1203730.CrossRefPubMedPubMedCentral
67.
Carpenter J, Kenward M: Multiple Imputation and its Application. 2013, West Sussex: John Wiley & Sons LtdCrossRef
68.
Egger M, Davey Smith G, Schneider M, Minder C: Bias in meta-analysis detected by a simple, graphical test. BMJ. 1997, 315 (7109): 629-634. 10.1136/bmj.315.7109.629.CrossRefPubMedPubMedCentral
69.
70.
Peters JL, Sutton AJ, Jones DR, Abrams KR, Rushton L: Contour-enhanced meta-analysis funnel plots help distinguish publication bias from other causes of asymmetry. J Clin Epidemiol. 2008, 61 (10): 991-996. 10.1016/j.jclinepi.2007.11.010.CrossRefPubMed
71.
Ioannidis JP, Trikalinos TA: The appropriateness of asymmetry tests for publication bias in meta-analyses: a large survey. CMAJ. 2007, 176 (8): 1091-1096. 10.1503/cmaj.060410.CrossRefPubMedPubMedCentral
72.
Duval S, Tweedie R: Trim and fill: A simple funnel-plot-based method of testing and adjusting for publication bias in meta-analysis. Biometrics. 2000, 56 (2): 455-463. 10.1111/j.0006-341X.2000.00455.x.CrossRefPubMed
73.
Jüni P, Nartey L, Reichenbach S, Sterchi R, Dieppe PA, Egger M: Risk of cardiovascular events and rofecoxib: cumulative meta-analysis. Lancet. 2004, 364 (9450): 2021-2029. 10.1016/S0140-6736(04)17514-4.CrossRefPubMed
74.
Wood J, Freemantle N, King M, Nazareth I: Trap of trends to statistical significance: likelihood of near significant P value becoming more significant with extra data. BMJ. 2014, 348: g2215-10.1136/bmj.g2215.CrossRefPubMed
75.
Bolland MJ, Grey A, Gamble GD, Reid IR: The effect of vitamin D supplementation on skeletal, vascular, or cancer outcomes: a trial sequential meta-analysis. Lancet Diab Endocrinol. 2014, 2 (4): 307-320. 10.1016/S2213-8587(13)70212-2.CrossRef
76.
Bjelakovic G, Nikolova D, Gluud LL, Simonetti RG, Gluud C: Antioxidant supplements for prevention of mortality in healthy participants and patients with various diseases. Cochrane Database Syst Rev. 2012, 3: CD007176-PubMed
77.
Guyatt GH, Oxman AD, Kunz R, Brozek J, Alonso-Coello P, Rind D, Devereaux PJ, Montori VM, Freyschuss B, Vist G, Jaeschke R, Williams JW, Murad MH, Sinclair D, Falck-Ytter Y, Meerpohl J, Whittington C, Thorlund K, Andrews J, Schunemann HJ: GRADE guidelines 6. Rating the quality of evidence--imprecision. J Clin Epidemiol. 2011, 64 (12): 1283-1293. 10.1016/j.jclinepi.2011.01.012.CrossRefPubMed
78.
Bauer P, Kohne K: Evaluation of experiments with adaptive interim analyses. Biometrics. 1994, 50 (4): 1029-1041. 10.2307/2533441.CrossRefPubMed
79.
Metadata
Title
Thresholds for statistical and clinical significance in systematic reviews with meta-analytic methods
Authors
Janus Christian Jakobsen
Jørn Wetterslev
Per Winkel
Theis Lange
Christian Gluud
Publication date
01-12-2014
Publisher
BioMed Central
Published in
BMC Medical Research Methodology / Issue 1/2014
Electronic ISSN: 1471-2288
DOI
https://doi.org/10.1186/1471-2288-14-120

Other articles of this Issue 1/2014

BMC Medical Research Methodology 1/2014 Go to the issue

Research article

External validation of multivariable prediction models: a systematic review of methodological conduct and reporting

Research article

Presenting simulation results in a nested loop plot

Research article

Evidence-based decision-making in infectious diseases epidemiology, prevention and control: matching research questions to study designs and quality appraisal tools

Research article

Investigation of publication bias in meta-analyses of diagnostic test accuracy: a meta-epidemiological study

Research article

Nurse awareness of clinical research: a survey in a Japanese University Hospital

Debate

Choosing sensitivity analyses for randomised trials: principles