Top

Systematic Reviews

Published in:

Open Access 01-12-2022 | Protocol

Major mistakes and errors in the use of Trial Sequential Analysis in systematic reviews or meta-analyses – protocol for a systematic review

Authors: Christian Gunge Riberholt, Markus Harboe Olsen, Joachim Birch Milan, Christian Gluud

Published in: Systematic Reviews | Issue 1/2022

Abstract

Background

Adequately conducted systematic reviews with meta-analyses are considered the highest level of evidence and thus directly defines many clinical guidelines. However, the risks of type I and II errors in meta-analyses are substantial. Trial Sequential Analysis is a method for controlling these risks. Erroneous use of the method might lead to research waste or misleading conclusions.

Methods

The current protocol describes a systematic review aimed to identify common and major mistakes and errors in the use of Trial Sequential Analysis by evaluating published systematic reviews and meta-analyses that include this method. We plan to include all studies using Trial Sequential Analysis published from January 2018 to January 2022, an estimated 400 to 600 publications. We will search Medical Literature Analysis and Retrieval System Online and the Cochrane Database of Systematic Reviews, including studies with all types of participants, interventions, and outcomes. Two independent reviewers will screen titles and abstracts, include relevant full text articles, extract data from the studies into a predefined checklist, and evaluate the methodological quality of the study using the AMSTAR 2, assessing the methodological quality of the systematic reviews.

Discussion

This protocol follows the Preferred Reporting Items for Systematic Reviews and Meta-Analysis Protocols (PRISMA-P). The identified mistakes and errors will be published in peer reviewed articles and form the basis of a reviewed guideline for the use of Trial Sequential Analysis. Appropriately controlling for type I and II errors might reduce research waste and improve quality and precision of the evidence that clinical guidelines are based upon.

Available only for authorised users

Garattini S, Jakobsen JC, Wetterslev J, Bertelé V, Banzi R, Rath A, et al. Evidence-based clinical practice: Overview of threats to the validity of evidence and how to minimise them. Eur J Intern Med. 2016;32:13–21. https://doi.org/10.1016/j.ejim.2016.03.020.CrossRefPubMed

Higgins JPT, Thomas J, Chandler J, Cumpston M, Li T, Page MJ, Welch VA (editors). Cochrane Handbook for Systematic Reviews of Interventions version 6.3. Updated February 2022. Cochrane; 2022. Available from http://www.training.cochrane.org/handbook.

Ioannidis JPA. The mass production of redundant, misleading, and conflicted systematic reviews and meta-analyses. Milbank Q. 2016;94:485–514. https://doi.org/10.1111/1468-0009.12210.CrossRefPubMedPubMedCentral

Brok J, Thorlund K, Gluud C, Wetterslev J. Trial sequential analysis reveals insufficient information size and potentially false positive results in many meta-analyses. J Clin Epidemiol. 2008;61:763–9. https://doi.org/10.1016/j.jclinepi.2007.10.007.CrossRefPubMed

Jakobsen JC, Wetterslev J, Winkel P, Lange T, Gluud C. Thresholds for statistical and clinical significance in systematic reviews with meta-analytic methods. BMC Med Res Methodol. 2014;14:120.CrossRef

Shrier I, Christensen R, Juhl C, Beyene J. Meta-analysis on continuous outcomes in minimal important difference units: an application with appropriate variance calculations. J Clin Epidemiol. 2016;80:57–67. https://doi.org/10.1016/j.jclinepi.2016.07.012.CrossRefPubMed

Roberts I, Ker K, Edwards P, Beecher D, Manno D, Sydenham E. The knowledge system underpinning healthcare is not fit for purpose and must change. BMJ. 2015;350:h2463. https://doi.org/10.1136/bmj.h2463.CrossRefPubMed

Bender R, Bunce C, Clarke M, Gates S, Lange S, Pace NL, et al. Attention should be given to multiplicity issues in systematic reviews. J Clin Epidemiol. 2008;61:857–65. https://doi.org/10.1016/j.jclinepi.2008.03.004.CrossRefPubMed

Turner RM, Bird SM, Higgins JPT. The impact of study size on meta-analyses: Examination of underpowered studies in Cochrane reviews. PLOS ONE. 2013;8:e59202. https://doi.org/10.1371/journal.pone.0059202.CrossRefPubMedPubMedCentral

10.

Imberger G, Vejlby AD, Hansen SB, Møller AM, Wetterslev J. Statistical multiplicity in systematic reviews of anaesthesia interventions: a quantification and comparison between Cochrane and non-Cochrane reviews. PLOS ONE. 2011;6:e28422. https://doi.org/10.1371/JOURNAL.PONE.0028422.CrossRefPubMedPubMedCentral

11.

Kjaergard LL, Villumsen J, Gluud C. Reported methodologic quality and discrepancies between large and small randomized trials in meta-analyses. Ann Intern Med. 2001;135:982–9. https://doi.org/10.7326/0003-4819-135-11-200112040-00010.CrossRefPubMed

12.

Wetterslev J, Thorlund K, Brok J, Gluud C. Estimating required information size by quantifying diversity in random-effects model meta-analyses. BMC Med Res Methodol. 2009;9:86. https://doi.org/10.1186/1471-2288-9-86.CrossRefPubMedPubMedCentral

13.

Claire R, Gluud C, Berlin I, Coleman T, Leonardi-Bee J. Using Trial Sequential Analysis for estimating the sample sizes of further trials: example using smoking cessation intervention. BMC Med Res Methodol. 2020;20:284. https://doi.org/10.1186/s12874-020-01169-7.CrossRefPubMedPubMedCentral

14.

Wetterslev J, Thorlund K, Brok J, Gluud C. Trial sequential analysis may establish when firm evidence is reached in cumulative meta-analysis. J Clin Epidemiol. 2008;61:64–75. https://doi.org/10.1016/j.jclinepi.2007.03.013.

15.

Wetterslev J, Jakobsen JC, Gluud C. Trial Sequential Analysis in systematic reviews with meta-analysis. BMC Med Res Methodol. 2017;17:1–18. https://doi.org/10.1186/s12874-017-0315-7.

16.

Guyatt GH, Oxman AD, Kunz R, Brozek J, Alonso-Coello P, Rind D, et al. GRADE guidelines 6. Rating the quality of evidence–imprecision. J Clin Epidemiol. 2011;64:1283–93. https://doi.org/10.1016/j.jclinepi.2011.01.012.CrossRefPubMed

17.

Castellini G, Bruschettini M, Gianola S, Gluud C, Moja L. Assessing imprecision in Cochrane systematic reviews: A comparison of GRADE and Trial Sequential Analysis. Syst Rev. 2018;7:110. https://doi.org/10.1186/s13643-018-0770-1.CrossRefPubMedPubMedCentral

18.

Gartlehner G, Nussbaumer-Streit B, Wagner G, Patel S, Swinson-Evans T, Dobrescu A, et al. Increased risks for random errors are common in outcomes graded as high certainty of evidence. J Clin Epidemiol. 2019;106:50–9. https://doi.org/10.1016/j.jclinepi.2018.10.009.CrossRefPubMed

19.

Thorlund K, Wetterslev J, Brok J, Gluud C. Trial sequential analyses of six meta-analyses considering heterogeneity and trial weight. In: Corroboree. Abstracts of the 13th Cochrane Colloquium; 2005 22-26 Oct; Melbourne, Australia. 2005. https://abstracts.cochrane.org/2005-melbourne/trial-sequential-analyses-six-meta-analyses-considering-heterogeneity-and-trial.

20.

Thorlund K, Devereaux PJ, Wetterslev J, Guyatt G, Ioannidis JPA, Thabane L, et al. Can trial sequential monitoring boundaries reduce spurious inferences from meta-analyses? Int J Epidemiol. 2009;38:276–86. https://doi.org/10.1093/ije/dyn179.CrossRefPubMed

21.

Copenhagen Trial Unit, Centre for Clinical Intervention Research. TSA Trial Sequential Analysis. 0.9.5.10 Beta. [Computer program]. Copenhagen Trial Unit, Centre for Clinical Intervention Research 2022. https://www.ctu.dk/tsa/downloads/. Accessed 26 Apr 2022.

22.

Brok J, Thorlund K, Wetterslev J, Gluud C. Apparently conclusive meta-analyses may be inconclusive–Trial sequential analysis adjustment of random error risk due to repetitive testing of accumulating data in apparently conclusive neonatal meta-analyses. Int J Epidemiol. 2009;38:287–98. https://doi.org/10.1093/ije/dyn188.CrossRefPubMed

23.

Moher D, Shamseer L, Clarke M, Ghersi D, Liberatî A, Petticrew M, et al. Preferred reporting items for systematic review and meta-analysis protocols (PRISMA-P) 2015 statement. Syst Rev. 2015;4(1):1. https://doi.org/10.1186/2046-4053-4-1.CrossRefPubMedPubMedCentral

24.

Covidence systematic review software. Veritas Health Innovation, Melbourne, Australia. https://www.covidence.org. Accessed 3 Jun 2021.

25.

Shea BJ, Reeves BC, Wells G, Thuku M, Hamel C, Moran J, et al. AMSTAR 2: a critical appraisal tool for systematic reviews that include randomised or non-randomised studies of healthcare interventions, or both. BMJ. 2017;358:4008. https://doi.org/10.1136/bmj.j4008.CrossRef

26.

Moher D, Liberati A, Tetzlaff J, Altman DG; PRISMA Group. Preferred reporting items for systematic reviews and meta-analyses: The PRISMA statement. PLOS MED. 2009;6:e1000097. https://doi.org/10.1371/journal.pmed.1000097.

27.

Harris PA, Taylor R, Thielke R, Payne J, Gonzalez N, Conde JG. Research electronic data capture (REDCap)— a metadata-driven methodology and workflow process for providing translational research informatics support. J Biomed Inform. 2009;42(2):377–81. https://doi.org/10.1016/j.jbi.2008.08.010.

28.

Harris PA, Taylor R, Thielke R, Payne J, Gonzalez N, Conde JG. Research electronic data capture (REDCap)-A metadata-driven methodology and workflow process for providing translational research informatics support. J Biomed Inform. 2009;42:377–81.CrossRef

29.

Harris PA, Taylor R, Minor BL, Elliott V, Fernandez M, O’Neal L, et al. The REDCap consortium: Building an international community of software platform partners. J Biomed Inform. 2019;95:103208.CrossRef

30.

Thorlund K, Engstrøm J, Wetterslev J, Brok J, Imberger G, Gluud C. User manual for Trial Sequential Analysis (TSA) [pdf]. 2nd ed. Copenhagen: Copenhagen Trial Unit, Centre for Clinical Intervention Research, 2017: pp 1–119. www.ctu.dk/tsa/files/tsa_manual.pdf.

31.

Schünemann H, Brożek J, Guyatt G, Oxman A (editors). GRADE handbook for grading quality of evidence and strength of recommendations. Updated October 2013. The GRADE Working Group; 2013. Available from http://www.guidelinedevelopment.org/handbook. Accessed 27 Apr 2022.

32.

Juul S, Nielsen EE, Feinberg J, Siddiqui F, Jørgensen CK, Barot E, et al. Interventions for treatment of COVID-19: a living systematic review with meta-analyses and trial sequential analyses (The LIVING Project). PLOS MED. 2020;17(9):e1003293. https://doi.org/10.1371/journal.pmed.1003293.CrossRefPubMedPubMedCentral

33.

Korang SK, Juul S, Nielsen EE, Feinberg J, Siddiqui F, Ong G, et al. Vaccines to prevent COVID-19: a protocol for a living systematic review with network meta-analysis including individual patient data (The LIVING VACCINE Project). Syst Rev. 2020;9:262. https://doi.org/10.1186/s13643-020-01516-1.CrossRefPubMedPubMedCentral

34.

Bangalore S, Wetterslev J, Pranesh S, Sawhney S, Gluud C, Messerli FH. Perioperative β blockers in patients having non-cardiac surgery: a meta-analysis. Lancet. 2008;372:1962–76. https://doi.org/10.1016/S0140-6736(08)61560-3.CrossRefPubMed

35.

Nielsen N, Friberg H, Gluud C, Herlitz J, Wetterslev J. Hypothermia after cardiac arrest should be further evaluated-A systematic review of randomised trials with meta-analysis and trial sequential analysis. Int J Cardiol. 2011;151:333–41. https://doi.org/10.1016/j.ijcard.2010.06.008.CrossRefPubMed

36.

Cochrane Scientific Committee. Trial Sequential Analysis or sequential meta-analysis. 2017. https://methods.cochrane.org/sites/default/files/public/uploads/2017_1_18_may_scientific_committee_agenda_docs.pdf. Accessed 15 May 2021.

37.

Cochrane Scientific Comittee. Should Cochrane apply error-adjustment methods when conducting repeated meta-analyses? 2018. https://methods.cochrane.org/sites/default/files/public/uploads/tsa_expert_panel_guidance_and_recommendation_final.pdf. Accessed 15 May 2021.

38.

Thomas J, Askie LM, Berlin JA, Elliott JH, Ghersi D, Simmonds M, et al. Chapter 22: Prospective approaches to accumulating evidence. In: Higgins JPT, Thomas J, Chandler J, Cumpston M, Li T, Page MJ, Welch VA (editors). Cochrane Handbook for Systematic Reviews of Interventions version 6.3. Updated February 2022. Cochrane; 2022. Available from http://www.training.cochrane.org/handbook.

Title: Major mistakes and errors in the use of Trial Sequential Analysis in systematic reviews or meta-analyses – protocol for a systematic review
Authors: Christian Gunge Riberholt
Markus Harboe Olsen
Joachim Birch Milan
Christian Gluud
Publication date: 01-12-2022
Publisher: BioMed Central
Published in: Systematic Reviews / Issue 1/2022
Electronic ISSN: 2046-4053
DOI: https://doi.org/10.1186/s13643-022-01987-4

Keynote webinar | Spotlight on medication adherence

Springer Medicine

Major mistakes and errors in the use of Trial Sequential Analysis in systematic reviews or meta-analyses – protocol for a systematic review

Abstract

Background

Methods

Discussion

Keynote webinar | Spotlight on medication adherence

Springer Medicine

Abstract

Background

Methods

Discussion

Please log in to get access to this content

Other articles of this Issue 1/2022

Extracellular vesicles for ischemia/reperfusion injury-induced acute kidney injury: a systematic review and meta-analysis of data from animal models

Rethinking mental wellness among adolescents: an integrative review protocol of mental health components

Age-specific information resources to address the needs of young people with stroke: a scoping review protocol

Acceptance and use of complementary and alternative medicine among medical specialists: a 15-year systematic review and data synthesis

Reducing systematic review burden using Deduklick: a novel, automated, reliable, and explainable deduplication algorithm to foster medical research

Authors’ rebuttal to Integrated Risk Information System (IRIS) response to “Assessing risk of bias in human environmental epidemiology studies using three tools: different conclusions from different tools”