Skip to main content
Key Specialties
Cardiology Diabetology Endocrinology Gastroenterology Geriatrics and Gerontology Gynecology and Obstetrics Hematology Infectious Disease Internal Medicine Nephrology Neurology Oncology Pediatrics Respiratory Medicine Rheumatology Surgery
Congress Coverage
2024 ACC Congress 2023 ESMO Congress 2023 EASD Congress 2023 ERS Congress 2023 ESC Congress 2023 EHA Congress 2023 ASCO Annual Meeting
Clinical Collections
Advances in Alzheimer’s

Keynote webinar | Spotlight on sleep in brain health

LIVE: Thursday 2nd May 2024, 18:00-19:30 (CEST)

Quality sleep is essential for health. But what happens to our brains when sleep patterns are disturbed? Join our experts to explore the interplay between sleep disruption and neurological diseases, and the questions that you need to be asking your patients to help you prevent the harmful effects of sleep deprivation.
ADVANCED SEARCH
Log in
Top
Systematic Reviews
Published in: Systematic Reviews 1/2018

Open Access 01-12-2018 | Methodology

Assessing imprecision in Cochrane systematic reviews: a comparison of GRADE and Trial Sequential Analysis

Authors: Greta Castellini, Matteo Bruschettini, Silvia Gianola, Christian Gluud, Lorenzo Moja

Published in: Systematic Reviews | Issue 1/2018

Login to get access

Abstract

Background

The evaluation of imprecision is a key dimension of the grading of the confidence in the estimate. Grading of Recommendations Assessment, Development and Evaluation (GRADE) gives recommendations on how to downgrade evidence for imprecision, but authors vary in their use. Trial Sequential Analysis (TSA) has been advocated for a more reliable assessment of imprecision. We aimed to evaluate reporting of and adherence to GRADE and to compare the assessment of imprecision of intervention effects assessed by GRADE and TSA in Cochrane systematic reviews.

Methods

In this cross-sectional study, we included 100 Cochrane reviews irrespective of type of intervention with a key dichotomous outcome meta-analyzed and assessed by GRADE. The methods and results sections of each review were assessed for adequacy of imprecision evaluation. We re-analyzed imprecision following the GRADE Handbook and the TSA Manual.

Results

Overall, only 13.0% of reviews stated the criteria they applied to assess imprecision. The most common dimensions were the 95% width of the confidence intervals and the optimal information size. Review authors downgraded 48.0% of key outcomes due to imprecision. When imprecision was re-analyzed following the GRADE Handbook, 64% of outcomes were downgraded. Agreement between review authors’ assessment and assessment by the authors of this study was moderate (kappa 0.43, 95% confidence interval [CI] 0.23 to 0.58). TSA downgraded 69.0% outcomes due to imprecision. Agreement between review authors’ GRADE assessment and TSA, irrespective of downgrading levels, was moderate (kappa 0.43, 95% CI 0.21 to 0.57). Agreement between our GRADE assessment following the Handbook and TSA was substantial (kappa 0.66, 95% CI 0.49 to 0.79).

Conclusions

In a sample of Cochrane reviews, methods for assessing imprecision were rarely reported. GRADE according to Handbook guidelines and TSA led to more severe judgment of imprecision rather than GRADE adopted by reviews’ authors. Cochrane initiatives to improve adherence to GRADE Handbook are warranted. TSA may transparently assist in such development.
Appendix
Available only for authorised users
Literature
1.
Guyatt GH, Oxman AD, Vist GE, Kunz R, Falck-Ytter Y, Alonso-Coello P, Schunemann HJ. GRADE: an emerging consensus on rating quality of evidence and strength of recommendations. BMJ. 2008;336:924–6.CrossRefPubMedPubMedCentral
2.
Guyatt GH, Oxman AD, Kunz R, Brozek J, Alonso-Coello P, Rind D, Devereaux PJ, Montori VM, Freyschuss B, Vist G, et al. GRADE guidelines 6. Rating the quality of evidence--imprecision. J Clin Epidemiol. 2011;64:1283–93.CrossRefPubMed
3.
Guyatt G, Oxman AD, Akl EA, Kunz R, Vist G, Brozek J, Norris S, Falck-Ytter Y, Glasziou P, DeBeer H, et al. GRADE guidelines: 1. Introduction-GRADE evidence profiles and summary of findings tables. J Clin Epidemiol. 2011;64:383–94.CrossRefPubMed
4.
Pandis N, Fleming PS, Worthington H, Salanti G. The quality of the evidence according to GRADE is predominantly low or very low in oral health systematic reviews. PLoS One. 2015;10:e0131644.CrossRefPubMedPubMedCentral
5.
Schunemann HJ. Interpreting GRADE's levels of certainty or quality of the evidence: GRADE for statisticians, considering review information size or less emphasis on imprecision? J Clin Epidemiol. 2016;75:6–15.CrossRefPubMed
6.
7.
Anttila S, Persson J, Vareman N, Sahlin NE. Conclusiveness resolves the conflict between quality of evidence and imprecision in GRADE. J Clin Epidemiol. 2016;75:1–5.CrossRefPubMed
8.
9.
Thorlund K, Imberger G, Walsh M, Chu R, Gluud C, Wetterslev J, Guyatt G, Devereaux PJ, Thabane L. The number of patients and events required to limit the risk of overestimation of intervention effects in meta-analysis—a simulation study. PLoS One. 2011;6:e25491.CrossRefPubMedPubMedCentral
10.
Jakobsen JC, Wetterslev J, Winkel P, Lange T, Gluud C. Thresholds for statistical and clinical significance in systematic reviews with meta-analytic methods. BMC Med Res Methodol. 2014;14:120.CrossRefPubMedPubMedCentral
11.
Higgins JP, Whitehead A, Simmonds M. Sequential methods for random-effects meta-analysis. Stat Med. 2011;30:903–21.CrossRefPubMed
12.
Simmonds M, Salanti G, McKenzie J, Elliott J, Living Systematic Review N. Living systematic reviews: 3. Statistical methods for updating meta-analyses. J Clin Epidemiol. 2017;91:38–46.CrossRefPubMed
13.
Wetterslev J, Jakobsen JC, Gluud C. Trial Sequential Analysis in systematic reviews with meta-analysis. BMC Med Res Methodol. 2017;17:39.
14.
Jakobsen JC, Gluud C, Winkel P, Lange T, Wetterslev J. The thresholds for statistical and clinical significance—a five-step procedure for evaluation of intervention effects in randomised clinical trials. BMC Med Res Methodol. 2014;14:34.CrossRefPubMedPubMedCentral
15.
16.
17.
Chapter 5.2.4.2 Imprecision in in systematic reviews in Schünemann H BJ, Guyatt G, Oxman A, editors. GRADE handbook for grading quality of evidence and strength of recommendations. Updated October 2013. The GRADE Working Group, 2013. Available from http://​guidelinedevelop​ment.​org/​handbook. Accessed Jan 2017.
18.
19.
Wetterslev J, Thorlund K, Brok J, Gluud C. Estimating required information size by quantifying diversity in random-effects model meta-analyses. BMC Med Res Methodol. 2009;9:86.CrossRefPubMedPubMedCentral
20.
Wetterslev J, Thorlund K, Brok J, Gluud C. Trial Sequential Analysis may establish when firm evidence is reached in cumulative meta-analysis. J Clin Epidemiol. 2008;61:64–75.CrossRefPubMed
21.
Watson PF, Petrie A. Method agreement analysis: a review of correct methodology. Theriogenology. 2010;73:1167–79.CrossRefPubMed
22.
Landis JR, Koch GG. The measurement of observer agreement for categorical data. Biometrics. 1977;33:159–74.CrossRefPubMed
23.
StataCorp. Stata Statistical Software: Release 8. College Station: StataCorp LP; 2003.
24.
Page MJ, Shamseer L, Altman DG, Tetzlaff J, Sampson M, Tricco AC, Catala-Lopez F, Li L, Reid EK, Sarkis-Onofre R, Moher D. Epidemiology and reporting characteristics of systematic reviews of biomedical research: a cross-sectional study. PLoS Med. 2016;13:e1002028.CrossRefPubMedPubMedCentral
25.
Copay AG, Subach BR, Glassman SD, Polly DW Jr, Schuler TC. Understanding the minimum clinically important difference: a review of concepts and methods. Spine J. 2007;7:541–6.CrossRefPubMed
26.
Armijo-Olivo S, Warren S, Fuentes J, Magee DJ. Clinical relevance vs. statistical significance: using neck outcomes in patients with temporomandibular disorders as an example. Man Ther. 2011;16:563–72.CrossRefPubMed
27.
Elliott JH, Synnot A, Turner T, Simmonds M, Akl EA, McDonald S, Salanti G, Meerpohl J, MacLehose H, Hilton J, et al. Living systematic review: 1. Introduction-the why, what, when, and how. J Clin Epidemiol. 2017;91:23–30.CrossRefPubMed
28.
Riva N, Puljak L, Moja L, Ageno W, Schunemann H, Magrini N, Squizzato A. Multiple overlapping systematic reviews facilitate the origin of disputes: the case of thrombolytic therapy for pulmonary embolism. J Clin Epidemiol. 2017;97:1–13.CrossRefPubMed
29.
Moher D, Liberati A, Tetzlaff J, Altman DG, Group P. Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. PLoS Med. 2009;6:e1000097.CrossRefPubMedPubMedCentral
30.
Moher D, Shamseer L, Clarke M, Ghersi D, Liberati A, Petticrew M, Shekelle P, Stewart LA, Group P-P. Preferred reporting items for systematic review and meta-analysis protocols (PRISMA-P) 2015 statement. Syst Rev. 2015;4(1).
31.
Pocock SJ, Stone GW. The primary outcome is positive—is that good enough? N Engl J Med. 2016;375:971–9.CrossRefPubMed
32.
Metadata
Title
Assessing imprecision in Cochrane systematic reviews: a comparison of GRADE and Trial Sequential Analysis
Authors
Greta Castellini
Matteo Bruschettini
Silvia Gianola
Christian Gluud
Lorenzo Moja
Publication date
01-12-2018
Publisher
BioMed Central
Published in
Systematic Reviews / Issue 1/2018
Electronic ISSN: 2046-4053
DOI
https://doi.org/10.1186/s13643-018-0770-1

Other articles of this Issue 1/2018

Systematic Reviews 1/2018 Go to the issue

Protocol

A methodological approach to identify the most reliable human milk collection method for compositional analysis: a systematic review protocol

Protocol

The prevalence of Lynch syndrome in women with endometrial cancer: a systematic review protocol

Protocol

Protocol for a systematic review of methods and cost-effectiveness findings of economic evaluations of obesity prevention and/or treatment interventions in children and adolescents

Research

Systematic review and meta-analysis of diagnostic accuracy of detection of any level of diabetic retinopathy using digital retinal imaging

Research

The effect of carbetocin compared to misoprostol in management of the third stage of labor and prevention of postpartum hemorrhage: a systematic review

Protocol

(Update of a) systematic review on the impact of elective early term (< 39th gestational week) caesarean sections on maternal and neonatal health - a protocol