Top

Published in:

Open Access 01-12-2019 | Research article

Assessments of attrition bias in Cochrane systematic reviews are highly inconsistent and thus hindering trial comparability

Authors: Andrija Babic, Ruzica Tokalic, João Amílcar Silva Cunha, Ivana Novak, Jelena Suto, Marin Vidak, Ivana Miosic, Ivana Vuka, Tina Poklepovic Pericic, Livia Puljak

Published in: BMC Medical Research Methodology | Issue 1/2019

Abstract

Background

An important part of the systematic review methodology is appraisal of the risk of bias in included studies. Cochrane systematic reviews are considered golden standard regarding systematic review methodology, but Cochrane’s instructions for assessing risk of attrition bias are vague, which may lead to inconsistencies in authors’ assessments. The aim of this study was to analyze consistency of judgments and support for judgments of attrition bias in Cochrane reviews of interventions published in the Cochrane Database of Systematic Reviews (CDSR).

Methods

We analyzed Cochrane reviews published from July 2015 to June 2016 in the CDSR. We extracted data on number of included trials, judgment of attrition risk of bias for each included trial (low, unclear or high) and accompanying support for the judgment (supporting explanation). We also assessed how many Cochrane reviews had different judgments for the same supporting explanations.

Results

In the main analysis we included 10,292 judgments and supporting explanations for attrition bias from 729 Cochrane reviews. We categorized supporting explanations for those judgments into four categories and we found that most of the supporting explanations were unclear. Numerical indicators for percent of attrition, as well as statistics related to attrition were judged very differently. One third of Cochrane review authors had more than one category of supporting explanation; some had up to four different categories. Inconsistencies were found even with the number of judgments, names of risk of bias domains and different judgments for the same supporting explanations in the same Cochrane review.

Conclusion

We found very high inconsistency in methods of appraising risk of attrition bias in recent Cochrane reviews. Systematic review authors need clear guidance about different categories they should assess and judgments for those explanations. Clear instructions about appraising risk of attrition bias will improve reliability of the Cochrane’s risk of bias tool, help authors in making decisions about risk of bias and help in making reliable decisions in healthcare.

Available only for authorised users

Tanjong-Ghogomu E, Tugwell P, Welch V. Evidence-based medicine and the Cochrane collaboration. Bull NYU Hosp Jt Dis. 2009;67(2):198–205.PubMed

Gluud LL. Bias in clinical intervention research. Am J Epidemiol. 2006;163(6):493–501.CrossRef

Higgins J, Green S: Cochrane handbook for systematic reviews of interventions version 5.1.0 [updated march 2011]. The Cochrane Collaboration, 2011. Available from https://training.cochrane.org/handbook. Accessed 30 Mar 2019.

Dumville JC, Torgerson DJ, Hewitt CE. Reporting attrition in randomised controlled trials. BMJ. 2006;332(7547):969–71.CrossRef

Akl EA, Briel M, You JJ, Sun X, Johnston BC, Busse JW, Mulla S, Lamontagne F, Bassler D, Vera C, et al. Potential impact on estimated treatment effects of information lost to follow-up in randomised controlled trials (LOST-IT): systematic review. BMJ. 2012;344:e2809.CrossRef

da Costa BR, Beckett B, Diaz A, Resta NM, Johnston BC, Egger M, Juni P, Armijo-Olivo S. Effect of standardized training on the reliability of the Cochrane risk of bias assessment tool: a prospective study. Syst Rev. 2017;6(1):44.CrossRef

Hollis S, Campbell F. What is meant by intention to treat analysis? Survey of published randomised controlled trials. BMJ. 1999;319(7211):670–4.CrossRef

Bell ML, Fiero M, Horton NJ, Hsu CH. Handling missing data in RCTs; a review of the top medical journals. BMC Med Res Methodol. 2014;14:118.CrossRef

Sackett DL, Richardson WS, Rosenberg W, Haynes RB: Evidence based medicine: how to practice and teach EBM. New York, NY, USA: Churchill Livingstone. 1997.

10.

Hartling L, Hamm MP, Milne A, Vandermeer B, Santaguida PL, Ansari M, Tsertsvadze A, Hempel S, Shekelle P, Dryden DM. Testing the risk of bias tool showed low reliability between individual reviewers and across consensus assessments of reviewer pairs. J Clin Epidemiol. 2013;66(9):973–81.CrossRef

11.

da Costa BR, Hilfiker R, Egger M. PEDro's bias: summary quality scores should not be used in meta-analysis. J Clin Epidemiol. 2013;66(1):75–7.CrossRef

12.

Propadalo I, Tranfic M, Vuka I, Barcot O, Poklepovic Pericic T, Puljak L. In Cochrane reviews risk of bias assessments for allocation concealment was frequently not in line with Cochrane’s handbook guidance. J Clin Epidemiol. 2019;106(10–17).

13.

Babic A, Pijuk A, Brazdilova L, Georgieva Y, Raposo Pereira MA, Poklepovic Pericic T, Puljak L: Judgments of other bias in Cochrane systematic reviews of interventions are highly inconsistent and thus hindering use and comparability of evidence. BioRxiv 2018, 366591; doi: https://doi.org/https://doi.org/10.1101/366591.

14.

Barcot O, Boric M, Poklepovic Pericic T, Cavar M, Dosenovic S, Vuka I, Puljak L: Judgments of risk of bias associated with random sequence generation in trials included in Cochrane systematic reviews are frequently erroneous. BioRxiv 2018, 366674; doi: https://doi.org/https://doi.org/10.1101/366674.

15.

Higgins PT, Sterne JAC, Savovic J, Page MJ, Hrobjartsson A, Boutron I, Reeves B, Eldridge S: A revised tool for assessing risk of bias in randomized trials in: Chandler J, McKenzie J, Boutron I, Welch V (editors). Cochrane methods. Cochrane database of systematic reviews 2016, issue 10 (Suppl 1). dx.doi.org/https://doi.org/10.1002/14651858.CD201601 . 2016.

Title: Assessments of attrition bias in Cochrane systematic reviews are highly inconsistent and thus hindering trial comparability
Authors: Andrija Babic
Ruzica Tokalic
João Amílcar Silva Cunha
Ivana Novak
Jelena Suto
Marin Vidak
Ivana Miosic
Ivana Vuka
Tina Poklepovic Pericic
Livia Puljak
Publication date: 01-12-2019
Publisher: BioMed Central
Published in: BMC Medical Research Methodology / Issue 1/2019
Electronic ISSN: 1471-2288
DOI: https://doi.org/10.1186/s12874-019-0717-9

At a glance: The STEP trials

Springer Medicine

Assessments of attrition bias in Cochrane systematic reviews are highly inconsistent and thus hindering trial comparability

Abstract

Background

Methods

Results

Conclusion

At a glance: The STEP trials

Springer Medicine

Abstract

Background

Methods

Results

Conclusion

Please log in to get access to this content

Other articles of this Issue 1/2019

A methodological systematic review of meta-ethnography conduct to articulate the complex analytical phases

Adherence to reporting guidelines increases the number of citations: the argument for including a methodologist in the editorial process and peer-review

Evaluating antimalarial efficacy in single-armed and comparative drug trials using competing risk survival analysis: a simulation study

Correction to: Feasibility of identifying important changes in care management resulting from cardiovascular magnetic resonance (CMR) using hospital episode data in patients who activate the primary percutaneous coronary intervention (PPCI) pathway

Use of saliva to monitor meningococcal vaccine responses: proposing a threshold in saliva as surrogate of protection

Does the provision of a DVD-based audio-visual presentation improve recruitment in a clinical trial? A randomised trial of DVD trial invitations