Top

Published in:

Open Access 01-12-2014 | Protocol

Impact of missing participant data for dichotomous outcomes on pooled effect estimates in systematic reviews: a protocol for a methodological study

Authors: Elie A Akl, Lara A Kahale, Arnav Agarwal, Nada Al-Matari, Shanil Ebrahim, Paul Elias Alexander, Matthias Briel, Romina Brignardello-Petersen, Jason W Busse, Batoul Diab, Alfonso Iorio, Joey Kwong, Ling Li, Luciane Cruz Lopes, Reem Mustafa, Ignacio Neumann, Kari AO Tikkinen, Per Olav Vandvik, Yuqing Zhang, Pablo Alonso-Coello, Gordon Guyatt

Published in: Systematic Reviews | Issue 1/2014

Abstract

Background

There is no consensus on how authors conducting meta-analysis should deal with trial participants with missing outcome data. The objectives of this study are to assess in Cochrane and non-Cochrane systematic reviews: (1) which categories of trial participants the systematic review authors consider as having missing participant data (MPD), (2) how trialists reported on participants with missing outcome data in trials, (3) whether systematic reviewer authors actually dealt with MPD in their meta-analyses of dichotomous outcomes consistently with their reported methods, and (4) the impact of different methods of dealing with MPD on pooled effect estimates in meta-analyses of dichotomous outcomes.

Methods/Design

We will conduct a methodological study of Cochrane and non-Cochrane systematic reviews. Eligible systematic reviews will include a group-level meta-analysis of a patient-important dichotomous efficacy outcome, with a statistically significant effect estimate. Teams of two reviewers will determine eligibility and subsequently extract information from each eligible systematic review in duplicate and independently, using standardized, pre-piloted forms. The teams will then use a similar process to extract information from the trials included in the meta-analyses of interest. We will assess first which categories of trial participants the systematic reviewers consider as having MPD. Second, we will assess how trialists reported on participants with missing outcome data in trials. Third, we will compare what systematic reviewers report having done, and what they actually did, in dealing with MPD in their meta-analysis. Fourth, we will conduct imputation studies to assess the effects of different methods of dealing with MPD on the pooled effect estimates of meta-analyses. We will specifically calculate for each method (1) the percentage of systematic reviews that lose statistical significance and (2) the mean change of effect estimates across systematic reviews.

Discussion

The impact of different methods of dealing with MPD on pooled effect estimates will help judge the associated risk of bias in systematic reviews. Our findings will inform recommendations regarding what assumptions for MPD should be used to test the robustness of meta-analytical results.

Available only for authorised users

Akl EA, Briel M, You JJ, Sun X, Johnston BC, Busse JW, Mulla S, Lamontagne F, Bassler D, Vera C, Alshurafa M, Katsios CM, Zhou Q, Cukierman-Yaffe T, Gangji A, Mills EJ, Walter SD, Cook DJ, Schunemann HJ, Altman DG, Guyatt GH: Potential impact on estimated treatment effects of information lost to follow-up in randomised controlled trials (LOST-IT): systematic review. BMJ. 2012, 344: e2809-10.1136/bmj.e2809.CrossRefPubMed

Alonso-Coello P, Carrasco-Labra A, Brignardello-Petersen R, Neumann I, Akl EA, Sun X, Johnston BC, Briel M, Busse JW, Glujovsky D, Granados CE, Iorio A, Irfan A, Garcia LM, Mustafa RA, Ramirez-Morera A, Sola I, Tikkinen KA, Ebrahim S, Vandvik PO, Zhang Y, Selva A, Sanabria AJ, Zazueta OE, Vernooij RW, Schunemann HJ, Guyatt GH: A methodological survey of the analysis, reporting and interpretation of absolute risk reduction in systematic revieWs (ARROW): a study protocol. Syst Rev. 2013, 2: 113-10.1186/2046-4053-2-113.CrossRefPubMedPubMedCentral

Higgins JP, Altman DG, Gotzsche PC, Juni P, Moher D, Oxman AD, Savovic J, Schulz KF, Weeks L, Sterne JA, Cochrane Bias Methods G, Cochrane Statistical Methods G: The Cochrane collaboration’s tool for assessing risk of bias in randomised trials. BMJ. 2011, 343: d5928-10.1136/bmj.d5928.CrossRefPubMedPubMedCentral

Savovic J, Weeks L, Sterne JA, Turner L, Altman DG, Moher D, Higgins JP: Evaluation of the Cochrane collaboration’s tool for assessing the risk of bias in randomized trials: focus groups, online survey, proposed recommendations and their implementation. Syst Rev. 2014, 3: 37-10.1186/2046-4053-3-37.CrossRefPubMedPubMedCentral

Akl EA, Johnston BC, Alonso-Coello P, Neumann I, Ebrahim S, Briel M, Cook DJ, Guyatt GH: Addressing dichotomous data for participants excluded from trial analysis: a guide for systematic reviewers. PLoS One. 2013, 8: e57132-10.1371/journal.pone.0057132.CrossRefPubMedPubMedCentral

Ebrahim S, Akl EA, Mustafa RA, Sun X, Walter SD, Heels-Ansdell D, Alonso-Coello P, Johnston BC, Guyatt GH: Addressing continuous data for participants excluded from trial analysis: a guide for systematic reviewers. Clin Epidemiol. 2013, 66: 1014-1021. e1011. 10.1016/j.jclinepi.2013.03.014.CrossRef

Ebrahim S, Johnston BC, Akl EA, Mustafa RA, Sun X, Walter SD, Heels-Ansdell D, Alonso-Coello P, Guyatt GH: Addressing continuous data measured with different instruments for participants excluded from trial analysis: a guide for systematic reviewers. J Clin Epidemiol. 2014, 67: 560-570. 10.1016/j.jclinepi.2013.11.014.CrossRefPubMed

Akl EA, Briel M, You JJ, Lamontagne F, Gangji A, Cukierman-Yaffe T, Alshurafa M, Sun X, Nerenberg KA, Johnston BC, Vera C, Mills EJ, Bassler D, Salazar A, Bhatnagar N, Busse JW, Khalid Z, Walter S, Cook DJ, Schunemann HJ, Altman DG, Guyatt GH: LOST to follow-up information in trials (LOST-IT): a protocol on the potential impact. Trials. 2009, 10: 40-10.1186/1745-6215-10-40.CrossRefPubMedPubMedCentral

Guyatt G, Oxman AD, Akl EA, Kunz R, Vist G, Brozek J, Norris S, Falck-Ytter Y, Glasziou P, DeBeer H, Jaeschke R, Rind D, Meerpohl J, Dahm P, Schunemann HJ: GRADE guidelines: 1. Introduction-GRADE evidence profiles and summary of findings tables. J Clin Epidemiol. 2011, 64: 383-394. 10.1016/j.jclinepi.2010.04.026.CrossRefPubMed

10.

Cramer H: Mathematical Methods of Statistics. 1946, Princeton: Princeton University Press

11.

Shea BJ, Grimshaw JM, Wells GA, Boers M, Andersson N, Hamel C, Porter AC, Tugwell P, Moher D, Bouter LM: Development of AMSTAR: a measurement tool to assess the methodological quality of systematic reviews. BMC Med Res Methodol. 2007, 7: 10-10.1186/1471-2288-7-10.CrossRefPubMedPubMedCentral

12.

White IR, Higgins JP, Wood AM: Allowing for uncertainty due to missing data in meta-analysis—part 1: two-stage methods. Stat Med. 2008, 27: 711-727. 10.1002/sim.3008.CrossRefPubMed

13.

White IR, Welton NJ, Wood AM, Ades AE, Higgins JP: Allowing for uncertainty due to missing data in meta-analysis—part 2: hierarchical models. Book Allowing for Uncertainty due to Missing Data in Meta-Analysis - Part 2: Hierarchical Models, Volume 27. 2008, 728-745.

14.

Ferreira-Gonzalez I, Busse JW, Heels-Ansdell D, Montori VM, Akl EA, Bryant DM, Alonso-Coello P, Alonso J, Worster A, Upadhye S, Jaeschke R, Schunemann HJ, Permanyer-Miralda G, Pacheco-Huergo V, Domingo-Salvany A, Wu P, Mills EJ, Guyatt GH: Problems with use of composite end points in cardiovascular trials: systematic review of randomised controlled trials. BMJ. 2007, 334: 786-10.1136/bmj.39136.682083.AE.CrossRefPubMedPubMedCentral

15.

Sun X, Briel M, Busse JW, You JJ, Akl EA, Mejza F, Bala MM, Bassler D, Mertz D, Diaz-Granados N, Vandvik PO, Malaga G, Srinathan SK, Dahm P, Johnston BC, Alonso-Coello P, Hassouneh B, Truong J, Dattani ND, Walter SD, Heels-Ansdell D, Bhatnagar N, Altman DG, Guyatt GH: The influence of study characteristics on reporting of subgroup analyses in randomised controlled trials: systematic review. BMJ. 2011, 342: d1569-10.1136/bmj.d1569.CrossRefPubMed

16.

Sun X, Briel M, Busse JW, You JJ, Akl EA, Mejza F, Bala MM, Bassler D, Mertz D, Diaz-Granados N, Vandvik PO, Malaga G, Srinathan SK, Dahm P, Johnston BC, Alonso-Coello P, Hassouneh B, Walter SD, Heels-Ansdell D, Bhatnagar N, Altman DG, Guyatt GH: Credibility of claims of subgroup effects in randomised controlled trials: systematic review. BMJ. 2012, 344: e1553-10.1136/bmj.e1553.CrossRefPubMed

17.

Bassler D, Briel M, Montori VM, Lane M, Glasziou P, Zhou Q, Heels-Ansdell D, Walter SD, Guyatt GH, Flynn DN, Elamin MB, Murad MH, Abu Elnour NO, Lampropulos JF, Sood A, Mullan RJ, Erwin PJ, Bankhead CR, Perera R, Ruiz Culebro C, You JJ, Mulla SM, Kaur J, Nerenberg KA, Schunemann H, Cook DJ, Lutz K, Ribic CM, Vale N, Stopit-Study Group: Stopping randomized trials early for benefit and estimation of treatment effects: systematic review and meta-regression analysis. JAMA. 2010, 303: 1180-1187. 10.1001/jama.2010.310.CrossRefPubMed

18.

Kasenda B, von Elm E, You J, Blumle A, Tomonaga Y, Saccilotto R, Amstutz A, Bengough T, Meerpohl JJ, Stegert M, Tikkinen KA, Neumann I, Carrasco-Labra A, Faulhaber M, Mulla SM, Mertz D, Akl EA, Bassler D, Busse JW, Ferreira-Gonzalez I, Lamontagne F, Nordmann A, Gloy V, Raatz H, Moja L, Rosenthal R, Ebrahim S, Schandelmaier S, Xin S, Vandvik PO: Prevalence, characteristics, and publication of discontinued randomized trials. JAMA. 2014, 311: 1045-1051. 10.1001/jama.2014.1361.CrossRefPubMed

19.

Hopewell S, Boutron I, Altman DG, Ravaud P: Incorporation of assessments of risk of bias of primary studies in systematic reviews of randomised trials: a cross-sectional study. BMJ Open. 2013, 3: e003342-CrossRefPubMedPubMedCentral

20.

Moseley AM, Elkins MR, Herbert RD, Maher CG, Sherrington C: Cochrane reviews used more rigorous methods than non-cochrane reviews: survey of systematic reviews in physiotherapy. J Clin Epidemiol. 2009, 62: 1021-1030. 10.1016/j.jclinepi.2008.09.018.CrossRefPubMed

21.

Briel M, Lane M, Montori VM, Bassler D, Glasziou P, Malaga G, Akl EA, Ferreira-Gonzalez I, Alonso-Coello P, Urrutia G, Kunz R, Culebro CR, da Silva SA, Flynn DN, Elamin MB, Strahm B, Murad MH, Djulbegovic B, Adhikari NK, Mills EJ, Gwadry-Sridhar F, Kirpalani H, Soares HP, Abu Elnour NO, You JJ, Karanicolas PJ, Bucher HC, Lampropulos JF, Nordmann AJ, Burns KE: Stopping randomized trials early for benefit: a protocol of the study of trial policy of interim truncation-2 (STOPIT-2). Trials. 2009, 10: 49-10.1186/1745-6215-10-49.CrossRefPubMedPubMedCentral

22.

Kasenda B, von Elm EB, You J, Blumle A, Tomonaga Y, Saccilotto R, Amstutz A, Bengough T, Meerpohl J, Stegert M, Tikkinen KA, Neumann I, Carrasco-Labra A, Faulhaber M, Mulla S, Mertz D, Akl EA, Bassler D, Busse JW, Ferreira-Gonzalez I, Lamontagne F, Nordmann A, Rosenthal R, Schandelmaier S, Sun X, Vandvik PO, Johnston BC, Walter MA, Burnand B, Schwenkglenks M: Learning from failure–rationale and design for a study about discontinuation of randomized trials (DISCO study). BMC Med Res Methodol. 2012, 12: 131-10.1186/1471-2288-12-131.CrossRefPubMedPubMedCentral

23.

Sun X, Briel M, Busse JW, Akl EA, You JJ, Mejza F, Bala M, Diaz-Granados N, Bassler D, Mertz D, Srinathan SK, Vandvik PO, Malaga G, Alshurafa M, Dahm P, Alonso-Coello P, Heels-Ansdell DM, Bhatnagar N, Johnston BC, Wang L, Walter SD, Altman DG, Guyatt GH: Subgroup analysis of trials is rarely easy (SATIRE): a study protocol for a systematic review to characterize the analysis, reporting, and claim of subgroup effects in randomized trials. Trials. 2009, 10: 101-10.1186/1745-6215-10-101.CrossRefPubMedPubMedCentral

24.

Wu D, Akl EA, Guyatt GH, Devereaux PJ, Brignardello-Petersen R, Prediger B, Patel K, Patel N, Lu T, Zhang Y, Falavigna M, Santesso N, Mustafa RA, Zhou Q, Briel M, Schunemann HJ: Methodological survey of designed uneven randomization trials (DU-RANDOM): a protocol. Trials. 2014, 15: 33-10.1186/1745-6215-15-33.CrossRefPubMedPubMedCentral

Title: Impact of missing participant data for dichotomous outcomes on pooled effect estimates in systematic reviews: a protocol for a methodological study
Authors: Elie A Akl
Lara A Kahale
Arnav Agarwal
Nada Al-Matari
Shanil Ebrahim
Paul Elias Alexander
Matthias Briel
Romina Brignardello-Petersen
Jason W Busse
Batoul Diab
Alfonso Iorio
Joey Kwong
Ling Li
Luciane Cruz Lopes
Reem Mustafa
Ignacio Neumann
Kari AO Tikkinen
Per Olav Vandvik
Yuqing Zhang
Pablo Alonso-Coello
Gordon Guyatt
Publication date: 01-12-2014
Publisher: BioMed Central
Published in: Systematic Reviews / Issue 1/2014
Electronic ISSN: 2046-4053
DOI: https://doi.org/10.1186/2046-4053-3-137

Keynote webinar | Spotlight on medication adherence

Springer Medicine

Impact of missing participant data for dichotomous outcomes on pooled effect estimates in systematic reviews: a protocol for a methodological study

Abstract

Background

Methods/Design

Discussion

Keynote webinar | Spotlight on medication adherence

Springer Medicine

Abstract

Background

Methods/Design

Discussion

Please log in to get access to this content

Other articles of this Issue 1/2014

Seeing the forests and the trees—innovative approaches to exploring heterogeneity in systematic reviews of complex interventions to enhance health system decision-making: a protocol

A systematic review protocol: social network analysis of tobacco use

The efficacy of resilience training programs: a systematic review protocol

Community pharmacy interventions for public health priorities: protocol for a systematic review of community pharmacy-delivered smoking, alcohol and weight management interventions

A systematic review of shared decision making interventions in chronic conditions: a review protocol

Trends in child and adolescent obesity prevalence according to socioeconomic position: protocol for a systematic review