Skip to main content
Top
Published in: BMC Medical Research Methodology 1/2014

Open Access 01-12-2014 | Research article

Methods for calculating confidence and credible intervals for the residual between-study variance in random effects meta-regression models

Authors: Dan Jackson, Rebecca Turner, Kirsty Rhodes, Wolfgang Viechtbauer

Published in: BMC Medical Research Methodology | Issue 1/2014

Login to get access

Abstract

Background

Meta-regression is becoming increasingly used to model study level covariate effects. However this type of statistical analysis presents many difficulties and challenges. Here two methods for calculating confidence intervals for the magnitude of the residual between-study variance in random effects meta-regression models are developed. A further suggestion for calculating credible intervals using informative prior distributions for the residual between-study variance is presented.

Methods

Two recently proposed and, under the assumptions of the random effects model, exact methods for constructing confidence intervals for the between-study variance in random effects meta-analyses are extended to the meta-regression setting. The use of Generalised Cochran heterogeneity statistics is extended to the meta-regression setting and a Newton-Raphson procedure is developed to implement the Q profile method for meta-analysis and meta-regression. WinBUGS is used to implement informative priors for the residual between-study variance in the context of Bayesian meta-regressions.

Results

Results are obtained for two contrasting examples, where the first example involves a binary covariate and the second involves a continuous covariate. Intervals for the residual between-study variance are wide for both examples.

Conclusions

Statistical methods, and R computer software, are available to compute exact confidence intervals for the residual between-study variance under the random effects model for meta-regression. These frequentist methods are almost as easily implemented as their established counterparts for meta-analysis. Bayesian meta-regressions are also easily performed by analysts who are comfortable using WinBUGS. Estimates of the residual between-study variance in random effects meta-regressions should be routinely reported and accompanied by some measure of their uncertainty. Confidence and/or credible intervals are well-suited to this purpose.
Appendix
Available only for authorised users
Literature
1.
go back to reference Knapp G, Hartung J: Improved tests for a random effects meta-regression with a single covariate. Stat Med. 2003, 22: 2693-2710. 10.1002/sim.1482.CrossRefPubMed Knapp G, Hartung J: Improved tests for a random effects meta-regression with a single covariate. Stat Med. 2003, 22: 2693-2710. 10.1002/sim.1482.CrossRefPubMed
2.
go back to reference Sharp S, Thompson SG: Explaining heterogeneity in meta-analysis: a comparison of methods. Stat Med. 1999, 18: 2693-2708. 10.1002/(SICI)1097-0258(19991030)18:20<2693::AID-SIM235>3.0.CO;2-V.CrossRefPubMed Sharp S, Thompson SG: Explaining heterogeneity in meta-analysis: a comparison of methods. Stat Med. 1999, 18: 2693-2708. 10.1002/(SICI)1097-0258(19991030)18:20<2693::AID-SIM235>3.0.CO;2-V.CrossRefPubMed
3.
go back to reference van Houwelingen HC, Arends LR, Stijnen T: Advanced methods in meta-analysis: multivariate approach and meta-regression. Stat Med. 2002, 21: 589-624. 10.1002/sim.1040.CrossRefPubMed van Houwelingen HC, Arends LR, Stijnen T: Advanced methods in meta-analysis: multivariate approach and meta-regression. Stat Med. 2002, 21: 589-624. 10.1002/sim.1040.CrossRefPubMed
4.
go back to reference Baker WL, White CM, Cappelleri JC, Kluger J, Coleman CI: Understanding heterogeneity in meta-analysis: the role of meta-regression. Int J Clin Pract. 2009, 63: 1426-1434. 10.1111/j.1742-1241.2009.02168.x.CrossRefPubMed Baker WL, White CM, Cappelleri JC, Kluger J, Coleman CI: Understanding heterogeneity in meta-analysis: the role of meta-regression. Int J Clin Pract. 2009, 63: 1426-1434. 10.1111/j.1742-1241.2009.02168.x.CrossRefPubMed
5.
go back to reference Jackson D: The significance level of meta-regression’s standard hypothesis test. Commun Stat Theory Methods. 2008, 37: 1576-1590. 10.1080/03610920801893863.CrossRef Jackson D: The significance level of meta-regression’s standard hypothesis test. Commun Stat Theory Methods. 2008, 37: 1576-1590. 10.1080/03610920801893863.CrossRef
6.
go back to reference Thompson SG, Higgins JTP: How should meta-regression analyses be undertaken and interpreted?. Stat Med. 2002, 21: 1559-1573. 10.1002/sim.1187.CrossRefPubMed Thompson SG, Higgins JTP: How should meta-regression analyses be undertaken and interpreted?. Stat Med. 2002, 21: 1559-1573. 10.1002/sim.1187.CrossRefPubMed
7.
go back to reference Biggerstaff BJ, Jackson D: The exact distribution of Cochran’s heterogeneity statistic in one-way random effects meta-analysis. Stat Med. 2008, 27: 6093-6110. 10.1002/sim.3428.CrossRefPubMed Biggerstaff BJ, Jackson D: The exact distribution of Cochran’s heterogeneity statistic in one-way random effects meta-analysis. Stat Med. 2008, 27: 6093-6110. 10.1002/sim.3428.CrossRefPubMed
8.
go back to reference Jackson D: Confidence intervals for the between-study variance in random effects meta-analysis using generalised Cochran heterogeneity statistics. Res Synth Methods. 2013, 4: 220-229.CrossRefPubMed Jackson D: Confidence intervals for the between-study variance in random effects meta-analysis using generalised Cochran heterogeneity statistics. Res Synth Methods. 2013, 4: 220-229.CrossRefPubMed
9.
go back to reference Viechtbauer W: Confidence intervals for the amount of heterogeneity in meta-analysis. Stat Med. 2007, 26: 37-52. 10.1002/sim.2514.CrossRefPubMed Viechtbauer W: Confidence intervals for the amount of heterogeneity in meta-analysis. Stat Med. 2007, 26: 37-52. 10.1002/sim.2514.CrossRefPubMed
10.
go back to reference Turner RM, Davey J, Clarke MJ, Thompson SG, Higgins JPT: Predicting the extent of heterogeneity in meta-analysis, using empirical data from the Cochrane database of systematic reviews. Int J Epidemiol. 2012, 41: 818-827. 10.1093/ije/dys041.CrossRefPubMedPubMedCentral Turner RM, Davey J, Clarke MJ, Thompson SG, Higgins JPT: Predicting the extent of heterogeneity in meta-analysis, using empirical data from the Cochrane database of systematic reviews. Int J Epidemiol. 2012, 41: 818-827. 10.1093/ije/dys041.CrossRefPubMedPubMedCentral
11.
go back to reference Viechtbauer W: Analysis of moderator effects in meta-analysis. Best Practices in Quantitative Methods. Edited by: Osborne JW, Osborne JW. 2008, Thousand Oaks, California, USA: Sage Publications, Viechtbauer W: Analysis of moderator effects in meta-analysis. Best Practices in Quantitative Methods. Edited by: Osborne JW, Osborne JW. 2008, Thousand Oaks, California, USA: Sage Publications,
12.
go back to reference Panityakul T, Bumrungsup C, Knapp G: On estimating residual heterogeneity in random effects meta-regression: a comparative study. J Stat Theory Appl. 2013, 12: 253-265. 10.2991/jsta.2013.12.3.4.CrossRef Panityakul T, Bumrungsup C, Knapp G: On estimating residual heterogeneity in random effects meta-regression: a comparative study. J Stat Theory Appl. 2013, 12: 253-265. 10.2991/jsta.2013.12.3.4.CrossRef
13.
go back to reference DerSimonian R, Kacker R: Random-effects model for meta-analysis of clinical trials: An update. Contemp Clin Trials. 2007, 28: 105-114. 10.1016/j.cct.2006.04.004.CrossRefPubMed DerSimonian R, Kacker R: Random-effects model for meta-analysis of clinical trials: An update. Contemp Clin Trials. 2007, 28: 105-114. 10.1016/j.cct.2006.04.004.CrossRefPubMed
14.
go back to reference Zhang F: Matrix Theory (2nd edition). 2010, New York: Springer Zhang F: Matrix Theory (2nd edition). 2010, New York: Springer
15.
go back to reference Cassella G, Berger Rl: Statistical Inference. 2002, Pacific Grove, USA: Duxbury Press Cassella G, Berger Rl: Statistical Inference. 2002, Pacific Grove, USA: Duxbury Press
16.
go back to reference Farebrother RW: Algorithm AS 204: the distribution of a positive linear combination ofχ 2random variables. Appl Stat. 1984, 33: 332-339. 10.2307/2347721.CrossRef Farebrother RW: Algorithm AS 204: the distribution of a positive linear combination ofχ 2random variables. Appl Stat. 1984, 33: 332-339. 10.2307/2347721.CrossRef
17.
go back to reference DerSimonian R, Laird N: Meta-analysis in clinical trials. Control Clin Trials. 1986, 7: 177-188. 10.1016/0197-2456(86)90046-2.CrossRefPubMed DerSimonian R, Laird N: Meta-analysis in clinical trials. Control Clin Trials. 1986, 7: 177-188. 10.1016/0197-2456(86)90046-2.CrossRefPubMed
18.
go back to reference Searle SR: Linear models. 1971, New York, USA: Wiley Searle SR: Linear models. 1971, New York, USA: Wiley
19.
go back to reference Biggerstaff BJ, Tweedie R: Incorporating variability in estimates of heterogeneity in the random effects model in meta-analysis. Stat Med. 1997, 16: 753-768. 10.1002/(SICI)1097-0258(19970415)16:7<753::AID-SIM494>3.0.CO;2-G.CrossRefPubMed Biggerstaff BJ, Tweedie R: Incorporating variability in estimates of heterogeneity in the random effects model in meta-analysis. Stat Med. 1997, 16: 753-768. 10.1002/(SICI)1097-0258(19970415)16:7<753::AID-SIM494>3.0.CO;2-G.CrossRefPubMed
20.
go back to reference Hartung J, Knapp G: On confidence intervals for the among-group variance in the one-way random effects model with unequal error variances. J Stat Plann Inference. 2005, 127: 157-177. 10.1016/j.jspi.2003.09.032.CrossRef Hartung J, Knapp G: On confidence intervals for the among-group variance in the one-way random effects model with unequal error variances. J Stat Plann Inference. 2005, 127: 157-177. 10.1016/j.jspi.2003.09.032.CrossRef
21.
go back to reference Knapp G, Biggerstaff BJ, Hartung J: Assessing the amount of heterogeneity in random-effects meta-analysis. Biom J. 2006, 48: 271-285. 10.1002/bimj.200510175.CrossRefPubMed Knapp G, Biggerstaff BJ, Hartung J: Assessing the amount of heterogeneity in random-effects meta-analysis. Biom J. 2006, 48: 271-285. 10.1002/bimj.200510175.CrossRefPubMed
22.
go back to reference Paule RC, Mandel J: Consensus values and weighting factors. J Res National Bureau Stand. 1982, 87: 377-385. 10.6028/jres.087.022.CrossRef Paule RC, Mandel J: Consensus values and weighting factors. J Res National Bureau Stand. 1982, 87: 377-385. 10.6028/jres.087.022.CrossRef
23.
go back to reference Berkey CS, Hoaglin DC, Mosteller F, Colditz GA: A random-effects regression model for meta-analysis. Stat Med. 1995, 14: 395-411. 10.1002/sim.4780140406.CrossRefPubMed Berkey CS, Hoaglin DC, Mosteller F, Colditz GA: A random-effects regression model for meta-analysis. Stat Med. 1995, 14: 395-411. 10.1002/sim.4780140406.CrossRefPubMed
25.
go back to reference Lambert PC, Sutton AJ, Burton PR, Abrams KR, Jones DR: How vague is vague? A simulation study of the impact of the use of vague prior distributions in MCMC using WinBUGS. Stat Med. 2005, 24: 2401-2428. 10.1002/sim.2112.CrossRefPubMed Lambert PC, Sutton AJ, Burton PR, Abrams KR, Jones DR: How vague is vague? A simulation study of the impact of the use of vague prior distributions in MCMC using WinBUGS. Stat Med. 2005, 24: 2401-2428. 10.1002/sim.2112.CrossRefPubMed
26.
go back to reference Pullenayegum EM: An informed reference prior for between-study heterogeneity in meta-analyses of binary outcomes. Stat Med. 2011, 30: 3082-3094. 10.1002/sim.4326.CrossRefPubMed Pullenayegum EM: An informed reference prior for between-study heterogeneity in meta-analyses of binary outcomes. Stat Med. 2011, 30: 3082-3094. 10.1002/sim.4326.CrossRefPubMed
27.
go back to reference Turner RM, Jackson D, Wei Y, Thompson SG, Higgins JPT: Predictive distributions for between-study heterogeneity and methods for their application in Bayesian meta-analysis. Undergoing peer review. Turner RM, Jackson D, Wei Y, Thompson SG, Higgins JPT: Predictive distributions for between-study heterogeneity and methods for their application in Bayesian meta-analysis. Undergoing peer review.
28.
go back to reference Rhodes KM, Turner RM, Higgins JPT: Predictive distributions were developed for the extent of heterogeneity in a meta-analysis, using continuous outcome data from the Cochrane database of systematic reviews. J Clin Epidemiol. (Epub ahead of print), Rhodes KM, Turner RM, Higgins JPT: Predictive distributions were developed for the extent of heterogeneity in a meta-analysis, using continuous outcome data from the Cochrane database of systematic reviews. J Clin Epidemiol. (Epub ahead of print),
29.
go back to reference Lunn DJ, Thomas A, Best N, Spiegelhalter D: WinBUGS–a Bayesian modelling framework: concepts, structure, and extensibility. Stat Comput. 2000, 10: 325-337. 10.1023/A:1008929526011.CrossRef Lunn DJ, Thomas A, Best N, Spiegelhalter D: WinBUGS–a Bayesian modelling framework: concepts, structure, and extensibility. Stat Comput. 2000, 10: 325-337. 10.1023/A:1008929526011.CrossRef
30.
go back to reference Weil K, Hooper L, Afzal Z, Esposito M, Worthington HV, van Wijk A, Coulthard P: Paracetamol for pain relief after surgical removal of lower wisdom teeth. Cochrane Database Syst Rev. 2007, (3): Art. No. CD004487. doi:10.1002/14651858.CD004487.pub2, Weil K, Hooper L, Afzal Z, Esposito M, Worthington HV, van Wijk A, Coulthard P: Paracetamol for pain relief after surgical removal of lower wisdom teeth. Cochrane Database Syst Rev. 2007, (3): Art. No. CD004487. doi:10.1002/14651858.CD004487.pub2,
31.
go back to reference Brooks SP, Gelman A: Alternative methods for monitoring convergence of iterative simulations. J Comput Graphical Stat. 1998, 7: 434-455. Brooks SP, Gelman A: Alternative methods for monitoring convergence of iterative simulations. J Comput Graphical Stat. 1998, 7: 434-455.
32.
go back to reference Foster C, Kaur A, Wedatilake T, HillsdonM: Interventions for promoting physical activity. Cochrane Database Syst Rev. 2005, (1): Art. No. CD003180. doi:10.1002/14651858.CD003180.pub2, Foster C, Kaur A, Wedatilake T, HillsdonM: Interventions for promoting physical activity. Cochrane Database Syst Rev. 2005, (1): Art. No. CD003180. doi:10.1002/14651858.CD003180.pub2,
33.
go back to reference Baker R, Jackson D: Inference for meta-analysis with a suspected temporal trend. Biom J. 2010, 52: 538-551. 10.1002/bimj.200900307.CrossRefPubMed Baker R, Jackson D: Inference for meta-analysis with a suspected temporal trend. Biom J. 2010, 52: 538-551. 10.1002/bimj.200900307.CrossRefPubMed
34.
go back to reference Lu G, Welton NJ, Higgins JPT, White IR, Ades AE: Linear inference for mixed treatment comparison meta-analysis: a two-stage approach. Res Synth Methods. 2011, 2: 43-60. 10.1002/jrsm.34.CrossRefPubMed Lu G, Welton NJ, Higgins JPT, White IR, Ades AE: Linear inference for mixed treatment comparison meta-analysis: a two-stage approach. Res Synth Methods. 2011, 2: 43-60. 10.1002/jrsm.34.CrossRefPubMed
35.
go back to reference Lu G, Ades AE: Modeling between-trial variance structure in mixed treatment comparisons. Biostatistics. 2009, 10: 792-805. 10.1093/biostatistics/kxp032.CrossRefPubMed Lu G, Ades AE: Modeling between-trial variance structure in mixed treatment comparisons. Biostatistics. 2009, 10: 792-805. 10.1093/biostatistics/kxp032.CrossRefPubMed
36.
go back to reference Thorlund K, Thadane L, Mills EJ: Modelling heterogeneity variances in multiple treatment comparison meta-analysis–Are informative priors the better solution?. BMC Res Methodol. 2013, 13: 2-10.1186/1471-2288-13-2.CrossRef Thorlund K, Thadane L, Mills EJ: Modelling heterogeneity variances in multiple treatment comparison meta-analysis–Are informative priors the better solution?. BMC Res Methodol. 2013, 13: 2-10.1186/1471-2288-13-2.CrossRef
37.
go back to reference Jackson D, Barrett JK, Rice S, White IR, Higgins JPT: A design-by-treatment interaction model for network meta-analysis with random inconsistency effects. Stat Med. 2014, 33: 3639-3654. 10.1002/sim.6188.CrossRefPubMedPubMedCentral Jackson D, Barrett JK, Rice S, White IR, Higgins JPT: A design-by-treatment interaction model for network meta-analysis with random inconsistency effects. Stat Med. 2014, 33: 3639-3654. 10.1002/sim.6188.CrossRefPubMedPubMedCentral
38.
go back to reference White IR, Barrett JK, Jackson D, Higgins JPT: Consistency and inconsistency in network meta-analysis: model estimation using multivariate meta-regression. Res Synth Methods. 2012, 3: 111-125. 10.1002/jrsm.1045.CrossRefPubMedPubMedCentral White IR, Barrett JK, Jackson D, Higgins JPT: Consistency and inconsistency in network meta-analysis: model estimation using multivariate meta-regression. Res Synth Methods. 2012, 3: 111-125. 10.1002/jrsm.1045.CrossRefPubMedPubMedCentral
39.
go back to reference Xiong T, Turner R, Wei Y, Neal D, Lyratzopoulos G, Higgins J: Comparative efficacy and safety of treatments for localized prostate cancer: an application of network meta-analysis. BMJ Open. 2014, 4: e004285-10.1136/bmjopen-2013-004285.CrossRefPubMedPubMedCentral Xiong T, Turner R, Wei Y, Neal D, Lyratzopoulos G, Higgins J: Comparative efficacy and safety of treatments for localized prostate cancer: an application of network meta-analysis. BMJ Open. 2014, 4: e004285-10.1136/bmjopen-2013-004285.CrossRefPubMedPubMedCentral
Metadata
Title
Methods for calculating confidence and credible intervals for the residual between-study variance in random effects meta-regression models
Authors
Dan Jackson
Rebecca Turner
Kirsty Rhodes
Wolfgang Viechtbauer
Publication date
01-12-2014
Publisher
BioMed Central
Published in
BMC Medical Research Methodology / Issue 1/2014
Electronic ISSN: 1471-2288
DOI
https://doi.org/10.1186/1471-2288-14-103

Other articles of this Issue 1/2014

BMC Medical Research Methodology 1/2014 Go to the issue