Top

Published in:

Open Access 01-12-2019 | Review

A Guide to Conducting a Meta-Analysis with Non-Independent Effect Sizes

Author: Mike W.-L. Cheung

Published in: Neuropsychology Review | Issue 4/2019

Abstract

Conventional meta-analytic procedures assume that effect sizes are independent. When effect sizes are not independent, conclusions based on these conventional procedures can be misleading or even wrong. Traditional approaches, such as averaging the effect sizes and selecting one effect size per study, are usually used to avoid the dependence of the effect sizes. These ad-hoc approaches, however, may lead to missed opportunities to utilize all available data to address the relevant research questions. Both multivariate meta-analysis and three-level meta-analysis have been proposed to handle non-independent effect sizes. This paper gives a brief introduction to these new techniques for applied researchers. The first objective is to highlight the benefits of using these methods to address non-independent effect sizes. The second objective is to illustrate how to apply these techniques with real data in R and Mplus. Researchers may modify the sample R and Mplus code to fit their data.

Available only for authorised users

Abramovitch, A., Anholt, G., Raveh-Gottfried, S., Hamo, N., & Abramowitz, J. S. (2018). Meta-analysis of intelligence quotient (IQ) in obsessive-compulsive disorder. Neuropsychology Review, 28(1), 111–120. https://doi.org/10.1007/s11065-017-9358-0 CrossRefPubMed

Albarracín, D., Cuijpers, P., Eastwick, P. W., Johnson, B. T., Roisman, G. I., Sinatra, G. M., & Verhaeghen, P. (2018). Editorial. Psychological Bulletin, 144(3), 223–226. https://doi.org/10.1037/bul0000147 CrossRefPubMed

Anderson, S. F., & Maxwell, S. E. (2016). There’s more than one way to conduct a replication study: Beyond statistical significance. Psychological Methods, 21(1), 1–12. https://doi.org/10.1037/met0000051 CrossRefPubMed

Belleville, S., Fouquet, C., Hudon, C., Zomahoun, H. T. V., Croteau, J., & Consortium for the Early Identification of Alzheimer’s disease-Quebec. (2017). Neuropsychological measures that predict progression from mild cognitive impairment to Alzheimer’s type dementia in older adults: A systematic review and meta-analysis. Neuropsychology Review, 27(4), 328–353. https://doi.org/10.1007/s11065-017-9361-5 CrossRefPubMedPubMedCentral

Borenstein, M., Hedges, L. V., Higgins, J. P. T., & Rothstein, H. R. (2009). Introduction to meta-analysis. Chichester, West Sussex, U.K.; Hoboken: John Wiley & Sons.

Borenstein, M., Higgins, J. P. T., Hedges, L. V., & Rothstein, H. R. (2017). Basics of meta-analysis: I2 is not an absolute measure of heterogeneity. Research Synthesis Methods, 8(1), 5–18. https://doi.org/10.1002/jrsm.1230 CrossRefPubMed

Burmester, B., Leathem, J., & Merrick, P. (2016). Subjective cognitive complaints and objective cognitive function in aging: A systematic review and meta-analysis of recent cross-sectional findings. Neuropsychology Review, 26(4), 376–393. https://doi.org/10.1007/s11065-016-9332-2 CrossRefPubMed

Card, N. A. (2012). Applied meta-analysis for social science research. New York: The Guilford Press.

Cheung, M. W.-L. (2013). Multivariate meta-analysis as structural equation models. Structural Equation Modeling: A Multidisciplinary Journal, 20(3), 429–454. https://doi.org/10.1080/10705511.2013.797827 CrossRef

Cheung, M. W.-L. (2014). Modeling dependent effect sizes with three-level meta-analyses: A structural equation modeling approach. Psychological Methods, 19(2), 211–229. https://doi.org/10.1037/a0032968 CrossRefPubMed

Cheung, M. W.-L. (2015a). Meta-analysis: A structural equation modeling approach. Chichester, West Sussex: John Wiley & Sons, Inc..

Cheung, M. W.-L. (2015b). metaSEM: An R package for meta-analysis using structural equation modeling. Frontiers in Psychology, 5(1521). https://doi.org/10.3389/fpsyg.2014.01521

Cheung, M. W.-L. (2018). Computing multivariate effect sizes and their sampling covariance matrices with structural equation modeling: Theory, examples, and computer simulations. Frontiers in Psychology, 9(1387). https://doi.org/10.3389/fpsyg.2018.01387

Cheung, M. W.-L., & Vijayakumar, R. (2016). A guide to conducting a meta-analysis. Neuropsychology Review, 26(2), 121–128. https://doi.org/10.1007/s11065-016-9319-z CrossRefPubMed

Cooper, H. M., Hedges, L. V., & Valentine, J. C. (2009). The handbook of research synthesis and meta-analysis (2nd ed.). New York: Russell Sage Foundation.

Demidenko, E. (2013). Mixed models: Theory and applications with R (2nd ed.). Hoboken, N.J: Wiley-Interscience.

Fernández-Castilla, B., Maes, M., Declercq, L., Jamshidi, L., Beretvas, S. N., Onghena, P., & den Noortgate, W. V. (2018). A demonstration and evaluation of the use of cross-classified random-effects models for meta-analysis. Behavior Research Methods, 1–19. https://doi.org/10.3758/s13428-018-1063-2 CrossRef

Fischer, R., & Boer, D. (2011). What is more important for national well-being: Money or autonomy? A meta-analysis of well-being, burnout, and anxiety across 63 societies. Journal of Personality and Social Psychology, 101(1), 164–184. https://doi.org/10.1037/a0023663 CrossRefPubMed

Fischer, R., Hanke, K., & Sibley, C. G. (2012). Cultural and institutional determinants of social dominance orientation: A cross-cultural meta-analysis of 27 societies. Political Psychology, 33(4), 437–467. https://doi.org/10.1111/j.1467-9221.2012.00884.x CrossRef

Gleser, L. J., & Olkin, I. (2009). Stochastically dependent effect sizes. In H. Cooper, L. V. Hedges, & J. C. Valentine (Eds.), The handbook of research synthesis and meta-analysis (2nd ed., pp. 357–376). New York: Russell Sage Foundation.

Goldstein, H. (2011). Multilevel statistical models (4th ed.). Hoboken, N.J: Wiley.

Gurevitch, J., Koricheva, J., Nakagawa, S., & Stewart, G. (2018). Meta-analysis and the science of research synthesis. Nature, 555(7695), 175–182. https://doi.org/10.1038/nature25753 CrossRefPubMed

Hedges, L. V., & Olkin, I. (1985). Statistical methods for meta-analysis. Orlando, FL: Academic Press.

Hedges, L. V., & Schauer, J. M. (2018). Statistical analyses for studying replication: Meta-analytic perspectives. Psychological Methods. https://doi.org/10.1037/met0000189 CrossRef

Hedges, L. V., Tipton, E., & Johnson, M. C. (2010). Robust variance estimation in meta-regression with dependent effect size estimates. Research Synthesis Methods, 1(1), 39–65. https://doi.org/10.1002/jrsm.5 CrossRefPubMed

Higgins, J. P. T., & Thompson, S. G. (2002). Quantifying heterogeneity in a meta-analysis. Statistics in Medicine, 21(11), 1539–1558. https://doi.org/10.1002/sim.1186 CrossRefPubMed

Hox, J. J. (2010). Multilevel analysis: Techniques and applications (2nd ed.). New York: Routledge.CrossRef

Hunt, M. (1997). How science takes stock: The story of meta-analysis. New York: Russell Sage Foundation.

Ishak, K. J., Platt, R. W., Joseph, L., & Hanley, J. A. (2008). Impact of approximating or ignoring within-study covariances in multivariate meta-analyses. Statistics in Medicine, 27(5), 670–686. https://doi.org/10.1002/sim.2913 CrossRefPubMed

Jackson, D., Riley, R., & White, I. R. (2011). Multivariate meta-analysis: Potential and promise. Statistics in Medicine, 30(20), 2481–2498. https://doi.org/10.1002/sim.4172 CrossRefPubMedPubMedCentral

Konstantopoulos, S. (2011). Fixed effects and variance components estimation in three-level meta-analysis. Research Synthesis Methods, 2(1), 61–76. https://doi.org/10.1002/jrsm.35 CrossRefPubMed

Li, J., & Fine, J. P. (2011). Assessing the dependence of sensitivity and specificity on prevalence in meta-analysis. Biostatistics, 12(4), 710–722. https://doi.org/10.1093/biostatistics/kxr008 CrossRefPubMedPubMedCentral

López-López, J. A., Van den Noortgate, W., Tanner-Smith, E. E., Wilson, S. J., & Lipsey, M. W. (2017). Assessing meta-regression methods for examining moderator relationships with dependent effect sizes: A Monte Carlo simulation. Research Synthesis Methods, 8(4), 435–450. https://doi.org/10.1002/jrsm.1245 CrossRefPubMed

Loring, D. W., & Bowden, S. C. (2016). Editorial. Neuropsychology Review, 26(1), 1–2. https://doi.org/10.1007/s11065-015-9314-9 CrossRefPubMed

Mauger, C., Lancelot, C., Roy, A., Coutant, R., Cantisano, N., & Gall, D. L. (2018). Executive functions in children and adolescents with turner syndrome: A systematic review and meta-analysis. Neuropsychology Review, 28(2), 188–215. https://doi.org/10.1007/s11065-018-9372-x CrossRefPubMed

Mewborn, C. M., Lindbergh, C. A., & Stephen Miller, L. (2017). Cognitive interventions for cognitively healthy, mildly impaired, and mixed samples of older adults: A systematic review and meta-analysis of randomized-controlled trials. Neuropsychology Review, 27(4), 403–439. https://doi.org/10.1007/s11065-017-9350-8 CrossRefPubMed

Moeyaert, M., Ugille, M., Beretvas, S. N., Ferron, J., Bunuan, R., & den Noortgate, W. V. (2017). Methods for dealing with multiple outcomes in meta-analysis: A comparison between averaging effect sizes, robust variance estimation and multilevel meta-analysis. International Journal of Social Research Methodology, 20(6), 559–572. https://doi.org/10.1080/13645579.2016.1252189 CrossRef

Moeyaert, M., Ugille, M., Ferron, J. M., Beretvas, S. N., & Van den Noortgate, W. (2013). The three-level synthesis of standardized single-subject experimental data: A Monte Carlo simulation study. Multivariate Behavioral Research, 48(5), 719–748. https://doi.org/10.1080/00273171.2013.816621 CrossRefPubMed

Muthén, B. O., & Muthén, L. K. (2017). Mplus user’s guide (8th ed.). Los Angeles, CA: Muthén & Muthén.

Nam, I.-S., Mengersen, K., & Garthwaite, P. (2003). Multivariate meta-analysis. Statistics in Medicine, 22(14), 2309–2333. https://doi.org/10.1002/sim.1410 CrossRefPubMed

Open Science Collaboration. (2012). An open, large-scale, collaborative effort to estimate the reproducibility of psychological science. Perspectives on Psychological Science, 7(6), 657–660. https://doi.org/10.1177/1745691612462588 CrossRef

Open Science Collaboration. (2015). Estimating the reproducibility of psychological. Science, 349(6251), aac4716. https://doi.org/10.1126/science.aac4716 CrossRef

Prado, C. E., Watt, S., & Crowe, S. F. (2018). A meta-analysis of the effects of antidepressants on cognitive functioning in depressed and non-depressed samples. Neuropsychology Review, 28(1), 32–72. https://doi.org/10.1007/s11065-018-9369-5 CrossRefPubMed

R Development Core Team. (2019). R: A language and environment for statistical computing. Vienna: Austria Retrieved from http://www.R-project.org/

Raudenbush, S. W., Becker, B. J., & Kalaian, H. (1988). Modeling multivariate effect sizes. Psychological Bulletin, 103(1), 111–120. https://doi.org/10.1037/0033-2909.103.1.111 CrossRef

Raudenbush, S. W., & Bryk, A. S. (2002). Hierarchical linear models: Applications and data analysis methods. Thousand Oaks: Sage Publications.

Riley, R. D. (2009). Multivariate meta-analysis: The effect of ignoring within-study correlation. Journal of the Royal Statistical Society: Series A (Statistics in Society), 172(4), 789–811. https://doi.org/10.1111/j.1467-985X.2008.00593.x CrossRef

Riley, R. D., Thompson, J. R., & Abrams, K. R. (2008). An alternative model for bivariate random-effects meta-analysis when the within-study correlations are unknown. Biostatistics, 9(1), 172–186. https://doi.org/10.1093/biostatistics/kxm023 CrossRefPubMed

Scammacca, N., Roberts, G., & Stuebing, K. K. (2014). Meta-analysis with complex research designs dealing with dependence from multiple measures and multiple group comparisons. Review of Educational Research, 84(3), 328–364. https://doi.org/10.3102/0034654313500826 CrossRefPubMed

Sherman, D. S., Mauser, J., Nuno, M., & Sherzai, D. (2017). The efficacy of cognitive intervention in mild cognitive impairment (MCI): A meta-analysis of outcomes on neuropsychological measures. Neuropsychology Review, 27(4), 440–484. https://doi.org/10.1007/s11065-017-9363-3 CrossRefPubMedPubMedCentral

Shin, I.-S. (2009). Same author and same data dependence in meta-analysis (Ph.D.). the Florida State University, United States -- Florida.

Stadler, M., Becker, N., Gödker, M., Leutner, D., & Greiff, S. (2015). Complex problem solving and intelligence: A meta-analysis. Intelligence, 53, 92–101. https://doi.org/10.1016/j.intell.2015.09.005 CrossRef

Timm, N. H. (1999). A note on testing for multivariate effect sizes. Journal of Educational and Behavioral Statistics, 24(2), 132–145. https://doi.org/10.3102/10769986024002132 CrossRef

Tipton, E. (2015). Small sample adjustments for robust variance estimation with meta-regression. Psychological Methods, 20(3), 375–393. https://doi.org/10.1037/met0000011 CrossRefPubMed

Van den Noortgate, W., López-López, J. A., Marín-Martínez, F., & Sánchez-Meca, J. (2013). Three-level meta-analysis of dependent effect sizes. Behavior Research Methods, 45(2), 576–594. https://doi.org/10.3758/s13428-012-0261-6 CrossRefPubMed

Viechtbauer, W. (2010). Conducting meta-analyses in R with the metafor package. Journal of Statistical Software, 36(3), 1–48. https://doi.org/10.18637/jss.v036.i03 CrossRef

Weissberger, G. H., Strong, J. V., Stefanidis, K. B., Summers, M. J., Bondi, M. W., & Stricker, N. H. (2017). Diagnostic accuracy of memory measures in Alzheimer’s dementia and mild cognitive impairment: A systematic review and meta-analysis. Neuropsychology Review, 27(4), 354–388. https://doi.org/10.1007/s11065-017-9360-6 CrossRefPubMedPubMedCentral

Title: A Guide to Conducting a Meta-Analysis with Non-Independent Effect Sizes
Author: Mike W.-L. Cheung
Publication date: 01-12-2019
Publisher: Springer US
Published in: Neuropsychology Review / Issue 4/2019
Print ISSN: 1040-7308
Electronic ISSN: 1573-6660
DOI: https://doi.org/10.1007/s11065-019-09415-6

Keynote webinar | Spotlight on medication adherence

Springer Medicine

A Guide to Conducting a Meta-Analysis with Non-Independent Effect Sizes

Abstract

Keynote webinar | Spotlight on medication adherence

Springer Medicine

Abstract

Please log in to get access to this content

Other articles of this Issue 4/2019

To Watch is to Work: a Review of NeuroImaging Data on Tool Use Observation Network

The Cerebellum in Frontotemporal Dementia: a Meta-Analysis of Neuroimaging Studies

Visuospatial Neglect - a Theory-Informed Overview of Current and Emerging Strategies and a Systematic Review on the Therapeutic Use of Non-invasive Brain Stimulation

Caregiver- and Clinician-Reported Adaptive Functioning in Rett Syndrome: a Systematic Review and Evaluation of Measurement Strategies

Towards a Better Understanding of Cognitive Deficits in Absence Epilepsy: a Systematic Review and Meta-Analysis

A Systematic Review of Neurocognitive Functioning in Behçet’s Disease