Top

Published in:

01-06-2016 | Review

A Guide to Conducting a Meta-Analysis

Authors: Mike W.-L. Cheung, Ranjith Vijayakumar

Published in: Neuropsychology Review | Issue 2/2016

Abstract

Meta-analysis is widely accepted as the preferred method to synthesize research findings in various disciplines. This paper provides an introduction to when and how to conduct a meta-analysis. Several practical questions, such as advantages of meta-analysis over conventional narrative review and the number of studies required for a meta-analysis, are addressed. Common meta-analytic models are then introduced. An artificial dataset is used to illustrate how a meta-analysis is conducted in several software packages. The paper concludes with some common pitfalls of meta-analysis and their solutions. The primary goal of this paper is to provide a summary background to readers who would like to conduct their first meta-analytic study.

Aytug, Z. G., Rothstein, H. R., Zhou, W., & Kern, M. C. (2012). Revealed or concealed? Transparency of procedures, decisions, and judgment calls in meta-analyses. Organizational Research Methods, 15(1), 103–133. doi:10.1177/1094428111403495.CrossRef

Bax, L., Yu, L.-M., Ikeda, N., & Moons, K. G. (2007). A systematic comparison of software dedicated to meta-analysis of causal studies. BMC Medical Research Methodology, 7(1), 40. doi:10.1186/1471-2288-7-40.CrossRefPubMedPubMedCentral

Borenstein, M. (2009). Effect sizes for continuous data. In H. Cooper, L. V. Hedges, & J. C. Valentine (Eds.), The handbook of research synthesis and meta-analysis (2nd ed., pp. 221–235). New York: Russell Sage Foundation.

Borenstein, M., Hedges, L. V., & Rothstein, H. R. (2005). Comprehensive meta-analysis (version 2). Englewood NJ: Biostat.

Borenstein, M., Hedges, L. V., Higgins, J. P. T., & Rothstein, H. R. (2009). Introduction to meta-analysis. Chichester, West Sussex, U.K.; Hoboken: John Wiley & Sons.CrossRef

Borenstein, M., Hedges, L. V., Higgins, J. P. T., & Rothstein, H. R. (2010). A basic introduction to fixed-effect and random-effects models for meta-analysis. Research Synthesis Methods, 1(2), 97–111. doi:10.1002/jrsm.12.CrossRefPubMed

Chalmers, I., Hedges, L. V., & Cooper, H. (2002). A brief history of research synthesis. Evaluation & the Health Professions, 25(1), 12–37. doi:10.1177/0163278702025001003.CrossRef

Cheung, M. W.-L. (2013). Multivariate meta-analysis as structural equation models. Structural Equation Modeling: A Multidisciplinary Journal, 20(3), 429–454. doi:10.1080/10705511.2013.797827.CrossRef

Cheung, M. W.-L. (2014). Modeling dependent effect sizes with three-level meta-analyses: a structural equation modeling approach. Psychological Methods, 19(2), 211–229. doi:10.1037/a0032968.CrossRefPubMed

Cheung, M. W.-L. (2015a). Meta-analysis: A structural equation modeling approach. Chichester, West Sussex: John Wiley & Sons, Inc..CrossRef

Cheung, M. W.-L. (2015b). metaSEM: an R package for meta-analysis using structural equation modeling. Frontiers in Psychology, 5(1521). doi:10.3389/fpsyg.2014.01521.

Cheung, M. W.-L., Ho, R. C. M., Lim, Y., & Mak, A. (2012). Conducting a meta-analysis: Basics and good practices. International Journal of Rheumatic Diseases, 15(2), 129–135. doi:10.1111/j.1756-185X.2012.01712.x

Cooper, H., & Hedges, L. V. (2009). Research synthesis as a scientific process. In H. Cooper, L. V. Hedges, & J. C. Valentine (Eds.), The handbook of research synthesis and meta-analysis (2nd ed., pp. 3–16). New York: Russell Sage Foundation.

Cooper, H., Maxwell, S., Stone, A., & Sher, K. (2008). Reporting standards for research in psychology why do we need them? What might they be? American Psychologist, 63(9), 839–851. doi:10.1037/0003-066X.63.9.839.CrossRef

Cooper, H. M., Hedges, L. V., & Valentine, J. C. (2009). The handbook of research synthesis and meta-analysis (2nd ed.). New York: Russell Sage Foundation.

Davey, J., Turner, R. M., Clarke, M. J., & Higgins, J. P. (2011). Characteristics of meta-analyses and their component studies in the Cochrane database of systematic reviews: a cross-sectional, descriptive analysis. BMC Medical Research Methodology, 11, 160. doi:10.1186/1471-2288-11-160.CrossRefPubMedPubMedCentral

Dewey, M. (2015). CRAN task view: Meta-analysis. Retrieved from http://CRAN.R-project.org/view=MetaAnalysis

Dickersin, K., Chan, S., Chalmersx, T. C., Sacks, H. S., & Smith, H. (1987). Publication bias and clinical trials. Controlled Clinical Trials, 8(4), 343–353. doi:10.1016/0197-2456(87)90155-3.CrossRefPubMed

Easterbrook, P. J., Gopalan, R., Berlin, J. A., & Matthews, D. R. (1991). Publication bias in clinical research. The Lancet, 337(8746), 867–872. doi:10.1016/0140-6736(91)90201-Y.CrossRef

Eysenck, H. J. (1978). An exercise in mega-silliness. American Psychologist, 33(5), 517.CrossRef

Ferguson, C. J., & Brannick, M. T. (2012). Publication bias in psychological science: prevalence, methods for identifying and controlling, and implications for the use of meta-analyses. Psychological Methods, 17(1), 120–128. doi:10.1037/a0024445.CrossRefPubMed

Fioravanti, M., Carlone, O., Vitale, B., Cinti, M. E., & Clare, L. (2005). A meta-analysis of cognitive deficits in adults with a diagnosis of schizophrenia. Neuropsychology Review, 15(2), 73–95. doi:10.1007/s11065-005-6254-9.CrossRefPubMed

Fleiss, J. L., & Berlin, J. A. (2009). Effect sizes for dichotomous data. In H. M. Cooper, L. V. Hedges, & J. C. Valentine (Eds.), The handbook of research synthesis and meta-analysis (2nd ed., pp. 237–253). New York: Russell Sage Foundation.

Francis, G. (2013). Replication, statistical consistency, and publication bias. Journal of Mathematical Psychology, 57(5), 153–169. doi:10.1016/j.jmp.2013.02.003.CrossRef

Glass, G. V. (1976). Primary, secondary, and meta-analysis of research. Educational Researcher, 5(10), 3–8. doi:10.2307/1174772.CrossRef

Harwell, M. (1997). An empirical study of Hedge’s homogeneity test. Psychological Methods, 2(2), 219–231. doi:10.1037/1082-989X.2.2.219.CrossRef

Hedges, L. V., & Olkin, I. (1985). Statistical methods for meta-analysis. Orlando, FL: Academic Press.

Hedges, L. V., & Pigott, T. D. (2001). The power of statistical tests in meta-analysis. Psychological Methods, 6(3), 203–217. doi:10.1037/1082-989X.6.3.203.CrossRefPubMed

Hedges, L. V., & Pigott, T. D. (2004). The power of statistical tests for moderators in meta-analysis. Psychological Methods, 9(4), 426–445. doi:10.1037/1082-989X.9.4.426.CrossRefPubMed

Higgins, J. P. T., & Thompson, S. G. (2002). Quantifying heterogeneity in a meta-analysis. Statistics in Medicine, 21(11), 1539–1558. doi:10.1002/sim.1186.CrossRefPubMed

Higgins, J. P. T., Thompson, S. G., Deeks, J. J., & Altman, D. G. (2003). Measuring inconsistency in meta-analyses. British Medical Journal, 327(7414), 557–560.CrossRefPubMedPubMedCentral

Huedo-Medina, T. B., Sanchez-Meca, J., Marin-Martinez, F., & Botella, J. (2006). Assessing heterogeneity in meta-analysis. Q statistic or I2 index? Psychological Methods June 2006, 11(2), 193–206.PubMed

Hunt, M. (1997). How science takes stock: The story of meta-analysis. New York: Russell Sage Foundation.

Koricheva, J., Gurevitch, J., & Mengersen, K. (Eds.) (2013). Handbook of meta-analysis in ecology and evolution. Princeton: Princeton University Press.

Kupfersmid, J., & Fiala, M. (1991). A survey of attitudes and behaviors of authors who publish in psychology and education journals. American Psychologist, 46(3), 249–250. doi:10.1037/0003-066X.46.3.249.CrossRef

Liberati, A., Altman, D. G., Tetzlaff, J., Mulrow, C., Gotzsche, P. C., Ioannidis, J. P. A., et al. (2009). The PRISMA statement for reporting systematic reviews and meta-analyses of studies that evaluate healthcare interventions: explanation and elaboration. BMJ, 339(jul21 1), b2700–b2700. doi:10.1136/bmj.b2700.CrossRefPubMedPubMedCentral

Lindsay, D. S. (2015). Replication in psychological science. Psychological Science, 26(12), 1827–1832. doi:10.1177/0956797615616374.CrossRefPubMed

Lipsey, M. W., & Wilson, D. (2000). Practical meta-analysis. Sage Publications, Inc ,Thousand Oaks.

Littell, J. H., Corcoran, J., & Pillai, V. (2008). Systematic reviews and meta-analysis. Oxford University Press. Retrieved from http://www.oxfordscholarship.com/view/10.1093/acprof:oso/9780195326543.001.0001/acprof-9780195326543

Lucas, R. E., & Brent Donnellan, M. (2013). Improving the replicability and reproducibility of research published in the journal of research in personality. Journal of Research in Personality, 47(4), 453–454. doi:10.1016/j.jrp.2013.05.002.CrossRef

Maxwell, S. E., Lau, M. Y., & Howard, G. S. (2015). Is psychology suffering from a replication crisis? What does “failure to replicate” really mean? American Psychologist, 70(6), 487–498. doi:10.1037/a0039400.CrossRefPubMed

Morris, S. B., & DeShon, R. P. (2002). Combining effect size estimates in meta-analysis with repeated measures and independent-groups designs. Psychological Methods, 7(1), 105–125. doi:10.1037/1082-989X.7.1.105.CrossRefPubMed

National Research Council (1992). Combining information: statistical issues and opportunities for research. Washington, DC: National Academies Press.

O’Rourke, K. (2007). An historical perspective on meta-analysis: dealing quantitatively with varying study results. JRSM, 100(12), 579–582. doi:10.1258/jrsm.100.12.579.CrossRef

Open Science Collaboration (2012). An open, large-scale, collaborative effort to estimate the reproducibility of psychological science. Perspectives on Psychological Science, 7(6), 657–660. doi:10.1177/1745691612462588.CrossRef

Open Science Collaboration (2015). Estimating the reproducibility of psychological. Science, 349(6251), aac4716. doi:10.1126/science.aac4716.CrossRef

Palmer, T. M., & Sterne, J. A. C. (Eds.) (2016). Meta-analysis: an updated collection from the Stata journal (2nd ed.). College Station: Stata Press.

Pearson, K. (1904). Report on certain enteric fever inoculation statistics. BMJ, 2(2288), 1243–1246. doi:10.1136/bmj.2.2288.1243.CrossRef

Pigott, T. D. (2012). Advances in meta-analysis. New York: Springer. Retrieved from http://www.springer.com.libproxy1.nus.edu.sg/statistics/social+sciences+%26+law/book/978-1-4614-2277-8

R Development Core Team. (2016). R: a language and environment for statistical computing. Vienna, Austria. Retrieved from http://www.R-project.org/

Reed, J. G., & Baxter, P. M. (2009). Using reference databases. In H. Cooper, L. V. Hedges, & J. C. Valentine (Eds.), The handbook of research synthesis and meta-analysis (2nd ed., pp. 73–101). New York: Russell Sage Foundation.

Rothstein, H. R., & Bushman, B. J. (2012). Publication bias in psychological science: comment on Ferguson and Brannick (2012). Psychological Methods, 17(1), 129–136. doi:10.1037/a0027128.CrossRefPubMed

Rothstein, H. R., Sutton, A. J., & Borenstein, M. (2005). Publication bias in meta-analysis: Prevention, assessment and adjustments. Chichester: John Wiley and Sons.CrossRef

Schmidt, F. L., & Hunter, J. E. (2015). Methods of meta-analysis: Correcting error and bias in research findings (3rd ed.). Thousand Oaks, CA: Sage.

Shadish, W. R. (2015). Introduction to the special issue on the origins of modern meta-analysis. Research Synthesis Methods, 6(3), 219–220. doi:10.1002/jrsm.1148.CrossRefPubMed

Smith, M. L., & Glass, G. V. (1977). Meta-analysis of psychotherapy outcome studies. American Psychologist, 32(9), 752–760.CrossRefPubMed

Song, F., Parekh, S., Hooper, L., Loke, Y., Ryder, J., Sutton, A., et al. (2010). Dissemination and publication of research findings: an updated review of related biases. Health Technology Assessment, 14(8). doi:10.3310/hta14080.

Sterne, J. A. C., Egger, M., & Sutton, A. J. (2001). Meta-analysis software. In M. Egger, G. D. Smith, & D. G. Altman (Eds.), Systematic Reviews in Health Care: Meta-Analysis in Context (pp. 336–346). London: BMJ Publishing Group. Retrieved from doi:10.1002/9780470693926.ch17/summary

Valentine, J. C., Pigott, T. D., & Rothstein, H. R. (2010). How many studies do you need? A primer on statistical power for meta-analysis. Journal of Educational and Behavioral Statistics, 35(2), 215–247. doi:10.3102/1076998609346961.CrossRef

Viechtbauer, W. (2007). Hypothesis tests for population heterogeneity in meta-analysis. British Journal of Mathematical and Statistical Psychology, 60(1), 29–60. doi:10.1348/000711005X64042.CrossRefPubMed

Wallace, B. C., Schmid, C. H., Lau, J., & Trikalinos, T. A. (2009). Meta-analyst: software for meta-analysis of binary, continuous and diagnostic data. BMC Medical Research Methodology, 9(1), 80. doi:10.1186/1471-2288-9-80.CrossRefPubMedPubMedCentral

Weare, K., & Nind, M. (2011). Mental health promotion and problem prevention in schools: what does the evidence say? Health Promotion International, 26(suppl 1), i29–i69. doi:10.1093/heapro/dar075.CrossRefPubMed

Title: A Guide to Conducting a Meta-Analysis
Authors: Mike W.-L. Cheung
Ranjith Vijayakumar
Publication date: 01-06-2016
Publisher: Springer US
Published in: Neuropsychology Review / Issue 2/2016
Print ISSN: 1040-7308
Electronic ISSN: 1573-6660
DOI: https://doi.org/10.1007/s11065-016-9319-z

At a glance: The STEP trials

Springer Medicine

A Guide to Conducting a Meta-Analysis

Abstract

At a glance: The STEP trials

Springer Medicine

Abstract

Please log in to get access to this content

Other articles of this Issue 2/2016

Resilience and Other Possible Outcomes After Mild Traumatic Brain Injury: a Systematic Review

Editorial

Functional Disability in Mild Cognitive Impairment: A Systematic Review and Meta-Analysis

A Neuropsychologist’s Guide To Undertaking a Systematic Review for Publication: Making the most of PRISMA Guidelines

Weighing the Evidence: A Systematic Review on Long-Term Neurocognitive Effects of Cannabis Use in Abstinent Adolescents and Adults

Social Cognition in Multiple Sclerosis: a Meta-Analysis