Top

Published in:

01-04-2013

Methods for Synthesizing Findings on Moderation Effects Across Multiple Randomized Trials

Authors: C. Hendricks Brown, Zili Sloboda, Fabrizio Faggiano, Brent Teasdale, Ferdinand Keller, Gregor Burkhart, Federica Vigna-Taglianti, George Howe, Katherine Masyn, Wei Wang, Bengt Muthén, Peggy Stephens, Scott Grey, Tatiana Perrino, Prevention Science and Methodology Group

Published in: Prevention Science | Issue 2/2013

Abstract

This paper presents new methods for synthesizing results from subgroup and moderation analyses across different randomized trials. We demonstrate that such a synthesis generally results in additional power to detect significant moderation findings above what one would find in a single trial. Three general methods for conducting synthesis analyses are discussed, with two methods, integrative data analysis and parallel analyses, sharing a large advantage over traditional methods available in meta-analysis. We present a broad class of analytic models to examine moderation effects across trials that can be used to assess their overall effect and explain sources of heterogeneity, and present ways to disentangle differences across trials due to individual differences, contextual level differences, intervention, and trial design.

Our argument below provides a partial justification due to space; the complete proof involves formulas for power based on noncentrality parameters, which in turn depend on sample size.

The development in this part of the text is limited to interactions involving a binary covariate. The power for detecting a linear interaction with a continuous baseline measure can be compared to that of the main effect once a common calibration of “effect size” is established. Our choice is to scale the treatment variable to have the same variance as that of the continuous variable. The regression coefficient of the interaction term measures the difference in response under intervention and control for two covariate values separated by 1 standard deviation, i.e. \( E{S_{{Inter}}} = E(Y|T = 1,X = 1) - E(Y|T = 0,X = 1) - \left\{ {E(Y|T = 1,X = 0) - E(Y|T = 0) - E(Y|T = 0,X = 0)} \right\} \).To achieve the same power for detecting an effect size, ES _ME for the main effect in a trial with equal allocations to intervention and control, we require \( E{S_{{Inter}}} = 2E{S_{{ME}}} \) _. This is the identical result for the case of a dichotomous moderator variable presented in the text.

In this argument we have ignored the differences in smaller degrees of freedom needed to test for this interaction effect across trials; nevertheless, the relationship ICC < 4(M-1)/N is still a very conservative bound.

Asparouhov, T., & Muthén, B. O. (2008). Multilevel mixture models. In G. R. Hancock & K. M. Samuelsen (Eds.), Advances in latent variable mixture models (pp. 27–51). Charlotte, NC: Information Age.

Barker, L. E., Luman, E. T., McCauley, M. M., & Chu, S. Y. (2002). Assessing equivalence: An alternative to the use of difference tests for measuring disparities in vaccination coverage. American Journal of Epidemiology, 156, 1056–1061.PubMedCrossRef

Bauer, D. J., & Hussong, A. M. (2009). Psychometric approaches for developing commensurate measures across independent studies: Traditional and new models. Psychological Methods, 14, 101–125.PubMedCrossRef

Benjamini, Y., & Hochberg, Y. (1995). Controlling the false discovery rate—a practical and powerful approach to multiple testing. Journal of the Royal Statistical Society Series B-(Statistical Methodology), 57, 289–300.

Berlin, J. A., Santanna, J., Schmid, C. H., Szczech, L. A., & Feldman, H. I. (2002). Individual patient-versus group-level data meta-regressions for the investigation of treatment effect modifiers: Ecological bias rears its ugly head. Statistics in Medicine, 21, 371–387.PubMedCrossRef

Bridge, J. A., Iyengar, S., Salary, C. B., Barbe, R. P., Birmaher, B., Pincus, H. A., et al. (2007). Clinical response and risk for reported suicidal ideation and suicide attempts in pediatric antidepressant treatment: A meta-analysis of randomized controlled trials. Journal of the American Medical Association, 297, 1683–1696.PubMedCrossRef

Brown, C. H. (1993). Analyzing preventive trials with generalized additive models. American Journal of Community Psychology, 21, 635–664.PubMedCrossRef

Brown, C. H., & Liao, J. (1999). Principles for designing randomized preventive trials in mental health: An emerging developmental epidemiology paradigm. American Journal of Community Psychology, 27, 673–710.PubMedCrossRef

Brown, E. C., Hawkins, J. D., Arthur, M. W., Briney, J. S., & Abbott, R. D. (2007). Effects of Communities That Care on prevention services systems: Findings from the community youth development study at 1.5 years. Prevention Science, 8, 180–191.PubMedCrossRef

Brown, C. H., Wang, W., Kellam, S. G., Muthén, B. O., Petras, H., Toyinbo, P., et al. (2008a). Methods for testing theory and evaluating impact in randomized field trials: Intent-to-treat analyses for integrating the perspectives of person, place, and time. Drug and Alcohol Dependence, 95, S74–S104.CrossRef

Brown, C. H., Wang, W., & Sandler, I. (2008b). Examining how context changes intervention impact: The use of effect sizes in multilevel meta-analysis. Child Development Perspectives, 2, 198–205.CrossRef

Coie, J. D., Watt, N. F., West, S. G., Hawkins, J. D., Asarnow, J. R., Markman, H. J., et al. (1993). The science of prevention: A conceptual framework and some directions for a national research program. American Psychologist, 48, 1013–1022.PubMedCrossRef

Cook, T. D. (1992). Meta-analysis for explanation: A casebook. New York: Russell Sage.

Cooper, H. (1999). Synthesizing research: A guide for literature reviews (3rd ed.). Thousand Oaks, CA: Sage.

Cooper, H. M. (2010). Research synthesis and meta-analysis: A step-by-step approach (4th ed.). Los Angeles: Sage.

Cooper, H., & Patall, E. A. (2009). The relative benefits of meta-analysis conducted with individual participant data versus aggregated data. Psychological Methods, 14, 165–176.PubMedCrossRef

Cooper, H. M., Hedges, L. V., & Valentine, J. C. (2009). The handbook of research synthesis and meta-analysis (2nd ed.). New York: Russell Sage.

Curran, P. J., & Hussong, A. M. (2009). Integrative data analysis: The simultaneous analysis of multiple data sets. Psychological Methods, 14, 81–100.PubMedCrossRef

Curran, P. J., Hussong, A. M., Cai, L., Huang, W., Chassin, L., Sher, K. J., et al. (2008). Pooling data from multiple longitudinal studies: The role of item response theory in integrative data analysis. Developmental Psychology, 44, 365–380.PubMedCrossRef

DerSimonian, R., & Laird, N. (1986). Meta-analysis in clinical trials. Controlled Clinical Trials, 7, 177–188.PubMedCrossRef

Dillman Carpentier, F. R., Mauricio, A. M., Gonzales, N. A., Millsap, R. E., Meza, C. M., Dumka, L. E., et al. (2007). Engaging Mexican origin families in a school-based preventive intervention. Journal of Primary Prevention, 28, 521–546.PubMedCrossRef

Dishion, T. J., Spracklen, K. M., Andrews, D. W., & Patterson, G. R. (1996). Deviancy training in male adolescents friendships. Behavior Therapy, 27, 373–390.CrossRef

Dishion, T. J., McCord, J., & Poulin, F. (1999). When interventions harm. Peer groups and problem behavior. American Psychologist, 54, 755–764.PubMedCrossRef

Dishion, T. J., Burraston, B., & Poulin, F. (2001). Peer group dynamics associated with iatrogenic effects in group interventions with high-risk young adolescents. In C. Erdley & D. W. Nangle (Eds.), Damon’s new directions in child development: The role of friendship in psychological adjustment (pp. 79–92). San Francisco: Jossey-Bass.

Dolan, L. J., Kellam, S. G., Brown, C. H., Werthamer-Larsson, L., Rebok, G. W., Mayer, L. S., et al. (1993). The short-term impact of two classroom-based preventive interventions on aggressive and shy behaviors and poor achievement. Journal of Applied Developmental Psychology, 14, 317–345.CrossRef

Durlak, J. A., & Wells, A. M. (1997). Primary prevention mental health programs for children and adolescents: A meta-analytic review. American Journal of Community Psychology, 25, 115–152.PubMedCrossRef

Elliott, D. S., & Mihalic, S. (2004). Issues in disseminating and replicating effective prevention programs. Prevention Science, 5, 47–53.PubMedCrossRef

Ennett, S. T., Tobler, N. S., Ringwalt, C. L., & Flewelling, R. L. (1994). How effective is drug abuse resistance education? A meta-analysis of Project DARE outcome evaluations. American Journal of Public Health, 84, 1394–1401.PubMedCrossRef

Faggiano, F., Vigna-Taglianti, F., Versino, E., Zambon, A., Borraccino, A., & Lemma, P. (2005). School-based prevention for illicit drugs’ use. Cochrane Database of Systematic Reviews, 2, Art. No.: CD003020.

Faggiano, F., Vigna-Taglianti, F. D., Versino, E., Zambon, A., Borraccino, A., & Lemma, P. (2008). School-based prevention for illicit drugs use: A systematic review. Preventive Medicine, 46, 385–396.PubMedCrossRef

Flay, B. R., Biglan, A., Boruch, R. F., Castro, F. G., Gottfredson, D., Kellam, S., et al. (2005). Standards of evidence: Criteria for efficacy, effectiveness and dissemination. Prevention Science, 6, 151–175.PubMedCrossRef

Fournier, J. C., DeRubeis, R. J., Hollen, S. D., Dimdjian, S., Amsterdam, J. D., Shelton, R. C., et al. (2010). Antidepressant drug effects and depression severity: Patient-level meta-analysis. Journal of the American Medical Association, 303, 47–53.PubMedCrossRef

Glass, G. V. (1976). Primary, secondary, and meta-analysis of research. Educational Researcher, 5, 3–8.

Glass, G. V. (1977). Integrating findings: The meta-analysis of research. Review of Research in Education, 5, 351–379.

Glass, T. A., & McAtee, M. J. (2006). Behavioral science at the crossroads in public health: Extending horizons, envisioning the future. Social Science & Medicine, 62, 1650–1671.CrossRef

Glass, G. V., & Smith, M. L. (1978). Meta-analysis of research on the relationship of class-size and achievement: The Class Size and Instruction Project. National Institute of Education (DHEW), Washington, DC.

Glass, G. V., McGaw, B., & Smith, M. L. (1981). Meta-analysis in social research. Beverly Hills, CA: Sage.

Hastie, T., & Tibshirani, R. (1990). Generalized additive models (1st ed.). London: Chapman and Hall.

Hedges, L. V., & Olkin, I. (1985). Statistical methods for meta-analysis. Orlando, FL: Academic.

Higgins, J. P. T., & Green, S. (Eds.). (2008). Cochrane Handbook for Systematic Reviews of Interventions Version 5.0.2 [updated September 2009]: The Cochrane Collaboration. Retrieved from http://www.cochrane-handbook.org

Higgins, J. P., Whitehead, A., Turner, R. M., Omar, R. Z., & Thompson, S. G. (2001). Meta-analysis of continuous outcome data from individual patients. Statistics in Medicine, 20, 2219–2241.PubMedCrossRef

Hofer, S. M., & Piccinin, A. M. (2009). Integrative data analysis through coordination of measurement and analysis protocol across independent longitudinal studies. Psychological Methods, 14, 150–164.PubMedCrossRef

Horowitz, J. L., & Garber, J. (2006). The prevention of depressive symptoms in children and adolescents: A meta-analytic review. Journal of Consulting and Clinical Psychology, 74, 401–415.PubMedCrossRef

Howe, G. W., Reiss, D., & Yuh, J. (2002). Can prevention trials test theories of etiology? Development and Psychopathology, 14, 673–693.PubMedCrossRef

Hussong, A., Cai, L., Curran, P., Flora, D., Chassin, L., & Zucker, R. (2008). Disaggregating the distal, proximal, and time-varying effects of parent alcoholism on children’s internalizing symptoms. Journal of Abnormal Child Psychology, 36, 335–346.PubMedCrossRef

Ialongo, N. S., Werthamer, L., Kellam, S. G., Brown, C. H., Wang, S., & Lin, Y. (1999). Proximal impact of two first-grade preventive interventions on the early risk behaviors for later substance abuse, depression, and antisocial behavior. American Journal of Community Psychology, 27, 599–641.PubMedCrossRef

Ialongo, N., Poduska, J., Werthamer, L., & Kellam, S. (2001). The distal impact of two first-grade preventive interventions on conduct problems and disorder in early adolescence. Journal of Emotional & Behavioral Disorders, 9, 146–160.CrossRef

Jané-Llopis, E., Hosman, C., Jenkins, R., & Anderson, P. (2003). Predictors of efficacy in depression prevention programmes. Meta-analysis. British Journal of Psychiatry, 183, 384–397.PubMedCrossRef

Kellam, S. G., & Langevin, D. J. (2003). A framework for understanding “evidence” in prevention research and programs. Prevention Science, 4, 137–153.PubMedCrossRef

Kellam, S. G., Koretz, D., & Moscicki, E. K. (1999). Core elements of developmental epidemiologically based prevention research. American Journal of Community Psychology, 27, 463–482.PubMedCrossRef

Kellam, S. G., Brown, C. H., Poduska, J. M., Ialongo, N. S., Wang, W., Toyinbo, P., et al. (2008). Effects of a universal classroom behavior management program in first and second grades on young adult behavioral, psychiatric, and social outcomes. Drug and Alcohol Dependence, 95, S5–S28.PubMedCrossRef

Klein, A., & Moosbrugger, H. (2000). Maximum likelihood estimation of latent interaction effects with the LMS method. Psychometrika, 65, 457–474.CrossRef

Klein, A. G., & Muthén, B. (2006). Modeling heterogeneity of growth depending on initial status. Journal of Educational and Behavioral Statistics, 31, 357–375.CrossRef

Kraemer, H. C., Wilson, G. T., Fairburn, C. G., Agras, W. S., Kraemer, H. C., Wilson, G. T., et al. (2002). Mediators and moderators of treatment effects in randomized clinical trials. Archives of General Psychiatry, 59, 877–883.PubMedCrossRef

Lipsey, M. W. (2003). Those confounded moderators in meta-analysis: Good, bad, and ugly. Annals of the American Academy of Political and Social Science, 587, 69–81.CrossRef

Mosteller, F., & Colditz, G. A. (1996). Understanding research synthesis (meta-analysis). Annual Review of Public Health, 17, 1–23.PubMedCrossRef

Muthén, B. (2001). Latent variable mixture modeling. In G. A. Marcoulides & R. E. Schumacker (Eds.), New developments and techniques in structural equation modeling (pp. 1–33). Hillsdale, NJ: Lawrence Erlbaum Associates.

Muthén, B., & Asparouhov, T. (2003a). Advances in latent variable modeling, Part I: Integrating multilevel and structural equation modeling using Mplus. Manucript in Preparation.

Muthén, B., & Asparouhov, T. (2003b). Modeling interactions between latent and observed continuous variables using maximum-likelihood estimation in Mplus. Mplus Web Notes, #6. Retrieved from http://www.statmodel.com/download/webnotes/webnote6.pdf

Muthén, B. O., & Asparouhov, T. (2008). Growth mixture modeling: Analysis with non-Gaussian random effects. In G. Fitzmaurice, M. Davidian, G. Verbeke, & G. Molenberghs (Eds.), Longitudinal data analysis: A handbook of modern statistical methods (pp. 143–165). Boca Raton, FL: Chapman & Hall/CRC.

Muthén, B. O., & Asparouhov, T. (2009). Multilevel regression mixture analysis. Journal of the Royal Statistical Society, 172, 639–657.CrossRef

Muthén, L. K., & Muthén, B. O. (1999). Mplus user’s guide version 2 (2nd ed.). Los Angeles: Authors.

Muthén, B., & Muthén, L. K. (2000). Integrating person-centered and variable-centered analyses: Growth mixture modeling with latent trajectory classes. Alcoholism: Clinical & Experimental Research, 24, 882–891.CrossRef

Muthén, L. K., & Muthén, B. O. (2007). Mplus: Statistical analysis with latent variables: User’s guide (4th ed.). Los Angeles: Authors.

Muthén, B. O., & Shedden, K. (1999). Finite mixture modeling with mixture outcomes using the EM algorithm. Biometrics, 55, 463–469.PubMedCrossRef

O’Connell, M. E., Boat, T., & Warner, K. E. (Eds.). (2009). Mental, emotional, and behavioral disorders among young people: Progress and possibilities. Washington, DC: National Academy.

Shadish, W. R., & Sweeney, R. B. (1991). Mediators and moderators in meta-analysis: There’s a reason we don’t let dodo birds tell us which psychotherapies should have prizes. Journal of Consulting and Clinical Psychology, 59, 883–893.PubMedCrossRef

Shrout, P. E. (2009). Short and long views of integrative data analysis: Comments on contributions to the special issue. Psychological Methods, 14, 177–181.PubMedCrossRef

Sloboda, Z., Stephens, R. C., Stephens, P. C., Grey, S. F., Teasdale, B., Hawthorne, R. D., et al. (2009). The Adolescent Substance Abuse Prevention Study: A randomized field trial of a universal substance abuse prevention program. Drug and Alcohol Dependence, 102, 1–10.PubMedCrossRef

Smith, M. L., & Glass, G. V. (1977). Meta-analysis of psychotherapy outcome studies. American Psychologist, 32, 752–760.PubMedCrossRef

Tein, J.-Y., Sandler, I. N., MacKinnon, D. P., & Wolchik, S. A. (2004). How did it work? Who did it work for? Mediation in the context of a moderated prevention effect for children of divorce. Journal of Consulting & Clinical Psychology, 72, 617–624.CrossRef

Tobler, N. S. (1986). Meta-analysis of 143 adolescent drug prevention programs: Quantitative outcome results of program participants compared to a control or comparison group. Journal of Drug Issues, 16, 537–567.

Van Horn, M. L., Jaki, T., Masyn, K., Ramey, S. L., Smith, J. A., & Antaramian, S. (2009). Detecting differential effects: Applying regression mixture models to identify variations in the influence of family resources on academic achievement. Developmental Psychology, 45, 1298–313.PubMedCrossRef

Wilson, D. B., & Lipsey, M. W. (2003). The role of method in treatment effectiveness research: Evidence from meta-analysis. In A. E. Kazdin (Ed.), Methodological issues & strategies in clinical research (3rd ed., pp. 589–615). Washington, DC: American Psychological Association.

Wolchik, S., Sandler, I., Weiss, L., & Winslow, E. (2007). New Beginnings: An empirically-based intervention program for divorced mothers to help children adjust to divorce. In J. M. Briesmeister & C. E. Schaefer (Eds.), Handbook of parent training: Helping parents prevent and solve problem behaviors (3rd ed., pp. 25–66). Hoboken, NJ: John Wiley & Sons.

Title: Methods for Synthesizing Findings on Moderation Effects Across Multiple Randomized Trials
Authors: C. Hendricks Brown
Zili Sloboda
Fabrizio Faggiano
Brent Teasdale
Ferdinand Keller
Gregor Burkhart
Federica Vigna-Taglianti
George Howe
Katherine Masyn
Wei Wang
Bengt Muthén
Peggy Stephens
Scott Grey
Tatiana Perrino
Prevention Science and Methodology Group
Publication date: 01-04-2013
Publisher: Springer US
Published in: Prevention Science / Issue 2/2013
Print ISSN: 1389-4986
Electronic ISSN: 1573-6695
DOI: https://doi.org/10.1007/s11121-011-0207-8

Keynote webinar | Spotlight on medication adherence

Springer Medicine

Methods for Synthesizing Findings on Moderation Effects Across Multiple Randomized Trials

Abstract

Keynote webinar | Spotlight on medication adherence

Springer Medicine

Abstract

Please log in to get access to this content

Other articles of this Issue 2/2013

Introduction to the Special Issue: Subgroup Analysis in Prevention and Intervention Research

Exploring Connections Between Moderators and Mediators: Commentary on Subgroup Analyses in Intervention Research

Erratum to: Subgroups Analysis when Treatment and Moderators are Time-varying

Detecting Moderator Effects Using Subgroup Analyses

When is the Story in the Subgroups?

Commentary on Subgroup Analysis in Intervention Research: Opportunities for the Public Health Approach to Violence Prevention