Top

Published in:

01-02-2018

Addressing Methodologic Challenges and Minimizing Threats to Validity in Synthesizing Findings from Individual-Level Data Across Longitudinal Randomized Trials

Authors: Ahnalee Brincks, Samantha Montag, George W. Howe, Shi Huang, Juned Siddique, Soyeon Ahn, Irwin N. Sandler, Hilda Pantin, C. Hendricks Brown

Published in: Prevention Science | Special Issue 1/2018

Abstract

Integrative Data Analysis (IDA) encompasses a collection of methods for data synthesis that pools participant-level data across multiple studies. Compared with single-study analyses, IDA provides larger sample sizes, better representation of participant characteristics, and often increased statistical power. Many of the methods currently available for IDA have focused on examining developmental changes using longitudinal observational studies employing different measures across time and study. However, IDA can also be useful in synthesizing across multiple randomized clinical trials to improve our understanding of the comprehensive effectiveness of interventions, as well as mediators and moderators of those effects. The pooling of data from randomized clinical trials presents a number of methodological challenges, and we discuss ways to examine potential threats to internal and external validity. Using as an illustration a synthesis of 19 randomized clinical trials on the prevention of adolescent depression, we articulate IDA methods that can be used to minimize threats to internal validity, including (1) heterogeneity in the outcome measures across trials, (2) heterogeneity in the follow-up assessments across trials, (3) heterogeneity in the sample characteristics across trials, (4) heterogeneity in the comparison conditions across trials, and (5) heterogeneity in the impact trajectories. We also demonstrate a technique for minimizing threats to external validity in synthesis analysis that may result from non-availability of some trial datasets. The proposed methods rely heavily on latent variable modeling extensions of the latent growth curve model, as well as missing data procedures. The goal is to provide strategies for researchers considering IDA.

Available only for authorised users

Barch, D. M., Gotlib, I. H., Bilder, R. M., Pine, D. S., Smoller, J. W., Brown, C. H., …, Farber, G. K. (2016). Common measures for national institute of mental health funded research. Biological Psychiatry, 79, e91–e96. doi:10.1016/j.biopsych.2015.07.006.

Bauer, D. J., & Hussong, A. M. (2009). Psychometric approaches for developing commensurate measures across independent studies: Traditional and new models. Psychological Methods, 14, 101–125. doi:10.1037/a0015583.CrossRefPubMedPubMedCentral

Bauer, D. J., Howard, A. L., Baldasaro, R. E., Curran, P. J., Hussong, A. M., Chassin, L., & Zucker, R. A. (2013). A trifactor model for integrating ratings across multiple informants. Psychological Methods, 18, 475–493. doi:10.1037/a0032475.CrossRefPubMedPubMedCentral

Bollen, K., & Lennox, R. (1991). Conventional wisdom on measurement: a structural equation perspective. Psychological Bulletin, 110, 305–314.CrossRef

Brown, C. H. (1983). Asymptotic comparison of missing data procedures for estimating factor loadings. Psychometrika, 48, 269–291.CrossRef

Brown, C. H. (1990). Protecting against nonrandomly missing data in longitudinal studies. Biometrics, 46, 143–155.CrossRefPubMed

Brown, C. H., Wyman, P. A., Brinales, J. M., & Gibbons, R. D. (2007). The role of randomized trials in testing interventions for the prevention of youth suicide. International Review of Psychiatry, 19, 617–631. doi:10.1080/09540260701797779.CrossRefPubMed

Brown, C. H., Wang, W., & Sandler, I. (2008). Examining how context changes intervention impact: the use of effect sizes in multilevel meta-analysis. Child Development Perspectives, 2, 198–205. doi:10.1111/j.1750-8606.2008.00065.x.CrossRefPubMedPubMedCentral

Brown, C. H., Sloboda, Z., Faggiano, F., Teasdale, B., Keller, F., Burhart, G., …, Prevention Science and Methodology Group (2013). Methods for synthesizing findings on moderation effects across multiple randomized trials. Prevention Science, 14, 144–156. doi:10.1007/s11121-011-0207-8.CrossRefPubMedPubMedCentral

Brown, C. H., Brincks, A., Huang, S., Perrino, T., Cruden, G., Pantin, H., …, Sandler, I. (2016). Two-year impact of prevention programs on adolescent depression: An integrative data analysis approach. Prevention Science, 1–21.

Curran, P. J. (2009). The seemingly quixotic pursuit of a cumulative psychological science: Introduction to the special issue. Psychological Methods, 14, 77–80. doi:10.1037/a0015972.CrossRefPubMedPubMedCentral

Curran, P. J., & Hussong, A. M. (2009). Integrative data analysis: The simultaneous analysis of multiple data sets. Psychological Methods, 14, 81–100. doi:10.1037/a0015914.CrossRefPubMedPubMedCentral

Curran, P. J., Hussong, A. M., Cai, L., Huang, W., Chassin, L., Sher, K. J., & Zucker, R. A. (2008). Pooling data from multiple longitudinal studies: the role of item response theory in integrative data analysis. Developmental Psychology, 44, 365–380. doi:10.1037/0012-1649.44.2.365.CrossRefPubMedPubMedCentral

Dagne, G., Brown, C. H., Howe, G., Kellam, S., & Liu, L. (2016). Testing moderation in network meta-analysis with individual participant data. Statistics in Medicine, 35, 2485–2502. doi:10.1002/sim.6883.CrossRefPubMedPubMedCentral

Edwards, M. C., & Wirth, R. J. (2012). Valid measurement without factorial invariance: a longitudinal example. In J. R. Harring, G. R. Hancock, J. R. Harring, & G. R. Hancock (Eds.), Advances in longitudinal methods in the social and behavioral sciences (pp. 289–311). Charlotte: IAP Information Age Publishing.

Gibbons, R. D., & Hedeker, D. R. (1992). Full-information item bi-factor analysis. Psychometrika, 57, 423–436.CrossRef

Gibbons, R. D., Weiss, D. J., Pilkonis, P. A., Frank, E., Moore, T., Kim, J. B., & Kupfer, D. J. (2012). Development of a computerized adaptive test for depression. Archives of General Psychiatry, 69, 1104–1112. doi:10.1001/archgenpsychiatry.2012.14.CrossRefPubMedPubMedCentral

Hancock, G. R., Kuo, W.-L., & Lawrence, F. R. (2001). An illustration of second-order latent growth models. Structural Equation Modeling, 8, 470–489. doi:10.1207/S15328007SEM0803_7.CrossRef

Hedges, L. V., & Vevea, J. L. (1998). Fixed- and random-effects models in meta-analysis. Psychological Methods, 3, 486–504. doi:10.1037/1082-989X.3.4.486.CrossRef

Hendershot, T., Pan, H., Haines, J., Harlan, W. R., Marazita, M. L., McCarty, C. A., …, Hamilton, C. M. (2015). Using the PhenX Toolkit to add standard measures to a study. Current Protocols in Human Genetics, 86, 1.21.21–17. doi:10.1002/0471142905.hg0121s86.

Howe, G. W., Dagne, G., Brincks, A., & Beardslee, W. (2016). Evaluating construct equivalence and harmonizing measurement of adolescent depression when synthesizing results across multiple studies. Submitted for publicaton.

Hussong, A. M., Curran, P. J., & Bauer, D. J. (2013). Integrative data analysis in clinical psychology research. Annual Review of Clinical Psychology, 9, 61–89. doi:10.1146/annurev-clinpsy-050212-185522.CrossRefPubMedPubMedCentral

Lavori, P. W., Brown, C. H., Duan, N., Gibbons, R. D., & Greenhouse, J. (2008). Missing data in longitudinal clinical trials. Part A. Design and conceptual issues. Psychiatric Annals, 38, 784–792.CrossRefPubMedPubMedCentral

McArdle, J. J. (1988). Dynamic but structural equation modeling of repeated measures data. In J. R. Nesselroade & R. B. Cattell (Eds.), Handbook of multivariate experimental psychology (pp. 561–614). New York: Plenum Press.CrossRef

McCarty, C. A., Huggins, W., Aiello, A. E., Bilder, R. M., Hariri, A., Jernigan, T. L., …, Zeng, Y. (2014). PhenX RISING: Real world implementation and sharing of PhenX measures. BMC Medical Genomics, 7, 16. doi:10.1186/1755-8794-7-16.

Muthén, B. O., & Brown, C. H. (2009). Estimating drug effects in the presence of placebo response: Causal inference using growth mixture modeling. Statistics in Medicine, 28, 3363–3395. doi:10.1002/sim.3721.CrossRefPubMedPubMedCentral

Muthén, B. O., Brown, C. H., Masyn, K., Jo, B., Khoo, S. T., Yang, C. C., …, Liao, J. (2002). General growth mixture modeling for randomized preventive interventions. Biostatistics, 3, 459–475. doi:10.1093/biostatistics/3.4.459.

Muthén, B. O., Brown, C. H., Leuchter, A., & Hunter, A. (2010). General approaches to analysis of course: Applying growth mixture modeling to randomized trials of depression medication. In P. E. Shrout, K. M. Keyes, & K. Ornstein (Eds.), Causality and psychopathology: finding the determinants of disorders and their cures (pp. 159–178). Washington: American Psychiatric Publishing.

Pan, H., Tryka, K. A., Vreeman, D. J., Huggins, W., Phillips, M. J., Mehta, J. P., …, Ramos, E. M. (2012). Using PhenX measures to identify opportunities for cross-study analysis. Human Mutation, 33, 849–857. doi: 10.1002/humu.22074.

Pathak, J., Pan, H., Wang, J., Kashyap, S., Schad, P. A., Hamilton, C. M., …, Chute, C. G. (2011). Evaluating phenotypic data elements for genetics and epidemiological research: Experiences from the eMERGE and PhenX Network Projects. AMIA Summits on Translational Science Proceedings, 2011, 41–45.

Perrino, T., Howe, G., Sperling, A., Beardslee, W., Sandler, I., Shern, D., Brown, C. H.(2013). Advancing science through collaborative data sharing and synthesis. Perspectives on Psychological Science, 8(4), 433–444. doi:10.1177/1745691613491579.

Perrino, T., Beardslee, W., Bernal, G., Brincks, A., Cruden, G., Howe, G., …, Brown, C. H. (2015). Toward scientific equity for the prevention of depression and depressive symptoms in vulnerable youth. Prevention Science, 16, 642–651. doi:10.1007/s11121-014-0518-7.

Raudenbush, S. W., & Bryk, A. S. (2002). Hierarchical linear models: applications and data analysis methods. Thousand Oaks: Sage Publications.

Royall, R. M., & Cumberland, W. G. (1981). An empirical study of the ratio estimator and estimators of its variance. Journal of the American Statistical Association, 76, 66–77.CrossRef

Rubin, D. B. (1976). Inference and missing data. Biometrika, 63, 581–592.CrossRef

Sandler, I. N., Wolchik, S. A., Cruden, G., Mahrer, N. E., Ahn, S., Brincks, A., & Brown, C. H. (2014). Overview of meta-analyses of the prevention of mental health, substance use, and conduct problems. Annual Review of Clinical Psychology, 10, 243–273. doi:10.1146/annurev-clinpsy-050212-185524.CrossRefPubMedPubMedCentral

Siddique, J., Brown, C. H., Hedeker, D., Duan, N., Gibbons, R. D., Miranda, J., & Lavori, P. W. (2008). Missing data in longitudinal trials—part B, analytic issues. Psychiatric Annals, 38, 793–801.CrossRefPubMedPubMedCentral

Siddique, J., Crespi, C. M., Gibbons, R. D., & Green, B. L. (2011). Using latent variable modeling and multiple imputation to calibrate rater bias in diagnosis assessment. Statistics in Medicine, 30, 160–174. doi:10.1002/sim.4109.CrossRefPubMed

Siddique, J., Reiter, J. P., Brincks, A., Gibbons, R. D., Crespi, C. M., & Brown, C. H. (2015). Multiple imputation for harmonizing longitudinal non-commensurate measures in individual participant data meta-analysis. Statistics in Medicine, 34, 3399–3414. doi:10.1002/sim.6562.CrossRefPubMedPubMedCentral

Siddique, J., de Chavez, P. J., Howe, G., Cruden, G., & Brown, C. H. (2016). Limitations in using multiple imputation to harmonize individual participant data for meta-analysis. Prevention Science, 1–14.

Tisak, J., & Meredith, W. (1990). Descriptive and associative developmental models. In A. Von Eye (Ed.), Statistical methods in longitudinal research (Vol. II, pp. 387–406). San Diego: Academic Press.

Wang, C.-P., Brown, C. H., & Bandeen-Roche, K. (2005). Residual diagnostics for growth mixture models: Examining the impact of a preventive intervention on multiple trajectories of aggressive behavior. Journal of the American Statistical Association, 100, 1054–1076. doi:10.1198/016214505000000501.CrossRef

Title: Addressing Methodologic Challenges and Minimizing Threats to Validity in Synthesizing Findings from Individual-Level Data Across Longitudinal Randomized Trials
Authors: Ahnalee Brincks
Samantha Montag
George W. Howe
Shi Huang
Juned Siddique
Soyeon Ahn
Irwin N. Sandler
Hilda Pantin
C. Hendricks Brown
Publication date: 01-02-2018
Publisher: Springer US
Published in: Prevention Science / Issue Special Issue 1/2018
Print ISSN: 1389-4986
Electronic ISSN: 1573-6695
DOI: https://doi.org/10.1007/s11121-017-0769-1

ACC 2024 Congress

Springer Medicine

Addressing Methodologic Challenges and Minimizing Threats to Validity in Synthesizing Findings from Individual-Level Data Across Longitudinal Randomized Trials

Abstract

ACC 2024 Congress

Springer Medicine

Abstract

Please log in to get access to this content

Other articles of this Special Issue 1/2018

The Family Check Up and Adolescent Depression: An Examination of Treatment Responders and Non-Responders

Prevention of Depression in At-Risk Adolescents: Moderators of Long-term Response

Identifying Moderators of Response to the Penn Resiliency Program: A Synthesis Study

Strength in Numbers

Two-Year Impact of Prevention Programs on Adolescent Depression: an Integrative Data Analysis Approach

Attendance Patterns and Links to Non-Response on Child Report of Internalizing among Mexican-Americans Randomized to a Universal Preventive Intervention