Top

Published in:

01-08-2020 | Climate Change

Sustaining an Evidence-Based Program Over Time: Moderators of Sustainability and the Role of the Getting to Outcomes® Implementation Support Intervention

Authors: Joie Acosta, Matthew Chinman, Patricia A. Ebener, Patrick S. Malone, Jill S. Cannon, Elizabeth J. D’Amico

Published in: Prevention Science | Issue 6/2020

Abstract

Problematic rates of alcohol, e-cigarette, and other drug use among US adolescents highlight the need for effective implementation of evidence-based programs (EBPs), yet schools and community organizations have great difficulty implementing and sustaining EBPs. Although a growing number of studies show that implementation support interventions can improve EBP implementation, the literature on how to improve sustainability through implementation support is limited. This randomized controlled trial advances the literature by testing the effects of one such implementation intervention—Getting To Outcomes (GTO)—on sustainability of CHOICE, an after-school EBP for preventing substance use among middle-school students. CHOICE implementation was tracked for 2 years after GTO support ended across 29 Boys and Girls Club sites in the greater Los Angeles area. Predictors of sustainability were identified for a set of key tasks targeted by the GTO approach (e.g., goal setting, evaluation, collectively called “GTO performance”) and for CHOICE fidelity using a series of path models. One year after GTO support ended, we found no differences between GTO and control sites on CHOICE fidelity. GTO performance was also similar between groups; however, GTO sites were superior in conducting evaluation. Better GTO performance predicted better CHOICE fidelity. Two years after GTO support ended, GTO sites were significantly more likely to sustain CHOICE implementation when compared with control sites. This study suggests that using an implementation support intervention like GTO can help low-resource settings continue to sustain their EBP implementation to help them get the most out of their investment. ClinicalTrials.gov Identifier: NCT02135991.

Available only for authorised users

Coefficient ω is a measure of internal consistency on the same scale as coefficient alpha but is less biased, has fewer problems than alpha, and has CIs to more accurately evaluate reliability (Dunn et al. 2014).

Aarons, G. A., Sommerfeld, D. H., & Walrath-Greene, C. M. (2009). Evidence-based practice implementation: The impact of public versus private sector organization type on organizational support, provider attitudes, and adoption of evidence-based practice. Implementation Science, 4, 83. https://doi.org/10.1186/1748-5908-4-83.CrossRefPubMed

Acosta, J., Chinman, M., Ebener, P., Malone, P. S., Paddock, S., Phillips, A., et al. (2013). An intervention to improve program implementation: Findings from a two-year cluster randomized trial of assets-getting to outcomes. Implementation Science : IS, 8, 87–87. https://doi.org/10.1186/1748-5908-8-87.CrossRefPubMed

Benjamini, Y., & Hochberg, Y. (1995). Controlling the false discovery rate: a practical and powerful approach to multiple testing. Journal of the Royal Statistical Society: Series B (Methodological), 57(1), 289–300.

Campbell, M. K., Carr, C., DeVellis, B., Switzer, B., Biddle, A., Amamoo, M. A., et al. (2009). A randomized trial of tailoring and motivational interviewing to promote fruit and vegetable consumption for cancer prevention and control. [journal article]. Annals of Behavioral Medicine, 38, 71. https://doi.org/10.1007/s12160-009-9140-5.CrossRefPubMedPubMedCentral

Cannon, J. S., Gilbert, M., Ebener, P., Malone, P. S., Reardon, C. M., Acosta, J., & Chinman, M. (2019). Influence of an implementation support intervention on barriers and facilitators to delivery of a substance use prevention program. Prevention Science. https://doi.org/10.1007/s11121-019-01037-x.

Chinman, M., Hunter, S. B., Ebener, P., Paddock, S. M., Stillman, L., Imm, P., et al. (2008a). The getting to outcomes demonstration and evaluation: An illustration of the prevention support system. [journal article]. American Journal of Community Psychology, 41, 206–224. https://doi.org/10.1007/s10464-008-9163-2.CrossRefPubMedPubMedCentral

Chinman, M., Hunter, S. B., Ebener, P., Paddock, S. M., Stillman, L., Imm, P., et al. (2008b). The getting to outcomes demonstration and evaluation: An illustration of the prevention support system. American Journal of Community Psychology, 41, 206–224. https://doi.org/10.1007/s10464-008-9163-2.CrossRefPubMedPubMedCentral

Chinman, M., Tremain, B., Imm, P., & Wandersman, A. (2009). Strengthening prevention performance using technology: A formative evaluation of interactive getting to outcomes. The American Journal of Orthopsychiatry, 79, 469–481. https://doi.org/10.1037/a0016705.CrossRefPubMedPubMedCentral

Chinman, M., Acosta, J., Ebener, P., Malone, P. S., & Slaughter, M. E. (2016). Can implementation support help community-based settings better deliver evidence-based sexual health promotion programs? A randomized trial of getting to outcomes®. [journal article]. Implementation Science, 11, 78. https://doi.org/10.1186/s13012-016-0446-y.CrossRefPubMed

Chinman, M., Acosta, J., Ebener, P., Malone, P. S., & Slaughter, M. E. (2018a). A cluster-randomized trial of getting to outcomes' impact on sexual health outcomes in community-based settings. Prevention Science, 19, 437–448. https://doi.org/10.1007/s11121-017-0845-6.CrossRefPubMedPubMedCentral

Chinman, M., Acosta, J., Ebener, P., Hunter, S., Imm, P., & Wandersman, A. (2019). Dissemination of evidence-based prevention interventions and policies. In Z. Sloboda, H. Petras, E. Robertson, & R. Hingson (Eds.), Prevention of substance use (pp. 367–383). Cham: Springer International Publishing.CrossRef

Cicchetti, D. (1994). Guidelines, criteria, and rules of thumb for evaluating normed and standardized assessment instrument in psychology (Vol. 6).

Cooper, B. R., Bumbarger, B. K., & Moore, J. E. (2015). Sustaining evidence-based prevention programs: correlates in a large-scale dissemination initiative. Prevention Science, 16, 145–57.

D'Amico, E. J., Tucker, J. S., Miles, J. N. V., Zhou, A. J., Shih, R. A., & Green Jr., H. D. (2012). Preventing alcohol use with a voluntary after-school program for middle school students: Results from a cluster randomized controlled trial of CHOICE. Prevention Science, 13, 415–425. https://doi.org/10.1007/s11121-011-0269-7.CrossRefPubMedPubMedCentral

Damschroder, L. J., & Lowery, J. C. (2013). Evaluation of a large-scale weight management program using the Consolidated Framework for Implementation Research (CFIR). Implementation Science, 8, 51. https://doi.org/10.1186/1748-5908-8-51.CrossRefPubMed

Drug Enforcement Administration. (2017). National drug threat assessment. Washington (DC): Department of Justice (US).

Dunn, T. J., Baguley, T., & Brunsden, V. (2014). From alpha to omega: A practical solution to the pervasive problem of internal consistency estimation. British Journal of Psychology, 105, 399–412. https://doi.org/10.1111/bjop.12046.CrossRefPubMed

Elwert, F., & Winship, C. (2014). Endogenous selection bias: The problem of conditioning on a collider variable. Annual Review of Sociology, 40, 31–53.

Gloppen, K. M., Arthur, M. W., Hawkins, J. D., & Shapiro, V. B. (2012). Sustainability of the communities that care prevention system by coalitions participating in the community youth development study. The Journal of Adolescent Health : Official Publication of the Society for Adolescent Medicine, 51, 259–264. https://doi.org/10.1016/j.jadohealth.2011.12.018.CrossRef

Hailemariam, M., Bustos, T., Montgomery, B., Barajas, R., Evans, L. B., & Drahota, A. (2019). Evidence-based intervention sustainability strategies: A systematic review. Implementation Science, 14, 57–68.CrossRef

Johnson, K., Collins, D., Shamblen, S., Kenworthy, T., & Wandersman, A. (2017). Long-term sustainability of evidence-based prevention interventions and community coalitions survival: A five and one-half year follow-up study. Prevention Science, 18, 610–621. https://doi.org/10.1007/s11121-017-0784-2.CrossRefPubMed

Johnston, L. D., Miech, R. A., O'Malley, P. M., Bachman, J. G., Schulenberg, J. E., & Patrick, M. E. (2019). Monitoring the future national survey results on drug use, 1975–2018: Overview, Key Findings on Adolescent Drug Use. Institute for Social Research.

Kirchner, J. E., Ritchie, M. J., Pitcock, J. A., Parker, L. E., Curran, G. M., & Fortney, J. C. (2014). Outcomes of a partnered facilitation strategy to implement primary care-mental health. Journal of General Internal Medicine, 29, 904–912. https://doi.org/10.1007/s11606-014-3027-2.CrossRefPubMedPubMedCentral

Krippendorff, K. (2004). Content analysis: An introduction to its methodology (2nd ed.). Sage. Organizational Research Methods, 13, 392–394.

Moore, J. E., Mascarenhas, A., Bain, J., & Straus, S. E. (2017). Developing a comprehensive definition of sustainability. Implementation Science, 12, 110–117.CrossRef

Moyers, T., Martin, T., Manuel, J., Miller, W., & Ernst, D. (2010). Revised global scales: Motivational interviewing treatment integrity 3.1. 1 (MITI 3.1. 1). Unpublished manuscript, University of New Mexico, Albuquerque, NM.

Peterson, A. E., Bond, G. R., Drake, R. E., McHugo, G. J., Jones, A. M., & Williams, J. (2014). Predicting the long-term sustainability of evidence-based practices in mental health care: An 8-year longitudinal analysis. The Journal of Behavioral Health Services & Research, 41, 337–346. https://doi.org/10.1007/s11414-013-9347-x.CrossRef

Pollak, K. I., Coffman, C. J., Alexander, S. C., Østbye, T., Lyna, P., Tulsky, J. A., et al. (2014). Weight's up? Predictors of weight-related communication during primary care visits with overweight adolescents. Patient Education and Counseling, 96, 327–332. https://doi.org/10.1016/j.pec.2014.07.025.CrossRefPubMedPubMedCentral

Powell, B. J., Waltz, T. J., Chinman, M. J., Damschroder, L., Smith, J. L., Matthieu, M. M., Proctor, E. K., & Kirchner, J. E. (2015). A refined compilation of implementation strategies: Results from the expert recommendations for implementing change (ERIC) project. Implementation Science, 10, 21–34.CrossRef

Rycroft-Malone, J., Harvey, G., Seers, K., Kitson, A., McCormack, B., & Titchen, A. (2004). An exploration of the factors that influence the implementation of evidence into practice. Journal of Clinical Nursing, 13, 913–924. https://doi.org/10.1111/j.1365-2702.2004.01007.x.CrossRefPubMed

Sacks, J. J., Gonzales, K. R., Bouchery, E. E., Tomedi, L. E., & Brewer, R. D. (2015). 2010 national and state costs of excessive alcohol consumption. American Journal of Preventive Medicine, 49, e73–e79.CrossRef

Shelton, R. C., & Lee, M. (2019). Sustaining evidence-based interventions and policies: Recent innovations and future directions in implementation science. Annual Review of Public Health, 39, 55–76.CrossRef

Shelton, R. C., Cooper, B. R., & Stirman, S. W. (2018). The sustainability of evidence-based interventions and practices in public health and health care. Annual Review of Public Health, 39, 55–76.CrossRef

Spoth, R., Guyll, M., Redmond, C., Greenberg, M., & Feinberg, M. (2011). Six-year sustainability of evidence-based intervention implementation quality by community-university partnerships: The PROSPER study. [journal article]. American Journal of Community Psychology, 48, 412–425. https://doi.org/10.1007/s10464-011-9430-5.CrossRefPubMedPubMedCentral

Stirman, S., Kimberly, J., Cook, N., Calloway, A., Castro, F., & Charns, M. (2012). The sustainability of new programs and innovations: A review of the empirical literature and recommendations for future research. Implementation Science, 7, 17. https://doi.org/10.1186/1748-5908-7-17.CrossRef

Textor, J., van der Zander, B., Gilthorpe, M. K., Liskiewicz, M., & Ellison, G. T. H. (2015). Welcome to DAGitty [on-line]. Accessed November 6, 2018, from http://dagitty.net/

Textor, J., van der Zander, B., Gilthorpe, M. K., Liskiewicz, M., & Ellison, G. T. H. (2016). Robust causal inference using directed acyclic graphs: The R package ‘dagitty’. International Journal of Epidemiology, 45, 1887–1894.PubMed

Turrisi, R., Larimer, M. E., Mallett, K. A., Kilmer, J. R., Ray, A. E., Mastroleo, N. R., et al. (2009). A randomized clinical trial evaluating a combined alcohol intervention for high-risk college students. Journal of Studies on Alcohol and Drugs, 70, 555–567.CrossRef

Wandersman, A., & Florin, P. (2003). Community interventions and effective prevention. The American Psychologist, 58, 441–448.CrossRef

Welsh, J. A., Chilenski, S. M., Johnson, L., Greenberg, M. T., & Spoth, R. L. (2016). Pathways to sustainability: 8-year follow-up from the PROSPER project. The Journal of Primary Prevention, 37, 263–286.CrossRef

Wiseman, S. H., Chinman, M., Ebener, P. A., Hunter, S., Imm, P. S., & Wandersman, A. H. (2007). Getting to outcomes: 10 steps for achieving results-based accountability. Santa Monica, CA: RAND.

Title: Sustaining an Evidence-Based Program Over Time: Moderators of Sustainability and the Role of the Getting to Outcomes® Implementation Support Intervention
Authors: Joie Acosta
Matthew Chinman
Patricia A. Ebener
Patrick S. Malone
Jill S. Cannon
Elizabeth J. D’Amico
Publication date: 01-08-2020
Publisher: Springer US
Keyword: Climate Change
Published in: Prevention Science / Issue 6/2020
Print ISSN: 1389-4986
Electronic ISSN: 1573-6695
DOI: https://doi.org/10.1007/s11121-020-01118-2

Keynote webinar | Spotlight on medication adherence

Springer Medicine

Sustaining an Evidence-Based Program Over Time: Moderators of Sustainability and the Role of the Getting to Outcomes® Implementation Support Intervention

Abstract

Keynote webinar | Spotlight on medication adherence

Springer Medicine

Abstract

Please log in to get access to this content

Other articles of this Issue 6/2020

Bystander Intervention as a Prevention Strategy for Campus Sexual Violence: Perceptions of Historically Minoritized College Students

If You Host It Online, Who Will (and Will Not) Come? Individual and Partner Enrollment in a Web-Based Intervention for Distressed Couples

Developmental Differences in the Association of Peer Relationships with Traumatic Stress Symptoms

The Role of Future Orientation and Self-determination on American Indian Adolescents’ Intentions to Use Alcohol and Marijuana

The User-Program Interaction: How Teacher Experience Shapes the Relationship Between Intervention Packaging and Fidelity to a State-Adopted Health Curriculum

Effects of ePREP and OurRelationship on Low-Income Couples’ Mental Health and Health Behaviors: a Randomized Controlled Trial