Top

Implementation Science

Published in:

Open Access 01-12-2018 | Research

Testing implementation support for evidence-based programs in community settings: a replication cluster-randomized trial of Getting To Outcomes®

Authors: Matthew Chinman, Patricia Ebener, Patrick S. Malone, Jill Cannon, Elizabeth J. D’Amico, Joie Acosta

Published in: Implementation Science | Issue 1/2018

Abstract

Background

Community organizations can have difficulty implementing evidence-based prevention programs. More research is needed on implementation support interventions designed to help these organizations implement programs with quality.

Methods

Preparing to Run Effective Programs (PREP) is a randomized controlled trial testing Getting To Outcomes (GTO), a 2-year implementation support intervention. It compares 15 Boys and Girls Club sites implementing CHOICE (control group), a five-session evidence-based alcohol and drug prevention program, with 14 similar sites implementing CHOICE supported by GTO (intervention group). PREP replicates a previous GTO study that had the same design, but featured a teen pregnancy prevention program instead. All sites received typical CHOICE training. Fourteen intervention sites received GTO manuals, training, and onsite technical assistance to help practitioners complete implementation best practices specified by GTO (i.e., GTO steps). During the first year, technical assistance providers helped the intervention group adopt, plan, and deliver CHOICE. Then, this group was trained on evaluation and quality improvement steps of GTO using feedback reports summarizing their own data, which yielded revised plans for subsequent implementation of CHOICE. This paper presents results regarding GTO’s impact on CHOICE fidelity (adherence, quality of delivery, dosage) and the proximal outcomes of the youth participants (aged 10–14)—attitudes and intentions regarding cigarettes, alcohol, and marijuana use. Fidelity was assessed at all sites by observer ratings and attendance logs. Proximal outcomes were assessed via survey at baseline, 3, and 6 months.

Results

After 1 year, fidelity and proximal outcomes were similar between Intervention and control groups. After 2 years (which included GTO quality improvement activities that took place between years 1 and 2), intervention sites had higher ratings of CHOICE adherence and quality of delivery (dosage remained similar). Proximal outcomes did not differ between groups in either year, although there was universally high endorsement of prosocial responses to those outcomes from the start.

Conclusions

Findings suggest that systematic implementation support provided by GTO can help community organizations achieve better fidelity. Findings replicate the implementation results from a previous GTO study using the same design, but with a different evidence-based program and different fidelity measures. Although proximal outcomes did not change, in large part due to ceiling effects, the implementation findings suggest GTO can support a variety of programs.

Trial registration

This project is registered at ClinicalTrials.gov with number NCT02135991. The trial was first registered on May 12, 2014.

Johnston LD, O’Malley PM, Miech RA, Bachman JG, Schulenberg JE. Monitoring the future national survey results on drug use, 1975–2015: overview, key findings on adolescent drug use. Ann Arbor, MI: Institute for Social Research, The University of Michigan; 2016.

U.S. Department of Health and Human Services. Facing Addiction in America. The Surgeon General’s Report on Alcohol, Drugs, and Health. Washington, DC: U.S. Department of Health and Human Services; 2016.

Green L. From research to ‘best practices’ in other settings and populations. Am J Health Behav. 2001;25(3):165–78.CrossRef

Wandersman A, Florin P. Community interventions and effective prevention: bringing researchers/evaluators, funders and practitioners together for accountability. Am Psychol. 2003;58(6/7):441–8.CrossRef

Ennett ST, Ringwalt CL, Thorne J, Rohrbach LA, Vincus A, Simons-Rudolph A, et al. A comparison of current practice in school-based substance abuse prevention programs with meta-analysis findings. Prev Sci. 2003;4:1–14.CrossRef

Hallfors D, Godette D. Will the 'principles of effectiveness' improve prevention practice? Early findings from a diffusion study. Health Educ Res. 2002;17(4):461–70.CrossRef

Kumar R, O'Malley PM, Johnston LD, Laetz VB. Alcohol, tobacco, and other drug use prevention programs in U.S. schools: a descriptive summary. Prev Sci. 2013;14(6):581–92. https://doi.org/10.1007/s11121-012-0340-z.CrossRefPubMedPubMedCentral

Ringwalt C, Hanley S, Vincus AA, Ennett ST, Rohrbach LA, Bowling JM. The prevalence of effective substance use prevention curricula in the nation's high schools. J Prim Prev. 2008;29(6):479–88. https://doi.org/10.1007/s10935-008-0158-4.CrossRefPubMedPubMedCentral

Ringwalt C, Vincus AA, Hanley S, Ennett ST, Bowling JM, Rohrbach LA. The prevalence of evidence-based drug use prevention curricula in U.S. middle schools in 2005. Prev Sci. 2009;10(1):33–40. https://doi.org/10.1007/s11121-008-0112-y.CrossRefPubMedPubMedCentral

10.

Chinman M, Acosta J, Hunter S, Ebener P. Getting To Outcomes®: evidence of empowerment evaluation and evaluation capacity building at work. In: Fetterman DM, Kaftarian S, Wandersman A, editors. Empowerment evaluation: knowledge and tools for self-assessment, evaluation capacity building and accountability. 2nd ed. Thousand Oaks, CA: Sage; 2015. p. 316–36.

11.

D'Amico EJ, Tucker JS, Miles JNV, Zhou AJ, Shih RA, Green HDJ. Preventing alcohol use with a voluntary after school program for middle school students: results from a cluster randomized controlled trial of CHOICE. Prev Sci. 2012;13:415–25.CrossRef

12.

Chinman M, Acosta J, Ebener P, Malone PS, Slaughter ME. Can implementation support help community-based settings better deliver evidence-based sexual health promotion programs? A randomized trial of Getting To Outcomes(R). Implement Sci. 2016;11(1):78.CrossRef

13.

Jemmott JB, Jemmott LS, Fong GT. Abstinence and safer sex HIV risk-reduction interventions for African American adolescents: a randomized controlled trial. J Am Med Assoc. 1998;279:1529–36.CrossRef

14.

Livet M, Wandersman A. Organizational functioning: facilitating effective interventions and increasing the odds of programming success. In: Fetterman DM, Wandersman A, editors. Empowerment evaluation in practice. New York: Guilford; 2005. p. 123–54.

15.

Ajzen I, Fishbein M. Attitude-behavior relations: a theoretical analysis and review of empirical research. Psychol Bull. 1977;84:888–918.CrossRef

16.

Bandura A. Health promotion by social cognitive means. Health Educ Behav. 2004;31(2):143–64.CrossRef

17.

Fishbein M, Ajzen I. Attitudes toward objects as predictive of single and multiple behavioral criteria. Psychol Rev. 1974;81:59–74.CrossRef

18.

Fishbein M, Ajzen I. Belief, attitude, intention, and behavior: an introduction to theory and research. Reading, MA: Addison-Wesley; 1975.

19.

Acosta J, Chinman M, Ebener P, Malone PS, Paddock S, Phillips A, et al. An intervention to improve program implementation: findings from a two-year cluster randomized trial of assets-Getting To Outcomes. Implement Sci. 2013;8:87.CrossRef

20.

Smelson DA, Chinman M, McCarthy S, Hannah G, Sawh L, Glickman M. A cluster randomized hybrid type III trial testing an implementation support strategy to facilitate the use of an evidence-based practice in VA homeless programs. Implement Sci. 2015;10:79. https://doi.org/10.1186/s13012-015-0267-4.CrossRefPubMedPubMedCentral

21.

Durlak JA, DuPre EP. Implementation matters: a review of research on the influence of implementation on program outcomes and the factors affecting implementation. Am J Community Psychol. 2008;41:327–50.CrossRef

22.

Chinman M, Imm P, Wandersman A. Getting to Outcomes 2004: Promoting Accountability Through Methods and Tools for Planning, Implementation, and Evaluation. Santa Monica, CA: RAND Corporation; 2004. Contract No.: TR-101

23.

Wandersman A, Imm P, Chinman M, Kaftarian S. Getting to outcomes: a results-based approach to accountability. Eval Program Plann. 2000;23:389–95.CrossRef

24.

Chinman M, Hunter SB, Ebener P, Paddock S, Stillman L, Imm P, et al. The Getting To Outcomes demonstration and evaluation: an illustration of the prevention support system. Am J Community Psychol. 2008;41:206–24.CrossRef

25.

Chinman M, Acosta JD, Ebener P, Sigel C, Keith J. Getting To Outcomes: a guide for teen pregnancy prevention. Santa Monica, CA: RAND Corporation; 2016.CrossRef

26.

Imm P, Chinman M, Wandersman A, Rosenbloom D, Guckenburg S, Leis R. Using the “Getting to Outcomes” Approach to Help Communities Prevent Underage Drinking. Santa Monica, CA: RAND Corporation; 2007. Contract No.: RB-9262

27.

Fisher D, Imm P, Chinman M, Wandersman A. Getting To Outcomes with developmental assets: ten steps to measuring success in youth programs and communities Minneapolis: MN: Search Institute; 2006.

28.

Chinman M, Tremain B, Imm P, Wandersman A. Strengthening prevention performance using technology: a formative evaluation of interactive Getting To Outcomes. Am J Orthopsychiatry. 2009;79:469–81 PMC2859836.CrossRef

29.

Chinman M, Acosta J, Ebener P, Malone PS, Slaughter M. A cluster-randomized trial of Getting To Outcomes’ impact on sexual health outcomes in community-based settings. Prev Sci. In Press.

30.

Hawkins JD, Oesterle S, Brown EC, Arthur MW, Abbott RD, Fagan AA, et al. Results of a type 2 translational research trial to prevent adolescent drug use and delinquency: a test of communities that care. Arch Pediatr Adolesc Med. 2009;163(9):789–98. https://doi.org/10.1001/archpediatrics.2009.141.CrossRefPubMedPubMedCentral

31.

Spoth R, Redmond C, Shin C, Greenberg M, Clair S, Feinberg M. Substance-use outcomes at 18 months past baseline: the PROSPER Community-University Partnership Trial. Am J Prev Med. 2007;32(5):395–402. https://doi.org/10.1016/j.amepre.2007.01.014.CrossRefPubMedPubMedCentral

32.

Rohrbach LA, Gunning M, Sun P, Sussman S. The project towards no drug abuse (TND) dissemination trial: implementation fidelity and immediate outcomes. Prev Sci. 2010;11(1):77–88. https://doi.org/10.1007/s11121-009-0151-z.CrossRefPubMedPubMedCentral

33.

Miller WR, Rollnick S. Motivational interviewing: preparing people for change. 2nd ed. New York: Guilford Press; 2002.

34.

Rollnick S, Miller WR, Butler CC. Motivational interviewing in health care: helping patients change behavior. New York: Guilford Press; 2008.

35.

D'Amico EJ, Ellickson PL, Wagner EF, Turrisi R, Fromme K, Ghosh-Dastidar B, et al. Developmental considerations for substance use interventions from middle school through college. Alcohol Clin Exp Res. 2005;29:474–83.CrossRef

36.

Chinman M, Acosta J, Ebener P, Burkhart Q, Clifford M, Corsello M, et al. Intervening with practitioners to improve the quality of prevention: one year findings from a randomized trial of assets-Getting To Outcomes. J Prim Prev. 2013;34:173–91.CrossRef

37.

Chinman M, Ebener P, Burkhart Q, Osilla KC, Imm P, Paddock SM, et al. Evaluating the impact of getting to outcomes-underage drinking on prevention capacity and alcohol merchant attitudes and selling behaviors. Prev Sci. 2014;15(4):485–96. https://doi.org/10.1007/s11121-013-0389-3.CrossRefPubMedPubMedCentral

38.

39.

40.

41.

Chinman M, Acosta J, Ebener P, Malone PS, Slaughter ME. A cluster-randomized trial of Getting To Outcomes’ impact on sexual health outcomes in community-based settings. Prev Sci. 2018;19(4):437–48. https://doi.org/10.1007/s11121-017-0845-6.CrossRefPubMed

42.

D'Amico EJ, Green HDJ, Miles JNV, Zhou AJ, Tucker JA, Shih RA. Voluntary after school alcohol and drug programs: if you build it right, they will come. J Res Adolesc. 2012;22(3):571–82.CrossRef

43.

Aarons GA, Glisson C, Hoagwood K, Kelleher K, Landsverk J, Cafri G. Psychometric properties and U.S. national norms of the evidence-based practice attitude scale (EBPAS). Psychol Assess. 2010;22(2):356–65. https://doi.org/10.1037/a0019188.CrossRefPubMed

44.

McDonald RP. Test theory: a unified treatment. Mahwah: Lawrence Erlbaum; 1999.

45.

Dunn TJ, Baguley T, Brunsden V. From alpha to omega: a practical solution to the pervasive problem of internal consistency estimation. Br J Psychol. 2014;105(3):399–412.CrossRef

46.

Krippendorff K. Content analysis: an introduction to its methodology. 2nd ed. Thousand Oaks, CA: Sage; 2004.

47.

Bandura A. Self-efficacy: toward a unifying theory of behavioral change. Psychol Rev. 1977;84:191–215.CrossRef

48.

Kahneman D, Tversky A. Choices, values, and frames. New York: Cambridge University Press; 2000.

49.

Bandura A. Self-efficacy: the exercise of control. New York: Freeman; 1997.

50.

Miller WR, Rollnick S. Motivational interviewing: helping people change. 3rd ed. New York: Guilford Press; 2012.

51.

D'Amico EJ, Edelen M. Pilot test of Project CHOICE: a voluntary after school intervention for middle school youth. Psychol Addict Behav. 2007;21(4):592–8.CrossRef

52.

Dusenbury L, Brannigan R, Hansen W, Walsh J, Falco M. Quality of implementation: developing measures crucial to understanding the diffusion of preventive interventions. Health Educ Behav. 2005;20:308–13.

53.

Moyers TB, Martin T, Manuel JK, Miller WR, Ernst D. Revised Global Scales: Motivational Interviewing Treatment Integrity 3.1.1 (MITI 3.1.1). Albuquerque: NM University of New Mexico; 2010.

54.

Madson MB, Campbell TC. Measures of fidelity in motivational enhancement: a systematic review. J Subst Abus Treat. 2006;31(1):67–73. https://doi.org/10.1016/j.jsat.2006.03.010.CrossRef

55.

Moyers TB, Martin T, Manuel JK, Hendrickson SM, Miller WR. Assessing competence in the use of motivational interviewing. J Subst Abus Treat. 2005;28(1):19–26. https://doi.org/10.1016/j.jsat.2004.11.001.CrossRef

56.

Cambell M, Carr C, DeVellis B, Switzer B, Biddle A, Amamoo MA, et al. A randomized trial of tailoring and motivational interviewing to promote fruit and vegetable consumption for cancer prevention and control. Ann Behav Med. 2009;38(2):71–85.CrossRef

57.

Martino S, Ball SA, Nich C, Frankforter TL, Carroll KM. Community program therapist adherence and competence in motivational enhancement therapy. Drug Alcohol Depend. 2008;96(1):37–48.CrossRef

58.

Parsons JT, Golub SA, Rosof E, Holder C. Motivational interviewing and cognitive-behavioral intervention to improve HIV medication adherence among hazardous drinkers: a randomized controlled trial. J Accuir Immune Defic. 2007;46(4):443.CrossRef

59.

Turrisi R, Larimer ME, Mallett KA, Kilmer JR, Ray AE, Mastroleo NR, et al. A randomized clinical trial evaluating a combined alcohol intervention for high-risk college students. J Stud Alcohol Drugs. 2009;70(4):555.CrossRef

60.

McCambridge J, Day M, Thomas BA, Strang J. Fidelity to motivational interviewing and subsequent cannabis cessation among adolescents. Addict Behav. 2011;36(7):749–54.CrossRef

61.

Pollak KI, Coffman CJ, Alexander SC, Ostbye T, Lyna P, Tulsky JA, et al. Predictors of weight-related communication during primary care visits with overweight adolescents. Patient Educ Couns. 2014;96(3):327–32.CrossRef

62.

Woodin EM, Sotskova A, O'Leary KD. Do motivational interviewing behaviors predict reductions in partner aggression for men and women? Behav Res Ther. 2012;50:79–84.CrossRef

63.

Cicchetti DV. Guidelines, criteria, and rules of thumb for evaluating normed and standardized assessment instruments in psychology. Psychol Assess. 1994;94:284–90.CrossRef

64.

Johnston LD, O'Malley PM, Miech RA, Bachman JG, Schulenberg JE. Monitoring the future national survey results on drug use, 1975–2016: overview, key findings on adolescent drug use. Ann Arbor: Institute for Social Research, The University of Michigan; 2017.

65.

D'Amico EJ, Hunter SB, Miles JN, Ewing BA, Osilla KC. A randomized controlled trial of a group motivational interviewing intervention for adolescents with a first time alcohol or drug offense. J Subst Abus Treat. 2013;45(5):400–8. https://doi.org/10.1016/j.jsat.2013.06.005.CrossRef

66.

D'Amico EJ, Miles JN, Stern SA, Meredith LS. Brief motivational interviewing for teens at risk of substance use consequences: a randomized pilot study in a primary care clinic. J Subst Abus Treat. 2008;35(1):53–61. https://doi.org/10.1016/j.jsat.2007.08.008.CrossRef

67.

D'Amico EJ, Parast L, Meredith LS, Ewing BA, Shadel WG, Stein BD. Screening in Primary Care: What Is the Best Way to Identify At-Risk Youth for Substance Use? Pediatrics. 2016;138(6). https://doi.org/10.1542/peds.2016-1717.CrossRef

68.

D'Amico EJ, Miles JN, Tucker JS. Gateway to curiosity: medical marijuana ads and intention and use during middle school. Psychol Addict Behav. 2015;29(3):613–9. https://doi.org/10.1037/adb0000094.CrossRefPubMedPubMedCentral

69.

D'Amico EJ, Parast L, Shadel WG, Meredith LS, Seelam R, Stein BD. Brief motivational interviewing intervention to reduce alcohol and marijuana use for at-risk adolescents in primary care: a randomized clinical trial. J Consult Clin Psychol. In Press.

70.

D’Amico EJ, McCarthy DM. Escalation and initiation of younger adolescents’ substance use: the impact of perceived peer use. J Adolesc Health. 2006;39:481–7.CrossRef

71.

D'Amico EJ, Miles JNV, Stern SA, Meredith LS. Brief motivational interviewing for teens at risk of substance use consequences: a randomized pilot study in a primary care clinic. J Subst Abus Treat. 2008;35:53–61.CrossRef

72.

Ellickson PL, Bell RM. Drug prevention in junior high: a multi-site longitudinal test. Science. 1990;247:1265–372.CrossRef

73.

Lakens D. Calculate Partial Eta Omega Epsilon Squared.xlsx. 2013. https://osf.io/kzsyg/ Accessed 15 Apr 2016.

74.

Lakens D. Calculating_Effect_Sizes.xlsx (v.4). 2013. https://osf.io/vbdah/. Accessed 22 Nov 2017.

75.

Hess MR, Kromrey JD. Effect_CI: a SAS® macro for constructing confidence intervals around standardized mean differences. 2003. http://analytics.ncsu.edu/sesug/2003/SD04-Hess.pdf. Accessed 16 Apr 2016.

76.

Benjamini Y, Hochberg B. Controlling the false discovery rate: a practical and powerful approach to multiple testing. J R Stat Soc. 1995;57(1):289–300.

77.

Fagan AA, Hanson K, Hawkins JD, Arthur MW. Bridging science to practice: achieving prevention program implementation fidelity in the community youth development study. Am J Community Psychol. 2008;41:235–49.CrossRef

78.

Rohrbach LA, Gunning M, Sun P, Sussman S. The project towards no drug abuse (TND) dissemination trial: implementation fidelity and immediate outcomes. Prev Sci. 2010;11:77–88.CrossRef

79.

Spoth R, Guyll M, Lillehoj CJ, Redmond C. PROSPER study of evidence-based intervention implementation quality by community–university partnerships. J Community Psychol. 2007;35(8):981–99.CrossRef

80.

D'Amico EJ, Houck JM, Hunter SB, Miles JN, Osilla KC, Ewing BA. Group motivational interviewing for adolescents: change talk and alcohol and marijuana outcomes. J Consult Clin Psychol. 2015;83(1):68–80. https://doi.org/10.1037/a0038155.CrossRefPubMed

81.

Mueller T, Tevendale HD, Fuller TR, House LD, Romero LM, Brittain A, et al. Teen pregnancy prevention: implementation of a multicomponent Community-Wide Approach. J Adolesc Health. 2017;60(3S):S9–S17. https://doi.org/10.1016/j.jadohealth.2016.11.002.CrossRefPubMed

82.

Ellickson PL, Martino SC, Collins RL. Marijuana use from adolescence to young adulthood: multiple developmental trajectories and their associated outcomes. Health Psychol. 2004;23(3):299–307. https://doi.org/10.1037/0278-6133.23.3.299.CrossRefPubMed

83.

Jackson KM, Sher KJ, Schulenberg JE. Conjoint developmental trajectories of young adult substance use. Alcohol Clin Exp Res. 2008;32(5):723–37. https://doi.org/10.1111/j.1530-0277.2008.00643.x.CrossRefPubMedPubMedCentral

84.

Schulenberg JE, Merline AC, Johnston LD, O'Malley PM, Bachman JG, Laetz VB. Trajectories of marijuana use during the transition to adulthood: the big picture based on National Panel Data. J Drug Issues. 2005;35(2):255–79.CrossRef

85.

Setoh P, Qin L, Zhang X, Pomerantz EM. The social self in early adolescence: two longitudinal investigations in the United States and China. Dev Psychol. 2015;51(7):949–61. https://doi.org/10.1037/a0039354.CrossRefPubMed

86.

Tucker JS, Ellickson PL, Orlando M, Klein DJ. Cigarette smoking from adolescence to young adulthood: women's developmental trajectories and associates outcomes. Womens Health Issues. 2006;16(1):30–7. https://doi.org/10.1016/j.whi.2005.08.004.CrossRefPubMed

87.

Tucker JS, Orlando M, Ellickson PL. Patterns and correlates of binge drinking trajectories from early adolescence to young adulthood. Health Psychol. 2003;22(1):79–87.CrossRef

88.

Vega WA, Chen KW, Williams J. Smoking, drugs, and other behavioral health problems among multiethnic adolescents in the NHSDA. Addict Behav. 2007;32(9):1949–56. https://doi.org/10.1016/j.addbeh.2006.12.022.CrossRefPubMed

89.

Chervin DD, Philliber S, Brindis CD, Chadwick AE, Revels ML, Kamin SL, et al. Community capacity building in CDC’s community coalition partnership programs for the prevention of teen pregnancy. J Adolesc Health. 2005;37:S11–S9.CrossRef

90.

Lesesne CA, Lewis KM, White CP, White DC, Green DC, Duffy JL, et al. Promoting science-based approaches to teen pregnancy prevention: proactively engaging the three systems of the Interactive Systems Framework. Am J Community Psychol. 2008;41:379–92.CrossRef

91.

Philliber S, Nolte K. Implementation science: promoting science-based approaches to prevent teen pregnancy. Prev Sci. 2008;9(3):166–77.CrossRef

92.

Zief S, Shapiro R, Strong D. How states are implementing evidence-based teen pregnancy prevention programs through the personal responsibility education program. Washington, DC: Office of Planning, Research and Evaluation, Administration for Children and Families; 2014.

Title: Testing implementation support for evidence-based programs in community settings: a replication cluster-randomized trial of Getting To Outcomes®
Authors: Matthew Chinman
Patricia Ebener
Patrick S. Malone
Jill Cannon
Elizabeth J. D’Amico
Joie Acosta
Publication date: 01-12-2018
Publisher: BioMed Central
Published in: Implementation Science / Issue 1/2018
Electronic ISSN: 1748-5908
DOI: https://doi.org/10.1186/s13012-018-0825-7

At a glance: The ONWARDS insulin icodec trials

Springer Medicine

Testing implementation support for evidence-based programs in community settings: a replication cluster-randomized trial of Getting To Outcomes®

Abstract

Background

Methods

Results

Conclusions

Trial registration

At a glance: The ONWARDS insulin icodec trials

Springer Medicine

Abstract

Background

Methods

Results

Conclusions

Trial registration

Please log in to get access to this content

Other articles of this Issue 1/2018

Testing the organizational theory of innovation implementation effectiveness in a community pharmacy medication management program: a hurdle regression analysis

Correction to: Updated clinical guidelines experience major reporting limitations

Evaluating the impact of multilevel evidence-based implementation strategies to enhance provider recommendation on human papillomavirus vaccination rates among an empaneled primary care patient population: a study protocol for a stepped-wedge cluster randomized trial

Standardizing an approach to the evaluation of implementation science proposals

A systematic review of trials evaluating success factors of interventions with computerised clinical decision support

How do clinicians use implementation tools to apply breast cancer screening guidelines to practice?