Top

Health Research Policy and Systems

Published in:

Open Access 01-12-2017 | Research

A mixed-methods study of system-level sustainability of evidence-based practices in 12 large-scale implementation initiatives

Authors: Ashley T. Scudder, Sarah M. Taber-Thomas, Kristen Schaffner, Joy R. Pemberton, Leah Hunter, Amy D. Herschell

Published in: Health Research Policy and Systems | Issue 1/2017

Abstract

Background

In recent decades, evidence-based practices (EBPs) have been broadly promoted in community behavioural health systems in the United States of America, yet reported EBP penetration rates remain low. Determining how to systematically sustain EBPs in complex, multi-level service systems has important implications for public health. This study examined factors impacting the sustainability of parent-child interaction therapy (PCIT) in large-scale initiatives in order to identify potential predictors of sustainment.

Methods

A mixed-methods approach to data collection was used. Qualitative interviews and quantitative surveys examining sustainability processes and outcomes were completed by participants from 12 large-scale initiatives.

Results

Sustainment strategies fell into nine categories, including infrastructure, training, marketing, integration and building partnerships. Strategies involving integration of PCIT into existing practices and quality monitoring predicted sustainment, while financing also emerged as a key factor.

Conclusions

The reported factors and strategies impacting sustainability varied across initiatives; however, integration into existing practices, monitoring quality and financing appear central to high levels of sustainability of PCIT in community-based systems. More detailed examination of the progression of specific activities related to these strategies may aide in identifying priorities to include in strategic planning of future large-scale initiatives.

Trial registration

ClinicalTrials.gov ID NCT02543359; Protocol number PRO12060529.

Available only for authorised users

McHugh RK, Barlow DH. The dissemination and implementation of evidence-based psychological treatments: a review of current efforts. Am Psychol. 2010;65(2):73–84. doi:10.1037/a0018121.CrossRefPubMed

Brownson RC, Colditz GA, Proctor EK. Dissemination and implementation research in heath: translating science to practice. New York: Oxford University Press; 2012.CrossRef

Tibbits MK, Bumbarger BK, Kyler SJ, Perkins DF. Sustaining evidence based interventions under real-world conditions: Results from a large-scale diffusion project. Prev Sci. 2010;11(3):252–62. doi:10.1007/s11121-010-0170-9.CrossRefPubMedPubMedCentral

Hanson RF, Gros KS, Davidson TM, Barr S, Cogen J, Deblinger E, et al. National trainers’ perspectives on challenges to implementation of an empirically-supported mental health treatment. Adm Policy Ment Health. 2013;41(4):522–34. doi:10.1007/s10488-013-0492-6.CrossRef

Sigel BA, Benton AH, Lynch CE, Kramer TL. Characteristics of 17 statewide initiatives to disseminate Trauma-Focused Cognitive-Behavioral Therapy (TF-CBT). Psychol Trauma. 2013;5(5):323–33. doi:10.1037/a0029095.CrossRef

Beveridge RM, Fowles TR, Masse JJ, McGoron L, Smith MA, Parrish BP, et al. State-wide dissemination and implementation of Parent-Child Interaction Therapy (PCIT): application of theory. Child Youth Serv Rev. 2015;48:38–48. doi:10.1016/j.childyouth.2014.11.013.CrossRef

Shediac-Rizkallah MC, Bone LR. Planning for the sustainability of community-based health programs: conceptual frameworks and future directions for research, practice and policy. Health Educ Res. 1998;13(1):87–108. doi:10.1093/her/13.1.87.CrossRefPubMed

Scheirer MA, Dearing JW. An agenda for research on the sustainability of public health programs. Am J Public Health. 2011;101(11):2059–67.CrossRefPubMedPubMedCentral

Pluye P, Potvin L, Denis J. Making public health programs last: conceptualizing sustainability. Eval Program Plann. 2004;27(2):121–33. doi:10.1016/j.evalprogplan.2004.01.001.CrossRef

10.

Stirman SW, Kimberly J, Cook N, Calloway A, Castro F, Charns M. The sustainability of new programs and innovations: a review of the empirical literature and recommendations for future research. Implement Sci. 2012;7:17. doi:10.1186/1748-5908-7-17.CrossRef

11.

Chambers DA, Glasglow RE, Strange KC. The dynamic sustainability framework: addressing the paradox of sustainment amid ongoing change. Implement Sci. 2013;8:117. doi:10.1186/1748-5908-8-117.CrossRefPubMedPubMedCentral

12.

Scheirer MA. Linking sustainability research to intervention types. Am J Public Health. 2013;103(4):e73–80. doi:10.2105/AJPH.2012.300976.CrossRefPubMedPubMedCentral

13.

Johnson K, Hays C, Center H, Daley C. Building capacity and sustainable prevention innovations: a sustainability planning model. Eval Program Plann. 2004;27(2):135–49. doi:10.1016/j.evalprogplan.2004.01.002.CrossRef

14.

Massatti RR, Sweeny HA, Panzano PC, Roth D. The de-adoption of Innovative Mental Health Practices (IMHP): Why organizations choose not to sustain an IMHP. Admin Policy Mental Health Mental Health Serv Res. 2008;35(1-2):50–65. doi:10.1007/s10488-007-0141-z.CrossRef

15.

Luke DA, Calhoun A, Robichaux C, Elliot MB, Moreland-Russel S. The program sustainability assessment tool: a new instrument for public health programs. Prev Chronic Dis. 2014;11:130184.CrossRefPubMed

16.

Aarons GA, Hurlburt SM. Advancing a conceptual model of evidence-based practice implementation in public service sectors. Admin Policy Ment Health. 2011;38(1):4–23. doi:10.1007/s10488-010-0327-7.CrossRef

17.

Bond GR, Drake RE, McHugo GJ, Peterson AE, Jones AM, Williams J. Long-term sustainability of evidence-based practices in community mental health agencies. Admin Policy Mental Health Mental Health Serve Res. 2014;41(2):228–36. doi:10.1007/s10488-012-0461-5.CrossRef

18.

Eyberg SM, Funderburk BW, Hembree-Kigin T, McNeil CB, Querido JG, Hood KK. Parent-child interaction therapy with behavior problem children: one and two year maintenance of treatment effects in the family. Child Fam Behav Ther. 2001;23(4):1–20. doi:10.1207/s15374424jccp2002_5.CrossRef

19.

Chadwick Center on Children and Families. Closing the quality chasm in child abuse treatment: identifying and disseminating best practices. San Diego: Children’s Hospital-San Diego; 2004.

20.

Scudder AT, Herschell AD. Building an evidence-base for the training of evidence-based treatments in community settings: use of an expert-informed approach. Child Youth Serv Rev. 2015;55:84–92. doi:10.1016/j.childyouth.2015.05.003.CrossRefPubMedPubMedCentral

21.

Scudder AT, Herschell AT, McNeil CB. Parent-child interaction therapy for children with disruptive behavior disorders. In: Reddy LA, Files-Hall TM, Schaefer CE, editors. Empirically based play interventions for children. 2nd ed. Washington, DC: American Psychological Association; 2016. p. 159–80.CrossRef

22.

Palinkas LA, Horwitz SM, Chamberlain P, Hurlburt M, Landsverk J. Mixed method designs in mental health services research: a review. Psychiatr Serv. 2011;62(3):255–63. doi:10.1176/ps.62.3.pss6203_0255.CrossRefPubMed

23.

Strauss A, Corbin J. Basics of Qualitative Research: Techniques and Procedures for Developing Grounded Theory. 2nd ed. Thousand Oaks: Sage; 1998.

24.

Creswell JW. Research Design: Qualitative, Quantitative, and Mixed Methods Approaches. Thousand Oaks: Sage Publications; 2013.

25.

Sigel BA, Kramer TL, Conners-Burrow NA, Church JK, Worley KB, Mitrani NA. Statewide dissemination of trauma-focused cognitive-behavioral therapy (TF-CBT). Child Youth Serv Rev. 2013;35(6):1023–29. doi:10.1016/j.childyouth.2013.03.012.CrossRef

26.

MacQueen KM, McLelland E, Kay K, Milstein B. Codebook development for team-based qualitative analysis. Cult Anthropol Methods. 1998;10(2):31–6.

27.

Muhr T. ATLAS.ti (Version 7). Berlin: Scientific Software Development; 2006.

28.

Bakeman R, Gottman JM. Observing interaction: an introduction to sequential analysis. 2nd ed. New York: Cambridge University Press; 1997.CrossRef

29.

Scheirer MA, Hartling G, Hagerman D. Defining sustainability outcomes of health programs: Illustrations from an on-line survey. Eval Program Plann. 2008;31(4):335–46. doi:10.1016/j.evalprogplan.2008.08.004.CrossRefPubMed

30.

Swain K, Whitley R, McHugo GJ, Drake RE. The sustainability of evidence based practices in routine mental health agencies. Community Ment Health J. 2010;46(2):119–29. doi:10.1007/s10597-009-9202-y.CrossRefPubMed

31.

Hodas G, Herschell AD, Mrozowski S. The Use of Time-out in Parent-Child Interaction Therapy in Pennsylvania. Policy Statement Clarification from Pennsylvania Office of Mental Health and Substance Abuse Services. 2013.

32.

Troutman B. The effects of foster care placement on young children’s mental health: risks and opportunities. 2011. https://www.healthcare.uiowa.edu/icmh/child/documents/Effectsoffostercareplacementonyoungchildren.pdf. Accessed 1 Jan 2016.

33.

Savaya R, Spiro S, Elran-Barak R. Sustainability of social programs: a comparative case study analysis. Am J Eval. 2008;29(4):478–93. doi:10.1177/1098214008325126.CrossRef

34.

Steckler A, Goodman RM. How to institutionalize health promotion programs. Am J Health Promotion. 1989;3(4):34–43.CrossRefPubMed

35.

Yin RK. Life histories of innovations: how new practices become routinized. Public Adm Rev. 1981;41(1):21–8. doi:10.2307/975720.CrossRef

36.

Leonard-Barton D. Implementation as mutual adaptation of technology and organization. Res Policy. 1988;17:251–67. doi:10.1016/0048-7333(88)90006-6.CrossRef

37.

Finnerity MT, Rapp CA, Bond GR, Lynde DW, Ganju V, Goldman HH. The State Health Authority Yardstick (SHAY). Community Mental Health J. 2009;45(3):228–36. doi:10.1007/s10597-009-9181-z.CrossRef

38.

Washington State Institute for Public Policy. Child Welfare Benefit-Cost Results. 2015. http://www.wsipp.wa.gov/BenefitCost. Accessed 1 Jan 2016.

39.

Bruns EJ, Kerns SE, Pullmann MD, Hensley SW, Lutterman T, Hoagwood KE. Research, data, and evidence-based treatment use in state behavioral health systems, 2001-2012. Psychiatric Serv. 2016;67(5):496–503. doi:10.1176/appi.ps.201500014.CrossRef

40.

Cohen D. Effect of the exclusion of behavioral health from health information technology (HIT) legislation on the future of integrated health care. J Behav Health Serv Res. 2015;42(4):534–9. doi:10.1007/s11414-014-9407-x.CrossRefPubMed

41.

Travis JK, Brestan-Knight E. A pilot study examining trainee treatment session fidelity when parent-child interaction therapy (PCIT) is implemented in community settings. J Behav Health Serv Res. 2013;40(3):342–54. doi:10.1007/s11414-013-9326-2.CrossRefPubMed

42.

Herschell AD, Scudder AT, Schaffner KF, Slagel LA. Feasibility and effectiveness of parent-child interaction therapy with victims of domestic violence: a pilot study. J Child Family Stud. 2016;26(1):271–83. doi:10.1007/s10826-016-0546-y.CrossRef

43.

Schoenwald SK, Garland AF, Chapman JE, Frazier SL, Sheidow AJ, Southam-Gerow MA. Toward the effective and efficient measurement of implementation fidelity. Admin Policy Mental Health. 2011;38(1):32–43. doi:10.1007/s10488-010-0321-0.CrossRef

44.

Kimberly JR, Evanisko MJ. Organizational innovation: the influence of individual, organizational, and contextual factors on hospital adoption of technological and administrative innovations. Acad Manage J. 1981;24(4):689–713. doi:10.2307/256170.CrossRefPubMed

45.

Scheirer MA. Is sustainability possible? A review and commentary on empirical studies of program sustainability. Am J Eval. 2005;26(3):320–47. doi:10.1177/1098214005278752.CrossRef

46.

Markiewicz J, Ebert L, Ling D, Amaya-Jackson L, Kisiel C. Learning Collaborative Toolkit. Durham: National Center for Child Traumatic Stress; 2006. http://www.nctsn.org/resources/learning-collaborative-toolkit%20. Accessed 1 Jan 2016.

Title: A mixed-methods study of system-level sustainability of evidence-based practices in 12 large-scale implementation initiatives
Authors: Ashley T. Scudder
Sarah M. Taber-Thomas
Kristen Schaffner
Joy R. Pemberton
Leah Hunter
Amy D. Herschell
Publication date: 01-12-2017
Publisher: BioMed Central
Published in: Health Research Policy and Systems / Issue 1/2017
Electronic ISSN: 1478-4505
DOI: https://doi.org/10.1186/s12961-017-0230-8

Keynote webinar | Spotlight on medication adherence

Springer Medicine

A mixed-methods study of system-level sustainability of evidence-based practices in 12 large-scale implementation initiatives

Abstract

Background

Methods

Results

Conclusions

Trial registration

Keynote webinar | Spotlight on medication adherence

Springer Medicine

Abstract

Background

Methods

Results

Conclusions

Trial registration

Please log in to get access to this content

Other articles of this Issue 1/2017

Knowledge mobilisation for policy development: implementing systems approaches through participatory dynamic simulation modelling

Incorporating sex, gender and vulnerable populations in a large multisite health research programme: The Ontario Pharmacy Evidence Network as a case study

Setting strategy for system change: using concept mapping to prioritise national action for chronic disease prevention

Integrating consumer engagement in health and medical research – an Australian framework

Alliance members’ roles in collective field-building: an assessment of leadership and championship within the Population Health Intervention Research Initiative for Canada

Examples of sex/gender sensitivity in epidemiological research: results of an evaluation of original articles published in JECH 2006–2014