Skip to main content
Top

Open Access 03-02-2025

Training and Technical Assistance Increase the Fidelity of Implementation of a Universal Prevention Initiative in Rural Schools: Results from a 3-Year Cluster-Randomized Trial

Authors: Hannah G. Calvert, Christopher M. Fleming, Michaela Lowe, Teri Lewis, Carl F. Siebert, Ashley Havlicak, Nate Anderson, Tate Castleton, Lindsey Turner

Published in: Prevention Science

Login to get access

Abstract

The need for well-implemented evidence-based interventions (EBIs) for the prevention of behavioral issues among children and adolescents is substantial. In rural areas, the need often matches or surpasses that of urban areas. Schools have a wide reach for prevention-focused EBIs. However, implementation in rural schools is often hindered by limited resources and capacity. Rural School Support Strategies (RS3) are a bundle of implementation supports that address implementation challenges in rural settings. They include providing additional leadership and coaching training, individualized technical assistance (mostly virtual), and monthly meetings of a virtual learning collaborative. A cluster-randomized Hybrid Type 3 implementation-effectiveness trial tested RS3 for implementing school-wide positive behavioral interventions and supports (PBIS), a universal prevention approach to improving student behavior, academic outcomes, and school climate. Forty rural schools received a multi-day training on PBIS each summer for 3 years. Half were randomized to also receive RS3 support. Linear and logistic regression models examined the effect of treatment condition and dosage of support on implementation fidelity for Tier 1 (universal) PBIS. Condition and dosage (number of hours) of support increased the odds of schools achieving the 70% threshold for adequate implementation fidelity. In the first year, the higher dosage of technical assistance events increased the likelihood of schools reaching fidelity, whereas later in the trial, the higher dosage of attendance at the virtual learning collaborative sessions yielded significant improvements in fidelity. Implications for accelerating the implementation of universal prevention initiatives in schools—particularly in rural settings—are discussed. This study was prospectively registered on ClinicalTrials.gov (NCT03736395), on November 9, 2018.
Appendix
Available only for authorised users
Literature
go back to reference Algozzine, B., Barrett, S., Eber, L., Horner, R., Lewis, T., Putnam, B., Swain-Bradway, J., McIntosh, K., & Sugai, G. (2019). School-wide PBIS Tiered Fidelity Inventory. OSEP Technical Assistance Center on Positive Behavioral Interventions and Supports. http://www.pbis.org. Algozzine, B., Barrett, S., Eber, L., Horner, R., Lewis, T., Putnam, B., Swain-Bradway, J., McIntosh, K., & Sugai, G. (2019). School-wide PBIS Tiered Fidelity Inventory. OSEP Technical Assistance Center on Positive Behavioral Interventions and Supports. http://​www.​pbis.​org.
go back to reference Baffsky, R., Ivers, R., Cullen, P., McGillivray, L., Werner-Seidler, A., Calear, A. L., Batterham, P. J., Toumbourou, J. W., Stokes, R., Kotselas, P., Prendergast, T., & Torok, M. (2023b). Co-design and development of implementation strategies: Enhancing the PAX Good Behaviour Game in Australian Schools. Journal of Prevention, 44(6), 679–704. https://doi.org/10.1007/s10935-023-00749-9CrossRefPubMed Baffsky, R., Ivers, R., Cullen, P., McGillivray, L., Werner-Seidler, A., Calear, A. L., Batterham, P. J., Toumbourou, J. W., Stokes, R., Kotselas, P., Prendergast, T., & Torok, M. (2023b). Co-design and development of implementation strategies: Enhancing the PAX Good Behaviour Game in Australian Schools. Journal of Prevention, 44(6), 679–704. https://​doi.​org/​10.​1007/​s10935-023-00749-9CrossRefPubMed
go back to reference Bradshaw, C. P. (2023). Leveraging implementation science to improve the scale-up of school mental health programming. In S.W. Evans, J. S. Owens, C. P. Bradshaw, & M. D. Weist (Eds.), Handbook of school mental health: Innovations in science and practice (pp. 409–417). Springer. https://doi.org/10.1007/978-3-031-20006-9 Bradshaw, C. P. (2023). Leveraging implementation science to improve the scale-up of school mental health programming. In S.W. Evans, J. S. Owens, C. P. Bradshaw, & M. D. Weist (Eds.), Handbook of school mental health: Innovations in science and practice (pp. 409–417). Springer. https://​doi.​org/​10.​1007/​978-3-031-20006-9
go back to reference Bradshaw, C. P., Koth, C. W., Bevans, K. B., Ialongo, N., & Leaf, P. J. (2008). The impact of school-wide positive behavioral interventions and supports (PBIS) on the organizational health of elementary schools. School Psychology Quarterly, 23(4), 462–473. https://doi.org/10.1037/a0012883CrossRef Bradshaw, C. P., Koth, C. W., Bevans, K. B., Ialongo, N., & Leaf, P. J. (2008). The impact of school-wide positive behavioral interventions and supports (PBIS) on the organizational health of elementary schools. School Psychology Quarterly, 23(4), 462–473. https://​doi.​org/​10.​1037/​a0012883CrossRef
go back to reference Bradshaw, C. P., Mitchell, M. M., & Leaf, P. J. (2010). Examining the effects of schoolwide positive behavioral interventions and supports on student outcomes: Results from a randomized controlled effectiveness trial in elementary schools. Journal of Positive Behavior Interventions, 12(3), 133–148. https://doi.org/10.1177/1098300709334798CrossRef Bradshaw, C. P., Mitchell, M. M., & Leaf, P. J. (2010). Examining the effects of schoolwide positive behavioral interventions and supports on student outcomes: Results from a randomized controlled effectiveness trial in elementary schools. Journal of Positive Behavior Interventions, 12(3), 133–148. https://​doi.​org/​10.​1177/​1098300709334798​CrossRef
go back to reference Fixsen, D. L., Naoom, S. F., Blase, K. A., Friedman, R. M., & Wallace, F. (2005). Implementation research: A synthesis of the literature. Tampa, FL: University of South Florida, Louis de la Parte Florida Mental Health Institute, The National Implementation Research Network. Fixsen, D. L., Naoom, S. F., Blase, K. A., Friedman, R. M., & Wallace, F. (2005). Implementation research: A synthesis of the literature. Tampa, FL: University of South Florida, Louis de la Parte Florida Mental Health Institute, The National Implementation Research Network.
go back to reference Grasley-Boy, N. M., Gage, N. A., Lombardo, M., Anderson, L., & Rila, A. (2022). School-wide positive behavior interventions and supports in rural and urban California schools: Effects on fidelity of implementation and suspension outcomes. Rural Special Education Quarterly, 41(2), 84–94. https://doi.org/10.1177/87568705221092766CrossRef Grasley-Boy, N. M., Gage, N. A., Lombardo, M., Anderson, L., & Rila, A. (2022). School-wide positive behavior interventions and supports in rural and urban California schools: Effects on fidelity of implementation and suspension outcomes. Rural Special Education Quarterly, 41(2), 84–94. https://​doi.​org/​10.​1177/​8756870522109276​6CrossRef
go back to reference Horner, R. H., Sugai, G., Smolkowski, K., Eber, L., Nakasato, J., Todd, A. W., & Esperanza, J. (2009). A randomized, wait-list controlled effectiveness trial assessing school-wide positive behavior support in elementary schools. Journal of Positive Behavior Interventions, 11(3), 133–144. https://doi.org/10.1177/1098300709332067CrossRef Horner, R. H., Sugai, G., Smolkowski, K., Eber, L., Nakasato, J., Todd, A. W., & Esperanza, J. (2009). A randomized, wait-list controlled effectiveness trial assessing school-wide positive behavior support in elementary schools. Journal of Positive Behavior Interventions, 11(3), 133–144. https://​doi.​org/​10.​1177/​1098300709332067​CrossRef
go back to reference Metz, A., Albers, B., Burke, K., Bartley, L., Louison, L., Ward, C., & Farley, A. (2021). Implementation practice in human service systems: Understanding the principles and competencies of professionals who support implementation. Human Service Organizations: Management, Leadership & Governance, 45(3), 238–259. https://doi.org/10.1080/23303131.2021.1895401CrossRef Metz, A., Albers, B., Burke, K., Bartley, L., Louison, L., Ward, C., & Farley, A. (2021). Implementation practice in human service systems: Understanding the principles and competencies of professionals who support implementation. Human Service Organizations: Management, Leadership & Governance, 45(3), 238–259. https://​doi.​org/​10.​1080/​23303131.​2021.​1895401CrossRef
go back to reference Olson, J. R., Coldiron, J. S., Parigoris, R. M., Zabel, M. D., Matarese, M., & Bruns, E. J. (2020). Developing an evidence-based technical assistance model: A process evaluation of the National Training and Technical Assistance Center for child, youth, and family mental health. The Journal of Behavioral Health Services & Research, 47(3), 312–330. https://doi.org/10.1007/s11414-020-09686-5CrossRef Olson, J. R., Coldiron, J. S., Parigoris, R. M., Zabel, M. D., Matarese, M., & Bruns, E. J. (2020). Developing an evidence-based technical assistance model: A process evaluation of the National Training and Technical Assistance Center for child, youth, and family mental health. The Journal of Behavioral Health Services & Research, 47(3), 312–330. https://​doi.​org/​10.​1007/​s11414-020-09686-5CrossRef
go back to reference Orenstein, S., Connors, E., Fields, P., Cushing, K., Yarnell, J., Bohnenkamp, J., Hoover, S., & Lever, N. (2023). Advancing school mental health quality through national learning communities. In S. W. Evans, J. S. Owens, C. P. Bradshaw, & M. D. Weist (Eds.), Handbook of School Mental Health: Innovations in Science and Practice (pp. 215–231). Springer. https://doi.org/10.1007/978-3-031-20006-9_15 Orenstein, S., Connors, E., Fields, P., Cushing, K., Yarnell, J., Bohnenkamp, J., Hoover, S., & Lever, N. (2023). Advancing school mental health quality through national learning communities. In S. W. Evans, J. S. Owens, C. P. Bradshaw, & M. D. Weist (Eds.), Handbook of School Mental Health: Innovations in Science and Practice (pp. 215–231). Springer. https://​doi.​org/​10.​1007/​978-3-031-20006-9_​15
go back to reference Orkin, A. M., Gill, P. J., Ghersi, D., Campbell, L., Sugarman, J., Emsley, R., Steg, P. G., Weijer, C., Simes, J., Rombey, T., Williams, H. C., Wittes, J., Moher, D., Richards, D. P., Kasamon, Y., Getz, K., Hopewell, S., Dickersin, K., Wu, T., … CONSERVE Group. (2021). Guidelines for reporting trial protocols and completed trials modified due to the COVID-19 pandemic and other extenuating circumstances: The CONSERVE 2021 Statement. JAMA, 326(3), 257–265. https://doi.org/10.1001/jama.2021.9941 Orkin, A. M., Gill, P. J., Ghersi, D., Campbell, L., Sugarman, J., Emsley, R., Steg, P. G., Weijer, C., Simes, J., Rombey, T., Williams, H. C., Wittes, J., Moher, D., Richards, D. P., Kasamon, Y., Getz, K., Hopewell, S., Dickersin, K., Wu, T., … CONSERVE Group. (2021). Guidelines for reporting trial protocols and completed trials modified due to the COVID-19 pandemic and other extenuating circumstances: The CONSERVE 2021 Statement. JAMA, 326(3), 257–265. https://​doi.​org/​10.​1001/​jama.​2021.​9941
go back to reference Pas, E. T., Kaiser, L., & Owens, J. S. (2023). Innovative approaches to coaching teachers in implementing Tier 1 and Tier 2 classroom interventions. In S. W. Evans, J. S. Owens, C. P. Bradshaw, & M. D. Weist (Eds.), Handbook of School Mental Health: Innovations in Science and Practice (pp. 343–355). Springer.CrossRef Pas, E. T., Kaiser, L., & Owens, J. S. (2023). Innovative approaches to coaching teachers in implementing Tier 1 and Tier 2 classroom interventions. In S. W. Evans, J. S. Owens, C. P. Bradshaw, & M. D. Weist (Eds.), Handbook of School Mental Health: Innovations in Science and Practice (pp. 343–355). Springer.CrossRef
go back to reference Sugai, G., Lewis-Palmer, T., Todd, A., & Horner, R. H. (2001). School-wide evaluation tool. University of Oregon. Sugai, G., Lewis-Palmer, T., Todd, A., & Horner, R. H. (2001). School-wide evaluation tool. University of Oregon.
go back to reference Turner, L., Calvert, H. G., Fleming, C. M., Lewis, T., Siebert, C., Anderson, N., Castleton, T., Havlicak, A., & McQuilkin, M. (2022). Study protocol for a cluster-randomized trial of a bundle of implementation support strategies to improve the fidelity of implementation of schoolwide positive behavioral interventions and supports in rural schools. Contemporary Clinical Trials Communications, 28, 100949. https://doi.org/10.1016/j.conctc.2022.100949CrossRefPubMedPubMedCentral Turner, L., Calvert, H. G., Fleming, C. M., Lewis, T., Siebert, C., Anderson, N., Castleton, T., Havlicak, A., & McQuilkin, M. (2022). Study protocol for a cluster-randomized trial of a bundle of implementation support strategies to improve the fidelity of implementation of schoolwide positive behavioral interventions and supports in rural schools. Contemporary Clinical Trials Communications, 28, 100949. https://​doi.​org/​10.​1016/​j.​conctc.​2022.​100949CrossRefPubMedPubMedCentral
go back to reference Wandersman, A., Chien, V. H., & Katz, J. (2012). Toward an evidence-based system for innovation support for implementing innovations with quality: Tools, training, technical assistance, and quality assurance/quality improvement. American Journal of Community Psychology, 50(3–4), 445–459. https://doi.org/10.1007/s10464-012-9509-7CrossRefPubMed Wandersman, A., Chien, V. H., & Katz, J. (2012). Toward an evidence-based system for innovation support for implementing innovations with quality: Tools, training, technical assistance, and quality assurance/quality improvement. American Journal of Community Psychology, 50(3–4), 445–459. https://​doi.​org/​10.​1007/​s10464-012-9509-7CrossRefPubMed
go back to reference Wandersman, A., Duffy, J., Flaspohler, P., Noonan, R., Lubell, K., Stillman, L., Blachman, M., Dunville, R., & Saul, J. (2008). Bridging the gap between prevention research and practice: The interactive systems framework for dissemination and implementation. American Journal of Community Psychology, 41(3), 171–181. https://doi.org/10.1007/s10464-008-9174-zCrossRefPubMed Wandersman, A., Duffy, J., Flaspohler, P., Noonan, R., Lubell, K., Stillman, L., Blachman, M., Dunville, R., & Saul, J. (2008). Bridging the gap between prevention research and practice: The interactive systems framework for dissemination and implementation. American Journal of Community Psychology, 41(3), 171–181. https://​doi.​org/​10.​1007/​s10464-008-9174-zCrossRefPubMed
Metadata
Title
Training and Technical Assistance Increase the Fidelity of Implementation of a Universal Prevention Initiative in Rural Schools: Results from a 3-Year Cluster-Randomized Trial
Authors
Hannah G. Calvert
Christopher M. Fleming
Michaela Lowe
Teri Lewis
Carl F. Siebert
Ashley Havlicak
Nate Anderson
Tate Castleton
Lindsey Turner
Publication date
03-02-2025
Publisher
Springer US
Published in
Prevention Science
Print ISSN: 1389-4986
Electronic ISSN: 1573-6695
DOI
https://doi.org/10.1007/s11121-025-01776-0