Top

Administration and Policy in Mental Health and Mental Health Services Research

Published in:

01-05-2018 | Original Article

Exploring Provider Use of a Digital Implementation Support System for School Mental Health: A Pilot Study

Authors: Melanie Livet, Mary Yannayon, Kelly Sheppard, Kelly Kocher, James Upright, Janey McMillen

Published in: Administration and Policy in Mental Health and Mental Health Services Research | Issue 3/2018

Abstract

This pilot study explored provider use of an online system, Centervention, to support the delivery of empirically supported school-based mental health interventions (ESIs); and associations between components of this system [resources, training, technical assistance (TA), feedback loops], implementation indicators, and student outcomes. Multilevel modeling data were collected from 39 providers implementing ESIs with 758 students. Training, TA, and progress monitoring predicted ESI adherence, and perceived value of resources and TA influenced student responsiveness. Greater adherence was predictive of better socio-emotional outcomes. Interviews with 15 providers illuminated how they used these four Centervention support strategies. Implications for digital implementation support research are discussed.

Available only for authorised users

Initially, the project team was planning on capturing provider use of this variable. Unfortunately, usage data on resources were unavailable due to unanticipated issues with the data usage tracker for the Resource Center pages. The Resource Center pages were not all coded to drive a change in the URL when clicked on by the user; in other words, the number of clicks on Resource Center pages could not be accurately captured. Provider perceptions of the value of ESI resources were included instead, based on previous research supporting that quality of resources are important for ESI implementation.

The kappa value was based on the two coders coding the presence or absence of 67 codes in each of the transcripts included in the 20% check (i.e., 3 interviews). The unit of analysis was the provider/transcript; consequently, the kappa value was computed based on 201 as the total number of opportunities to agree on presence or absence of a particular code. Kappa values ranged from 0.81 to 1 per transcript/provider.

The inclusion of the implementation support strategies reduced the conditional variance at the group level from 0.024 to 0.022, while variance at the provider level was reduced from 0.043 to 0.012.

The inclusion of the implementation support strategies reduced the conditional variance at both the group level (from 0.13 to 0.083) and the provider level (from 0.14 to 0.089).

As could be expected, the addition of adherence, engagement, and attendance to the model did not alter the conditional variance at the student level (0.35 to 0.35) (attendance, the only student-level variable, had minimal variability). However, the variance at the group level decreased from 0.51 to 0.48.

3C Institute. Centervention. Retrieved October 31, 2014, from https://centervention.org.

Anderson, N. B. (2006). Evidence-based practice in psychology. American Psychologist, 61(4), 271–285.CrossRef

Baron, M., & Graham, C. (2007). Identifying key factors in the adoption of innovative practices. Technology and Teacher Education Annual, 18(2), 772.

Berkel, C., Mauricio, A. M., Schoenfelder, E., & Sandler, I. N. (2011). Putting the pieces together: An integrated model of program implementation. Prevention Science, 12, 23–33. doi:10.1007/s11121-010-0186-1.CrossRefPubMedPubMedCentral

Bickman, L., Douglas, S. R., De Andrade, A. R. V., Tomlinson, M., Gleacher, A., Olin, S., & Hoagwood, K. (2016). Implementing a measurement feedback system: A tale of two sites. Administration and Policy in Mental Health and Mental Health Services Research, 43(3), 410–425. doi:10.1007/s10488-015-0647-8.CrossRefPubMedPubMedCentral

Bickman, L., Kelley, S., Breda, C., De Andrade, A., & Riemer, M. (2011). Effects of routine feedback to clinicians on youth mental health outcomes: A randomized trial. Psychiatric Services, 62(12), 1423–1429. doi:10.1176/appi.ps.002052011.CrossRefPubMed

Bickman, L., Lyon, A. R., & Wolpert, M. (2016). Achieving precision mental health through effective assessment, monitoring, and feedback processes: Introduction to the special issue. Administration and Policy in Mental Health and Mental Health Services Research, 43(3), 271–276. doi:10.1007/s10488-016-0718-5.CrossRefPubMedPubMedCentral

Brightwood, L. H., & DeRosier, M. E. (2007). Lifestories for kids: Enhancing character development and social skills through storytelling (Grades 3–5). Cary, NC: 3C Institute for Social Development.

Brightwood, L. H., DeRosier, M. E., & Maschauer, E. L. (2012). Adventures in emotional literacy (AEL): Building social relationships through enhanced communication, cooperation, and confidence (Grade 3). Cary, NC: 3C Institute for Social Development.

Brightwood, L. H., DeRosier, M. E., Maschauer, E. L., & Wilson, M. E. (2009). Adventures in emotional literacy (AEL): Building social relationships through enhanced communication, cooperation, and confidence (Grades K-2). Cary, NC: 3C Institute for Social Development.

Bumbarger, B., & Perkins, D. F. (2008). After randomized trials: Issues related to dissemination of evidence-based interventions. Journal of Children’s Services, 3, 53–61. doi:10.1108/17466660200800012.

Craig, A. B., Leary, K. A., Parker, J., McMillen, J. S., & DeRosier, M. E. (2015). Effectiveness of an emotional literacy intervention for the elementary classroom. (Unpublished manuscript).

Crosse, S., Williams, B., Hagen, C. A., Harmon, M., Ristow, L., DiGaetano, R., … & Derzon, J. H. (2011). Prevalence and implementation fidelity of research-based prevention programs in public schools (Final Report). Washington, D.C.: U.S. Department of Education, Office of Planning, Evaluation and Policy Development, Policy and Program Studies Service. Retrieved from http://files.eric.ed.gov/fulltext/ED529062.pdf.

DeRosier, M. E. (2004). Building relationships and combating bullying: Effectiveness of a school-based social skills group intervention. Journal of Clinical Child and Adolescent Psychology, 33, 125–130. doi:10.1207/S15374424JCCP3301_18.CrossRef

DeRosier, M. E. (2007). Social Skills Group Intervention (S.S.GRIN): Group interventions and exercises for enhancing children’s communication, cooperation, and confidence (Grades K-5). Cary, NC: 3C Institute for Social Development.

DeRosier, M. E., & Brightwood, L. H. (2007). Lifestories for kids: Enhancing character development and social skills through storytelling (Grades K-2). Cary, NC: 3C Institute for Social Development.

DeRosier, M. E., & Marcus, S. R. (2005). Building friendships and combating bullying: Effectiveness of S.S.GRIN at one-year follow-up. Journal of Clinical Child and Adolescent Psychology, 34, 140–150. doi:10.1207/s15374424jccp3401_13.CrossRefPubMed

DeRosier, M. E., & Mercer, S. (2007). Improving student social behavior: The effectiveness of a school-based character education program. Journal of Research in Character Education, 5, 131–148.

Domitrovich, C. E., Bradshaw, C. P., Poduska, J. M., Hoagwood, K., Buckley, J. A., Olin, S., … & Ialongo, N. S. (2008). Maximizing the implementation quality of evidence-based preventive interventions in schools: A conceptual framework. Advances in School Mental Health Promotion, 1, 6–28. doi:10.1080/1754730X.2008.9715730.CrossRefPubMedPubMedCentral

Durlak, J. A., & DuPre, E. P. (2008). Implementation matters: A review of the influence of implementation on program outcomes and the factors affecting implementation. American Journal of Community Psychology, 41, 327–350. doi:10.1007/s10464-008-9165-0.CrossRefPubMed

Farmer, E. M., Burns, B. J., Phillips, S. D., Angold, A., & Costello, E. J. (2003). Pathways into and through mental health services for children and adolescents. Psychiatric Services, 54(1), 60–66. doi:10.1176/appi.ps.54.1.60.CrossRefPubMed

Fixsen, D. L., Naoom, S. F., Blase, K. A., Friedman, R. M., & Wallace, F. (2005). Implementation research: A synthesis of the literature. Tampa, FL: University of South Florida, Louis de la Parte Florida Mental Health Institute, The National Implementation Research Network.

Forman, S. G., Olin, S. S., Hoagwood, K. E., Crowe, M., & Saka, N. (2009). Evidence-based interventions in schools: Developers’ views of implementation barriers and facilitators. School Mental Health, 1, 26–36. doi:10.1007/s12310-008-9002-5.CrossRef

Forman, S. G., Shapiro, E. S., Codding, R. S., Gonzales, J. E., Reddy, L. A., Rosenfield, S. A., … & Stoiber, K. C. (2013). Implementation science and school psychology. School Psychology Quarterly, 28, 77–100. doi:10.1037/spq0000019.CrossRefPubMed

Frambach, R. T., & Schillewaert, N. (2002). Organizational innovation adoption: A multi-level framework of determinants and opportunities for future research. Journal of Business Research, 55(2), 163–176. doi:10.1016/S0148-2963(00)00152-1.CrossRef

Gill, P., Stewart, K., Treasure, E., & Chadwick, B. (2008). Methods of data collection in qualitative research: Interviews and focus groups. British Dental Journal, 204, 291–295. doi:10.1038/bdj.2008.192.CrossRefPubMed

Gingiss, P. M., Roberts-Gray, C., & Boerm, M. (2006). Bridge-It: A system for predicting implementation fidelity for school-based tobacco prevention programs. Prevention Science, 7, 197–207. doi:10.1007/s11121-006-0038-1.CrossRefPubMed

Gresham, F. M., Cook, C. R., Collins, T., Dart, E., Rasetshwane, K., Truelson, E., & Grant, S. (2010). Developing a change-sensitive brief behavior rating scale as a progress monitoring tool for social behavior: An example using the Social Skills Rating System—Teacher Form. School Psychology Review, 39, 364–379.

Han, S. S., & Weiss, B. (2005). Sustainability of teacher implementation of school-based mental health programs. Journal of Abnormal Child Psychology, 33, 665–679. doi:10.1007/s10802-005-7646-2.CrossRefPubMed

Henderson, J. L., MacKay, S., & Peterson-Badali, M. (2006). Closing the research-practice gap: Factors affecting adoption and implementation of a children’s mental health program. Journal of Clinical Child and Adolescent Psychology, 35, 2–12. doi:10.1207/s15374424jccp3501_1.CrossRefPubMed

Herschell, A. D., Kolko, D. J., Baumann, B. L., & Davis, A. C. (2010). The role of therapist training in the implementation of psychosocial treatments: A review and critique with recommendations. Clinical Psychology Review, 30, 448–466. doi:10.1016/j.cpr.2010.02.005.CrossRefPubMedPubMedCentral

Hoagwood, K., Atkins, M., & Ialongo, N. (2013). Unpacking the black box of implementation: The next generation for policy, research, and practice. Administration and Policy in Mental Health, 40, 451–455. doi:10.1007/s10488-013-0512-6.CrossRefPubMedPubMedCentral

Hsieh, H. F., & Shannon, S. E. (2005). Three approaches to qualitative content analysis. Qualitative Health Research, 15, 1277–1288. doi:10.1177/1049732305276687.CrossRefPubMed

Hubley, S., Woodcock, E. A., Dimeff, L. A., & Dimidjian, S. (2015). Disseminating behavioural activation for depression via online training: Preliminary steps. Behavioural and Cognitive Psychotherapy, 43(02), 224–238. doi:10.1017/S1352465813000842.CrossRefPubMed

Jensen, P. S., & Foster, M. (2010). Closing the research to practice gap in children’s mental health: Structures, solutions, and strategies. Administration and Policy in Mental Health and Mental Health Services Research, 37, 111–119. doi:10.1007/s10488-010-0286-z.CrossRefPubMed

Jensen-Doss, A., & Hawley, K. M. (2010). Understanding barriers to evidence-based assessment: Clinician attitudes toward standardized assessment tools. Journal of Clinical Child and Adolescent Psychology, 39(6), 885–896. doi:10.1080/15374416.2010.517169.CrossRefPubMedPubMedCentral

Kreft, I. G., De Leeuw, J., & Aiken, L. S. (1995). The effect of different forms of centering in hierarchical linear models. Multivariate Behavioral Research, 30, 1–21.CrossRefPubMed

Langberg, J. M., & Smith, B. H. (2006). Developing evidence-based interventions for deployment into school settings: A case example highlighting key issues of efficacy and effectiveness. Evaluation and Program Planning, 29, 323–334. doi:10.1016/j.evalprogplan.2006.02.002.CrossRefPubMed

Langley, A. K., Nadeem, E., Kataoka, S. H., Stein, B. D., & Jaycox, L. H. (2010). Evidence-based mental health programs in schools: Barriers and facilitators of successful implementation. School Mental Health, 2, 105–113. doi:10.1007/s12310-010-9038-1.CrossRefPubMedPubMedCentral

Leeman, J., Calancie, L., Hartman, M. A., Escoffery, C. T., Herrmann, A. K., Tague, L. E., … & Samuel-Hodge, C. (2015). What strategies are used to build practitioners’ capacity to implement community-based interventions and are they effective? A systematic review. Implementation Science, 10, 80. doi:10.1186/s13012-015-0272-7.CrossRefPubMedPubMedCentral

Levin, B. L., Hennessy, K. D., & Petrila, J. (2010). Mental health services: A public health perspective (3rd edn.). New York: Oxford University Press.CrossRef

Lochman, J. E., Boxmeyer, C., Powell, N., Qu, L., Wells, K., & Windle, M. (2009). Dissemination of the Coping Power program: Importance of intensity of counselor training. Journal of Consulting and Clinical Psychology, 77, 397. doi:10.1037/a0014514.CrossRefPubMed

Lyon, A. R., & Lewis, C. C. (2016). Designing health information technologies for uptake: Development and implementation of measurement feedback systems in mental health service delivery. Administration and Policy in Mental Health and Mental Health Services Research, 43(3), 344–349. doi:10.1007/s10488-015-0704-3.CrossRefPubMedPubMedCentral

Lyon, A. R., Lewis, C. C., Boyd, M. R., Hendrix, E., & Liu, F. (2016a). Capabilities and characteristics of digital measurement feedback systems: Results from a comprehensive review. Administration and Policy in Mental Health and Mental Health Services Research, 43(3), 441–466. doi:10.1007/s10488-016-0719-4.CrossRefPubMedPubMedCentral

Lyon, A. R., Ludwig, K., Wasse, J. K., Bergstrom, A., Hendrix, E., & McCauley, E. (2016b). Determinants and functions of standardized assessment use among school mental health clinicians: A mixed methods evaluation. Administration and Policy in Mental Health and Mental Health Services Research, 43(1), 122–134. doi:10.1007/s10488-015-0626-0.CrossRefPubMedPubMedCentral

Lyon, A. R., Pullmann, M. D., Whitaker, K., Ludwig, K., Wasse, J. K., & McCauley, E. (2017). A digital feedback system to support implementation of measurement-based care by school-based mental health clinicians. Journal of Clinical Child and Adolescent Psychology, 00, 1–12. doi:10.1080/15374416.2017.1280808.CrossRef

Lyon, A. R., Wasse, J. K., Ludwig, K., Zachry, M., Bruns, E. J., Unützer, J., & McCauley, E. (2016c). The contextualized technology adaptation process (CTAP): Optimizing health information technology to improve mental health systems. Administration and Policy in Mental Health and Mental Health Services Research, 43(3), 394–409. doi:10.1007/s10488-015-0637-x.CrossRefPubMedPubMedCentral

McHugh, K. R., & Barlow, D. H. (2010). The dissemination and implementation of evidence-based psychological treatments: A review of current efforts. American Psychologist, 65, 73–84. doi:10.1037/a0018121.CrossRefPubMed

McIntyre, L. L., Gresham, F. M., Digennaro, F. D., & Reed, D. D. (2007). Treatment integrity of school-based interventions with children in Journal of Applied Behavioral Analysis studies from 1991 to 2005. Journal of Applied Behavior Analysis, 40, 659–672. doi:10.1901/jaba.2007.659-672.CrossRefPubMed

Meyers, D. C., Durlak, J. A., & Wandersman, A. (2012). The Quality Implementation Framework: A synthesis of critical steps in the implementation process. American Journal of Community Psychology, 50(3–4), 462–480. doi:10.1007/s10464-012-9522-x.CrossRefPubMed

Mihalic, S. F., Fagan, A. A., & Argamaso, S. (2008). Implementing the LifeSkills Training drug prevention program: Factors related to implementation fidelity. Implementation Science, 3, 1–16. doi:10.1186/1748-5908-3-5.CrossRef

Nadeem, E., Cappella, E., Holland, S., Coccaro, C., & Crisonino, G. (2016). Development and piloting of a classroom-focused measurement feedback system. Administration and Policy in Mental Health and Mental Health Services Research, 43(3), 379–393. doi:10.1007/s10488-015-0651-z.CrossRefPubMedPubMedCentral

O’Donnell, C. L. (2008). Defining, conceptualizing, and measuring fidelity of implementation and its relationship to outcomes in K–12 curriculum intervention research. Review of Educational Research, 78, 33–84. doi:10.3102/0034654307313793.CrossRef

Palinkas, L. A., Aarons, G. A., Horwitz, S., Chamberlain, P., Hurlburt, M., & Landsverk, J. (2011). Mixed method designs in implementation research. Administration and Policy in Mental Health and Mental Health Services Research, 38(1), 44–53. doi:10.1007/s10488-010-0314-z.CrossRefPubMed

Powell, B. J., McMillen, J. C., Hawley, K. M., & Proctor, E. K. (2013). Mental health clinicians’ motivation to invest in training: Results from a practice-based research network survey. Psychiatric Services, 64, 816–818. doi:10.1176/appi.ps.003602012.CrossRefPubMed

Powell, B. J., Proctor, E. K., & Glass, J. E. (2014). A systematic review of strategies for implementing empirically supported mental health interventions. Research on Social Work Practice, 24, 192–212. doi:10.1177/1049731513505778.CrossRefPubMed

Proctor, E. K., & Brownson, R. C. (2012). Measurement issues in dissemination and implementation research. In R. C. Brownson, G. A. Colditz & E. K. Proctor (Eds.), Dissemination and implementation research in health: Translating science to practice (pp. 261–280). New York, NY: Oxford University Press, Inc. doi:10.1093/acprof:oso/9780199751877.003.0013.CrossRef

Proctor, E. K., Powell, B. J., & McMillen, J. C. (2013). Implementation strategies: Recommendations for specifying and reporting. Implementation Science, 8(1), 1. doi:10.1186/1748-5908-8-139.CrossRef

Rohrbach, L. A., Gunning, M., Sun, P., & Sussman, S. (2010). The project towards no drug abuse (TND) dissemination trial: Implementation fidelity and immediate outcomes. Prevention Science, 11, 77–88. doi:10.1007/s11121-009-0151-z.CrossRefPubMedPubMedCentral

Shea, C. M., Jacobs, S. R., Esserman, D. A., Bruce, K., & Weiner, B. J. (2014). Organizational readiness for implementing change: A psychometric assessment of a new measure. Implementation Science, 9, 15. doi:10.1186/1748-5908-9-7.CrossRef

Sopko, K. M. (2006). School mental health services in the United States. InForum, 1–9. Available from National Association of State Directors of Special Education website: http://nasdse.org/DesktopModules/DNNspot-Store/ProductFiles/180_a75a2595-fb39-4f38-ac15-46b4d9bc45b6.pdf.

Tibbits, M. K., Bumbarger, B. K., Kyler, S. J., & Perkins, D. F. (2010). Sustaining evidence-based interventions under real-world conditions: Results from a large-scale diffusion project. Prevention Science, 11, 252–262. doi:10.1007/s11121-010-0170-9.CrossRefPubMedPubMedCentral

Todd, N. R., Allen, N. E., & Javdani, S. (2012). Multilevel modeling: Method and application for community-based research. In L. A. Jason & D. S. Glenwick (Eds.), Methodological approaches to community-based research (pp. 167–186). Washington D.C.: American Psychological Association. 10.1037/13492-010.CrossRef

Wandersman, A., Chien, V. H., & Katz, J. (2012). Toward an evidence-based system for innovation support for implementing innovations with quality: Tools, training, technical assistance, and quality assurance/quality improvement. American Journal of Community Psychology, 50, 445–459. doi:10.1007/s10464-012-9509-7.CrossRefPubMed

Weiner, B. J., Amick, H., & Lee, S.-Y. D. (2008). Conceptualization and measurement of organizational readiness for change: A review of the literature in health services research and other fields. Medical Care Research and Review, 65(4), 379–436. doi:10.1177/1077558708317802.CrossRefPubMed

Title: Exploring Provider Use of a Digital Implementation Support System for School Mental Health: A Pilot Study
Authors: Melanie Livet
Mary Yannayon
Kelly Sheppard
Kelly Kocher
James Upright
Janey McMillen
Publication date: 01-05-2018
Publisher: Springer US
Published in: Administration and Policy in Mental Health and Mental Health Services Research / Issue 3/2018
Print ISSN: 0894-587X
Electronic ISSN: 1573-3289
DOI: https://doi.org/10.1007/s10488-017-0829-7

Keynote webinar | Spotlight on medication adherence

Springer Medicine

Exploring Provider Use of a Digital Implementation Support System for School Mental Health: A Pilot Study

Abstract

Keynote webinar | Spotlight on medication adherence

Springer Medicine

Abstract

Please log in to get access to this content

Other articles of this Issue 3/2018

Impact of a Usual Source of Care on Health Care Use, Spending, and Quality Among Adults With Mental Health Conditions

Individualized Progress Measures Are More Acceptable to Clinicians Than Standardized Measures: Results of a National Survey

Impact of the New Mental Health Services on Rates of Suicide and Hospitalisations by Attempted Suicide, Psychiatric Problems, and Alcohol Problems in Brazil

Promoting Teachers’ Implementation of Classroom-Based Prevention Programming Through Coaching: The Mediating Role of the Coach–Teacher Relationship

Behavioural and Emotional Problems in Children and Educational Outcomes: A Dynamic Panel Data Analysis

Stepped Care Interpersonal Psychotherapy Treatment for Depressed Adolescents: A Pilot Study in Pediatric Clinics