Top

Published in:

01-01-2015

Observational Measures of Implementer Fidelity for a School-Based Preventive Intervention: Development, Reliability, and Validity

Authors: Wendi Cross, Jennifer West, Peter A. Wyman, Karen Schmeelk-Cone, Yinglin Xia, Xin Tu, Michael Teisl, C. Hendricks Brown, Marion Forgatch

Published in: Prevention Science | Issue 1/2015

Abstract

Current measures of implementer fidelity often fail to adequately measure core constructs of adherence and competence, and their relationship to outcomes can be mixed. To address these limitations, we used observational methods to assess these constructs and their relationships to proximal outcomes in a randomized trial of a school-based preventive intervention (Rochester Resilience Project) designed to strengthen emotion self-regulation skills in first–third graders with elevated aggressive–disruptive behaviors. Within the intervention group (n = 203), a subsample (n = 76) of students was selected to reflect the overall sample. Implementers were 10 paraprofessionals. Videotaped observations of three lessons from year 1 of the intervention (14 lessons) were coded for each implementer–child dyad on adherence (content) and competence (quality). Using multilevel modeling, we examined how much of the variance in the fidelity measures was attributed to implementer and to the child within implementer. Both measures had large and significant variance accounted for by implementer (competence, 68 %; adherence, 41 %); child within implementer did not account for significant variance indicating that ratings reflected stable qualities of the implementer rather than the child. Raw adherence and competence scores shared 46 % of variance (r = .68). Controlling for baseline differences and age, the amount (adherence) and quality (competence) of program delivered predicted children’s enhanced response to the intervention on both child and parent reports after 6 months, but not on teacher report of externalizing behavior. Our findings support the use of multiple observations for measuring fidelity and that adherence and competence are important components of fidelity which could be assessed by many programs using these methods.

Available only for authorised users

We use the generic term “implementer” and the program specific term “Mentor” interchangeably.

We explored including gender in the SEM models, but it did not improve the model fit or affect results, and there was no change when we removed gender from the model.

Aarons, G. A., Hurlburt, M., & Horwitz, S. M. (2010). Advancing a conceptual model of evidence-based practice implementation in public service sectors. Administration and Policy in Mental Health and Mental Health Services Research, 38, 4–23. doi:10.1007/s10488-010-0327-7.PubMedCentralCrossRef

Aarons, G. A., Green, A. E., Palinkas, L. A., Self-Brown, S., Whitaker, D. J., Lutzker, J. R., Silovsky, J. F., Hecht, D. B., & Chaffin, M. J. (2012). Dynamic adaptation process to implement an evidence-based child maltreatment intervention. Implementation Science, 7, 32. doi:10.1186/1748-5908-7-32.PubMedCentralCrossRefPubMed

American Psychological Association (1994). Diagnostic and statistical manual of mental disorders (5th ed.) Washington, DC: Author.

Breitenstein, S. M., Fogg, L., Garvey, C., Hill, C., Resnick, B., & Gross, D. (2010). Measuring implementation fidelity in a community-based parenting intervention. Nursing Research, 59, 158–165. doi:10.1097/NNR.0b013e3181dbb2e2.PubMedCentralCrossRefPubMed

Browne, M. W., & Cudeck, R. (1993). Alternative ways of assessing model fit. In K. A. Bollen & J. S. Long (Eds.), Testing structural models. Newbury Park: Sage.

Brownson, R. C., Colditz, G. A., & Proctor, E. K. (2012). Dissemination and implementation research in health: Translating science to practice. New York: Oxford University.CrossRef

Byrnes, H. F., Miller, B. A., Aalborg, A. E., Plasencia, A. V., & Keagy, C. D. (2010). Implementation fidelity in adolescent family-based prevention programs: Relationship to family engagement. Health Education Research, 25, 531–541. doi:10.1093/her/cyq006.PubMedCentralCrossRefPubMed

Carroll, K. M., Nich, C., Sifry, R. L., Nuro, K. F., Frankforter, T. L., Ball, S. A., Fenton, L., & Rounsaville, B. J. (2000). A general system for evaluating therapist adherence and competence in psychotherapy research in the addictions. Drug and Alcohol Dependence, 57, 225–238. doi:10.1016/S0376-8716(99)00049-6.CrossRefPubMed

Cicchetti, D. V. (1994). Guidelines, criteria, and rules of thumb for evaluating normed and standardized assessment instruments in psychology. Psychological Assessment, 6, 284–290. doi:10.1037/1040-3590.6.4.284.CrossRef

Cross, W., & West, J. (2011). Examining implementer fidelity: Conceptualising and measuring adherence and competence. Journal of Children’s Services, 6, 18–33. doi:10.5042/jcs.2011.0123.PubMedCentralCrossRefPubMed

Cross, W. & Wyman, P. A. (2004). Promoting resilient children initiative (PRCI): Mentor’s manual. Unpublished manuscript. University of Rochester: Authors.

Cross, W., Pisani, A, Schmeelk-Cone, K., Xia, Y., Tu, X., McMahon, M., Funkfah, J., & Gould. (2014). Fidelity assessment of the transfer of training in a suicide prevention program for crisis hotlines. Crisis: The Journal of Crisis Intervention and Suicide Prevention. In press.

Curran, G. M., Mukherjee, S., Allee, E., & Owen, R. R. (2008). A process for developing an implementation intervention: QUERI Series. Implementation Science, 3, 17. doi:10.1186/1748-5908-3-17.PubMedCentralCrossRefPubMed

Damschroder, L. J., Aron, D. C., Keith, R. E., Kirsh, S. R., Alexander, J. A., & Lowery, J. C. (2009). Fostering implementation of health services research findings into practice: A consolidated framework for advancing implementation. Implementation Science, 4, 50. doi:10.1186/1748-5908-4-50.PubMedCentralCrossRefPubMed

Dumas, J. E., Lynch, A. M., Laughlin, J. E., Smith, E. P., & Prinz, R. J. (2001). Promoting intervention fidelity: Conceptual issues, methods, and preliminary results from the EARLY ALLIANCE prevention trial. American Journal of Preventative Medicine, 20, 38–47. doi:10.1016/S0749-3797(00)00272-5.CrossRef

Durlak, J. A., & DuPre, E. P. (2008). Implementation matters: A review of research on the influence of implementation on program outcomes and the factors affecting implementation. American Journal of Community Psychology, 41, 327–350. doi:10.1007/s10464-008-9165-0.CrossRefPubMed

Fixsen, D. L., Naoom, S. F., Blase, K. A., Freidman, R. M., & Wallace, F. (2005). Implementation Research: A Synthesis of the Literature. Tampa: University of South Florida, Louis de la Parte Florida Mental Health Institute, The National Implementation Research Network.

Floyd, F., & Widaman, K. (1995). Factor analysis in the development and refinement of clinical assessment instruments. Psychological Assessment, 7, 286–299. doi:10.1037/1040-3590.7.3.286.CrossRef

Forgatch, M. S., & DeGarmo, D. S. (2011). Sustaining fidelity following the nationwide PMTO implementation in Norway. Prevention Science, 12, 235–246. doi:10.1007/s11121-011-0225-6.PubMedCentralCrossRefPubMed

Forgatch, M. S., DeGarmo, D. S., & Beldavs, Z. G. (2005). An efficacious theory-based intervention for stepfamilies. Behavior Therapy, 36, 357–365. doi:10.1016/S0005-7894(05)80117-0.PubMedCentralCrossRefPubMed

Glasgow, R. E., Vogt, T. M., & Boles, S. M. (1999). Evaluating the public health impact of health promotion interventions: The RE-AIM framework. American Journal of Public Health, 89, 1322–1327. doi:10.2105/AJPH.89.9.1322.PubMedCentralCrossRefPubMed

Glasgow, R. E., Vinson, C., Chambers, D., Khoury, M. J., Kaplan, R. M., & Hunter, C. (2012). National Institutes of Health approaches to dissemination and implementation science: Current and future directions. American Journal of Public Health, 102, 1274–1281. doi:10.2105/AJPH.2012.300755.PubMedCentralCrossRefPubMed

Gould, M. S., Cross, W., Pisani, A. R., Munfakh, J. L., & Kleinman, M. (2013). Impact of Applied Suicide Intervention Skills Training (ASIST) on national suicide prevention Lifeline counselor interventions and suicidal caller outcomes. Suicide and Life Threatening Behaviors, 43, 676–691. doi:10.1111/sltb.12049.CrossRef

Hamre, B. K., Justice, L. M., Pianta, R. C., Kilday, C., Sweeney, B., Downer, J. T., & Leach, A. (2010). Implementation fidelity of MyTeachingPartner literacy and language activities: Association with preschoolers’ language and literacy growth. Early Childhood Research Quarterly, 25, 329–347. doi:10.1016/j.ecresq.2009.07.002.CrossRef

Hepner, K. A., Hunter, S. B., Paddock, S. M., Zhou, A. J., & Watkins, K. E. (2011). Training addiction counselors to implement CBT for depression. Administration and Policy in Mental Health, 38, 313–323. doi:10.1007/s10488-011-0359-7.PubMedCentralCrossRefPubMed

Hogue, A., Henderson, C. E., Dauber, S., Barajas, P. C., Fried, A., & Liddle, H. A. (2008). Treatment adherence, competence, and outcome in individual and family therapy for adolescent behavior problems. Journal of Consulting & Clinical Psychology, 76, 544–555. doi:10.1037/0022-006X.76.4.544.CrossRef

Hu, L., & Bentler, P. M. (1999). Cutoff criteria for fit indexes in covariate structure analysis: Conventional criteria versus new alternatives. Structural Equation Modeling, 6, 1–55. doi:10.1080/10705519909540118.CrossRef

James, I. A., Blackburn, I. M., Milne, D. L., & Reichfelt, F. K. (2001). Moderators of trainee therapists’ competence in cognitive therapy. British Journal of Clinical Psychology 40, 131–141.

Kellam, S. G., Ling, X., Merisca, R., Brown, C. H., & Ialongo, N. (1998). The effect of the level of aggression in the first grade classroom on the course and malleability of aggressive behavior into middle school. Development and Psychopathology, 10, 165–186. doi:10.1017/S0954579498001564.CrossRefPubMed

Liber, J. M., van Widenfelt, B. M., van der Leeden, A. J., Goedhart, A. W., Utens, E. M., & Treffers, P. D. (2010). The relation of severity and comorbidity to treatment outcome with Cognitive Behavioral Therapy for childhood anxiety disorders. Journal of Abnormal Child Psychology, 38, 683–694. doi:10.1007/s10802-010-9394-1.PubMedCentralCrossRefPubMed

Lillehoj, C. J., Griffin, K. W., & Spoth, R. (2004). Program provider and observer ratings of school-based preventive intervention implementation: Agreement and relation to youth outcomes. Health Education & Behavior, 31, 242–257. doi:10.1177/1090198103260514.CrossRef

Moore, J. E., Beck, T. C., Sylvertsen, A., & Domitrovich, C. (2009). Making sense of implementation: Multiple dimensions, multiple sources, multiple methods. Washington, DC: Paper presented to the 17th Annual Meeting of the Society for Prevention Research.

Muthén, L. K., & Muthén, B. O. (2010). Mplus Version 6. Los Angeles: Muthén & Muthén.

Proctor, E., Silmere, H., Raghavan, R., Hovmand, P., Aarons, G., Bunger, A., Griffey, R., & Hensley, M. (2011). Outcomes for implementation research: Conceptual distinctions, measurement challenges, and research agenda. Administration and Policy in Mental Health, 38, 65–76. doi:10.1007/s10488-010-0319-7.PubMedCentralCrossRefPubMed

SAS. (2008). SAS, Version 9.2. Cary: SAS Institute Inc.

Schoenwald, S. K. (2011). It’s a bird, it’s a plane, it’s fidelity measurement in the real world. Clinical Psychology: Science and Practice, 18, 142–147. doi:10.1111%2Fj.1468-2850.2011.01245.x.

Schoenwald, S. K., & Garland, A. F. (2013). A review of treatment adherence measurement methods. Psychological Assessment, 25, 146–156. doi:10.1037/a0029715.PubMedCentralCrossRefPubMed

Schoenwald, S. K., Garland, A. F., Chapman, J. E., Frazier, S. L., Sheidow, A. J., & Southam-Gerow, M. A. (2011). Toward effective and efficient measurement of implementation fidelity. Administration and Policy in Mental Health and Mental Health Services Research, 38, 32–43. doi:10.1007/s10488-010-0321-0.PubMedCentralCrossRefPubMed

Smith, G. T., & McCarthy, D. M. (1995). Methodological considerations in the refinement of clinical assessment instruments. Psychological Assessment, 7, 300–308. doi:10.1037/1040-3590.7.3.300.CrossRef

Snyder, J., Reid, J., Stoolmiller, M., Howe, G., Brown, H., Dagne, G., & Cross, W. (2006). The role of behavior observation in measurement systems for randomized prevention trials. Prevention Science, 7, 43–56. doi:10.1007/s11121-005-0020-3.CrossRefPubMed

Stetler, C. B., McQueen, L., Demakis, J., & Mittman, B. S. (2008). An organizational framework and strategic implementation for systems-level change to enhance research-based practice: QUERI series. Implementation Science, 3, 30. doi:10.1186/1748-5908-3-30.PubMedCentralCrossRefPubMed

Streiner, D. L. (2003). Being inconsistent about consistency: When coefficient alpha does and doesn’t matter. Journal of Personality Assessment, 80, 217–222. doi:10.1207/S15327752JPA8003_01.CrossRefPubMed

Tabachnick, B. G., & Fidell, L. S. (2007). Using multivariate statistics (Vol. 5). Boston: Pearson Education, Inc.

Valla, J. P., Bergeron, L., & Smolla, N. (2000). The Dominic-R: A pictorial interview for 6-to 11-year old children. Journal of the American Academy of Child and Adolescent Psychiatry, 39, 85–93. doi:10.1097/00004583-200001000-00020.CrossRefPubMed

Waltz, J., Addis, M. E., Koerner, K., & Jacobson, N. S. (1993). Testing the integrity of a psychotherapy protocol: Assessment of adherence and competence. Journal of Consulting & Clinical Psychology, 61, 620–630. doi:10.1037/0022-006X.61.4.620.CrossRef

Webb, C. A., DeRubeis, R. J., & Barber, J. P. (2010). Therapist adherence/competence and treatment outcome: A meta-analytic review. Journal of Consulting and Clinical Psychology, 78, 200–211. doi:10.1037/a0018912.PubMedCentralCrossRefPubMed

Wells, M. G., Burlingame, G. M., & Lambert, M. J. (1996). Conceptualization and measurement of patient change during psychotherapy: Development of the Outcome Questionnaire and Youth Outcome Questionnaire. Psychotherapy: Theory, Research, Practice, Training, 33, 275–283. doi:10.1037/0033-3204.33.2.275.CrossRef

Werthamer-Larsson, L., Kellam, S., & Wheeler, L. (1991). Effect of first-grade classroom environment on shy behavior, aggressive problems, and concentration problems. American Journal of Community Psychology, 19, 585–601. doi:10.1007/BF00937993.CrossRefPubMed

Wyman, P. A., Cross, W., Hendricks Brown, C., Yu, Q., Tu, X., & Eberly, S. (2010). Intervention to strengthen emotional self-regulation in children with emerging mental health problems: Proximal impact on school behavior. Journal of Abnormal Child Psychology, 38, 707–720. doi:10.1007/s10802-010-9398-x.PubMedCentralCrossRefPubMed

Wyman, P. A., Cross, W., Wu, P., Schmeelk-Cone, K. West, J., Tu, X., Lu, Naiji & Brown, C. H. (2013). Intervention to strengthen emotion self-regulation in young urban children: Impact on internalizing and externalizing problems and mediating effects of self-regulation skills. (in review).

Title: Observational Measures of Implementer Fidelity for a School-Based Preventive Intervention: Development, Reliability, and Validity
Authors: Wendi Cross
Jennifer West
Peter A. Wyman
Karen Schmeelk-Cone
Yinglin Xia
Xin Tu
Michael Teisl
C. Hendricks Brown
Marion Forgatch
Publication date: 01-01-2015
Publisher: Springer US
Published in: Prevention Science / Issue 1/2015
Print ISSN: 1389-4986
Electronic ISSN: 1573-6695
DOI: https://doi.org/10.1007/s11121-014-0488-9

At a glance: The ONWARDS insulin icodec trials

Springer Medicine

Observational Measures of Implementer Fidelity for a School-Based Preventive Intervention: Development, Reliability, and Validity

Abstract

At a glance: The ONWARDS insulin icodec trials

Springer Medicine

Abstract

Please log in to get access to this content

Other articles of this Issue 1/2015

Adolescent Peer Networks and the Potential for the Diffusion of Intervention Effects

The Process of Adoption of Evidence-based Tobacco Use Prevention Programs in California Schools

Erratum to: Population-Based Prevention of Child Maltreatment:The U.S. Triple P System Population Trial

Understanding “Agency” in the Translation of a Health Promotion Program

Erratum to: Surviving or Thriving: Quality Assurance Mechanisms to Promote Innovation in the Development of Evidence-Based Parenting Interventions

Sustaining Evidence-Based Prevention Programs: Correlates in a Large-Scale Dissemination Initiative