Top

Published in:

01-01-2016

An Experimental Study of Procedures to Enhance Ratings of Fidelity to an Evidence-Based Family Intervention

Authors: Justin D. Smith, Thomas J. Dishion, Kimbree Brown, Karina Ramos, Naomi B. Knoble, Daniel S. Shaw, Melvin N. Wilson

Published in: Prevention Science | Issue 1/2016

Abstract

The valid and reliable assessment of fidelity is critical at all stages of intervention research and is particularly germane to interpreting the results of efficacy and implementation trials. Ratings of protocol adherence typically are reliable, but ratings of therapist competence are plagued by low reliability. Because family context and case conceptualization guide the therapist’s delivery of interventions, the reliability of fidelity ratings might be improved if the coder is privy to client context in the form of an ecological assessment. We conducted a randomized experiment to test this hypothesis. A subsample of 46 families with 5-year-old children from a multisite randomized trial who participated in the feedback session of the Family Check-Up (FCU) intervention were selected. We randomly assigned FCU feedback sessions to be rated for fidelity to the protocol using the COACH rating system either after the coder reviewed the results of a recent ecological assessment or had not. Inter-rater reliability estimates of fidelity ratings were meaningfully higher for the assessment information condition compared to the no-information condition. Importantly, the reliability of the COACH mean score was found to be statistically significantly higher in the information condition. These findings suggest that the reliability of observational ratings of fidelity, particularly when the competence or quality of delivery is considered, could be improved by providing assessment data to the coders. Our findings might be most applicable to assessment-driven interventions, where assessment data explicitly guides therapist’s selection of intervention strategies tailored to the family’s context and needs, but they could also apply to other intervention programs and observational coding of context-dependent therapy processes, such as the working alliance.

T scores ≥70 on the Externalizing Scale of the parent-report version of the Child Behavior Checklist (Achenbach 1991).

Achenbach, T. M. (1991). Manual for the child behavior checklist/4-18 and 1991 profile. Burlington: University of Vermont, Department of Psychiatry.

Berkel, C., Mauricio, A. M., Schoenfelder, E. N., & Sandler, I. N. (2011). Putting the pieces together: An integrated model of program implementation. Prevention Science, 12, 23–33. doi:10.1007/s11121-010-0186-1.PubMedCrossRef

Brody, G. H., Murry, V. M., Gerrard, M., Gibbons, F. X., Molgaard, V., McNair, L., & Neubaum-Carlan, E. (2004). The strong African american families program: translating research into prevention programming. Child Development, 75, 900–917. doi:10.2307/3696600.PubMedCrossRef

Carroll, C., Patterson, M., Wood, S., Booth, A., Rick, J., & Balain, S. (2007). A conceptual framework for implementation fidelity. Implementation Science, 2, 40. doi:10.1186/1748-5908-2-40.PubMedPubMedCentralCrossRef

Chiapa, A., Smith, J. D., Kim, H., Dishion, T. J., Shaw, D. S., & Wilson, M. N. (2015). The trajectory of fidelity in a multiyear trial of the family check-up predicts change in child problem behavior. Journal of Consulting and Clinical Psychology. doi:10.1037/ccp0000034.PubMedPubMedCentral

Cicchetti, D. V. (1994). Guidelines, criteria, and rules of thumb for evaluating normed and standardized assessment instruments in psychology. Psychological Assessment, 6, 284–290. doi:10.1037/1040-3590.6.4.284.CrossRef

Cross, W. F., & West, J. C. (2011). Examining implementer fidelity: Conceptualizing and measuring adherence and competence. Journal of Children’s Services, 6, 18–33. doi:10.5042/jcs.2011.0123.PubMedPubMedCentralCrossRef

Dishion, T. J., & Stormshak, E. A. (2007). Intervening in children's lives: An ecological, family-centered approach to mental health care. Washington, DC: American Psychological Association.CrossRef

Dishion, T. J., Shaw, D. S., Connell, A., Gardner, F. E. M., Weaver, C., & Wilson, M. (2008). The family check-up with high-risk indigent families: Preventing problem behavior by increasing parents’ positive behavior support in early childhood. Child Development, 79, 1395–1414. doi:10.1111/j.1467-8624.2008.01195.x.PubMedPubMedCentralCrossRef

Dishion, T.J., Knutson, N., Brauer, L., Gill, A., & Risso, J. (2010). Family Check-Up: COACH ratings manual. Unpublished coding manual. Available from the Child and Family Center, 6217. Eugene: University of Oregon OR 97403.

Forgatch, M. S., & DeGarmo, D. S. (2011). Sustaining fidelity following the nationwide PMTO™ implementation in Norway. Prevention Science, 12, 235–246. doi:10.1007/s11121-011-0225-6.PubMedPubMedCentralCrossRef

Forgatch, M. S., DeGarmo, D. S., & Beldavs, Z. G. (2005a). An efficacious theory-based intervention for stepfamilies. Behavior Therapy, 36, 357–365. doi:10.1016/S0005-7894(05)80117-0.PubMedPubMedCentralCrossRef

Forgatch, M. S., Patterson, G. R., & DeGarmo, D. S. (2005b). Evaluating fidelity: Predictive validity for a measure of competent adherence to the Oregon model of parent management training. Behavior Therapy, 36, 3–13. doi:10.1016/S0005-7894(05)80049-8.PubMedPubMedCentralCrossRef

Henggeler, S. W., Schoenwald, S. K., Liao, J. G., Letourneau, E. J., & Edwards, D. L. (2002). Transporting efficacious treatments to field settings: The link between supervisory practices and therapist fidelity in MST programs. Journal of Clinical Child & Adolescent Psychology, 31, 155–167. doi:10.1207/S15374424JCCP3102_02.CrossRef

Hogue, A., Liddle, H. A., Becker, D., & Johnson-Leckrone, J. (2002). Family-based prevention counseling for high-risk young adolescents: Immediate outcomes. Journal of Community Psychology, 30, 1–22. doi:10.1002/jcop.1047.CrossRef

Hogue, A., Liddle, H. A., Singer, A., & Leckrone, J. (2005). Intervention fidelity in family‐based prevention counseling for adolescent problem behaviors. Journal of Community Psychology, 33, 191–211. doi:10.1002/jcop.20031.CrossRef

Hogue, A., Henderson, C. E., Dauber, S., Barajas, P. C., Fried, A., & Liddle, H. A. (2008). Treatment adherence, competence, and outcome in individual and family therapy for adolescent behavior problems. Journal of Consulting and Clinical Psychology, 76, 544–555. doi:10.1037/0022-006x.76.4.544.PubMedPubMedCentralCrossRef

Huey, S. J., Jr., Henggeler, S. W., Brondino, M. J., & Pickrel, S. G. (2000). Mechanisms of change in multisystemic therapy: Reducing delinquent behavior through therapist adherence and improved family and peer functioning. Journal of Consulting and Clinical Psychology, 68, 451–467. doi:10.1037/0022-006x.68.3.451.PubMedCrossRef

Hukkelberg, S. S., & Ogden, T. (2013). Working alliance and treatment fidelity as predictors of externalizing problem behaviors in parent management training. Journal of Consulting and Clinical Psychology, 81, 1010–1020. doi:10.1037/a0033825.PubMedCrossRef

Institute of Medicine. (2001). Crossing the quality chasm: A new health system for the 21st century. Washington, DC: National Academy Press.

Kazdin, A. E. (2003). Research design in clinical psychology (4th ed.). Boston: Allyn & Bacon.

Knutson, N. M., Forgatch, M. S., & Rains, L. A. (2003). Fidelity of implementation rating system (FIMP): The training manual for PMTO. Eugene: Oregon Social Learning Center.

Landsverk, J. A., Brown, C. H., Chamberlain, P., Palinkas, L. A., Ogihara, M., Czaja, S., & Horwitz, S. M. (2012). Design and analysis in dissemination and implementation research. In R. C. Brownson, G. A. Colditz, & E. K. Proctor (Eds.), Dissemination and implementation research in health: Translating research to practice (pp. 225–260). New York: Oxford University Press.CrossRef

Lee, C.-Y., August, G., Realmuto, G., Horowitz, J., Bloomquist, M., & Klimes-Dougan, B. (2008). Fidelity at a distance: Assessing implementation fidelity of the early risers prevention program in a going-to-scale intervention trial. Prevention Science, 9, 215–229. doi:10.1007/s11121-008-0097-6.PubMedCrossRef

Meier, S. T. (1999). Training the practitioner-scientist: Bridging case conceptualization, assessment, and intervention. The Counseling Psychologist, 27, 846–869. doi:10.1177/0011000099276008.CrossRef

Miller, W. R., & Rollnick, S. (2002). Motivational interviewing: Preparing people for change (2nd ed.). New York: Guilford Press.

Muse, K., & McManus, F. (2013). A systematic review of methods for assessing competence in cognitive–behavioural therapy. Clinical Psychology Review, 33, 484–499. doi:10.1016/j.cpr.2013.01.010.PubMedCrossRef

Perepletchikova, F., & Kazdin, A. E. (2005). Treatment integrity and therapeutic change: Issues and research recommendations. Clinical Psychology: Science and Practice, 12, 365–383. doi:10.1093/Clipsy/Bpi045.

Prado, G., Pantin, H., Schwartz, S. J., Lupei, N. S., & Szapocznik, J. (2006). Predictors of engagement and retention into a parent-centered, ecodevelopmental HIV preventive intervention for Hispanic adolescents and their families. Journal of Pediatric Psychology, 31, 874–890. doi:10.1093/jpepsy/jsj046.PubMedCrossRef

R Core Team (2012). R: A language and environment for statistical computing. Vienna: R Foundation for Statistical Computing. Retrieved from http://www.R-project.org/.

Rao, C.R. (2009). Linear statistical inference and its applications (Vol. 22): Wiley.

Safran, J. D., Muran, J. C., Samstag, L. W., & Stevens, C. (2001). Repairing alliance ruptures. Psychotherapy: Theory, Research, Practice, Training, 38, 406–412. doi:10.1037/0033-3204.38.4.406.CrossRef

Schoenwald, S. K., & Henggeler, S. W. (2004). A public health perspective on the transport of evidence-based practices. Clinical Psychology: Science and Practice, 11, 360–363. doi:10.1093/clipsy.bph092.

Schoenwald, S. K., Garland, A. F., Chapman, J., Frazier, S., Sheidow, A., & Southam-Gerow, M. (2011). Toward the effective and efficient measurement of implementation fidelity. Administration and Policy in Mental Health and Mental Health Services Research, 38, 32–43. doi:10.1007/s10488-010-0321-0.PubMedPubMedCentralCrossRef

Shrout, P., & Fleiss, J. (1979). Intraclass correlations: Uses in assessing rater reliability. Psychological Bulletin, 86, 420–428. doi:10.1037/0033-2909.86.2.420.PubMedCrossRef

Silberschatz, G., Fretter, P. B., & Curtis, J. T. (1986). How do interpretations influence the process of psychotherapy? Journal of Consulting and Clinical Psychology, 54, 646–652. doi:10.1037/0022-006X.54.5.646.PubMedCrossRef

Smith, J. D., Dishion, T. J., Shaw, D. S., & Wilson, M. N. (2013). Indirect effects of fidelity to the family check-up on changes in parenting and early childhood problem behaviors. Journal of Consulting and Clinical Psychology, 81, 962–974. doi:10.1037/a0033950.PubMedCrossRef

Smith, J. D., Stormshak, E. A., & Kavanagh, K. (2014). Results of a pragmatic effectiveness-implementation hybrid trial of the family check-up in community mental health agencies. Administration and Policy in Mental Health and Mental Health Services Research, 42, 265–278. doi:10.1007/s10488-014-0566-0.CrossRef

SPSS Statistics Version 22.0.0 (2014). Chicago: SPSS Inc.

Webb, C. A., DeRubeis, R. J., & Barber, J. P. (2010). Therapist adherence/competence and treatment outcome: A meta-analytic review. Journal of Consulting and Clinical Psychology, 78, 200–211. doi:10.1037/a0018912.PubMedPubMedCentralCrossRef

Title: An Experimental Study of Procedures to Enhance Ratings of Fidelity to an Evidence-Based Family Intervention
Authors: Justin D. Smith
Thomas J. Dishion
Kimbree Brown
Karina Ramos
Naomi B. Knoble
Daniel S. Shaw
Melvin N. Wilson
Publication date: 01-01-2016
Publisher: Springer US
Published in: Prevention Science / Issue 1/2016
Print ISSN: 1389-4986
Electronic ISSN: 1573-6695
DOI: https://doi.org/10.1007/s11121-015-0589-0

At a glance: The STEP trials

Springer Medicine

An Experimental Study of Procedures to Enhance Ratings of Fidelity to an Evidence-Based Family Intervention

Abstract

At a glance: The STEP trials

Springer Medicine

Abstract

Please log in to get access to this content

Other articles of this Issue 1/2016

The Impact of Teachers’ Modifications of an Evidenced-Based HIV Prevention Intervention on Program Outcomes

Changing Latino Adolescents’ Substance Use Norms and Behaviors: the Effects of Synchronized Youth and Parent Drug Use Prevention Interventions

Long-Term Effects of the Life Skills Program IPSY on Substance Use: Results of a 4.5-Year Longitudinal Study

Pilot Test of an Adapted, Evidence-Based HIV Sexual Risk Reduction Intervention for Homeless Women

Alcohol Sales to Youth: Data from Rural Communities Within the Cherokee Nation

Feasibility and Reliability of a Coding System to Capture In-Session Group Behavior in Adolescents