Top

Administration and Policy in Mental Health and Mental Health Services Research

Published in:

01-09-2015 | Original Paper

Blending Qualitative and Computational Linguistics Methods for Fidelity Assessment: Experience with the Familias Unidas Preventive Intervention

Authors: Carlos Gallo, Hilda Pantin, Juan Villamar, Guillermo Prado, Maria Tapia, Mitsunori Ogihara, Gracelyn Cruden, C. Hendricks Brown

Published in: Administration and Policy in Mental Health and Mental Health Services Research | Issue 5/2015

Abstract

Careful fidelity monitoring and feedback are critical to implementing effective interventions. A wide range of procedures exist to assess fidelity; most are derived from observational assessments (Schoenwald and Garland, Psycholog Assess 25:146–156, 2013). However, these fidelity measures are resource intensive for research teams in efficacy/effectiveness trials, and are often unattainable or unmanageable for the host organization to rate when the program is implemented on a large scale. We present a first step towards automated processing of linguistic patterns in fidelity monitoring of a behavioral intervention using an innovative mixed methods approach to fidelity assessment that uses rule-based, computational linguistics to overcome major resource burdens. Data come from an effectiveness trial of the Familias Unidas intervention, an evidence-based, family-centered preventive intervention found to be efficacious in reducing conduct problems, substance use and HIV sexual risk behaviors among Hispanic youth. This computational approach focuses on “joining,” which measures the quality of the working alliance of the facilitator with the family. Quantitative assessments of reliability are provided. Kappa scores between a human rater and a machine rater for the new method for measuring joining reached 0.83. Early findings suggest that this approach can reduce the high cost of fidelity measurement and the time delay between fidelity assessment and feedback to facilitators; it also has the potential for improving the quality of intervention fidelity ratings.

Aarons, G. A., Hurlburt, M., & Horwitz, S. M. (2011). Advancing a conceptual model of evidence-based practice implementation in public service sectors. Administration and Policy in Mental Health, 38(1), 4–23. doi:10.1007/s10488-010-0327-7.PubMedCentralCrossRefPubMed

Alexander, J., Pugh, C., Parsons, B., & Sexton, T. (2000). Family functional therapy. In D. S. Elliot (Ed.), Blueprints for violence prevention: Book three (pp. 445–453). Boulder: Center for the Study and Prevention of Violence.

Allen, J. D., Linnan, L. A., & Emmons, K. M. (2012). Fidelity and its relationship to impementation effectiveness, adaptation, and dissemination. In R. C. Brownson, G. A. Colditz, & E. K. Proctor (Eds.), Dissemination and implementation in health: Translating science to practice (pp. 281–304). New York: Oxford University Press.CrossRef

Bahrammirzaee, A. (2010). A comparative survey of artificial intelligence applications in finance: Artificial neural networks, expert system and hybrid intelligent systems. Neural Computing and Applications, 19(8), 1165–1195. doi:10.1007/s00521-010-0362-z.CrossRef

Barber, J. P., Gallop, R., Crits-Christoph, P., Frank, A., Thase, M. E., Weiss, R. D., et al. (2006). The role of therapist adherence, therapist competence, and the alliance in predicting outcome of individual drug counseling: Results from the NIDA collaborative cocaine treatment study. Psychotherapy Research, 16, 240–299.CrossRef

Brown, C. H., Kellam, S. G., Kaupert, S., Muthén, B. O., Wang, W., Muthén, L., et al. (2012). Partnerships for the design, conduct, and analysis of effectiveness, and implementation research: Experiences of the prevention science and methodology group. Administration and Policy in Mental Health, 39(4), 301–316. doi:10.1007/s10488-011-0387-3.PubMedCentralCrossRefPubMed

Brown, C. H., Mohr, D. C., Gallo, C. G., Mader, C., Palinkas, L. A., Wingood, G., et al. (2013). A computational future for preventing HIV in minority communities: How advanced technology can improve implementation of effective programs. Journal of Acquired Immune Deficiency Syndromes, 63(1), S72–S84.PubMedCentralCrossRefPubMed

Busch, A. M., Kanter, J. W., Callaghan, G. M., Baruch, D. E., Weeks, C. E., & Berlin, K. S. (2009). A micro-process analysis of functional analytic psychotherapy’s mechanism of change. Behavior Therapy, 40(3), 280–290. doi:10.1016/j.beth.2008.07.003.CrossRefPubMed

Carletta, J. (1996). Assessing agreement on classification tasks: The kappa statistic. Computational Linguistics, 22(2), 249–254.

Chambers, D. A. (2008). Advancing the science of implementation: A workshop summary. Administration and Policy in Mental Health and Mental Health Services Research, 35(1–2), 3–10. doi:10.1007/s10488-007-0146-7.CrossRefPubMed

Chambers, D. (2012). The interactive systems framework for dissemination and implementation: Enhancing the opportunity for implementation science. American Journal of Community Psychology, 50(3–4), 282–284. doi:10.1007/s10464-012-9528-4.CrossRefPubMed

Chinman, M., Hunter, S., Ebener, P., Paddock, S., Stillman, L., Imm, P., et al. (2008). The getting to outcomes demonstration and evaluation: An illustration of the prevention support system. American Journal of Community Psychology, 41(3–4), 206–224. doi:10.1007/s10464-008-9163-2.PubMedCentralCrossRefPubMed

De Cooman, G., & Zaffalon, M. (2004). Updating beliefs with incomplete observations. Artificial Intelligence, 159(1–2), 75–125.CrossRef

Durlak, J., & DuPre, E. (2008). Implementation matters: A review of research on the influence of implementation on program outcomes and the factors affecting implementation. American Journal of Community Psychology, 41(3), 327–350. doi:10.1007/s10464-008-9165-0.CrossRefPubMed

Fixsen, D. L., Naoom, S. F., Blase, K. A., Friedman, R. M., & Wallace, F. (2005). Implementation research: A synthesis of the literature. (No. FMHI Publication #231). Tampa: University of South Florida, Louis de la Parte Florida Mental Health Institute.

Gallo, C. G., Aist, G., Allen, J. F., de Beaumont, W., Coria, S., Gegg-Harrison, W., & Swift, M. (2007). Continuous understanding in a multimodal dialogue corpus. Proceedings of DECALOG: The 2007 Workshop on the Semantics and Pragmatics of Dialogue, Trento (pp. 75–82).

Gallo, C. G., Jaeger, T. F., & Furth, K. (2010). A database for the exploration of Spanish planning. The Seventh International Conference on Language Resources and Evaluation (LREC). Valletta, Malta.

Hanson, R., Gros, K., Davidson, T., Barr, S., Cohen, J., Deblinger, E., & Ruggiero, K. (2013). National trainers’ perspectives on challenges to implementation of an empirically-supported mental health treatment. Administration and Policy in Mental Health and Mental Health Services Research, Online First, 1–13. doi:10.1007/s10488-013-0492-6.

Henggeler, S. W., & Borduin, C. M. (1990). Family therapy and beyond: A multisystemic approach to treating the behavior problems of children and adolescents. Pacific Grove: Brooks/Cole.

Henggeler, S. W., Schoenwald, S. K., Borduin, C. M., Rowland, M. D., & Cunningham, P. B. (1998). Multisystemic treatment of antisocial behavior in children and adolescents. Treatment manuals for practitioners. New York: Guilford Press.

Henggeler, S. W., Schoenwald, S. K., Liao, J. G., Letourneau, E. J., & Edwards, D. L. (2002). Transporting efficacious treatments to field settings: The link between supervisory practices and therapist fidelity in MST programs. Journal of Clinical Child and Adolescent Psychology: The Official Journal for the Society of Clinical Child and Adolescent Psychology, American Psychological Association, Division 53, 31(2), 155–167.CrossRef

Herschell, A. D. (2010). Fidelity in the field: Developing infrastructure and fine-tuning measurement. Clinical Psychology: Science and Practice, 17(3), 253–257. doi:10.1111/j.1468-2850.2010.01216.x.

Hogue, A., Dauber, S., Chinchilla, P., Fried, A., Henderson, C., Inclan, J., et al. (2008). Assessing fidelity in individual and family therapy for adolescent substance abuse. Journal of Substance Abuse Treatment, 35(2), 137–147.PubMedCentralCrossRefPubMed

Holmes, J. N., & Holmes, W. J. (2002). Speech synthesis and recognition 2e (HBK) (2nd ed.). New York: Taylor & Francis.

Huth, M., & Ryan, M. (2004). Logic in computer science: Modelling and reasoning about systems. Cambridge: Cambridge University Press.CrossRef

Inoue, M., Ogihara, M., Hanada, R., & Furuyama, N. (2010). Utility of gestural cues in indexing semantic miscommunication. Future Information Technology (FutureTech), 2010 5th International Conference on Future Information Technology (pp. 1–6).

Kellam, S. G., Mackenzie, A. C. L., Brown, C. H., Poduska, J. M., Petras, H., & Wilcox, H. C. (2011). The good behavior game and the future of prevention and treatment. Addiction Science and Clinical Practice, 6(1), 73–84.PubMedCentralPubMed

Landsverk, J., Brown, C. H., Chamberlain, P., Palinkas, L., Rolls Reutz, J., & Horwitz, S. M. (2012). Design and analysis in dissemination and implementation research. In R. C. Brownson, G. A. Colditz, & E. K. Proctor (Eds.), Dissemination and implementation research in health: Translating science to practice (pp. 225–260). London: Oxford University Press.CrossRef

Liddle, H. (2002). Multidimensional family therapy (MDFT) for adolescent cannabis users. Rockville: Center for Substance Abuse Treatment, Substance Abuse and Mental Health Services Administration.

Liddle, H. A., Rowe, C. L., Gonzalez, A., Henderson, C. E., Dakof, G. A., & Greenbaum, P. E. (2006). Changing provider practices, program environment, and improving outcomes by transporting multidimensional family therapy to an adolescent drug treatment setting. American Journal on Addictions, 15(Suppl), 102–112.CrossRefPubMed

Lopez, A. (2008). Statistical machine translation. ACM Computing Surveys (CSUR), 40(3), 8.CrossRef

Minuchin, S. (1974). Families and family therapy. Cambridge: Harvard University Press.

Minuchin, S., & Fishman, C. H. (1981). Family therapy techniques. Cambridge: Harvard University Press.

Orlinsky, D. E., Ronnestad, M. H., & Willutski, U. (2004). In Lambert M. J. (Ed.), Fifty years of psychotherapy process-outcome research: Continuity and change [Handbook of psychotherapy and behaviour change] (5th ed.). New York: John Wiley & Sons.

Overholser, J. C. (1995). Elements of the socratic method: IV. Disavowal of knowledge. Psychotherapy: Theory, research, practice. Training, 32(2), 283–292.

Palinkas, L. A., Aarons, G. A., Horwitz, S. M., Chamberlain, P., Hulburt, M., & Landsverk, J. (2011). Mixed method designs in implementation research. Administration & Policy in Mental Health and Mental Health Services, 38(1), 44–53.CrossRef

Pantin, H., Coatsworth, J. D., Feaster, D. J., Newman, F. L., Briones, E., Prado, G., et al. (2003). Familias unidas: The efficacy of an intervention to promote parental investment in Hispanic immigrant families. Prevention Science, 4(3), 189–201.CrossRefPubMed

Pantin, H., Prado, G., Lopez, B., Huang, S., Tapia, M. I., Schwartz, S. J., et al. (2009). A randomized controlled trial of familias unidas for Hispanic adolescents with behavior problems. Psychosomatic Medicine, 71(9), 987–995. doi:10.1097/PSY.0b013e3181bb2913.PubMedCentralCrossRefPubMed

Pantin, H., Schwartz, S. J., Sullivan, S., Prado, G., & Szapocznik, J. (2004). Ecodevelopmental HIV prevention programs for Hispanic adolescents. American Journal of Orthopsychiatry, 74(4), 545–558.PubMedCentralCrossRefPubMed

Poduska, J., Kellam, S. G., Brown, C. H., Ford, C., Windham, A., Keegan, N., et al. (2009). Study protocol for a group randomized controlled trial of a classroom-based intervention aimed at preventing early risk factors for drug abuse: Integrating effectiveness and implementation research. Implementation Science, 4, 56. doi:10.1186/1748-5908-4-56.PubMedCentralCrossRefPubMed

Popescu, A., Etzioni, O., & Kautz, H. (2003). Towards a theory of natural language interfaces to databases. Proceedings of the 8th International Conference on Intelligent User Interfaces, Miami (pp. 149–157).

Prado, G., Huang, S., Cordova, D., Malcolm, S., Estrada, Y., Cano, N., et al. (2013). Ecodevelopmental and intrapersonal moderators of a family based preventive intervention for hispanic youth: A latent profile analysis. Prevention Science: The Official Journal of the Society for Prevention Research, 14(3), 290–299. doi:10.1007/s11121-012-0326-x.CrossRef

Prado, G., & Pantin, H. (2011). Reducing substance use and HIV health disparities among Hispanic youth in the USA: The Familias Unidas program of research. Intervencion Psicosocial, 20(1), 63–73. doi:10.5093/in2011v20n1a6.PubMedCentralCrossRefPubMed

Prado, G., Pantin, H., Briones, E., Schwartz, S. J., Feaster, D., Huang, S., et al. (2007). A randomized controlled trial of a parent-centered intervention in preventing substance use and HIV risk behaviors in Hispanic adolescents. Journal of Consulting and Clinical Psychology, 75(6), 914–926. doi:10.1037/0022-006X.75.6.914.PubMedCentralCrossRefPubMed

Prado, G., Pantin, H., Huang, S., Cordova, D., Tapia, M. I., Velazquez, M. R., et al. (2012). Effects of a family intervention in reducing HIV risk behaviors among high-risk Hispanic adolescents: A randomized controlled trial. Archives of Pediatric and Adolescent Medicine, 166(2), 127–133. doi:10.1001/archpediatrics.2011.189.CrossRef

Prado, G., Pantin, H., Schwartz, S. J., Schwartz, S. J., Lupei, N. S., & Szapocznik, J. (2006). Predictors of engagement and retention into a parent-centered, ecodevelopmental HIV preventive intervention for Hispanic adolescents and their families. Journal of Pediatric Psychology, 31(9), 874–890. doi:10.1093/jpepsy/jsj046.CrossRefPubMed

Prinz, R. J., Sanders, M. R., Shapiro, C. J., Whitaker, D. J., & Lutzker, J. R. (2009). Population-based prevention of child maltreatment: The US triple p system population trial. Prevention Science, 10(1), 1–12. doi:10.1007/s11121-009-0123-3.PubMedCentralCrossRefPubMed

Real, K., & Poole, M. S. (2005). Innovation implementation: Conceptualization and measurement in organizational research. In R. Woodman, W. Pasmore & A. B. Shani (Eds.), Research in organizational change and development (15th ed., pp. 63–134). Online: Emerald Group Publishing Limited. doi:10.1016/S0897-3016(04)15003-9.

Robbins, M. S., Feaster, D. J., Horigian, V. E., Puccinelli, M. J., Henderson, C., & Szapocznik, J. (2011). Therapist adherence in brief strategic family therapy for adolescent drug abusers. Journal of Consulting and Clinical Psychology, 79(1), 43–53. doi:10.1037/a0022146.PubMedCentralCrossRefPubMed

Rowe, C., Rigter, H., Henderson, C., Gantner, A., Mos, K., Nielsen, P., et al. (2013). Implementation fidelity of multidimensional family therapy in an international trial. Journal of Substance Abuse Treatment, 44(4), 391–399.CrossRefPubMed

Schoenwald, S. K., Chapman, J. E., Kelleher, K., Hoagwood, K. E., Landsverk, J., Stevens, J., et al. (2008). A survey of the infrastructure for children’s mental health services: Implications for the implementation of empirically supported treatments (ESTs). Administration and Policy in Mental Health and Mental Health Services Research, 35(1–2), 84–97.CrossRefPubMed

Schoenwald, S. K., & Garland, A. F. (2013). A review of treatment adherence measurement methods. Psychological Assessment, 25, 146–156. doi:10.1037/a0029715.PubMedCentralCrossRefPubMed

Schoenwald, S. K., Garland, A. F., Chapman, J. E., Frazier, S. L., Sheidow, A. J., & Southam-Gerow, M. A. (2011). Toward the effective and efficient measurement of implementation fidelity. Administration and Policy in Mental Health, 38(1), 32–43. doi:10.1007/s10488-010-0321-0.PubMedCentralCrossRefPubMed

Suarez-Morales, L., Matthews, J., Martino, S., Ball, S. A., Rosa, C., Farentinos, C., et al. (2007). Issues in designing and implementing a Spanish-language multi-site clinical trial. American Journal on Addictions, 16(3), 206–215.PubMedCentralCrossRefPubMed

Szapocznik, J., & Coatsworth, J. D. (1999). An ecodevelopmental framework for organizing the influences on drug abuse: A developmental model of risk and protection. Drug abuse: Origins & interventions. Washington: American Psychological Association. doi:10.1037/10341-014.

Szapocznik, J., Hervis, O. E., & Schwartz, S. (2003). Brief strategic family therapy for adolescent drug abuse. (NIH publication no. 03-4751; NIDA Therapy Manuals for Drug Addiction Series ed. Rockville: National Institute of Drug Abuse.

Wall, L., Christiansen, T., & Orwant, J. (2000). Programming perl (3rd ed.). Sebastopol: O’Reilly Media.

Title: Blending Qualitative and Computational Linguistics Methods for Fidelity Assessment: Experience with the Familias Unidas Preventive Intervention
Authors: Carlos Gallo
Hilda Pantin
Juan Villamar
Guillermo Prado
Maria Tapia
Mitsunori Ogihara
Gracelyn Cruden
C. Hendricks Brown
Publication date: 01-09-2015
Publisher: Springer US
Published in: Administration and Policy in Mental Health and Mental Health Services Research / Issue 5/2015
Print ISSN: 0894-587X
Electronic ISSN: 1573-3289
DOI: https://doi.org/10.1007/s10488-014-0538-4

At a glance: The ONWARDS insulin icodec trials

Springer Medicine

Blending Qualitative and Computational Linguistics Methods for Fidelity Assessment: Experience with the Familias Unidas Preventive Intervention

Abstract

At a glance: The ONWARDS insulin icodec trials

Springer Medicine

Abstract

Please log in to get access to this content

Other articles of this Issue 5/2015

Clinicians’ Attitudes Towards Outcome and Process Monitoring: A Validation of the Outcome Measurement Questionnaire

Long-Term Clinical Outcomes from a Randomized Controlled Trial of Two Implementation Strategies to Promote Collaborative Care Attendance in Community Practices

Erratum to: Characteristics of People Who Use Telephone Counseling: Findings from Secondary Analysis of a Population-Based Study

Linking the Legislative Process to the Consequences of Realigning California’s Public Mental Health System

Optimal Design and Purposeful Sampling: Complementary Methodologies for Implementation Research

Co-responding Police-Mental Health Programs: A Review