Top

Administration and Policy in Mental Health and Mental Health Services Research

Open Access 29-03-2024 | Psychotherapy | Original Article

Measuring Alliance and Symptom Severity in Psychotherapy Transcripts Using Bert Topic Modeling

Authors: Christopher Lalk, Tobias Steinbrenner, Weronika Kania, Alexander Popko, Robin Wester, Jana Schaffrath, Steffen Eberhardt, Brian Schwartz, Wolfgang Lutz, Julian Rubel

Published in: Administration and Policy in Mental Health and Mental Health Services Research

Abstract

We aim to use topic modeling, an approach for discovering clusters of related words (“topics”), to predict symptom severity and therapeutic alliance in psychotherapy transcripts, while also identifying the most important topics and overarching themes for prediction. We analyzed 552 psychotherapy transcripts from 124 patients. Using BERTopic (Grootendorst, 2022), we extracted 250 topics each for patient and therapist speech. These topics were used to predict symptom severity and alliance with various competing machine-learning methods. Sensitivity analyses were calculated for a model based on 50 topics, LDA-based topic modeling, and a bigram model. Additionally, we grouped topics into themes using qualitative analysis and identified key topics and themes with eXplainable Artificial Intelligence (XAI). Symptom severity could be predicted with highest accuracy by patient topics (\(r\)=0.45, 95%-CI 0.40, 0.51), whereas alliance was better predicted by therapist topics (\(r\)=0.20, 95%-CI 0.16, 0.24). Drivers for symptom severity were themes related to health and negative experiences. Lower alliance was correlated with various themes, especially psychotherapy framework, income, and everyday life. This analysis shows the potential of using topic modeling in psychotherapy research allowing to predict several treatment-relevant metrics with reasonable accuracy. Further, the use of XAI allows for an analysis of the individual predictive value of topics and themes. Limitations entail heterogeneity across different topic modeling hyperparameters and a relatively small sample size.

Available only for authorised users

We also tested a combined model with patient and therapist topics, which did not improve the performance.

Aafjes-van Doorn, K., Porcerelli, J., & Müller-Frommeyer, L. C. (2020). Language style matching in psychotherapy: An implicit aspect of alliance. Journal of Counseling Psychology, 67(4), 509–522. https://doi.org/10.1037/cou0000433.CrossRefPubMed

Aafjes-van Doorn, K., Kamsteeg, C., Bate, J., & Aafjes, M. (2021). A scoping review of machine learning in psychotherapy research. Psychotherapy Research: Journal of the Society for Psychotherapy Research, 31(1), 92–116. https://doi.org/10.1080/10503307.2020.1808729.CrossRefPubMed

Ægisdóttir, S., White, M. J., Spengler, P. M., Maugherman, A. S., Anderson, L. A., Cook, R. S., Nichols, C. N., Lampropoulos, G. K., Walker, B. S., Cohen, G., & Rush, J. D. (2006). The Meta-Analysis of Clinical Judgment Project: Fifty-six years of Accumulated Research on Clinical Versus Statistical Prediction. The Counseling Psychologist, 34(3), 341–382. https://doi.org/10.1177/0011000005285875.CrossRef

Atkins, D. C., Rubin, T. N., Steyvers, M., Doeden, M. A., Baucom, B. R., & Christensen, A. (2012). Topic models: A novel method for modeling couple and family text data. Journal of Family Psychology, 26(5), 816–827. https://doi.org/10.1037/a0029607.CrossRefPubMedPubMedCentral

Atkins, D. C., Steyvers, M., Imel, Z. E., & Smyth, P. (2014). Scaling up the evaluation of psychotherapy: Evaluating motivational interviewing fidelity via statistical text classification. Implementation Science, 9, 49. https://doi.org/10.1186/1748-5908-9-49.CrossRefPubMedPubMedCentral

Atzil-Slonim, D., Juravski, D., Bar-Kalifa, E., Gilboa-Schechtman, E., Tuval-Mashiach, R., Shapira, N., & Goldberg, Y. (2021). Using topic models to identify clients’ functioning levels and alliance ruptures in psychotherapy. Psychotherapy, 58(2), 324–339. https://doi.org/10.1037/pst0000362.CrossRefPubMed

Baumeister, R. F., Bratslavsky, E., Finkenauer, C., & Vohs, K. D. (2001). Bad is stronger than good. Review of General Psychology, 5(4), 323–370. https://doi.org/10.1037/1089-2680.5.4.323.CrossRef

Bean, C. A. L., Summers, C. B., & Ciesla, J. A. (2022). Dampening of positive affect and depression: A meta-analysis of cross-sectional and longitudinal relationships. Behaviour Research and Therapy, 156, 104153. https://doi.org/10.1016/j.brat.2022.104153.CrossRefPubMed

Beck, A. T., Steer, R. A., Ball, R., & Ranieri, W. (1996). Comparison of Beck Depression inventories -IA and -II in psychiatric outpatients. Journal of Personality Assessment, 67(3), 588–597. https://doi.org/10.1207/s15327752jpa6703_13.CrossRefPubMed

Bird, S., Klein, E., & Loper, E. (2009). Natural language processing with Python. O’Reilly. https://ebookcentral.proquest.com/lib/kxp/detail.action?docID=443090.

Blei, D. M., Ng, A. Y., & Jordan, M. I. (2003). Latent dirichlet allocation. Journal of Machine Learning Research, 3(Jan), 993–1022.

Bordin, E. S. (1979). The generalizability of the psychoanalytic concept of the working alliance. Psychotherapy: Theory Research & Practice, 16(3), 252–260. https://doi.org/10.1037/h0085885.CrossRef

Breiman, L. (2001). Machine Learning, 45(1), 5–32. https://doi.org/10.1023/A:1010933404324.CrossRef

Burkhardt, H., Pullmann, M., Hull, T., Aren, P., & Cohen, T. (2022). Comparing emotion feature extraction approaches for predicting depression and anxiety. In A. Zirikly, D. Atzil-Slonim, M. Liakata, S. Bedrick, B. Desmet, M. Ireland, A. Lee, S. MacAvaney, M. Purver, R. Resnik, & A. Yates (Eds.), Proceedings of the Eighth Workshop on Computational Linguistics and Clinical Psychology (pp. 105–115). Association for Computational Linguistics. https://doi.org/10.18653/v1/2022.clpsych-1.9.

Castonguay, L. G., Barkham, M., Lutz, W., & McAleavey, A. A. (2013). Practice oriented research: Approaches and applications. In M. J. Lambert (Ed.), Bergin and Garfield’s Handbook of Psychotherapy and Behavior Change (6th ed., pp. 85–133). Wiley

Chen, T., & Guestrin, C. (2016). XGBoost. https://arxiv.org/pdf/1603.02754https://doi.org/10.1145/2939672.2939785.

R Core Team (2022). R: A Language and Environment for Statistical Computing (Version 4.2.2) [Computer software]. Vienna, Austria. https://www.R-project.org/.

Cortes, C., & Vapnik, V. (1995). Support-vector networks. Machine Learning, 20(3), 273–297. https://doi.org/10.1007/BF00994018.CrossRef

Crits-Christoph, P., Gibbons, M. B. C., & Mukherjee, D. (2021). Psychotherapy Process-Outcome Research. In M. Barkham, W. Lutz, & L. G. Castonguay (Eds.), Bergin and Garfield’s Handbook of Psychotherapy and Behavior Change (50th ed., pp. 298–340). Wiley.

Deerwester, S., Dumais, S. T., Furnas, G. W., Landauer, T. K., & Harshman, R. (1990). Indexing by latent semantic analysis. Journal of the American Society for Information Science, 41(6), 391–407.CrossRef

Delgadillo, J., de Jong, K., Lucock, M., Lutz, W., Rubel, J., Gilbody, S., Ali, S., Aguirre, E., Appleton, M., Nevin, J., O’Hayon, H., Patel, U., Sainty, A., Spencer, P., & McMillan, D. (2018). Feedback-informed treatment versus usual psychological treatment for depression and anxiety: A multisite, open-label, cluster randomised controlled trial. The Lancet Psychiatry, 5(7), 564–572. https://doi.org/10.1016/S2215-0366(18)30162-7.CrossRefPubMed

Derogatis, L. R., & Melisaratos, N. (1983). The brief Symptom Inventory: An introductory report. Psychological Medicine, 13(3), 595–605.CrossRefPubMed

Duncan, B. L., Miller, S. D., Sparks, J. A., Claud, D. A., & Reynolds, L. R. (2003). The Session Rating Scale: Preliminary Psychometric Properties of a Working Alliance Measure (Vol. 3). https://www.researchgate.net/profile/scott-miller-32/publication/254093433_the_session_rating_scale_preliminary_psychometric_properties_of_a_working_alliance_measure.

First, M. B., & Gibbon, M. (2004). The structured clinical interview for DSM-IV axis I disorders (SCID-I) and the structured clinical interview for DSM-IV axis II disorders (SCID-II). In M. J. Hilsenroth & D. L. Segal (Eds.), Comprehensive handbook of psychological assessment: Vol. 2 Personality assessment (pp. 134–143). Wiley. https://psycnet.apa.org/record/2004-12821-011.

Flückiger, C., Del Re, A. C., Wampold, B. E., & Horvath, A. O. (2018). The alliance in adult psychotherapy: A meta-analytic synthesis. Psychotherapy, 55(4), 316–340. https://doi.org/10.1037/pst0000172.CrossRefPubMed

Goldberg, S. B., Flemotomos, N., Martinez, V. R., Tanana, M. J., Kuo, P. B., Pace, B. T., Villatte, J. L., Georgiou, P. G., van Epps, J., Imel, Z. E., Narayanan, S. S., & Atkins, D. C. (2020). Machine learning and natural language processing in psychotherapy research: Alliance as example use case. Journal of Counseling Psychology, 67(4), 438–448. https://doi.org/10.1037/cou0000382.CrossRefPubMedPubMedCentral

Grootendorst, M. (2022, March 11). BERTopic: Neural topic modeling with a class-based TF-IDF procedure. https://arxiv.org/pdf/2203.05794.

Hajjem, A., Bellavance, F., & Larocque, D. (2014). Mixed-effects random forest for clustered data. Journal of Statistical Computation and Simulation, 84(6), 1313–1328. https://doi.org/10.1080/00949655.2012.741599.CrossRef

Hawkins, M., Gill, S. D., Batterham, R., Elsworth, G. R., & Osborne, R. H. (2017). The health literacy questionnaire (HLQ) at the patient-clinician interface: A qualitative study of what patients and clinicians mean by their HLQ scores. BMC Health Services Research, 17(1), 309. https://doi.org/10.1186/s12913-017-2254-8.CrossRefPubMedPubMedCentral

Hill, C. E., & Knox, S. (2021). Essentials of consensual qualitative research. Essentials of qualitative methods series. American Psychological Association.

Holzinger, A., Saranti, A., Molnar, C., Biecek, P., & Samek, W. (2022). Explainable AI Methods - A Brief Overview. In A. Holzinger (Ed.), Lecture notes in computer science Lecture notes in artificial intelligence: Vol. 13200. xxAI - beyond explainable AI: International workshop, held in conjunction with ICML 2020, July 18, 2020, Vienna, Austria: revised and extended papers (Vol. 13200, pp. 13–38). Springer. https://doi.org/10.1007/978-3-031-04083-2_2.

Iacovides, A., & Siamouli, M. (2008). Comorbid mental and somatic disorders: An epidemiological perspective. Current Opinion in Psychiatry, 21(4), 417–421. https://doi.org/10.1097/YCO.0b013e328303ba42.CrossRefPubMed

Imel, Z. E., Steyvers, M., & Atkins, D. C. (2015). Computational psychotherapy research: Scaling up the evaluation of patient-provider interactions. Psychotherapy (Chicago Ill), 52(1), 19–30. https://doi.org/10.1037/a0036841.CrossRefPubMed

Iob, E., Frank, P., Steptoe, A., & Fancourt, D. (2020). Levels of severity of depressive symptoms among at-risk groups in the UK during the COVID-19 pandemic. JAMA Network Open, 3(10), e2026064. https://doi.org/10.1001/jamanetworkopen.2020.26064CrossRefPubMedPubMedCentral

Kherwa, P., & Bansal, P. (2018). Topic modeling: A Comprehensive Review. ICST Transactions on Scalable Information Systems, 0(0), 159623. https://doi.org/10.4108/eai.13-7-2018.159623.CrossRef

Kroenke, K., Spitzer, R. L., & Williams, J. B. (2001). The PHQ-9: Validity of a brief depression severity measure. Journal of General Internal Medicine, 16(9), 606–613. https://doi.org/10.1046/j.1525-1497.2001.016009606.x[R. L.].CrossRefPubMedPubMedCentral

Lambert, M. J. (2021). The Efficacy and Effectiveness of Psychotherapy. In M. Barkham, W. Lutz, & L. G. Castonguay (Eds.), Bergin and Garfield’s Handbook of Psychotherapy and Behavior Change (50th ed., pp. 169–218). Wiley.

Lewinsohn, P. M., & Libet, J. (1972). Pleasant events, activity schedules, and depressions. Journal of Abnormal Psychology, 79(3), 291–295. https://doi.org/10.1037/h0033207.CrossRefPubMed

Lin, B., Bouneffouf, D., Cecchi, G., & Tejwani, R. (2022, April 13). Neural Topic Modeling of Psychotherapy Sessions. https://arxiv.org/pdf/2204.10189.

Lorenzo-Luaces, L., Rodriguez-Quintana, N., & Bailey, A. J. (2020). Double trouble: Do symptom severity and duration interact to predicting treatment outcomes in adolescent depression? Behaviour Research and Therapy, 131, 103637. https://doi.org/10.1016/j.brat.2020.103637.CrossRefPubMedPubMedCentral

Lundberg, S. M., & Lee, S. I. (2017). A Unified Approach to Interpreting Model Predictions. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, & R. Garnett (Eds.), Advances in Neural Information Processing Systems 30 (pp. 4765–4774). Curran Associates, Inc.

Lundberg, S. M., Erion, G., Chen, H., DeGrave, A., Prutkin, J. M., Nair, B., Katz, R., Himmelfarb, J., Bansal, N., & Lee, S. I. (2020). From local explanations to global understanding with explainable AI for trees. Nature Machine Intelligence, 2(1), 2522–5839.CrossRef

Lutz, W., Tholen, S., Schürch, E., & Berking, M. (2006). Reliabilität Von Kurzformen gängiger Psychometrischer Instrumente zur evaluation des therapeutischen fortschritts in Psychotherapie und psychiatrie. Diagnostica, 52(1), 11–25. https://doi.org/10.1026/0012-1924.52.1.11.CrossRef

Lutz, W., Castonguay, L. G., Lambert, M., & Barkham, M. (2021a). Traditions and New Beginnings: Historical and Current Perspectives on Research in Psychotherapy and Behavior Change. In M. Barkham, W. Lutz, & L. G. Castonguay (Eds.), Bergin and Garfield’s Handbook of Psychotherapy and Behavior Change (50th ed., pp. 3–18). Wiley.

Lutz, W., de Jong, K., Rubel, J. A., & Delgadillo, J. (2021b). Measuring, predicting, and tracking change in psychotherapy. In M. Barkham, W. Lutz, & L. G. Castonguay (Eds.), Bergin and Garfield’s Handbook of Psychotherapy and Behavior Change (50th ed., pp. 89–133). Wiley.

Lutz, W., Deisenhofer, A. K., Rubel, J., Bennemann, B., Giesemann, J., Poster, K., & Schwartz, B. (2022a). Prospective evaluation of a clinical decision support system in psychological therapy. Journal of Consulting and Clinical Psychology, 90(1), 90–106. https://doi.org/10.1037/ccp0000642.CrossRefPubMed

Lutz, W., Schwartz, B., & Delgadillo, J. (2022b). Measurement-based and Data-informed psychological therapy. Annual Review of Clinical Psychology, 18, 71–98. https://doi.org/10.1146/annurev-clinpsy-071720-014821.CrossRefPubMed

McInnes, L., & Healy, J. (2017). Accelerated Hierarchical Density Based Clustering. In Data Mining Workshops (ICDMW), 2017 IEEE International Conference on (pp. 33–42). IEEE.

Miao, Y., Yu, L., & Blunsom, P. (2016). Neural Variational Inference for Text Processing. In M. F. Balcan & K. Q. Weinberger (Eds.), Proceedings of Machine Learning Research, Proceedings of The 33rd International Conference on Machine Learning (pp. 1727–1736). PMLR. https://proceedings.mlr.press/v48/miao16.html.

Moberly, N. J., & Watkins, E. R. (2008). Ruminative self-focus and negative affect: An experience sampling study. Journal of Abnormal Psychology, 117(2), 314–323. https://doi.org/10.1037/0021-843X.117.2.314.CrossRefPubMedPubMedCentral

Montani, I., Honnibal, M., Boyd, A., van Landeghem, S., & Peters, H. (2023). explosion/spaCy: v3.7.2: Fixes for APIs and requirements [Computer software]. Zenodo.

Murphy, M. G., Rakes, S., & Harris, R. M. (2020). The Psychometric properties of the Session Rating Scale: A narrative review. Journal of Evidence-Based Social Work (2019), 17(3), 279–299. https://doi.org/10.1080/26408066.2020.1729281.CrossRefPubMed

Norcross, J. C., & Lambert, M. J. (2019). Evidence-based therapist contributions (3rd ed.). Psychotherapy relationships that work: Vol. 1. Oxford University Press. https://doi.org/10.1093/med-psych/9780190843953.001.0001.

O’Leary, D. E. (2013). Artificial Intelligence and Big Data. IEEE Intelligent Systems, 28(2), 96–99. https://doi.org/10.1109/MIS.2013.39.CrossRef

Paatero, P., & Tapper, U. (1994). Positive matrix factorization: A non-negative factor model with optimal utilization of error estimates of data values. Environmetrics, 5(2), 111–126. https://doi.org/10.1002/ENV.3170050203.CrossRef

Peng, H., Long, F., & Ding, C. (2005). Feature selection based on mutual information: Criteria of max-dependency, max-relevance, and min-redundancy. IEEE Transactions on Pattern Analysis and Machine Intelligence, 27(8), 1226–1238. https://doi.org/10.1109/TPAMI.2005.159.CrossRefPubMed

Python Software Foundation (2023). Python (Version 3.9) [Computer software]. https://www.python.org/.

Reese, R. J., Gillaspy, J. A., Owen, J. J., Flora, K. L., Cunningham, L. C., Archie, D., & Marsden, T. (2013). The influence of demand characteristics and social desirability on clients’ ratings of the therapeutic alliance. Journal of Clinical Psychology, 69(7), 696–709. https://doi.org/10.1002/jclp.21946.CrossRefPubMed

Rehurek, R., & Sojka, P. (2010). Software Framework for Topic Modelling with Large Corpora. In Proceedings of the LREC 2010 Workshop on New Challenges for NLP Frameworks (pp. 45–50). ELRA.

Reimers, N., & Gurevych, I. (2019). Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics. http://arxiv.org/abs/1908.1008.

Rubel, J. A., Rosenbaum, D., & Lutz, W. (2017). Patients’ in-session experiences and symptom change: Session-to-session effects on a within- and between-patient level. Behaviour Research and Therapy, 90, 58–66. https://doi.org/10.1016/j.brat.2016.12.007.CrossRefPubMed

Ryu, J., Heisig, S., McLaughlin, C., Katz, M., Mayberg, H. S., & Gu, X. (2023). A natural language processing approach reveals first-person pronoun usage and non-fluency as markers of therapeutic alliance in psychotherapy. IScience, 26(6), 106860. https://doi.org/10.1016/j.isci.2023.106860.CrossRefPubMedPubMedCentral

Sels, L., Tran, A., Greenaway, K. H., Verhofstadt, L., & Kalokerinos, E. K. (2021). The social functions of positive emotions. Current Opinion in Behavioral Sciences, 39, 41–45. https://doi.org/10.1016/j.cobeha.2020.12.009.CrossRef

Seshadri, R. (2023). featurewiz [Computer software]. AutoViML. https://github.com/AutoViML/featurewiz.

Shapley, L. S. (1951). Notes on the n-Person Game - II: The Value of an n-Person Game. Research Memorandum (RM-670). Santa Monica, Calif. RAND Corporation. https://www.rand.org/pubs/research_memoranda/RM670.html.

Sigrist, F. (2022). Gaussian process boosting. Journal of Machine Learning Research, 23(232), 1–46. http://jmlr.org/papers/v23/20-322.html.

Spitzer, R. L., Kroenke, K., Williams, J. B. W., & Löwe, B. (2006). A brief measure for assessing generalized anxiety disorder: The GAD-7. Archives of Internal Medicine, 166(10), 1092–1097. https://doi.org/10.1001/archinte.166.10.1092.CrossRefPubMed

Tibshirani, R., Hastie, T., Witten, D., & James, G. (2021). An introduction to statistical learning: With applications in R. Springer. https://doi.org/10.1007/978-1-0716-1418-1.CrossRef

van der Laan, M. J., Polley, E. C., & Hubbard, A. E. (2007). Super learner. Statistical Applications in Genetics and Molecular Biology, 6, Article25. https://doi.org/10.2202/1544-6115.1309.CrossRefPubMed

Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017, June 12). Attention Is All You Need. https://arxiv.org/pdf/1706.03762.

Whipple, J. L., Lambert, M. J., Vermeersch, D. A., Smart, D. W., Nielsen, S. L., & Hawkins, E. J. (2003). Improving the effects of psychotherapy: The use of early identification of treatment and problem-solving strategies in routine practice. Journal of Counseling Psychology, 50(1), 59–68. https://doi.org/10.1037/0022-0167.50.1.59.CrossRef

White, K. J., McCoy, M., Watkins, K., Chen, X., Koposko, J., & Mizuta, M. (2021). We don’t talk about that: Exploring money conversations of Black, Hispanic, and White households. Family and Consumer Sciences Research Journal, 49(4), 328–343. https://doi.org/10.1111/fcsr.12397.CrossRef

Zilcha-Mano, S., & Fisher, H. (2022). Distinct roles of state-like and trait-like patient–therapist alliance in psychotherapy. Nature Reviews Psychology, 1(4), 194–210. https://doi.org/10.1038/s44159-022-00029-z.CrossRef

Zou, H., & Hastie, T. (2005). Addendum: Regularization and Variable Selection Via the Elastic Net. Journal of the Royal Statistical Society Series B: Statistical Methodology, 67(5), 768. https://doi.org/10.1111/j.1467-9868.2005.00527.x.CrossRef

Title: Measuring Alliance and Symptom Severity in Psychotherapy Transcripts Using Bert Topic Modeling
Authors: Christopher Lalk
Tobias Steinbrenner
Weronika Kania
Alexander Popko
Robin Wester
Jana Schaffrath
Steffen Eberhardt
Brian Schwartz
Wolfgang Lutz
Julian Rubel
Publication date: 29-03-2024
Publisher: Springer US
Keywords: Psychotherapy
Artificial Intelligence
Published in: Administration and Policy in Mental Health and Mental Health Services Research
Print ISSN: 0894-587X
Electronic ISSN: 1573-3289
DOI: https://doi.org/10.1007/s10488-024-01356-4

Keynote webinar | Spotlight on medication adherence

Springer Medicine

Measuring Alliance and Symptom Severity in Psychotherapy Transcripts Using Bert Topic Modeling

Abstract

Keynote webinar | Spotlight on medication adherence

Springer Medicine

Abstract

Please log in to get access to this content