Top

Administration and Policy in Mental Health and Mental Health Services Research

Published in:

01-09-2017 | Original Paper

Reliability of Therapist Effects in Practice-Based Psychotherapy Research: A Guide for the Planning of Future Studies

Authors: Anne-Katharina Schiefele, Wolfgang Lutz, Michael Barkham, Julian Rubel, Jan Böhnke, Jaime Delgadillo, Mark Kopta, Dietmar Schulte, David Saxon, Stevan L. Nielsen, Michael J. Lambert

Published in: Administration and Policy in Mental Health and Mental Health Services Research | Issue 5/2017

Abstract

This paper aims to provide researchers with practical information on sample sizes for accurate estimations of therapist effects (TEs). The investigations are based on an integrated sample of 48,648 patients treated by 1800 therapists. Multilevel modeling and resampling were used to realize varying sample size conditions to generate empirical estimates of TEs. Sample size tables, including varying sample size conditions, were constructed and study examples given. This study gives an insight into the potential size of the TE and provides researchers with a practical guide to aid the planning of future studies in this field.

Available only for authorised users

To reduce the complexity of this paper, AIC values are not reported in detail but can be requested from the first author.

Because of the nested data structure, three sample size parameters must be considered when weighting the arithmetic mean: (1) number of patients (2) number of therapists (3) number of patients per therapist. The mean TE weighted for the number of patients is 7.2 %, the mean TE weighted for the number of therapists is 7.1 and 5.75 % if the mean TE is weighted by the mean number of patients per therapist.

Due to shortage of space and clarity, not all sample size conditions are displayed in Table 3.

The explained variance (R²) was calculated in accordance with the recommendations of Hox (2010).

Adelson, L. J., & Owen, J. (2012). Bringing the psychotherapist back: Basic concepts for reading articles examining therapist effects using multilevel modeling. Psychotherapy, 49(2), 152–162. doi:10.1037/a0023990.CrossRefPubMed

Baldwin, S. A., & Imel, Z. E. (2013). Therapist effects: Findings and methods. In M. J. Lambert (Ed.), Bergin and Garfield’s handbook of psychotherapy and behavior change (6th ed., pp. 258–297). New York, NY: Wiley.

Barkham, M., Margison, F., Leach, C., Lucock, M., Mellor-Clark, J., Evans, C.,…McGrath, G. (2001). Service profiling and outcomes benchmarking using the CORE-OM: Towards practice-based evidence in the psychological therapies. Journal of Consulting and Clinical Psychology, 69, 184–196. doi:10.1037/0022-006X.69.2.184.

Bates, D., Maechler, M., & Bolker, B. (2013). lme4: Linear mixed-effects models using s4classes [Software-Handbook]. (R package version 0.999999-2)

Bell, B. A., Morgan, G. B., Schoeneberger, J. A., Kromrey, J. D., & Ferron, J. M. (2014). How low can you go? An investigation of the influence of sample size and model complexity on point and interval estimates in two-level linear models. Methodology: European Journal of Research Methods for the Behavioral and Social Sciences, 10(1), 1–11. doi:10.1027/1614-2241/a000062.CrossRef

Beutler, L. E., Malik, A., Alimohamed, S., Harwood, T. M., Talebi, H., Noble, W., & Wong, E. (2004). Therapist variables. In M. J. Lambert (Ed.), Bergin and Garfield’s handbook of psychotherapy and behavior change (5th ed., pp. 227–306). New York, NY: Wiley.

Boswell, J. F., Castonguay, L. G., & Wasserman, R. H. (2010). Effects of psychotherapy training and intervention use on session outcome. Journal of Consulting and Clinical Psychology, 78, 717–723. doi:10.1037/a0020088.CrossRefPubMed

Burton, P., Gurrin, L., & Sly, P. (1998). Extending the simple linear regression model to account for correlated responses: An introduction to gear generalized estimating equations and multilevel mixed modelling. Statistics in Medicine, 17, 1261–1291.CrossRefPubMed

Crits-Christoph, P., Baranackie, K., Kurcias, J., Beck, A., Carroll, K., Perry, K.,… Zitrin, C. (1991). Meta-analysis of therapist effects in psychotherapy outcome studies. Psychotherapy Research, 1, 81–91. doi:10.1080/10503309112331335511.

Crits-Christoph, P., & Gallop, R. (2006). Therapist effects in the National Institute of Mental Health Treatment of Depression Collaborative Research Program and other psychotherapy studies. Psychotherapy Research, 16, 178–181. doi:10.1080/10503300500265025.CrossRef

De Jong, K., Moerbeek, M., & Van der Leeden, R. (2010). A priori power analysis in longitudinal three-level multilevel models: an example with therapist effects. Psychotherapy Research, 20(3), 273–284. doi:10.1080/10503300903376320.CrossRefPubMed

Derogatis, L. R. (1975). Brief symptom inventory. Baltimore: Clinical Psychometric Research.

Derogatis, L. R. (1977). SCL-90-R: Administration, Scoring and Procedures Manual I. Baltimore: Clinical Psychometric Research.

Dinger, U., Strack, M., Leichsenring, F., Wilmers, F., & Schauenburg, H. (2008). Therapist effects on outcome and alliance in inpatient psychotherapy. Journal of Clinical Psychology, 64, 344–354. doi:10.1002/jclp.20443.CrossRefPubMed

Ehlers, A., Grey, N., Wild, J., Stott, R., Liness, S., Deale, A.,… & Clark, D. M. (2013). Implementation of cognitive therapy for PTSD in routine clinical care: effectiveness and moderators of outcome in a consecutive sample. Behaviour Research and Therapy, 51, 742–752. doi:10.1016/j.brat.2013.08.006.

Eldridge, S. M., Ashby, D., & Kerry, S. (2006). Sample size for cluster randomized trials: effect of coefficient of variation of cluster size and analysis method. International Journal of Epidemiology, 35(5), 1292–1300. doi:10.1093/ije/dyl129.CrossRefPubMed

Elkin, I., Falconnier, L., & Martinovich, Z. (2007). Misrepresentations in Wampold and Bolt’s critique of Elkin, Falconnier, Martinovich, and Mahoney’s study of therapist effects. Psychotherapy Research, 17, 253–256. doi:10.1080/10503300601039816.CrossRef

Elkin, I., Falconnier, L., Martinovich, Z., & Mahoneya, C. (2006). Therapist effects in the National Institute of Mental Health Treatment of Depression Collaborative Research Program. Psychotherapy Research, 16, 144–160. doi:10.1080/10503300500268540.CrossRef

Elkin, I., Shea, M. T., Watkins, J. T., Imber, S. D., Sotsky, S. M., Collins, J. F., & Parloff, M. B. (1989). National Institute of Mental Health Treatment of Depression Collaborative Research Program. General effectiveness of treatments. Archives of General Psychiatry, 46, 971–982. doi:10.1001/archpsyc.1989.01810110013002.CrossRefPubMed

Evans, C., Connell, J., Barkham, M., Margison, F., McGrath, G., Mellor-Clark, J., & Audin, K. (2002). Toward a standardized brief outcome measure: Psychometric properties and utility of the CORE-OM. The British Journal of Psychiatry, 180, 51–60. doi:10.1192/bjp.180.1.51.CrossRefPubMed

Franke, G. (2000). BSI: Brief Symptom Inventory von L.R. Derogatis (Kurzform der SCL-90-R) – Deutsche Version. Beltz Test GmbH.

Gao, F., Earnest, A., Matchar, D. B., Campbell, M. J., & Machin, D. (2015). Sample size calculations for the design of cluster randomized trials: A summary of methodology. Contemporary Clinical Trials, 42, 41–50.CrossRefPubMed

Gardiner, J. C., Luo, Z., & Roman, L. A. (2009). Fixed effects, random effects and GEE: What are the differences? Statistics in Medicine, 28, 221–239. doi:10.1002/sim.3478.CrossRefPubMed

Garfield, S. L. (1997). The therapist as a neglected variable in psychotherapy research. Clinical Psychology: Science and Practice, 4, 40–43. doi:10.1111/j.1468-2850.1997.tb00097.x.CrossRef

Hofmann, S. G., & Barlow, D. H. (2014). Evidence-based psychological interventions and the common factors approach: the beginnings of a rapprochement? Psychotherapy, 5, 510–513. doi:10.1037/a0037045.CrossRef

Howard, K., Brill, P., Lueger, R., O’Mahoney, M. & Grissom, G. (1993b). Compass outpatient tracking assessment: Psychometric properties. Integra.

Howard, K., Lueger, R., Maling, M., & Martinovich, Z. (1993a). A phase model of psychotherapy outcome: Causal mediation of change. Journal of Consulting and Clinical Psychology, 61, 678–685. doi:10.1037/0022-006X.61.4.678.CrossRefPubMed

Howard, K. I., Moras, K., Brill, P. L., Martinovich, Z., & Lutz, W. (1996). Evaluation of psychotherapy: Efficacy, effectiveness, and patient progress. American Psychologist, 51, 1059–1064. doi:10.1037/0003-066X.51.10.1059.CrossRefPubMed

Hox, J. (2010). Multilevel analysis: Techniques and applications (2nd ed.). England: Routledge.CrossRef

Huppert, J. D., Bufka, L. F., Barlow, D. H., Gorman, J. M., Shear, M. K., & Woods, S. W. (2001). Therapists, therapist variables, and cognitive-behavioral therapy outcome in a multicenter trial for panic disorder. Journal of Consulting and Clinical Psychology, 69, 747–755. doi:10.1037/0022-006X.69.5.747.CrossRefPubMed

Kazdin, A. E., & Bass, D. (1989). Power to detect differences between alternative treatments in comparative psychotherapy outcome research. Journal of Consulting and Clinical Psychology, 57, 138–147. doi:10.1037/0022-006X.57.1.138.CrossRefPubMed

Kim, D., Wampold, B. E., & Bolt, D. M. (2006). Therapist effects in psychotherapy: A random-effects modeling of the National Institute of Mental Health Treatment of Depression Collaborative Research Program data. Psychotherapy Research, 16, 161–172. doi:10.1080/10503300500264911.CrossRef

Kopta, S., & Lowry, J. (2002). Psychometric evaluation of the behavioral health questionnaire-20: A brief instrument for assessing global mental health and the three phases of psychotherapy outcome. Psychotherapy Research, 12, 413–426. doi:10.1093/ptr/12.4.413.CrossRef

Kreft, I. G. G. (1996). Are multilevel techniques necessary? An overview, including imulation studies. Unpublished manuscript. Los Angeles: University of California, Department of Statistics.

Kroenke, K., Spitzer, R. L., & Williams, J. B. (2001). The PHQ-9: Validity of a brief depression severity measure. Journal of General Internal Medicine, 16, 606–613. doi:10.1046/j.1525-1497.2001.016009606.x.CrossRefPubMedPubMedCentral

Kuznetsova, A., Brockhoff, P. B., & Christensen, R. H. B. (2014). lmerTest: Tests for random and fixed effects for linear mixed effect models (lmer objects of lme4 package). (R package version 2.0-6)

Lambert, M. J. (1992). Psychotherapy outcome research: Implications for integrative and eclectic therapists. In J. C. Norcross & M. R. Goldfried (Eds.), Handbook of psychotherapy integration. New York: Basic Books.

Lambert, M. J. (2004). Administration and scoring manual for the OQ-45.2 (outcome questionnaire). OQ Measures, LLC.

Lambert, M. J., Hansen, N. B., Umphress, V., Lunnen, K., Okiishi, J., Burlingame, G. M., et al. (1996). Administration and scoring manual for the Outcome Questionnaire (OQ-45.2). Stevenson MD: American Professional Credentialing Services.

Lueger, R. J., Howard, K. I., Martinovich, Z., Lutz, W., Anderson, E. E., & Grissom, G. (2001). Assessing treatment progress of individual patients using expected treatment response models. Journal of Consulting and Clinical Psychology, 69, 150–158. doi:10.1037/0022-006X.69.2.150.CrossRefPubMed

Lutz, W., & Barkham, M. (2015). Therapist effects. The encyclopedia of clinical psychology. Blackwell: Wiley.

Lutz, W., Böhnke, J. R., & Köck, K. (2011). Lending an ear to feedback systems: Evaluation of recovery and non-response in psychotherapy in a German outpatient setting. Community Mental Health Journal, 47, 311–317. doi:10.1007/s10597-010-9307-3.CrossRefPubMed

Lutz, W., Leon, S. C., Martinovich, Z., Lyons, J. S., & Stiles, W. B. (2007). Therapist effects in outpatient psychotherapy: A three-level growth curve approach. Journal of Counseling Psychology, 54, 32–39. doi:10.1037/0022-0167.54.1.32.CrossRef

Lyons, J. S., Howard, K. I., O’Mahoney, M. T., & Lish, J. (1997). The measurement and management of clinical outcomes in mental health services. New York, NY: Wiley.

Maas, C. J. M., & Hox, J. J. (2005). Sufficient sample sizes for multilevel modeling. Methodology, 1, 86–92. doi:10.1027/1614-1881.1.3.86.CrossRef

Manea, L., Gilbody, S., & McMillan, D. (2012). Optimal cut-off scores for diagnosing depression with the Patient Health Questionnaire (PHQ-9): A meta-analysis. Canadian Medical Association Journal, 184(3), E191–E196. doi:10.1503/cmaj.110829.CrossRefPubMed

Moerbeek, M. (2014). Multilevel modeling in the context of growth modeling. Annals of Nutrition & Metabolism, 65, 121–128. doi:10.1159/000360485.CrossRef

Musca, S. C., Kamiejski, R., Nugier, A., Méot, A., Er-Rafiy, A., & Brauer, M. (2011). Data with hierarchical structure: impact of intraclass correlation and sample size on Type-I error. Frontiers in Psychology, 2, 1–6. doi:10.3389/fpsyg.2011.00074.CrossRef

National Institute for Health and Care Excellence (2011). Common mental health disorders: Identification and pathways to care. [CG123]. London: National Institute for Health and Care Excellence. Retrieved on 21/04/2015 from http://www.nice.org.uk/guidance/CG123.

Okiishi, J. C., Lambert, M. J., Eggett, D., Nielsen, L., & Dayton, D. D. (2006). An analysis of therapist treatment effects: Toward providing feedback to individual therapists on their clients' psychotherapy outcome. Journal of Clinical Psychology, 62(9), 1157–1172. doi:10.1002/jclp.20272.CrossRefPubMed

Okiishi, J., Lambert, M. J., Nielsen, S. L., & Ogles, B. M. (2003). Waiting for supershrink: An empirical analysis of therapist effects. Clinical Psychology & Psychotherapy, 10, 361–373. doi:10.1002/cpp.383.CrossRef

Owen, J., Tao, K., & Rodolfa, E. (2010). Microaggressions and women in short-term psychotherapy: Initial evidence. The Counseling Psychologist, 38, 923–946. doi:0.1177/0011000010376093.CrossRef

R Development Core Team (2014). R [Computer software]. Retrieved from http://www.R-project.org/.

Raudenbush, S., & Bryk, A. (2002). Hierarchical linear models (2nd ed.). Newbury Park, CA: Sage.

Ricks, D. F. (1974). Supershrink: Methods of a therapist judged successful on the basis of adult outcomes of adolescent patients. In D. F. Ricks, M. Roff, & A. Thomas (Eds.), Life history research in psychopathology (Vol. 3, pp. 275–297). Minneapolis: University of Minnesota Press.

Saxon, D., & Barkham, M. (2012). Patterns of therapist variability: Therapist effects and the contribution of patient severity and risk. Journal of Consulting and Clinical Psychology, 80, 535–546. doi:10.1037/a0028898.CrossRefPubMed

Shoukri, M. M., Asyali, M. H., & Donner, A. (2004). Sample size requirements for the design of reliability study: Review and new results. Statistical Methods in Medical Research, 13, 251–271. doi:10.1191/0962280204sm365ra.CrossRef

Wampold, B. E., & Bolt, D. M. (2006). Therapist effects: Clever ways to make them (and everything else) disappear. Psychotherapy Research, 16, 184–187. doi:10.1080/10503300500265181.CrossRef

Title: Reliability of Therapist Effects in Practice-Based Psychotherapy Research: A Guide for the Planning of Future Studies
Authors: Anne-Katharina Schiefele
Wolfgang Lutz
Michael Barkham
Julian Rubel
Jan Böhnke
Jaime Delgadillo
Mark Kopta
Dietmar Schulte
David Saxon
Stevan L. Nielsen
Michael J. Lambert
Publication date: 01-09-2017
Publisher: Springer US
Published in: Administration and Policy in Mental Health and Mental Health Services Research / Issue 5/2017
Print ISSN: 0894-587X
Electronic ISSN: 1573-3289
DOI: https://doi.org/10.1007/s10488-016-0736-3

Keynote webinar | Spotlight on medication adherence

Springer Medicine

Reliability of Therapist Effects in Practice-Based Psychotherapy Research: A Guide for the Planning of Future Studies

Abstract

Keynote webinar | Spotlight on medication adherence

Springer Medicine

Abstract

Please log in to get access to this content

Other articles of this Issue 5/2017

Therapist-Specific Factors and Psychotherapy Outcomes of Adult and Youth Clients Seen in a Psychology Training Clinic

Understanding the Therapist Contribution to Psychotherapy Outcome: A Meta-Analytic Approach

Harnessing the Therapist Effect in Patient-Centered Mental Health Care Decision Making

A 20-Site Comparison of Transition-in-Place Versus Traditional Transitional Housing Programs

The Therapist’s Role in Effective Marriage and Family Therapy Practice: The Case for Evidence Based Therapists

Leveraging After-School Programs to Minimize Risks for Internalizing Symptoms Among Urban Youth: Weaving Together Music Education and Social Development