Skip to main content
Key Specialties
Cardiology Diabetology Endocrinology Gastroenterology Geriatrics and Gerontology Gynecology and Obstetrics Hematology Infectious Disease Internal Medicine Nephrology Neurology Oncology Pediatrics Respiratory Medicine Rheumatology Surgery
Congress Coverage
2024 ACC Congress 2023 ESMO Congress 2023 EASD Congress 2023 ERS Congress 2023 ESC Congress 2023 EHA Congress 2023 ASCO Annual Meeting
Clinical Collections
Advances in Alzheimer’s

At a glance: The ONWARDS insulin icodec trials

A round-up of the ONWARDS phase 3 clinical trials evaluating the novel weekly insulin icodec in people with diabetes.
ADVANCED SEARCH
Log in
Top
Implementation Science
Published in: Implementation Science 1/2018

Open Access 01-12-2018 | Research

Objective coding of content and techniques in workplace-based supervision of an EBT in public mental health

Authors: Shannon Dorsey, Suzanne E. U. Kerns, Leah Lucid, Michael D. Pullmann, Julie P. Harrison, Lucy Berliner, Kelly Thompson, Esther Deblinger

Published in: Implementation Science | Issue 1/2018

Login to get access

Abstract

Background

Workplace-based clinical supervision as an implementation strategy to support evidence-based treatment (EBT) in public mental health has received limited research attention. A commonly provided infrastructure support, it may offer a relatively cost-neutral implementation strategy for organizations. However, research has not objectively examined workplace-based supervision of EBT and specifically how it might differ from EBT supervision provided in efficacy and effectiveness trials.

Methods

Data come from a descriptive study of supervision in the context of a state-funded EBT implementation effort. Verbal interactions from audio recordings of 438 supervision sessions between 28 supervisors and 70 clinicians from 17 public mental health organizations (in 23 offices) were objectively coded for presence and intensity coverage of 29 supervision strategies (16 content and 13 technique items), duration, and temporal focus. Random effects mixed models estimated proportion of variance in content and techniques attributable to the supervisor and clinician levels.

Results

Interrater reliability among coders was excellent. EBT cases averaged 12.4 min of supervision per session. Intensity of coverage for EBT content varied, with some discussed frequently at medium or high intensity (exposure) and others infrequently discussed or discussed only at low intensity (behavior management; assigning/reviewing client homework). Other than fidelity assessment, supervision techniques common in treatment trials (e.g., reviewing actual practice, behavioral rehearsal) were used rarely or primarily at low intensity. In general, EBT content clustered more at the clinician level; different techniques clustered at either the clinician or supervisor level.

Conclusions

Workplace-based clinical supervision may be a feasible implementation strategy for supporting EBT implementation, yet it differs from supervision in treatment trials. Time allotted per case is limited, compressing time for EBT coverage. Techniques that involve observation of clinician skills are rarely used. Workplace-based supervision content appears to be tailored to individual clinicians and driven to some degree by the individual supervisor. Our findings point to areas for intervention to enhance the potential of workplace-based supervision for implementation effectiveness.

Trial registration

NCT01800266, Clinical Trials, Retrospectively Registered (for this descriptive study; registration prior to any intervention [part of phase II RCT, this manuscript is only phase I descriptive results])
Appendix
Available only for authorised users
Literature
1.
Powell BJ, Waltz TJ, Chinman MJ, Damschroder LJ, Smith JL, Matthieu MM, et al. A refined compilation of implementation strategies: results from the Expert Recommendations for Implementing Change (ERIC) project. Implement Sci. 2015;10:21.CrossRefPubMedPubMedCentral
2.
Beidas RS, Kendall PC. Training therapists in evidence-based practice: a critical review of studies from a systems-contextual perspective. Clin Psychol Sci Pract. 2010;17:1–30.CrossRef
3.
Herschell AD, Kolko DJ, Baumann BL, Davis AC. The role of therapist training in the implementation of psychosocial treatments: a review and critique with recommendations. Clin Psychol Rev. 2010;30:448–66.CrossRefPubMedPubMedCentral
4.
Beidas RS, Edmunds JM, Marcus SC, Kendall PC. Training and consultation to promote implementation of an empirically supported treatment: a randomized trial. Psychiatr Serv. 2012;63:660–5.CrossRefPubMedPubMedCentral
5.
Sholomskas DE, Syracuse-Siewert G, Rounsaville BJ, Ball SA, Nuro KF, Carroll KM. We don’t train in vain: a dissemination trial of three strategies of training clinicians in cognitive–behavioral therapy. J Consult Clin Psychol. 2005;73:106–15.CrossRefPubMedPubMedCentral
6.
Rakovshik SG, McManus F, Vazquez-Montes M, Muse K, Ougrin D. Is supervision necessary? Examining the effects of internet-based CBT training with and without supervision. J Consult Clin Psychol. 2016;84:191–9.CrossRefPubMed
7.
Chorpita BF, Regan J. Dissemination of effective mental health treatment procedures: maximizing the return on a significant investment. Behav Res Ther. 2009;47:990–3.CrossRefPubMed
8.
Nadeem E, Gleacher A, Beidas RS. Consultation as an implementation strategy for evidence-based practices across multiple contexts: unpacking the black box. Adm Policy Ment Health Ment Health Serv Res. 2013;40:439–50.CrossRef
9.
Edmunds JM, Beidas RS, Kendall PC. Dissemination and implementation of evidence–based practices: Training and consultation as implementation strategies. Clin Psychol. Sci Pract 2013;2:152–65.
10.
Miller WR, Yahne CE, Moyers TB, Martinez J, Pirritano M. A randomized trial of methods to help clinicians learn motivational interviewing. J Consult Clin Psychol. 2004;72:1050–62.CrossRefPubMed
11.
Edmunds JM, Kendall PC, Ringle VA, Read KL, Brodman DM, Pimentel SS, et al. An examination of behavioral rehearsal during consultation as a predictor of training outcomes. Adm Policy Ment Health Ment Health Serv Res. 2013;40:456–66.CrossRef
12.
Stirman SW, Pontoski K, Creed T, Xhezo R, Evans AC, Beck AT, et al. A non-randomized comparison of strategies for consultation in a community-academic training program to implement an evidence-based psychotherapy. Adm Policy Ment Health Ment Health Serv Res. 2015;44:55–66.CrossRef
13.
Funderburk B, Chaffin M, Bard E, Shanley J, Bard D, Berliner L. Comparing client outcomes for two evidence-based treatment consultation strategies. J Clin Child Adolesc Psychol. 2014;44:730–41.CrossRefPubMed
14.
Bearman SK, Weisz JR, Chorpita BF, Hoagwood K, Ward A, Ugueto AM, et al. More practice, less preach? The role of supervision processes and therapist characteristics in EBP implementation. Admin Pol Ment Health. 2013;40:518–29.CrossRef
15.
Bearman SK, Schneiderman RL, Zoloth E. Building an evidence base for effective supervision practices: an analogue experiment of supervision to increase EBT fidelity. Adm Policy Ment Health Ment Health Serv Res. 2017;44:293–307.CrossRef
16.
Schoenwald SK, Chapman JE, Kelleher K, Hoagwood KE, Landsverk J, Stevens J, et al. A survey of the infrastructure for children’s mental health services: implications for the implementation of empirically supported treatments (ESTs). Adm Policy Ment Health Ment Health Serv Res. 2008;35:84–97.CrossRef
17.
Dorsey S, Pullmann MD, Deblinger E, Berliner L, Kerns SE, Thompson K, et al. Improving practice in community-based settings: a randomized trial of supervision—study protocol. Implement Sci. 2013;8:89.CrossRefPubMedPubMedCentral
18.
Dorsey S, Pullmann MD, Kerns SEU, Jungbluth N, Meza R, Thompson K, et al. The juggling act of supervision in community mental health: implications for supporting evidence-based treatment. Admin Pol Ment Health. 2017;44:838–52.
19.
Stewart RE, Adams DR, Mandell DS, Hadley TR, Evans AC, Rubin R, et al. The perfect storm: collision of the business of mental health and the implementation of evidence-based practices. Psychiatr Serv Wash DC. 2016;67:159–61.CrossRef
20.
Massatti R, Sweeney H, Panzano P, Roth D. The de-adoption of innovative mental health practices (IMHP): why organizations choose not to sustain an IMHP. Adm Policy Ment Health Ment Health Serv Res. 2008;35:50–65.CrossRef
21.
Stirman SW. The sustainability of new programs and innovations: a review of the empirical literature and recommendations for future research. Implement Sci. 2012;7:1–19.CrossRef
22.
Cohen JA, Mannarino AP. Disseminating and implementing trauma-focused CBT in community settings. Trauma Violence Abuse. 2008;9:214–26.CrossRefPubMed
23.
Dorsey S, Berliner L, Lyon AR, Pullmann MD, Murray LK. A statewide common elements initiative for children’s mental health. J Behav Health Serv Res. 2016;43:246–61.CrossRefPubMedPubMedCentral
24.
Sigel BA, Benton AH, Lynch CE, Kramer TL. Characteristics of 17 statewide initiatives to disseminate trauma-focused cognitive-behavioral therapy (TF-CBT). Psychol Trauma Theory Res Pract Policy. 2013;5:323–33.CrossRef
25.
Hoge MA, Migdole S, Farkas MS, Ponce AN, Hunnicutt C. Supervision in public sector behavioral health: a review. Clin Superv. 2011;30:183–203.CrossRef
26.
Accurso EC, Taylor RM, Garland AF. Evidence-based practices addressed in community-based children’s mental health clinical supervision. Train Educ Prof Psychol. 2011;5:88–96.PubMedPubMedCentral
27.
Roth AD, Pilling S, Turner J. Therapist training and supervision in clinical trials: implications for clinical practice. Behav Cogn Psychother. 2010;38:291–302.CrossRefPubMed
28.
Henggeler S, Schoenwald S, Borduin C, Rowland M, Cunningham P. Multisystemic treatment of antisocial behavior in children and adolescents. New York: Guilford Press; 1998.
29.
Ivers N, Jamtvedt G, Flottorp S, Young JM, Odgaard-Jensen J, French SD, et al. Audit and feedback: effects on professional practice and healthcare outcomes. Cochrane Dat Sys Rev. 2012;6:1–227.
30.
Henggeler SW, Schoenwald SK, Liao JG, Letourneau EJ, Edwards DL. Transporting efficacious treatments to field settings: the link between supervisory practices and therapist fidelity in MST programs. J Clin Child Adolesc Psychol. 2002;31:155–67.CrossRefPubMed
31.
Schoenwald SK, Sheidow AJ, Chapman JE. Clinical supervision in treatment transport: effects on adherence and outcomes. J Consult Clin Psychol. 2009;77:410–21.CrossRefPubMedPubMedCentral
32.
Bradshaw T, Butterworth A, Mairs H. Does structured clinical supervision during psychosocial intervention education enhance outcome for mental health nurses and the service users they work with? J Psychiatr Ment Health Nurs. 2007;14:4–12.CrossRefPubMed
33.
Schoenwald SK, Mehta TG, Frazier SL, Shernoff ES. Clinical supervision in effectiveness and implementation research. Clin Psychol Sci Pract. 2013;20:44–59.CrossRef
34.
Bickman L. Our quality-assurance methods aren’t so sure. Behav Healthc Tomorrow. 2000;9:41–2. 48PubMed
35.
Beidas RS, Marcus S, Aarons GA, Hoagwood KE, Schoenwald S, Evans AC, et al. Predictors of community therapists’ use of therapy techniques in a large public mental health system. JAMA Peds. 2015;169:374–82.CrossRef
36.
Brookman-Frazee L, Haine RA, Baker-Ericzén M, Zoffness R, Garland AF. Factors associated with use of evidence-based practice strategies in usual care youth psychotherapy. Adm Policy Ment Health Ment Health Serv Res. 2010;37:254–69.CrossRef
37.
Cohen JA, Mannarino AP, Deblinger E. Treating trauma and traumatic grief in children and adolescents. New York City: Guilford Press; 2006.
38.
McLeod BD. The therapy process observational coding system for child psychotherapy. Los Angeles: University of California, Los Angeles; 2001.
39.
McLeod BD, Weisz JR. The therapy process observational coding system for child psychotherapy strategies scale. J Clin Child Adolesc Psychol. 2010;39:436–43.CrossRefPubMed
40.
Garland AF, Brookman-Frazee L, Hurlburt MS, Accurso EC, Zoffness RJ, Haine-Schlagel R, et al. Mental health care for children with disruptive behavior problems: a view inside therapists’ offices. Psychiatr Serv. 2010;61:788–95.CrossRefPubMedPubMedCentral
41.
Hanson RF, Gros KS, Davidson TM, Barr S, Cohen J, Deblinger E, et al. National trainers’ perspectives on challenges to implementation of an empirically-supported mental health treatment. Admin Pol Ment Health. 2014;41:522–34.CrossRef
42.
Milne DL, & Reiser RP. A manual for evidence-based CBT supervision. Wiley; 2017.
43.
Milne D, Aylott H, Fitzpatrick H, Ellis MV. How does clinical supervision work? Using a “best evidence synthesis” approach to construct a basic model of supervision. Clin Superv. 2008;27:170–90.CrossRef
44.
Milne DL, Reiser RP, Cliffe T, Breese L, Boon A, Raine R, et al. A qualitative comparison of cognitive-behavioural and evidence-based clinical supervision. Cogn Behav Ther. 2011;4:152–66.
45.
Nakamura BJ, Selbo-Bruns A, Okamura K, Chang J, Slavin L, Shimabukuro S. Developing a systematic evaluation approach for training programs within a train-the-trainer model for youth cognitive behavior therapy. Behav Res Ther. 2014;53:10–9.CrossRefPubMed
46.
Cicchetti DV. Guidelines, criteria, and rules of thumb for evaluating normed and standardized assessment instruments in psychology. Psychol Assess. 1994;6:284–90.CrossRef
47.
Cicchetti DV, Feinstein AR. High agreement but low kappa: II. Resolving the paradoxes. J Clin Epidemiol. 1990;43:551–8.CrossRefPubMed
48.
Hutchinson TP. Kappa muddles together two sources of disagreement: tetrachoric correlation is preferable. Res Nurs Health. 1993;16:313–6.CrossRefPubMed
49.
Banerjee M, Capozzoli M, Mcsweeney L, Sinha D. Beyond kappa: a review of interrater agreement measures. Can J Stat. 1999;27:3–23.CrossRef
50.
Kolko DJ, Cohen JA, Mannarino AP, Baumann BL, Knudsen K. Community treatment of child sexual abuse: a survey of practitioners in the National Child Traumatic Stress Network. Adm Policy Ment Health Ment Health Serv Res. 2009;36:37–49.CrossRef
51.
Borntrager C, Chorpita BF, Higa-McMillan CK, Daleiden EL, Starace N. Usual care for trauma-exposed youth: are clinician-reported therapy techniques evidence-based? Child Youth Serv Rev. 2013;35:133–41.CrossRef
52.
Forgatch M, Patterson G, Gewirtz A. Looking forward: the promise of widespread implementation of parent training programs. Perspect Psychol Sci. 2013;8:682–94.CrossRefPubMedPubMedCentral
53.
Southam-Gerow MA, Weisz JR, Kendall PC. Youth with anxiety disorders in research and service clinics: examining client differences and similarities. J Clin Child Adolesc Psychol. 2003;32:375–85.CrossRefPubMed
54.
Lyon AR, Dorsey S, Pullmann M, Silbaugh-Cowdin J, Berliner L. Clinician use of standardized assessments following a common elements psychotherapy training and consultation program. Adm Policy Ment Health Ment Health Serv Res. 2014;42:47–60.CrossRef
55.
Beidas RS, Maclean JC, Fishman J, Dorsey S, Schoenwald SK, Mandell DS, Marcus SC. A randomized trial to identify accurate and cost-effective fidelity measurement methods for cognitivebehavioral therapy: Project FACTS study protocol. BMC Psychiatry. 2016;16(1):323. https://​doi.​org/​10.​1186/​s12888-016-1034-z.
56.
Schoenwald SK. It’s a bird, it’s a plane, it’s... fidelity measurement in the real world. Clin Psychol Sci Pract. 2011;18:142–7.CrossRef
57.
Schoenwald SK, Garland AF, Chapman JE, Frazier SL, Sheidow AJ, Southam-Gerow MA. Toward the effective and efficient measurement of implementation fidelity. Adm Policy Ment Health Ment Health Serv Res. 2011;38:32–43.CrossRef
58.
Wiltsey Stirman S, Gutner CA, Gamarra J, Vogt D, Suvak M, Watchen JS, et al. Leveraging routine clinical materials to assess fidelity to an evidence-based psychotherapy. In: Lewis CC, editor. 3rd Bienn. Conf. Soc. Implement. Res. Collab. SIRC 2015 Adv. Effic. Methodol. Community Partnersh. Team Sci. A12. Seattle, WA; 2016.
59.
Beidas RS, Cross W, Dorsey S. Show me, don’t tell me: behavioral rehearsal as a training and analogue fidelity tool. Cogn Behav Pract. 2014;21:1–11.CrossRefPubMed
60.
Kavanagh DJ, Spence S, Sturk H, Strong J, Wilson J, Worrall L, et al. Outcomes of training in supervision: randomised controlled trial. Aust Psychol. 2008;43:96–104.CrossRef
61.
Beidas RS, Marcus S, Wolk CB, Powell B, Aarons GA, Evans AC, et al. A prospective examination of clinician and supervisor turnover within the context of implementation of evidence-based practices in a publicly-funded mental health system. Adm Policy Ment Health Ment Health Serv Res. 2016;43:640–9.CrossRef
Metadata
Title
Objective coding of content and techniques in workplace-based supervision of an EBT in public mental health
Authors
Shannon Dorsey
Suzanne E. U. Kerns
Leah Lucid
Michael D. Pullmann
Julie P. Harrison
Lucy Berliner
Kelly Thompson
Esther Deblinger
Publication date
01-12-2018
Publisher
BioMed Central
Published in
Implementation Science / Issue 1/2018
Electronic ISSN: 1748-5908
DOI
https://doi.org/10.1186/s13012-017-0708-3

Other articles of this Issue 1/2018

Implementation Science 1/2018 Go to the issue

Correction

Correction to: barriers and facilitators for the management of vertigo: a qualitative study with primary care providers

Research

Improving care at scale: process evaluation of a multi-component quality improvement intervention to reduce mortality after emergency abdominal surgery (EPOCH trial)

Research

Implementation between text and work—a qualitative study of a readmission prevention program targeting elderly patients

Study protocol

Sustainable reduction of antibiotic-induced antimicrobial resistance (ARena) in German ambulatory care: study protocol of a cluster randomised trial

Correction

Correction to: Improving care at scale: process evaluation of a multi-component quality improvement intervention to reduce mortality after emergency abdominal surgery (EPOCH trial)

Research

Linking molar organizational climate and strategic implementation climate to clinicians’ use of evidence-based psychotherapy techniques: cross-sectional and lagged analyses from a 2-year observational study