Top

Administration and Policy in Mental Health and Mental Health Services Research

Published in:

01-05-2016 | Original Paper

Implementing a Measurement Feedback System: A Tale of Two Sites

Authors: Leonard Bickman, Susan R. Douglas, Ana Regina Vides De Andrade, Michele Tomlinson, Alissa Gleacher, Serene Olin, Kimberly Hoagwood

Published in: Administration and Policy in Mental Health and Mental Health Services Research | Issue 3/2016

Abstract

A randomized experiment was conducted in two outpatient clinics evaluating a measurement feedback system called contextualized feedback systems. The clinicians of 257 Youth 11–18 received feedback on progress in mental health symptoms and functioning either every 6 months or as soon as the youth’s, clinician’s or caregiver’s data were entered into the system. The ITT analysis showed that only one of the two participating clinics (Clinic R) had an enhanced outcome because of feedback, and only for the clinicians’ ratings of youth symptom severity on the SFSS. A dose–response effect was found only for Clinic R for both the client and clinician ratings. Implementation analyses showed that Clinic R had better implementation of the feedback intervention. Clinicians’ questionnaire completion rate and feedback viewing at Clinic R were 50 % higher than clinicians at Clinic U. The discussion focused on the differences in implementation at each site and how these differences may have contributed to the different outcomes of the experiment.

Note that this initial version was considered a beta version with software bugs identified and corrected throughout about the first year.

As noted the methods section, participation in the quality improvement initiative was required of agency staff. In addition, the agency issued a mandate about midway through the project for both sites requiring the use of CFS for all eligible cases with CFS implementation data to be used in staff performance evaluations.

American Psychological Association (APA) Presidential Task Force on Evidence-Based Practice. (2006). Evidence-based practice in psychology. American Psychologist, 61(4), 271–285.CrossRef

Arco, L. (1997). Improving program outcome with process-based feedback. Journal of Organizational Behavior Management, 17(1), 37–64.CrossRef

Arco, L. (2008). Feedback for improving staff training and performance in behavioral treatment programs. Behavioral Interventions, 23(1), 39–64.CrossRef

Athay, M. M., Riemer, M., & Bickman, L. (2012). The Symptoms and Functioning Severity Scale (SFSS): Psychometric evaluation and differences of youth, caregiver, and clinician ratings over time. Administration and Policy in Mental Health and Mental Health Services Research, 39(1–2), 13–29.CrossRefPubMedPubMedCentral

Baumberger, B. K. (2014). Understanding and promoting treatment integrity. In L. M. Hagermoser Sanetti & T. R. Kratochwill (Eds.), Treatment integrity: A foundation for evidence-based practice in applied psychology (pp. 35–54). Washington, DC: American Psychological Association.CrossRef

Bickman, L. (2008). Why don’t we have effective mental health services? Administration and Policy in Mental Health and Mental Health Services Research, 35, 437–439.CrossRefPubMed

Bickman, L. & Athay, M. I. (Eds.) (2012). Youth mental health measurement (special issue) Administration and Policy in Mental Health and Mental Health Services Research.

Bickman, L., Athay, M. M., Riemer, M., Lambert, E. W., Kelley, S. D., Breda, C., et al. (Eds.). (2010). Manual of the Peabody Treatment Progress Battery (2nd ed.). Nashville, TN: Vanderbilt University.

Bickman, L., Kelley, D. S., & Athay, M. (2012). The technology of measurement feedback systems. Couple and Family Psychology, 1(4), 274–284.CrossRefPubMedPubMedCentral

Bickman, L., Kelley, S. D., Breda, C., deAndrade, A. R., & Riemer, M. (2011). Effects of routine feedback to clinicians on mental health outcomes of youths: results of a randomized trial. Psychiatric Services, 62(12), 1423–1429.CrossRefPubMed

Bickman, L., Rosof, J., Salzer, M. S., Summerfelt, W. T., Noser, K., Wilson, S. J., & Karver, M. S. (2000). What information do clinicians value for monitoring adolescent client progress and outcome? Professional Psychology: Research and Practice, 31(1), 70–74.CrossRef

Boswell, J. F., Kraus, D. R., Miller, S. D., & Lambert, M. J. (2015). Implementing routine outcome monitoring in clinical practice: Benefits, challenges, and solutions. Psychotherapy research, 25(1), 6–19.CrossRefPubMed

Bruns, E., Hoagwood, K., & Hamilton, J. (2008). State implementation of evidence-based practice for youths, Part I: Responses to the state of the evidence. Journal of the American Academy of Child and Adolescent Psychiatry, 47(4), 369–373.CrossRefPubMed

Cebul, R. D. (2008). Using electronic medical records to measure and improve performance. Transactions of the American Clinical and Climatological Association, 119, 65–75. discussion 75–76.PubMedPubMedCentral

Chorpita, B., Bernstein, A., & Daleiden, E. (2008). Driving with roadmaps and dashboards: Using information resources to structure the decision models in service organizations. Administration and Policy in Mental Health and Mental Health Services Research, 35, 114–123.CrossRefPubMed

Chorpita, B. F., Bernstein, A. D., & Daleiden, E. L. (2011). Empirically guided coordination of multiple evidence-based treatments: An illustration of relevance mapping in children’s mental health services. Journal of Consulting and Clinical Psychology, 79, 470–480.CrossRefPubMed

Douglas, S., Button, S., & Casey, S. E. (2014). Implementing for sustainability: Promoting use of a measurement feedback system for innovation and quality improvement. Administration and Policy in Mental Health and Mental Health Services Research,. doi:10.1007/s10488-014-0607-8.PubMed

Duncan, E. A. S., & Murray, J. (2012). The barriers and facilitators to routine outcome measurement by allied health professionals in practice: A systematic review. BMC Health Services Research, 12, 96.CrossRefPubMedPubMedCentral

Duncan, K., & Pozehl, B. (2000). Effects of performance feedback on patient pain outcomes. Clinical Nursing Research, 9(4), 379–397.CrossRefPubMed

Fitzpatrick, M. (2012). Blurring practice–research boundaries using progress monitoring: A personal introduction to this issue of Canadian psychology. Canadian Psychology/Psychologie Canadienne, 53(2), 75–81.CrossRef

Fixsen, D. L., Naoom, S. F., Blase, K. A., Friedman, R. M., & Wallace, F. (2005). Implementation research: A synthesis of the literature. Tampa, FL: University of South Florida, Louis de la Parte Florida Mental Health Institute National Implementation Research Network (FMHI Publication #231).

Furman, C. E., Adamek, M. S., & Furman, A. G. (1992). The use of an auditory device to transmit feedback to student therapists. Journal of Music Therapy, 29(1), 40.CrossRef

Gleacher, A. A., Nadeem, E., Moy, A. J., Whited, A. L., Albano, A., Radigan, M., et al. (2011). Statewide CBT training for clinicians and supervisors treating youth: The New York State Evidence Based Treatment Dissemination Center. Journal of Emotional and Behavioral Disorders, 19(3), 182–192.CrossRef

Goebel, L. J. (1997). A peer review feedback method of promoting compliance with preventive care guidelines in a resident ambulatory care clinic. The Joint Commission Journal on Quality Improvement, 23(4), 196–202.PubMed

Goodman, J. D., McKay, J. R., & DePhilippis, D. (2013). Progress monitoring in mental health and addiction treatment: A means of improving care. Professional Psychology: Research and Practice, 44(4), 231–246.CrossRef

Hannan, C., Lambert, M., Harmon, C., Nielson, S., Smart, D., Shimokawa, K., et al. (2005). A lab test and algorithms for identifying clients at risk for treatment failure. Journal of Clinical Psychology, 61(2), 155–163.CrossRefPubMed

Hatfield,D., McCullough, L., Frantz, S. H. B., & Krieger, K., Do we know when our clients get worse? An investigation of therapists’ ability to detect negative client change. (2009). Clinical Psychology and Psychotherapy. doi:10.1002/cpp.656.

Hawley, K. M., & Weisz, J. R. (2003). Child, parent, and therapist (dis)agreement on target problems in outpatient therapy: The therapist’s dilemma and its implications. Journal of Consulting and Clinical Psychology, 71(1), 62–70.CrossRefPubMed

Holmboe, E., Scranton, R., Sumption, K., & Hawkins, R. (1998). Effect of medical record audit and feedback on residents’ compliance with preventive health care guidelines. Academic Medicine, 73(8), 901–903.CrossRefPubMed

Howe, A. (1996). Detecting psychological distress: can general practitioners improve their own performance? British Journal of General Practice, 46(408), 407–410.PubMedPubMedCentral

Jensen-Doss, A., & Hawley, K. M. (2010). Understanding barriers to evidence-based assessment: Clinician attitudes toward standardized assessment tools. Journal of Clinical Child and Adolescent Psychology, 39, 885–896.CrossRefPubMedPubMedCentral

Kluger, A. N., & DeNisi, A. (1996). The effects of feedback interventions on performance: A historical review, a meta-analysis, and a preliminary feedback intervention theory. Psychological Bulletin, 119(2), 254–284.CrossRef

Lambert, M. J. (2001). Psychotherapy outcome and quality improvement: introduction to the special section on patient-focused research. Journal of Consulting and Clinical Psychology, 69(2), 147–149.CrossRefPubMed

Lambert, M. J., Hansen, N. B., & Finch, A. E. (2001). Patient-focused research: using patient outcome data to enhance treatment effects. Journal of Consulting and Clinical Psychology, 69(2), 159–172.CrossRefPubMed

Lambert, M. J., Harmon, C., Slade, K., Whipple, J. L., & Hawkins, E. J. (2005). Providing feedback to psychotherapists on their patients’ progress: Clinical results and practice suggestions—Review. Journal of Clinical Psychology, 61(2), 165–174.CrossRefPubMed

Leshan, L. A., Fitzsimmons, M., Marbella, A., & Gottlieb, M. (1997). Increasing clinical prevention efforts in a family practice residency program through CQI methods. The Joint Commission Journal on Quality Improvement, 23(7), 391–400.PubMed

Love, S. M., Koob, J. J., & Hill, L. E. (2007). Meeting the challenges of evidence-based practice: Can mental health therapists evaluate their practice? Brief Treatment and Crisis Intervention, 7(3), 184–193.CrossRef

Martin, A. M., Fishman, R., Baxter, L., & Ford, T. (2011). Practitioners’ attitudes towards the use of standardized diagnostic assessment in routine practice: a qualitative study in two child and adolescent mental health services. Clinical Child Psychology and Psychiatry, 16(3), 407–420.CrossRefPubMed

Mazonson, P. D., Mathias, S. D., Fifer, S. K., Buesching, D. P., Malek, P., & Patrick, D. L. (1996). The mental health patient profile: does it change primary care physicians’ practice patterns? The Journal of the American Board of Family Practice, 9(5), 336–345.PubMed

Moløkken, K. J. & Jørgensen, M. (2003, September). A Review of Surveys on Software Effort Estimation International Symposium on Empirical Software Engineering (ISESE 2003).

Mortenson, B. P., & Witt, J. C. (1998). The use of weekly performance feedback to increase teacher implementation of a pre-referral academic intervention. School Psychology Review, 27, 613–627.

New Freedom Commission on Mental Health. (2003). Achieving the Promise: Transforming Mental Health Care in America. Final Report. DHHS Pub. No. SMA-03-3832. Rockville, MD.

Noell, G. H., & Gansle, K. A. (2014). The use of performance feedback to improve intervention implementation in schools. In L. M. Hagermoser Sanetti & T. R. Kratochwill (Eds.), Treatment integrity: A foundation for evidence-based practice in applied psychology (pp. 161–184). Washington, DC: American Psychological Association.CrossRef

Powell, B. J., McMillen, J. C., Proctor, E. K., Carpenter, C. R., Griffey, R. T., Bunger, A. C., et al. (2012). A compilation of strategies for implementing clinical innovations in health and mental health. Medical Care Research and Review, 69(2), 123–157.CrossRefPubMedPubMedCentral

Riemer, M., & Bickman, L. (2011). Using program theory to link social psychology and program evaluation. In M. M. Mark, S. I. Donaldson, & B. Campbell (Eds.), Social Psychology and Program/Policy Evaluation. New York: Guilford.

Riemer, M., Rosof-Williams, J., & Bickman, L. (2005). Theories related to changing clinician practice. Child and Adolescent Psychiatric Clinics of North America, 14, 241–254.CrossRefPubMed

Robinson, M. B., Thompson, E., & Black, N. A. (1996). Evaluation of the effectiveness of guidelines, audit and feedback: improving the use of intravenous thrombolysis in patients with suspected acute myocardial infarction. International Journal for Quality in Health Care, 8(3), 211–222.CrossRefPubMed

Rokstad, K., Straand, J., & Fugelli, P. (1995). Can drug treatment be improved by feedback on prescribing profiles combined with therapeutic recommendations? A prospective, controlled trial in general practice. Journal of Clinical Epidemiology, 48, 1061–1068.CrossRefPubMed

Rose, D. J., & Church, R. J. (1998). Learning to teach: the acquisition and maintenance of teaching skills. Journal of Behavioral Education, 8(1), 5–35.CrossRef

Rotheram-Borus, M. J., Swendeman, D., & Chorpita, B. F. (2012). Disruptive innovations for designing and diffusing evidence-based interventions. American Psychologist, 67(6), 463–476.CrossRefPubMedPubMedCentral

Sapyta, J., Riemer, M., & Bickman, L. (2005). Feedback to clinicians: Theory, research, and practice. Journal of Clinical Psychology, 61(2), 145–153.CrossRefPubMed

Southam-Gerow, M. A., & Prinstein, M. J. (2014). Evidence base updates: The evolution of the evaluation of psychological treatments for children and adolescents. Journal of Clinical Child and Adolescent Psychology, 43(1), 1–6.CrossRefPubMed

Tabenkin, H., Steinmetz, D., Eilat, Z., Heman, N., Dagan, B., & Epstein, L. (1995). A peer review programme to audit the management of hypertensive patients in family practices in Israel. Family Practice, 12(3), 309–312.CrossRefPubMed

Tuckman, B. W., & Yates, D. (1980). Evaluating the student feedback strategy for changing teacher style. The Journal of Educational Research, 74(2), 74–77.CrossRef

Useem, J. (May 31, 2014). Business school, disrupted. The New York Times.

Wager, K. A., Lee, F. W., & Glaser, J. P. (2009). Health care information systems: A practical approach for health care management (2nd ed.). San-Francisco: Jossey-Bass.

Weisz, J. R., Ng, M. Y., & Bearman, S. K. (2014). Odd couple? Reenvisioning the relation between science and practice in the dissemination-implementation era. Clinical Psychological Science, 2(1), 58–74.CrossRef

Wolpert, M., Curtis-Tyler, K., & Edbrooke-Childs, J. (2014). A qualitative exploration of patient and clinician views on patient reported outcome measures in child mental health and diabetes services. Administration and Policy in Mental Health and Mental Health Services Research, doi:10.1007/s10488-014-0586-9.

Title: Implementing a Measurement Feedback System: A Tale of Two Sites
Authors: Leonard Bickman
Susan R. Douglas
Ana Regina Vides De Andrade
Michele Tomlinson
Alissa Gleacher
Serene Olin
Kimberly Hoagwood
Publication date: 01-05-2016
Publisher: Springer US
Published in: Administration and Policy in Mental Health and Mental Health Services Research / Issue 3/2016
Print ISSN: 0894-587X
Electronic ISSN: 1573-3289
DOI: https://doi.org/10.1007/s10488-015-0647-8

At a glance: The STEP trials

Springer Medicine

Implementing a Measurement Feedback System: A Tale of Two Sites

Abstract

At a glance: The STEP trials

Springer Medicine

Abstract

Please log in to get access to this content

Other articles of this Issue 3/2016

Feedback from Outcome Measures and Treatment Effectiveness, Treatment Efficiency, and Collaborative Practice: A Systematic Review

A Qualitative Exploration of Patient and Clinician Views on Patient Reported Outcome Measures in Child Mental Health and Diabetes Services

Using Patient Reported Outcome Measures to Improve Service Effectiveness (UPROMISE): Training clinicians to Use Outcome Measures in Child Mental Health

Designing Health Information Technologies for Uptake: Development and Implementation of Measurement Feedback Systems in Mental Health Service Delivery

The Contextualized Technology Adaptation Process (CTAP): Optimizing Health Information Technology to Improve Mental Health Systems

When is Sessional Monitoring More Likely in Child and Adolescent Mental Health Services?