Top

Administration and Policy in Mental Health and Mental Health Services Research

Published in:

01-09-2009 | Original Paper

Individualization and Quality Improvement: Two New Scales to Complement Measurement of Program Fidelity

Authors: Gary R. Bond, Robert E. Drake, Charles A. Rapp, Gregory J. McHugo, Haiyi Xie

Published in: Administration and Policy in Mental Health and Mental Health Services Research | Issue 5/2009

Abstract

Fidelity scales have been widely used to assess program adherence to the principles of an evidence-based practice, but they do not measure important aspects of quality of care. Pragmatic scales measuring clinical quality of services are needed to complement fidelity scales measuring structural aspects of program implementation. As part of the instrumentation developed for the National Implementing Evidence-Based Practices Project, we piloted a new instrument with two 5-item quality scales, Individualization (a client-level quality scale) and Quality Improvement (an organizational-level quality scale). Pairs of independent fidelity assessors conducted fidelity reviews in 49 sites in 8 states at baseline and at four subsequent 6-month intervals over a 2-year follow-up period. The assessors followed a standardized protocol to administer these quality scales during daylong site visits; during these same visits they assessed programs on fidelity to the evidence-based practice that the site was seeking to implement. Assessors achieved acceptable interrater reliability for both Individualization and Quality Improvement. Principal components factor analysis confirmed the 2-scale structure. The two scales were modestly correlated with each other and with the evidence-based practice fidelity scales. Over the first year, Individualization and Quality Improvement improved, but showed little or no improvement during the last year of follow-up. The two newly developed scales showed adequate psychometric properties in this preliminary study, but further research is needed to assess their validity and utility in routine clinical practice.

Adams, N., & Grieder, D. (2004). Treatment planning for person-centered care: The road to mental health and addiction recovery. Amsterdam: Elsevier Academic Press.

Arns, P. G., & Linney, J. A. (1995). The relationship of service individualization to client functioning in programs for severely mentally ill persons. Community Mental Health Journal, 31, 127–137. doi:10.1007/BF02188762.PubMedCrossRef

Batalden, P. B., & Stoltz, P. K. (1993). A framework for the continual improvement of health care: Building and applying professional and improvement knowledge to test changes in daily work. The Joint Commission Journal on Quality Improvement, 19, 424–452.PubMed

Becker, D. R., Baker, S. R., Carlson, L., Flint, L., Howell, R., Lindsay, S., et al. (2007). Critical strategies for implementing supported employment. Journal of Vocational Rehabilitation, 27, 13–20.

Becker, D. R., Smith, J., Tanzman, B., Drake, R. E., & Tremblay, T. (2001). Fidelity of supported employment programs and employment outcomes. Psychiatric Services (Washington, D.C.), 52, 834–836. doi:10.1176/appi.ps.52.6.834.

Becker, D. R., Xie, H., McHugo, G. J., Halliday, J., & Martinez, R. A. (2006). What predicts supported employment program outcomes? Community Mental Health Journal, 42, 303–313. doi:10.1007/s10597-006-9037-8.PubMedCrossRef

Bond, G. R., Evans, L., Salyers, M. P., Williams, J., & Kim, H. K. (2000). Measurement of fidelity in psychiatric rehabilitation. Mental Health Services Research, 2, 75–87. doi:10.1023/A:1010153020697.PubMedCrossRef

Bond, G. R., Drake, R. E., McHugo, G. J., Rapp, C. A., & Whitley, R. (2009). Strategies for improving fidelity in the National Evidence-Based Practices Project. Research on Social Work Practice. doi:10.1177/1049731509335531.

Bond, G. R., McHugo, G. J., Becker, D. R., Rapp, C. A., & Whitley, R. (2008). Fidelity of supported employment: Lessons learned from the National Evidence-Based Practices Project. Psychiatric Rehabilitation Journal, 31, 300–305. doi:10.2975/31.4.2008.300.305.PubMedCrossRef

Bond, G. R., & Salyers, M. P. (2004). Prediction of outcome from the Dartmouth ACT fidelity scale. CNS Spectrums, 9, 937–942.PubMed

Bond, G. R., Vogler, K. M., Resnick, S. G., Evans, L. J., Drake, R. E., & Becker, D. R. (2001). Dimensions of supported employment: Factor structure of the IPS Fidelity Scale. Journal of Mental Health, 10, 383–393. doi:10.1080/09638230120041146.CrossRef

Brunette, M. F., Asher, D., Whitley, R., Lutz, W. J., Wieder, B. L., Jones, A. M., et al. (2008). Implementation of integrated dual disorders treatment: A qualitative analysis of facilitators and barriers. Psychiatric Services (Washington, D.C.), 59, 989–995. doi:10.1176/appi.ps.59.9.989.

Burns, T., Catty, J., Dash, M., Roberts, C., Lockwood, A., & Marshall, M. (2007). Use of intensive case management to reduce time in hospital in people with severe mental illness: Systematic review and meta-regression. British Medical Journal, 335, 336. doi:310.1136/bmj.39251.599259.599255. (18 August, published 13 July 2007).PubMedCrossRef

Campbell, D. T., & Fiske, D. W. (1959). Convergent and discriminant validation by the multitrait-multimethod matrix. Psychological Bulletin, 56, 81–105. doi:10.1037/h0046016.PubMedCrossRef

Catty, J., Lissouba, P., White, S., Becker, T., Drake, R. E., Fioritti, A., et al. (2008). Predictors of employment for people with severe mental illness: Results of an international six-centre RCT. The British Journal of Psychiatry, 192, 224–231. doi:10.1192/bjp.bp.107.041475.PubMedCrossRef

Clarke, S. P., Crowe, T. P., Oades, L. G., & Deane, F. P. (2009). Do goal-setting interventions improve the quality of goals in mental health services? Psychiatric Rehabilitation Journal, 32, 292–299. doi:10.2975/32.4.2009.292.299.PubMedCrossRef

Cook, J. A. (2004). Blazing trails: Using evidence-based practice and stakeholder consensus to enhance psychosocial rehabilitation services in Texas. Psychiatric Rehabilitation Journal, 27, 305–318. doi:10.2975/27.2004.305.306.PubMedCrossRef

DeLuca, N. (2007). Consensus on the key ingredients of person-centered treatment planning for adults with severe mental illness. Unpublished dissertation, Indiana University-Purdue University Indianapolis, Indianapolis.

Deming, W. E. (1986). Out of crisis. Cambridge, MA: Massachusetts Institute of Technology, Center for Advanced Engineering Study.

Drake, R. E., Bond, G. R., & Essock, S. M. (2009). Implementing evidence-based practices for people with schizophrenia. Schizophrenia Bulletin, 35, online.

Drake, R. E., Goldman, H. H., Leff, H. S., Lehman, A. F., Dixon, L., Mueser, K. T., et al. (2001). Implementing evidence-based practices in routine mental health service settings. Psychiatric Services (Washington, D.C.), 52, 179–182. doi:10.1176/appi.ps.52.2.179.

Drake, R. E., Rosenberg, S. D., Teague, G. B., Bartels, S. J., & Torrey, W. C. (2003). Fundamental principles of evidence-based medicine applied to mental health care. The Psychiatric Clinics of North America, 24, 811–820. doi:10.1016/S0193-953X(03)00063-7.CrossRef

Druss, B., Rosenheck, R., & Stolar, M. (1999). Patient satisfaction and administrative measures as indicators of the quality of mental health care. Psychiatric Services (Washington, D.C.), 50, 1053–1058.

Finnerty, M. T., Rapp, C. A., Bond, G. R., Lynde, D. W., Ganju, V. J., & Goldman, H. H. (2009). The State Health Authority Yardstick (SHAY). Community Mental Health Journal. doi:10.1007/s10597-009-9181-z.

Gowdy, E., & Rapp, C. A. (1989). Managerial behavior: The common denominators of successful community based programs. Psychosocial Rehabilitation Journal, 13(2), 31–51.

Hall, L. L., Graf, A. C., Fitzpatrick, M. J., Lane, T., & Birkel, R. C. (2003). Shattered lives: Results of a national survey of NAMI members living with mental illness and their families. Arlington, VA: NAMI/TRIAD (Treatment/Recovery Information and Advocacy Data Base).

Harkness, D. R. (1997). Testing interactional social work theory: A panel analysis of supervised practice and outcomes. The Clinical Supervisor, 15, 33–35. doi:10.1300/J001v15n01_03.CrossRef

Harkness, D. R., & Hensley, H. (1991). Changing the focus of social work supervision: Effects on client satisfaction and generalized contentment. Social Work, 36, 506–512.PubMed

Henggeler, S. W., Pickrel, S. G., & Brondino, M. J. (1999). Multisystemic treatment of substance-abusing and -dependent delinquents: Outcomes, treatment fidelity, and transportability. Mental Health Services Research, 1, 171–184. doi:10.1023/A:1022373813261.PubMedCrossRef

Hermann, R. C. (2005). Improving mental healthcare. A guide to measurement-based quality improvement. Washington, DC: American Psychiatric Publishing.

Jerrell, J. M., & Ridgely, M. S. (1999). Impact of robustness of program implementation on outcomes of clients in dual diagnosis programs. Psychiatric Services (Washington, D.C.), 50, 109–112.

Lehman, A. F., Kreyenbuhl, J., Buchanan, R. W., Dickerson, F. B., Dixon, L. B., Goldberg, R., et al. (2004). The schizophrenia Patient Outcomes Research Team (PORT): Updated treatment recommendations 2003. Schizophrenia Bulletin, 30, 193–217.PubMed

Lehman, A. F., & Steinwachs, D. M. (1998). Patterns of usual care for schizophrenia: Initial results from the Schizophrenia Patient Outcomes Research Team (PORT) client survey. Schizophrenia Bulletin, 24, 11–23.PubMed

Littell, R. C., Milliken, G. A., Stroup, W. W., Wolfinger, R. D., & Schabenberber, O. (2006). SAS system for mixed models (2nd ed.). Cary, NC: SAS Publishing.

Mancini, A. D., Moser, L. L., Whitley, R., McHugo, G. J., Bond, G. R., Finnerty, M. T., et al. (2009). Assertive community treatment: Facilitators and barriers to implementation in routine mental health settings. Psychiatric Services, 60(2), 189–195.PubMedCrossRef

Marshall, T., Rapp, C. A., Becker, D. R., & Bond, G. R. (2008). Key factors for implementing supported employment. Psychiatric Services (Washington, D.C.), 59, 886–892. doi:10.1176/appi.ps.59.8.886.

McGraw, K. O., & Wong, S. P. (1996). Forming inferences about some intraclass correlation coefficients. Psychological Methods, 1, 30–46. doi:10.1037/1082-989X.1.1.30.CrossRef

McGrew, J. H., Johannesen, J. K., Griss, M. E., Born, D. L., & Katuin, C. (2005). Performance-based funding of supported-employment: A multi-site controlled trial. Journal of Vocational Rehabilitation, 23, 81–99.

McHugo, G. J., Drake, R. E., Teague, G. B., & Xie, H. (1999). Fidelity to assertive community treatment and client outcomes in the New Hampshire Dual Disorders Study. Psychiatric Services (Washington, D.C.), 50, 818–824.

McHugo, G. J., Drake, R. E., Whitley, R., Bond, G. R., Campbell, K., Rapp, C. A., et al. (2007). Fidelity outcomes in the National Implementing Evidence-Based Practices Project. Psychiatric Services (Washington, D.C.), 58, 1279–1284. doi:10.1176/appi.ps.58.10.1279.

Mor Barak, M. E., Travis, D. J., Pyun, H., & Xie, B. (2009). The impact of supervision on worker outcomes: A meta-analysis. The Social Service Review, 83, 3–32. doi:10.1086/599028.CrossRef

Nadler, D. A. (1977). Feedback and organization development. Using data based methods. Reading: Addison-Wesley.

New Freedom Commission on Mental Health. (2003). Achieving the promise: Transforming mental health care in America. Final report. DHHS Pub. No. SMA-03-3832. Rockville, MD: Substance Abuse and Mental Health Services Administration.

Ogden, T., Hagen, K. A., Askeland, E., & Christensen, B. (2009). Implementing and evaluating evidence-based treatment of conduct problems in children and youth in Norway. Research on Social Work Practice. doi:10.1177/1049731509335530.

Poertner, J., & Rapp, C. A. (2007). Textbook of social administration: The consumer centered approach. Philadelphia, PA: Haworth Press.

Rapp, C. A., Etzel-Wise, D., Marty, D., Coffman, M., Carlson, L., Asher, D., et al. (2008). Evidence-based practice implementation strategies: Results of a qualitative study. Community Mental Health Journal, 44, 213–224. doi:10.1007/s10597-007-9109-4.PubMedCrossRef

Rapp, C. A., & Goscha, R. J. (2004). The principles of effective case management of mental health services. Psychiatric Rehabilitation Journal, 27, 319–333. doi:10.2975/27.2004.319.333.PubMedCrossRef

Salyers, M. P., Bond, G. R., Teague, G. B., Cox, J. F., Smith, M. E., Hick, M. L., et al. (2003). Is it ACT yet? Real-world examples of evaluating the degree of implementation for assertive community treatment. The Journal of Behavioral Health Services & Research, 30, 308–324.

SAS. (2008). SAS/STAT user’s guide, version 9.2. Cary, NC: SAS Institute Inc.

Shannon, H. S., Robson, L. S., & Sale, J. E. (2001). Creating safer and healthier workplaces: Role of organizational factors and job characteristics. American Journal of Industrial Medicine, 40, 319–334. doi:10.1002/ajim.1106.PubMedCrossRef

Simon, G., Rutter, C., Crosier, M., Scott, J., Operskalski, B. H., & Ludman, E. (2009). Are comparisons of consumer satisfaction with providers biased by nonresponse or case-mix differences. Psychiatric Services (Washington, D.C.), 60, 67–73. doi:10.1176/appi.ps.60.1.67.

Swift, J. K., & Callahan, J. L. (2009). The impact of client treatment preferences on outcome: A meta-analysis. Journal of Clinical Psychology, 65, 368–381.PubMedCrossRef

Taylor, M. S. (1987). The effects of feedback on the behavior of organizational personnel. Administration in Social Work, 11, 191–203. doi:10.1300/J147v11n03_18.CrossRef

Torrey, W. C., Lynde, D. W., & Gorman, P. (2005). Promoting the implementation of practices that are supported by research: The National Implementing Evidence-Based Practice Project. Child and Adolescent Psychiatric Clinics of North America, 14, 297–306. doi:10.1016/j.chc.2004.05.004.PubMedCrossRef

Tsai, J., & Bond, G. R. (2008). A comparison of electronic medical records to conventional paper records in community mental health centers. International Journal for Quality in Health Care, 20, 136–143. doi:10.1093/intqhc/mzm064.PubMedCrossRef

U.S. Surgeon General. (2000). Mental health: A report of the surgeon general - Executive summary. Rockville, MD: U.S. Department of Health and Human Services, Substance Abuse and Mental Health Services Administration, Center for Mental Health Services, National Institutes of Health, National Institute of Mental Health.

Wang, P. S., Demler, O., & Kessler, R. C. (2002). Adequacy of treatment for serious mental illness in the United States. American Journal of Public Health, 92, 92–98. doi:10.2105/AJPH.92.1.92.PubMedCrossRef

Young, A. S., Sullivan, G., Burnam, A., & Brook, R. H. (1998). Measuring the quality of outpatient treatment for schizophrenia. Archives of General Psychiatry, 55, 611–617. doi:10.1001/archpsyc.55.7.611.PubMedCrossRef

Title: Individualization and Quality Improvement: Two New Scales to Complement Measurement of Program Fidelity
Authors: Gary R. Bond
Robert E. Drake
Charles A. Rapp
Gregory J. McHugo
Haiyi Xie
Publication date: 01-09-2009
Publisher: Springer US
Published in: Administration and Policy in Mental Health and Mental Health Services Research / Issue 5/2009
Print ISSN: 0894-587X
Electronic ISSN: 1573-3289
DOI: https://doi.org/10.1007/s10488-009-0226-y

At a glance: The STEP trials

Springer Medicine

Individualization and Quality Improvement: Two New Scales to Complement Measurement of Program Fidelity

Abstract

At a glance: The STEP trials

Springer Medicine

Abstract

Please log in to get access to this content

Other articles of this Issue 5/2009

Capacity for Delivering Culturally Competent Mental Health Services in New Mexico: Results of a Statewide Agency Survey

Do Noncontingent Incentives Increase Survey Response Rates among Mental Health Providers? A Randomized Trial Comparison

Revisiting Practice-Based Research Networks as a Platform for Mental Health Services Research

Preference in Random Assignment: Implications for the Interpretation of Randomized Trials

Developing a Clinical Assessment Model Suited to use in an Agency Providing Short-Term Substance use Treatment: Findings from a Delphi Survey of Expert Opinion

Capacity for Delivering Culturally Competent Mental Health Services in New Mexico: Results of a Statewide Agency Survey