Top

Administration and Policy in Mental Health and Mental Health Services Research

Published in:

01-07-2008 | Original Paper

When Programs Benefit Some People More than Others: Tests of Differential Service Effectiveness

Authors: Cathaleene Macias, Danson R. Jones, William A. Hargreaves, Qi Wang, Charles F. Rodican, Paul J. Barreira, Paul B. Gold

Published in: Administration and Policy in Mental Health and Mental Health Services Research | Issue 4/2008

Abstract

Practitioners need to know for whom evidence-based services are most or least effective, but few services research studies provide this information. Using data from a randomized controlled comparison of supported employment findings for two multi-service psychiatric rehabilitation programs, we illustrate and compare procedures for measuring program-by-client characteristic interactions depicting differential program effectiveness, and then illustrate how a significant program-by-client interaction can explain overall program differences in service effectiveness. Interaction analyses based on cluster analysis-identified sample subgroups appear to provide statistically powerful and meaningful hypothesis tests that can aid in the interpretation of main effect findings and help to refine program theory.

Abel, K. M., Webb, R. T., Salmon, M. P., Wan, M. W., & Appleby, L. (2005). Prevalence and predictors of parenting outcomes in a cohort of mothers with schizophrenia admitted for joint mother and baby psychiatric care in England. Journal of Clinical Psychiatry, 66(6), 781–789.PubMedCrossRef

Ackerman, R. J. (1999). An interactional approach for pharmacopsychologists and psychologists: Gender concerns. Journal of Clinical Psychology in Medical Settings, 6(1), 39–61.CrossRef

Aguinis, H. (2004). Regression analysis for categorical moderators. New York: Guilford Press.

Aguinis, H., & Stone-Romero, E. F. (1997). Methodological artifacts in moderated multiple regression and their effects on statistical power. Journal of Applied Psychology, 82(1), 192–201.CrossRef

Aiken, L. S., & West, S. G. (1991). Multiple regression: Testing and interpreting interactions. Newbury Park, CA: Sage.

Allness, D., & Knoedler, W. (1998). The PACT model: A Manual for PACT Start-Up. Arlington, VA: National Alliance for the Mentally Ill.

Anderson, S. B. (1998). We are not alone: Fountain house and the development of clubhouse culture. New York, NY: Fountain House, Inc.

Anthony, W. A., Rogers, E. S., Cohen, M., & Davies, R. R. (1995). Relationships between psychiatric symptomatology, work skills, and future vocational performance. Psychiatric Services, 46(4), 353–358.PubMed

Aronson, E., Ellsworth, P., Carlsmith, J. M., & Gonzales, M. H. (1990). Methods of research in social psychology. New York: McGraw-Hill.

Batstra, L., Bos, E. H., & Neeleman, J. (2002). Quantifying psychiatric comorbidity: Lessons from chronic disease epidemiology. Social Psychiatry and Psychiatric Epidemiology, 37, 105–111.PubMedCrossRef

Beard, J. H., Propst, R. N., & Malamud, T. J. (1982). The fountain house model of psychiatric rehabilitation. Psychosocial Rehabilitation Journal, 5(1), 1–12.

Bekker, M. H. J. (2003). Investigating gender within health research is more than sex disaggregation of data: A multi-facet gender and health model. Psychology, Health and Medicine, 8(2), 231–243.CrossRef

Beutler, L., Brown, M., Crothers, L., Booker, K., & Seabrook, M. (1996). The dilemma of factitious demographic distinctions in psychological research. Journal of Consulting and Clinical Psychology, 64, 892–902.PubMedCrossRef

Bickman, L., Noser, K., & Summerfelt, W. T. (1999). Long-term effects of a system of care on children and adolescents. Journal of Behavioral Health Services and Research, 26(2), 185–202.PubMedCrossRef

Blankertz, L. (1998). The value and practicality of deliberate sampling for heterogeneity: A critical multiplist perspective. American JoumaI of Evaluation, 19(3), 307–324.

Bond, G. R., Vogler, K., Resnick, S. G., Evans, L., Drake, R., & Becker, D. (2001). Dimensions of supported employment: Factor structure of the IPS fidelity scale. Journal of Mental Health (UK), 10(4), 383–393.CrossRef

Bühringer, G. (2006). Allocating treatment options to patient profiles: Clinical art or science? Addiction, 101, 646–652.PubMedCrossRef

Burke-Miller, J., Cook, J. A., Grey, D. D., Razzano, L., Blyler, C. R., Leff, H. S., et al. (2006). Demographic characteristics and employment among people with severe mental illness in a multisite study. Community Mental Health Journal, 42(2), 143–159.PubMedCrossRef

Carey, K. B., Henson, J. M., Carey, M. P., & Maisto, S. A. (2007). Which heavy drinking college students benefit from a brief motivational intervention? Journal of Consulting and Clinical Psychology, 75(4), 663–669.PubMedCrossRef

Chambless, D. L., & Hollon, S. D. (1998). Defining empirically supported therapies. Journal of Consulting and Clinical Psychology, 66(1), 7–18.PubMedCrossRef

Chwastiak, L. A., Rosenheck, R. A., McEvoy, J. P., Keefe, R. S. E., Swartz, M. S., & Lieberman, J. A. (2006). Interrelationships of psychiatric symptom severity, medical comorbidity, and functioning in schizophrenia. Psychiatric Services, 57(8), 1102–1109.PubMedCrossRef

Clark, C., & Rich, A. R. (2003). Outcomes of homeless adults with mental illness in a housing program and in case management only. Psychiatric Services, 54(1), 78–83.PubMedCrossRef

Cook, J. A., Pickett-Schenk, S. A., Gray, D., Banghart, M., Rosenheck, R., & Randolph, F. (2001). Vocational outcomes among formerly homeless persons with severe mental illness in the ACCESS program. Psychiatric Services, 52(8), 1075–1080.PubMedCrossRef

Cook, J. A., Leff, H. S., Blyler, C. R., Gold, P. B., Goldberg, R. W., Mueser, K. T., et al. (2005). Results of a multisite randomized trial of supported employment interventions for individuals with severe mental illness. Archives of General Psychiatry, 62, 505–512.PubMedCrossRef

Dembling, B. P., Chen, D. T., & Vachon, L. (1999). Life expectancy and causes of death in a population treated for serious mental illness. Psychiatric Services, 50(8), 1036–1042.PubMed

Department of Labor. (1998). Job Training Program Act, Disability Grant Program Funded Under Title III, Section 323, and Title IV, Part D, Section 452 (Vol. 63): Federal Register.

Dickey, B., Normand, S.-L., Weiss, R., Drake, R., & Azeni, H. (2002). Medical morbidity, mental illness, and substance use disorders. Psychiatric Services, 53(7), 861–867.PubMedCrossRef

Dixon, L., Goldberg, R., Lehman, A., & McNary, S. (2001). The impact of health status on work, symptoms, and functional outcomes in severe mental illness. Journal of Nervous and Mental Disease, 189, 17–23.PubMedCrossRef

Drebing, C., Van Ormer, A., Krebs, C., Rosenheck, R., Rounsaville, B., Herz, L., et al. (2005). The impact of enhanced incentives on vocational rehabilitation outcomes for dually diagnosed veterans. Journal of Applied Behavior Analysis, 38(3), 359–372.PubMedCrossRef

Druss, B. G., Marcus, S. C., Rosenheck, R. A., Olfson, M., Tanielian, T., & Pincus, H. A. (2000). Understanding disability in mental and general medical conditions. American Journal of Psychiatry, 157(9), 1485–1491.PubMedCrossRef

Frey, J. L. (1994). Long term support: The critical element to sustaining competitive employment: Where do we begin? Psychosocial Rehabilitation Journal, 17(3), 127–133.

Goldberg, R. W., Lucksted, A., McNary, S., Gold, J. M., Dixon, L., & Lehman, A. (2001). Correlates of long-term unemployment among inner-city adults with serious and persistent mental illness. Psychiatric Services, 52(1), 101–103.PubMedCrossRef

Halvorsen, M. S., & Monsen, J. T. (2007). Self-image as a moderator of change in psychotherapy. Psychotherapy Research, 17(2), 205–217.CrossRef

Hendryx, M. S., Moore, R., Leeper, T., Reynolds, M., & Davis, S. (2001). An examination of methods for risk-adjustment of rehospitalization rates. Mental Health Services Research, 3(1), 15–24.PubMedCrossRef

Hendryx, M. S., & Teague, G. B. (2001). Comparing alternative risk-adjustment models. Journal of Behavioral Health Services and Research, 28(3), 247–257.PubMedCrossRef

Hodges, K., Xue, Y., & Wotring, J. (2004). Outcomes for children with problematic behavior in school and at home served by public mental health. Journal of Emotional and Behavioral Disorders, 12(2), 109–119.CrossRef

Howell, W., & Peterson, P. (2004). Uses of theory in randomized field trials. American Behavioral Scientist, 47(5), 634–657.CrossRef

James, G. M., Sugar, C. A., Desai, R., & Rosenheck, R. A. (2006). A comparison of outcomes among patients with schizophrenia in two mental health systems: A health state approach. Schizophrenia Research, 86, 309–320.PubMedCrossRef

Jensen, P., Arnold, E., Richters, J., Severe, J., Vereen, D., Vittiello, B., & Shiller, E. (1999). Moderators and mediators of treatment response for children with attention-deficit/hyperactivity disorder. Archives of General Psychiatry, 56, 1088–1096.CrossRef

Jones, D. R., Macias, C., Barreira, P. J., Fisher, W. H., Hargreaves, W. A., & Harding, C. M. (2004). Prevalence, severity, and co-occurrence of chronic physical health problems of persons with serious mental illness. Psychiatric Services, 55(11), 1250–1257.PubMedCrossRef

Judd, C. M., & Kenny, D. A. (1981). Estimating the effects of social interventions. New York: Cambridge University Press.

Kay, S. R., Fiszbein, A., & Opler, L. A. (1987). The positive and negative syndrome scale (PANSS) for schizophrenia. Schizophrenia Bulletin, 13, 261–276.PubMed

Kenny, D. A., Calsyn, R. J., Morse, G. A., Klinkenberg, W. D., Winter, J. P., & Trusty, M. L. (2004). Evaluation of treatment programs for persons with severe mental illness: Moderator and mediator effects. Evaluation Review, 28(4), 294–324.PubMedCrossRef

King, R. D., Gaines, L. S., Lambert, E. W., Summerfelt, W. T., & Bickman, L. (2000). The co-occurrence of psychiatric substance use diagnoses in adolescents in different service systems: Frequency, recognition, cost, and outcomes. Journal of Behavioral Health Services and Research, 27(4), 417–430.PubMedCrossRef

Kraemer, H. C., Stice, E., Kazdin, A., Offord, D., & Kupfer, D. (2001). How do risk factors work together? Mediators, moderators, and independent, overlapping, and proxy risk factors. American Journal of Psychiatry, 158(6), 848–856.PubMedCrossRef

Kraemer, H. C., Wilson, G. T., Fairburn, C. G., & Agras, W. S. (2002). Mediators and moderators of treatment effects in randomized clinical trials. Archives of General Psychiatry, 59, 877–883.PubMedCrossRef

Kraemer, H. C., Wilson, K. A., & Hayward, C. (2006). Lifetime prevalence and pseudocomorbidity in psychiatric research. Archives of General Psychiatry, 63, 604–608.PubMedCrossRef

Krause, M. S., & Howard, K. I. (2003). What random assignment does and does not do. Journal of Clinical Psychology, 59, 751–766.PubMedCrossRef

Kronick, R., Gilmer, T., Dreyfus, T., & Lee, L. (2000). Improving health-based payment for medicaid beneficiaries: CDPS. Health Care Financing Review, 21(3), 29–64.PubMed

Lehman, A. F., Goldberg, R. W., Dixon, L. B., NcNary, S., Postrado, L., Hackman, A., et al. (2002). Improving employment outcomes for persons with severe mental illnesses. Archives of General Psychiatry, 59(2), 165–172.PubMedCrossRef

Lipchik, G. L., Nicholson, R. A., & Penzien, D. B. (2005). Allocation of patients to conditions in headache clinical trials: Randomization, stratification, and treatment matching. Headache, 45, 419–428.PubMedCrossRef

Macias, C., DeCarlo, L., Wang, Q., Frey, J., & Barreira, P. (2001). Work interest as a predictor of competitive employment: Policy implications for psychiatric rehabilitation. Administration and Policy in Mental Health, 28(4), 279–297.PubMedCrossRef

Macias, C., Barreira, P., Hargreaves, W., Bickman, L., Fisher, W. H., & Aronson, E. (2005). Impact of referral source and study applicants’ preference for randomly assigned service on research enrollment, service engagement, and evaluative outcomes. American Journal of Psychiatry, 162(4), 781–787.PubMedCrossRef

Macias, C., Rodican, C. F., Hargreaves, W. A., Jones, D. R., Barreira, P. J., & Wang, Q. (2006). Supported employment outcomes of a randomized controlled trial of assertive community treatment and clubhouse models. Psychiatric Services, 57(10), 1406–1415.PubMedCrossRef

Maisto, S. A., Conigliaro, J., McNeil, M., Kraemer, K., & Kelley, M. E. (2001). The relationship between eligibility criteria for participation in alcohol brief intervention trials and other alcohol and health-related variables. American Journal of Addictions, 10(3), 218–231.CrossRef

McBride, C., Atkinson, L., Quilty, L. C., & Bagby, R. M. (2006). Attachment as moderator of treatment outcome in major depression: A randomized control trial of interpersonal psychotherapy versus cognitive behavior therapy. Journal of Consulting and Clinical Psychology, 74(6), 1041–1054.PubMedCrossRef

McKendrick, K., Sullivan, C., Banks, S., & Sacks, S. (2007). Modified therapeutic community treatment for offenders with MICA disorders: Antisocial personality disorder and treatment outcomes. Journal of Offender Rehabilitation, 44(2–3), 133–159.CrossRef

Miklowitz, D., & Clarkin, J. F. (1999). Balancing internal and external validity. Prevention and Treatment, 2, 1–4.

Mueser, K. T., Salyers, M. P., & Mueser, P. R. (2001). A prospective analysis of work in schizophrenia. Schizophrenia Bulletin, 27(2), 281–296.PubMed

Ogrodniczuk, J. S., Joyce, A. S., & Piper, W. E. (2007). Effect of patient dissatisfaction with the therapist on group therapy outcome. Clinical Psychology and Psychotherapy, 14(2), 126–134.CrossRef

Peck, L. R. (2005). Using cluster analysis in program evaluation. Evaluation Review, 29(2), 178–196.PubMedCrossRef

Pettinati, H. M., Volpicelli, J. R., Kranzler, H. R., Luck, G., Rukstalis, M. R., & Cnaan, A. (2000). Sertraline treatment for alcohol dependence: Interactive effects of medication and alcoholic subtype. Alcoholism: Clinical and Experimental Research, 24(7), 1041–1049.CrossRef

Propst, R. (1992). Standards for clubhouse programs: Why and how they were developed. Psychosocial Rehabilitation Journal, 16(2), 25–30.

Rapkin, B. D., & Dumont, K. A. (2000). Methods for identifying and assessing groups in health behavioral research. Addiction, 95(Supplement 3), S395–S417.PubMedCrossRef

Razzano, L., & Hamilton, M. M. (2005). Health-related barriers to employment among people with HIV/AIDS. Journal of Vocational Rehabilitation, 22(3), 179–188.

Razzano, L. A., Cook, J. A., Burke, J., Mueser, K. T., Pickett-Schenk, S., Grey, D. D., et al. (2005). Clinical factors associated with employment among people with severe mental illness: Findings from the employment intervention demonstration program. Journal of Nervous and Mental Disease, 193(11), 705–713.PubMedCrossRef

Regenold, M., Sherman, M., & Fenzel, M. (1999). Getting back to work: Self-efficacy as a predictor of employment outcome. Psychiatric Rehabilitation Journal, 22(4), 361–367.

Rothwell, P. M. (2005). External validity of randomized controlled trials: ‘‘To whom do the results of this trial apply?’’. Lancet, 365, 82–93.PubMedCrossRef

Rubin, W. V., & Panzano, P. C. (2002). Identifying meaningful subgroups of adults with severe mental illness. Psychiatric Services, 53(4), 452–457.PubMedCrossRef

Ruscio, A. M., & Holohan, D. R. (2006). Applying empirically supported treatments to complex cases: Ethical, empirical, and practical considerations. Clinical Psychology: Science and Practice, 13(2), 146–162.CrossRef

Salyers, M. P., McHugo, G. H., Cook, J. A., Razzano, L. A., Drake, R. E., & Mueser, K. T. (2001). Reliability of instruments in a cooperative, multisite study: Employment intervention demonstration program. Mental Health Services Research, 3(3), 129–139.PubMedCrossRef

Slade, E., & Salkever, D. (2001). Symptom effects on employment in a structural model of mental illness and treatment: Analysis of patients with schizophrenia. Journal of Mental Health Policy and Economics, 4(1), 25–34.PubMed

SPSS. (1999). Statistical Program for the Social Sciences (Version 11) [Windows]. Chicago.

Stein, L. I., & Test, M. A. (1980). Alternative to mental hospital treatment. I. Conceptual model, treatment program, and clinical evaluation. Archives of General Psychiatry, 37(4), 392–397.PubMed

Stone-Romero, E. F., & Anderson, L. E. (1994). Relative power of moderated multiple regression and the comparison of subgroup correlation coefficients for detecting moderating effects. Journal of Applied Psychology, 79(3), 354–359.CrossRef

Stout, C. E., & Hayes, R. A. (Eds.). (2005). Evidence-based practice: Methods, models, and tools for mental health professionals. New York, NY: John Wiley.

Trach, J. S. (1990). Supported employment program characteristics. In R. Rusch (Ed.), Supported employment: Models, methods and issues. Sycamore, IL: Sycamore Press.

Turner, E. H., Matthews, A. M., Linardatos, E., Tell, R. A., & Rosenthal, R. (2008). Selective publication of antidepressant trials and its influence on apparent efficacy. New England Journal of Medicine, 358, 252–260.PubMedCrossRef

Twamley, E., Jeste, D. V., & Lehman, A. (2003). Vocational rehabilitation in schizophrenia and other psychotic disorders: A literature review and meta-analysis of randomized controlled trials. Journal of Nervous and Mental Disease, 191(8), 515–523.PubMedCrossRef

Uehara, E., Srebnik, D., & Smukler, M. (2003). Statistical and consensus-based strategies for grouping consumers in mental health level-of-care schemes. Administration and Policy in Mental Health, 30(4), 287–306.PubMedCrossRef

Ward, J. H. (1963). Hierarchical grouping to optimize an objective function. Journal of the American Statistical Association, 58, 236–244.CrossRef

Wells, K. B. (1999). Treatment research at the crossroads: The scientific interface of clinical trials and effectiveness research. American Journal of Psychiatry, 156(1), 5–10.PubMed

West, S. G., & Aiken, L. S. (1997). Toward understanding individual effects in multicomponent prevention programs: Design and analysis strategies. In K. J. Bryant, M. Windle, & S. G. West (Eds.), The science of prevention: Methodological advances from alcohol and substance abuse research. Washington, DC: American Psychological Association.

Wewiorski, N. J., & Fabian, E. S. (2004). Association between demographic and diagnostic factors and employment outcomes for people with psychiatric disabilities: A synthesis of recent research. Mental Health Services Research, 6(1), 9–21.PubMedCrossRef

Workforce Investment Act of (1998): Public law 105–220.

Title: When Programs Benefit Some People More than Others: Tests of Differential Service Effectiveness
Authors: Cathaleene Macias
Danson R. Jones
William A. Hargreaves
Qi Wang
Charles F. Rodican
Paul J. Barreira
Paul B. Gold
Publication date: 01-07-2008
Publisher: Springer US
Published in: Administration and Policy in Mental Health and Mental Health Services Research / Issue 4/2008
Print ISSN: 0894-587X
Electronic ISSN: 1573-3289
DOI: https://doi.org/10.1007/s10488-008-0174-y

Keynote webinar | Spotlight on medication adherence

Springer Medicine

When Programs Benefit Some People More than Others: Tests of Differential Service Effectiveness

Abstract

Keynote webinar | Spotlight on medication adherence

Springer Medicine

Abstract

Please log in to get access to this content

Other articles of this Issue 4/2008

A Common Factors Approach to Improving the Mental Health Capacity of Pediatric Primary Care

What is a Mental Health Clinic? How to ask Parents about Help-seeking Contacts Within the Mental Health System

Evaluation of Treatment Programs for Dual Disorder Individuals: Modeling Longitudinal and Mediation Effects

Use of Psychiatric Advance Directives During Psychiatric Crisis Events

Engaging and Recruiting Counties in an Experiment on Implementing Evidence-based Practice in California

Social Disadvantage and the Mental Health of Military Personnel Returning from Iraq and Afghanistan