Top

Administration and Policy in Mental Health and Mental Health Services Research

Open Access 25-09-2024 | Original Article

Comparative Analysis of Independent Reviewer-Rated and Self-Rated Fidelity Scores in Individual Placement and Support Programs: Repeated Cross-Sectional Surveys

Authors: Sosei Yamaguchi, Junko Koike, Momoka Igarashi, Takayuki Kawaguchi, Takuma Shiozawa, Kaori Usui, Mai Iwanaga, Asami Matsunaga, Koji Yoshida, Sayaka Sato, Chiyo Fujii

Published in: Administration and Policy in Mental Health and Mental Health Services Research

Abstract

Fidelity assessments can contribute to maintaining the adherence to the individual placement and support (IPS) model, which enhances vocational outcomes for individuals with mental illness worldwide. While independent reviews are standard, self-assessments could broaden the implementation of IPS. This study aimed to evaluate reviewer-rated and self-rated fidelity assessments using the Japanese version of the Individualized Supported Employment Fidelity Scale (JiSEF), and to compare the two assessment methods in terms of their correlations with employment outcomes in Japan. Over the 3-year research period, fidelity assessments were conducted by independent reviewers and trained program staff members across 26 programs, totaling 58 assessments. Analyses involved kappa statistics for item-level comparison, the intra-class correlation coefficient (ICC) and paired t-test for the overall fidelity scores, and Pearson’s correlations to examine the relationship between the fidelity scores and program-level employment outcomes. Most individual JiSEF items demonstrated fair to good reliability between reviewer-rated and self-rated assessments. The ICC for the overall JiSEF scores between the two assessment methods was 0.756, yet the distribution of self-rated scores was more scattered compared with that of reviewer-rated scores. The mean total scores from self-assessments were significantly lower than those from reviewer assessments (t = 2.072, P = 0.043). While both sets of scores correlated significantly with employment rates (r = 0.640, P < 0.001 for reviewer assessments; r = 0.325, P = 0.013 for self-assessments), the correlation was stronger for reviewer ratings (z = 2.207, P = 0.027). Self-rated fidelity assessments offer several benefits. However, since independent reviews had a more normal distribution and higher correlation with employment outcome, they should remain the priority in fidelity assessments within the Japanese IPS framework.

Available only for authorised users

Adamus, C., Richter, D., Sutor, K., Zürcher, S. J., & Mötteli, S. (2024). Preference for competitive employment in people with mental disorders: A systematic review and meta-analysis of proportions. Journal of Occupational Rehabilitation. [Online ahead of print] https://doi.org/10.1007/s10926-024-10192-0

Becker, D. R., & Bond, G. R. (2020). Commentary on special issue on individual placement and support (IPS) international. Psychiatric Rehabilitation Journal, 43(1), 79–82. https://doi.org/10.1037/prj0000402CrossRefPubMed

Bond, G. R. (2013). Self-assessed fidelity: Proceed with caution. Psychiatric Services, 64(4), 393–394. https://doi.org/10.1176/appi.ps.640418CrossRefPubMed

Bond, G. R., Peterson, A. E., Becker, D. R., & Drake, R. E. (2012). Validation of the revised individual placement and support fidelity scale (IPS-25). Psychiatric Services, 63(8), 758–763. https://doi.org/10.1176/appi.ps.201100476CrossRefPubMed

Bond, G. R., Al-Abdulmunem, M., Marbacher, J., Christensen, T. N., Sveinsdottir, V., & Drake, R. E. (2023). A systematic review and meta-analysis of IPS supported employment for young adults with mental health conditions. Administration and Policy in Mental Health and Mental Health Services Research, 50(1), 160–172. https://doi.org/10.1007/s10488-022-01228-9CrossRefPubMed

Bond, G. R., Lockett, H., & van Weeghel, J. (2020). International growth of individual placement and support. Epidemiology and Psychiatric Sciences, 29, e183. https://doi.org/10.1017/S2045796020000955CrossRefPubMedPubMedCentral

Brinchmann, B., Widding-Havneraas, T., Modini, M., Rinaldi, M., Moe, C. F., McDaid, D., Park, A. L., Killackey, E., Harvey, S. B., & Mykletun, A. (2020). A meta-regression of the impact of policy on the efficacy of individual placement and support. Acta Psychiatrica Scandinavica, 141(3), 206–220. https://doi.org/10.1111/acps.13129CrossRefPubMed

de Winter, L., Couwenbergh, C., van Weeghel, J., Bergmans, C., & Bond, G. R. (2020). Fidelity and IPS: Does quality of implementation predict vocational outcomes over time for organizations treating persons with severe mental illness in the Netherlands? Social Psychiatry and Psychiatric Epidemiology, 55(12), 1607–1617. https://doi.org/10.1007/s00127-020-01890-0CrossRefPubMed

de Winter, L., Couwenbergh, C., van Weeghel, J., Sanches, S., Michon, H., & Bond, G. R. (2022). Who benefits from individual placement and support? A meta-analysis. Epidemiology and Psychiatric Sciences, 31, e50. https://doi.org/10.1017/S2045796022000300CrossRefPubMedPubMedCentral

Drake, R. E., Becker, D. R., & Bond, G. R. (2019). Introducing Individual Placement and Support (IPS) supported employment in Japan. Psychiatry and Clinical Neurosciences, 73(2), 47–49. https://doi.org/10.1111/pcn.12792CrossRefPubMed

Employment Security Bureau. (2023). 2023 Report on the results of the survey on employment of persons with disabilities. Ministry of Health, Labour and Welfare. https://www.mhlw.go.jp/content/11601000/001233719.pdf

Feinstein, A. R., & Cicchetti, D. V. (1990). High agreement but low Kappa: I. the problems of two paradoxes. Journal of Clinical Epidemiology, 43(6), 543–549. https://doi.org/10.1016/0895-4356(90)90158-L

Gu, A., & Yoo, H. I. (2019). vcemway: A one-stop solution for robust inference with multiway clustering. Stata Journal, 19(4), 900–912. https://doi.org/10.1177/1536867x19893637CrossRef

Gühne, U., Pabst, A., Löbner, M., Breilmann, J., Hasan, A., Falkai, P., Kilian, R., Allgöwer, A., Ajayi, K., Baumgärtner, J., Brieger, P., Frasch, K., Heres, S., Jäger, M., Küthmann, A., Putzhammer, A., Schneeweiß, B., Schwarz, M., Becker, T., Kösters, M., & Riedel-Heller, S. G. (2021). Employment status and desire for work in severe mental illness: Results from an observational, cross-sectional study. Social Psychiatry and Psychiatric Epidemiology, 56(9), 1657–1667. https://doi.org/10.1007/s00127-021-02088-8

Hayashi, T., Yamaguchi, S., & Sato, S. (2020). Implementing the individual placement and support model of supported employment in Japan: Barriers and strategies. Psychiatric Rehabilitation Journal, 43(1), 53–59. https://doi.org/10.1037/prj0000394CrossRefPubMed

Holm, M., Taipale, H., Tanskanen, A., Tiihonen, J., & Mitterdorfer-Rutz, E. (2021). Employment among people with schizophrenia or bipolar disorder: A population-based study using nationwide registers. Acta Psychiatrica Scandinavica, 143(1), 61–71. https://doi.org/10.1111/acps.13254CrossRefPubMed

Jónasson, H., van Weeghel, J., Koatz, D., Johnston, G., Bejerholm, U., & Fioritti, A. (2022). Boosting the development of individual placement and support in Europe. Epidemiology and Psychiatric Sciences, 31, e29. https://doi.org/10.1017/S2045796022000129CrossRefPubMedCentral

Kim, S. J., Bond, G. R., Becker, D. R., Swanson, S. J., & Langfitt-Reese, S. (2015). Predictive validity of the individual placement and support fidelity scale (IPS-25): A replication study. Journal of Vocational Rehabilitation, 43, 209–216. https://doi.org/10.3233/JVR-150770CrossRef

Klein, D. (2018). Implementing a general framework for assessing interrater agreement in Stata. Stata Journal, 18(4), 871–901. https://doi.org/10.1177/1536867x1801800408CrossRef

Koo, T. K., & Li, M. Y. (2016). A guideline of selecting and reporting intraclass correlation coefficients for reliability research. Journal of Chiropractic Medicine, 15(2), 155–163. https://doi.org/10.1016/j.jcm.2016.02.012CrossRefPubMedPubMedCentral

Landis, J. R., & Koch, G. G. (1977). The measurement of observer agreement for categorical data. Biometrics, 33(1), 159–174. https://doi.org/10.2307/2529310CrossRefPubMed

Lockett, H., Waghorn, G., Kydd, R., & Chant, D. (2016). Predictive validity of evidence-based practices in supported employment: A systematic review and meta-analysis. Mental Health Review Journal, 21(4), 261–281. https://doi.org/10.1108/MHRJ-12-2015-0040CrossRef

Luciano, A., & Meara, E. (2014). Employment status of people with mental illness: National survey data from 2009 and 2010. Psychiatric Services, 65(10), 1201–1209. https://doi.org/10.1176/appi.ps.201300335CrossRefPubMed

Luciano, A., Bond, G. R., Drake, R. E., & Becker, D. R. (2014). Is high fidelity to supported employment equally attainable in small and large communities? Community Mental Health Journal, 50(1), 46–50. https://doi.org/10.1007/s10597-013-9687-2CrossRefPubMed

Majuri, T., Alakokkare, A.-E., Haapea, M., Nordström, T., Miettunen, J., Jääskeläinen, E., & Ala-Mursula, L. (2023). Employment trajectories until midlife in schizophrenia and other psychoses: The Northern Finland Birth Cohort 1966. Social Psychiatry and Psychiatric Epidemiology, 58(1), 65–76. https://doi.org/10.1007/s00127-022-02327-6CrossRefPubMed

Makin, T. R., & Orban de Xivry, J.-J. (2019). Ten common statistical mistakes to watch out for when writing or reviewing a manuscript. eLife, 8, e48175. https://doi.org/10.7554/eLife.48175

Margolies, P. J., Humensky, J. L., Chiang, I. C., Covell, N. H., Broadway-Wilson, K., Gregory, R., Jewell, T. C., Scannevin, G., Jr., Baker, S., & Dixon, L. B. (2017). Is there a role for fidelity self-assessment in the individual placement and support model of supported employment? Psychiatric Services, 68(9), 975–978. https://doi.org/10.1176/appi.ps.201600264CrossRefPubMed

Margolies, P. J., Humensky, J. L., Chiang, I. C., Covell, N. H., Jewell, T. C., Broadway-Wilson, K., Gregory, R., Scannevin, G., Jr., & Dixon, L. B. (2018). Relationship between self-assessed fidelity and self-reported employment in the individual placement and support model of supported employment. Psychiatric Services, 69(5), 609–612. https://doi.org/10.1176/appi.ps.201700472CrossRefPubMed

Ministry of Health, Labour and Welfare. (2023). Summary of the health administration report for the fiscal year 2022. Ministry of Health, Labour and Welfare. https://www.mhlw.go.jp/toukei/saikin/hw/eisei_houkoku/22/dl/kekka1.pdf

Park, A. L., Rinaldi, M., Brinchmann, B., Killackey, E., Aars, N. A. P., Mykletun, A., & McDaid, D. (2022). Economic analyses of supported employment programmes for people with mental health conditions: A systematic review. European Psychiatry, 65(1), e51. https://doi.org/10.1192/j.eurpsy.2022.2309CrossRefPubMedPubMedCentral

Sasaki, N., Yamaguchi, S., Shimodaira, M., Sato, S., Taneda, A., Yoshida, K., & Ito, J. (2018). Development and validation of a Japanese fidelity scale for supported employment. Administration and Policy in Mental Health and Mental Health Services Research, 45(2), 318–327. https://doi.org/10.1007/s10488-017-0825-yCrossRefPubMed

Takata, T. (2003). Self-enhancement and self-criticism in Japanese culture: An experimental analysis. Journal of Cross-Cultural Psychology, 34(5), 542–551. https://doi.org/10.1177/0022022103256477CrossRef

Waghorn, G., van Veggel, R., Chant, D., & Lockett, H. (2018). The utility of item level fidelity scores for developing evidence based practices in supported employment. Journal of Vocational Rehabilitation, 48(3), 387–391. https://doi.org/10.3233/JVR-180946CrossRef

Yamagishi, T., Hashimoto, H., Cook, K. S., Kiyonari, T., Shinada, M., Mifune, N., Inukai, K., Takagishi, H., Horita, Y., & Li, Y. (2012). Modesty in self-presentation: A comparison between the USA and Japan. Asian Journal of Social Psychology, 15(1), 60–68. https://doi.org/10.1111/j.1467-839X.2011.01362.xCrossRef

Yamaguchi, S., Mizuno, M., Sato, S., Matsunaga, A., Sasaki, N., Shimodaira, M., & Fujii, C. (2020). Contents and intensity of services in low- and high-fidelity programs for supported employment: Results of a longitudinal survey. Psychiatric Services, 71(5), 472–479. https://doi.org/10.1176/appi.ps.201900255CrossRefPubMed

Yamaguchi, S., Sato, S., Ojio, Y., Shiozawa, T., Matsunaga, A., Taneda, A., Sawada, U., Yoshida, K., & Fujii, C. (2021). Assessing stable validity and reliability of the Japanese version of the individualized supported employment fidelity scale: A replication. Neuropsychopharmacology Reports, 41(2), 248–254. https://doi.org/10.1002/npr2.12172CrossRefPubMedPubMedCentral

Yamaguchi, S., Sato, S., Shiozawa, T., Matsunaga, A., Ojio, Y., & Fujii, C. (2022). Predictive association of low- and high-fidelity supported employment programs with multiple outcomes in a real-world setting: A prospective longitudinal multi-site study. Administration and Policy in Mental Health and Mental Health Services Research, 49(2), 255–266. https://doi.org/10.1007/s10488-021-01161-3CrossRefPubMed

Zheng, K., Stern, B. Z., Wafford, Q. E., & Kohli-Lynch, C. N. (2022). Trial-based economic evaluations of supported employment for adults with severe mental illness: A systematic review. Administration and Policy in Mental Health and Mental Health Services Research, 49(3), 440–452. https://doi.org/10.1007/s10488-021-01174-yCrossRefPubMed

Title: Comparative Analysis of Independent Reviewer-Rated and Self-Rated Fidelity Scores in Individual Placement and Support Programs: Repeated Cross-Sectional Surveys
Authors: Sosei Yamaguchi
Junko Koike
Momoka Igarashi
Takayuki Kawaguchi
Takuma Shiozawa
Kaori Usui
Mai Iwanaga
Asami Matsunaga
Koji Yoshida
Sayaka Sato
Chiyo Fujii
Publication date: 25-09-2024
Publisher: Springer US
Published in: Administration and Policy in Mental Health and Mental Health Services Research
Print ISSN: 0894-587X
Electronic ISSN: 1573-3289
DOI: https://doi.org/10.1007/s10488-024-01413-y

Keynote webinar | Spotlight on modern management of frailty

Springer Medicine

Comparative Analysis of Independent Reviewer-Rated and Self-Rated Fidelity Scores in Individual Placement and Support Programs: Repeated Cross-Sectional Surveys

Abstract

Keynote webinar | Spotlight on modern management of frailty

Springer Medicine

Abstract

Please log in to get access to this content