Top

Published in:

Open Access 01-12-2021

Critical examination of current response shift methods and proposal for advancing new methods

Authors: Véronique Sébille, Lisa M. Lix, Olawale F. Ayilara, Tolulope T. Sajobi, A. Cecile J. W. Janssens, Richard Sawatzky, Mirjam A. G. Sprangers, Mathilde G. E. Verdam, the Response Shift – in Sync Working Group

Published in: Quality of Life Research | Issue 12/2021

Abstract

Purpose

This work is part of an international, interdisciplinary initiative to synthesize research on response shift in results of patient-reported outcome measures. The objective is to critically examine current response shift methods. We additionally propose advancing new methods that address the limitations of extant methods.

Methods

Based on literature reviews, this critical examination comprises design-based, qualitative, individualized, and preference-based methods, latent variable models, and other statistical methods. We critically appraised their definition, operationalization, the type of response shift they can detect, whether they can adjust for and explain response shift, their assumptions, and alternative explanations. Overall limitations requiring new methods were identified.

Results

We examined 11 methods that aim to operationalize response shift, by assessing change in the meaning of one’s self-evaluation. Six of these methods distinguish between change in observed measurements (observed change) and change in the construct that was intended to be measured (target change). The methods use either (sub)group-based or individual-level analysis, or a combination. All methods have underlying assumptions to be met and alternative explanations for the inferred response shift effects. We highlighted the need to address the interpretation of the results as response shift and proposed advancing new methods handling individual variation in change over time and multiple time points.

Conclusion

No single response shift method is optimal; each method has strengths and limitations. Additionally, extra steps need to be taken to correctly interpret the results. Advancing new methods and conducting computer simulation studies that compare methods are recommended to move response shift research forward.

Available only for authorised users

Schwartz, C. E., & Rapkin, B. D. (2004). Reconsidering the psychometrics of quality of life assessment in light of response shift and appraisal. Health and Quality of Life Outcomes, 2(1), 16. https://doi.org/10.1186/1477-7525-2-16.CrossRefPubMedPubMedCentral

Golembiewski, R. T. (1976). Measuring change and persistence in human affairs: Types of change generated by OD designs. The Journal of Applied Behavioral Science, 12(2), 133–157. https://doi.org/10.1177/002188637601200201.CrossRef

Howard, G. S., Ralph, K. M., Gulanick, N. A., Maxwell, S. E., Nance, D. W., & Gerber, S. K. (1979). Internal invalidity in pretest-posttest self-report evaluations and a re-evaluation of retrospective pretests. Applied Psychological Measurement, 3(1), 1–23. https://doi.org/10.1177/014662167900300101.CrossRef

Sprangers, M. A., & Schwartz, C. E. (1999). Integrating response shift into health-related quality of life research: a theoretical model. Social Science & Medicine, 48(11), 1507–1515.CrossRef

Rapkin, B. D., & Schwartz, C. E. (2004). Toward a theoretical model of quality-of-life appraisal: Implications of findings from studies of response shift. Health and Quality of Life Outcomes, 2, 14. https://doi.org/10.1186/1477-7525-2-14.CrossRefPubMedPubMedCentral

Oort, F. J. (2005). Towards a formal definition of response shift (in reply to G.W. Donaldson). Quality of Life Research An International Journal of Quality of Life Aspects of Treatment, Care and Rehabilitation, 14(10), 2353–2355. https://doi.org/10.1007/s11136-005-3978-1.CrossRef

Oort, F. J., Visser, M. R. M., & Sprangers, M. A. G. (2009). Formal definitions of measurement bias and explanation bias clarify measurement and conceptual perspectives on response shift. Journal of Clinical Epidemiology, 62(11), 1126–1137. https://doi.org/10.1016/j.jclinepi.2009.03.013.CrossRefPubMed

Vanier, A., Oort, F. J., McClimans, L., Ow, N., Gulek, B. G., Böhnke, J. R., … & the Response Shift - in Sync Working Group. (n.d.). Response shift in patient-reported outcomes measures: A formal definition and a revised model., Manuscript under review (same issue).

Schwartz, C. E., Ahmed, S., Sawatzky, R., Sajobi, T., Mayo, N., Finkelstein, J., …, & Sprangers, M. A. G. (2013). Guidelines for secondary analysis in search of response shift. Quality of Life Research: An International Journal of Quality of Life Aspects of Treatment, Care and Rehabilitation, 22(10), 2663–2673. https://doi.org/10.1007/s11136-013-0402-0.CrossRef

10.

Schwartz, C. E., & Sprangers, M. A. G. (2010). Guidelines for improving the stringency of response shift research using the thentest. Quality of Life Research: An International Journal of Quality of Life Aspects of Treatment, Care and Rehabilitation, 19(4), 455–464. https://doi.org/10.1007/s11136-010-9585-9.CrossRef

11.

Sajobi, T. T., Brahmbatt, R., Lix, L. M., Zumbo, B. D., & Sawatzky, R. (2018). Scoping review of response shift methods: Current reporting practices and recommendations. Quality of Life Research: An International Journal of Quality of Life Aspects of Treatment, Care and Rehabilitation, 27(5), 1133–1146. https://doi.org/10.1007/s11136-017-1751-x.CrossRef

12.

Sawatzky, R., Sajobi, T. T., Brahmbhatt, R., Chan, E. K. H., Lix, L. M., & Zumbo, B. D. (2017). Longitudinal change in response processes: A response shift perspective. In B. D. Zumbo & A. M. Hubley (Eds.), Understanding and investigating response processes in validation research (pp. 251–276). Cham: Springer.CrossRef

13.

Mayo, N. E., Scott, S. C., Dendukuri, N., Ahmed, S., & Wood-Dauphinee, S. (2008). Identifying response shift statistically at the individual level. Quality of Life Research: An International Journal of Quality of Life Aspects of Treatment, Care and Rehabilitation, 17(4), 627–639. https://doi.org/10.1007/s11136-008-9329-2.CrossRef

14.

Li, Y., & Rapkin, B. (2009). Classification and regression tree uncovered hierarchy of psychosocial determinants underlying quality-of-life response shift in HIV/AIDS. Journal of Clinical Epidemiology, 62(11), 1138–1147. https://doi.org/10.1016/j.jclinepi.2009.03.021.CrossRefPubMedPubMedCentral

15.

Lix, L. M., Sajobi, T. T., Sawatzky, R., Liu, J., Mayo, N. E., Huang, Y., …, & Bernstein, C. N. (2013). Relative importance measures for reprioritization response shift. Quality of Life Research: An International Journal of Quality of Life Aspects of Treatment, Care and Rehabilitation, 22(4), 695–703. https://doi.org/10.1007/s11136-012-0198-3.CrossRef

16.

Guilleux, A., Blanchin, M., Vanier, A., Guillemin, F., Falissard, B., Schwartz, C. E., …, & Sébille, V. (2015). RespOnse Shift ALgorithm in Item response theory (ROSALI) for response shift detection with missing data in longitudinal patient-reported outcome studies. Quality of Life Research: An International Journal of Quality of Life Aspects of Treatment, Care and Rehabilitation, 24(3), 553–564. https://doi.org/10.1007/s11136-014-0876-4.CrossRef

17.

Gerlich, C., Schuler, M., Jelitte, M., Neuderth, S., Flentje, M., Graefen, M., …, & Faller, H. (2016). Prostate cancer patients’ quality of life assessments across the primary treatment trajectory: “True” change or response shift? Acta Oncologica (Stockholm, Sweden), 55(7), 814–820. https://doi.org/10.3109/0284186X.2015.1136749.CrossRef

18.

Reissmann, D. R., Erler, A., Hirsch, C., Sierwald, I., Machuca, C., & Schierz, O. (2018). Bias in retrospective assessment of perceived dental treatment effects when using the Oral Health Impact Profile. Quality of Life Research: An International Journal of Quality of Life Aspects of Treatment, Care and Rehabilitation, 27(3), 775–782. https://doi.org/10.1007/s11136-017-1725-z.CrossRef

19.

Visser, M. R. M., Oort, F. J., & Sprangers, M. A. G. (2005). Methods to detect response shift in quality of life data: A convergent validity study. Quality of Life Research: An International Journal of Quality of Life Aspects of Treatment, Care and Rehabilitation, 14(3), 629–639. https://doi.org/10.1007/s11136-004-2577-x.CrossRef

20.

Ahmed, S., Mayo, N. E., Wood-Dauphinee, S., Hanley, J. A., & Cohen, S. R. (2005). The structural equation modeling technique did not show a response shift, contrary to the results of the then test and the individualized approaches. Journal of Clinical Epidemiology, 58(11), 1125–1133. https://doi.org/10.1016/j.jclinepi.2005.03.003.CrossRefPubMed

21.

Schwartz, C. E., Sprangers, M. A. G., Oort, F. J., Ahmed, S., Bode, R., Li, Y., & Vollmer, T. (2011). Response shift in patients with multiple sclerosis: An application of three statistical techniques. Quality of Life Research: An International Journal of Quality of Life Aspects of Treatment, Care and Rehabilitation, 20(10), 1561–1572. https://doi.org/10.1007/s11136-011-0056-8.CrossRef

22.

Salmon, M., Blanchin, M., Rotonda, C., Guillemin, F., & Sébille, V. (2017). Identifying patterns of adaptation in breast cancer patients with cancer-related fatigue using response shift analyses at subgroup level. Cancer Medicine, 6(11), 2562–2575. https://doi.org/10.1002/cam4.1219.CrossRefPubMedPubMedCentral

23.

Preiß, M., Friedrich, M., Stolzenburg, J.-U., Zenger, M., & Hinz, A. (2019). Response shift effects in the assessment of urologic cancer patients’ quality of life. European Journal of Cancer Care, 28(4), e13027. https://doi.org/10.1111/ecc.13027.CrossRefPubMed

24.

Blanchin, M., Guilleux, A., Hardouin, J.-B., & Sébille, V. (2020). Comparison of structural equation modelling, item response theory and Rasch measurement theory-based methods for response shift detection at item level: A simulation study. Statistical Methods in Medical Research, 29(4), 1015–1029. https://doi.org/10.1177/0962280219884574.CrossRefPubMed

25.

Sprangers, M. A. G., Sajobi, Tolulope T., Vanier, A., Mayo, N. E., Sawatzky, R., Lix, L. M., et al. & The Response Shift—in Sync Working Group. (2021). Response shift in results of patient-reported outcome measures: A commentary to the response shift - in sync working group initiative. Quality of Life Research. https://doi.org/10.1007/s11136-020-02747-4.CrossRefPubMedPubMedCentral

26.

Sawatzky, R., Kwon, J.-Y., Barclay, R., Chauhan, C., Frank, L., van den Hout, W., et al. & The Response Shift—in Sync Working Group (2021). Implications of response shift for micro-, meso-, and macro-level healthcare decision-making using results of patient-reported outcome measures. Quality of Life Research,. https://doi.org/10.1007/s11136-021-02766-9.CrossRefPubMed

27.

Schwartz, C. E., & Sprangers, M. A. G. (1999). Methodological approaches for assessing response shift in longitudinal health-related quality-of-life research. Social Science & Medicine, 48(11), 1531–1548.CrossRef

28.

Barclay-Goddard, R., Epstein, J. D., & Mayo, N. E. (2009). Response shift: A brief overview and proposed research priorities. Quality of Life Research: An International Journal of Quality of Life Aspects of Treatment, Care and Rehabilitation, 18(3), 335–346. https://doi.org/10.1007/s11136-009-9450-x.CrossRef

29.

Sprangers, M. A., Van Dam, F. S., Broersen, J., Lodder, L., Wever, L., Visser, M. R., …, & Smets, E. M. (1999). Revealing response shift in longitudinal research on fatigue–the use of the thentest approach. Acta Oncologica (Stockholm, Sweden), 38(6), 709–718.CrossRef

30.

Beeken, R. J., Eiser, C., & Dalley, C. (2011). Health-related quality of life in haematopoietic stem cell transplant survivors: A qualitative study on the role of psychosocial variables and response shifts. Quality of Life Research: An International Journal of Quality of Life Aspects of Treatment, Care and Rehabilitation, 20(2), 153–160. https://doi.org/10.1007/s11136-010-9737-y.CrossRef

31.

Ahmed, S., Mayo, N. E., Wood-Dauphinee, S., Hanley, J. A., & Cohen, S. R. (2005). Using the Patient Generated Index to evaluate response shift post-stroke. Quality of Life Research: An International Journal of Quality of Life Aspects of Treatment, Care and Rehabilitation, 14(10), 2247–2257. https://doi.org/10.1007/s11136-005-8118-4.CrossRef

32.

Ring, L., Höfer, S., Heuston, F., Harris, D., & O’Boyle, C. A. (2005). Response shift masks the treatment impact on patient reported outcomes (PROs): The example of individual quality of life in edentulous patients. Health and Quality of Life Outcomes, 3, 55. https://doi.org/10.1186/1477-7525-3-55.CrossRefPubMedPubMedCentral

33.

Korfage, I. J., de Koning, H. J., & Essink-Bot, M.-L. (2007). Response shift due to diagnosis and primary treatment of localized prostate cancer: A then-test and a vignette study. Quality of Life Research, 16(10), 1627–1634. https://doi.org/10.1007/s11136-007-9265-6.CrossRefPubMedPubMedCentral

34.

Oort, F. J. (2005). Using structural equation modeling to detect response shifts and true change. Quality of Life Research: An International Journal of Quality of Life Aspects of Treatment, Care and Rehabilitation, 14(3), 587–598.CrossRef

35.

Boucekine, M., Boyer, L., Baumstarck, K., Millier, A., Ghattas, B., Auquier, P., & Toumi, M. (2015). Exploring the response shift effect on the quality of life of patients with schizophrenia: An application of the random forest method. Medical Decision Making: An International Journal of the Society for Medical Decision Making, 35(3), 388–397. https://doi.org/10.1177/0272989X14559273.CrossRef

36.

King-Kallimanis, B. L., Oort, F. J., & Garst, G. J. A. (2010). Using structural equation modelling to detect measurement bias and response shift in longitudinal data. AStA Advances in Statistical Analysis, 94(2), 139–156. https://doi.org/10.1007/s10182-010-0129-y.CrossRef

37.

Lix, L. M., Chan, E. K. H., Sawatzky, R., Sajobi, T. T., Liu, J., Hopman, W., & Mayo, N. (2016). Response shift and disease activity in inflammatory bowel disease. Quality of Life Research: An International Journal of Quality of Life Aspects of Treatment, Care and Rehabilitation, 25(7), 1751–1760. https://doi.org/10.1007/s11136-015-1188-z.CrossRef

38.

Jacqmin-Gadda, H., Sibillot, S., Proust, C., Molina, J.-M., & Thiébaut, R. (2007). Robustness of the linear mixed model to misspecified error distribution. Computational Statistics & Data Analysis, 51(10), 5142–5154. https://doi.org/10.1016/j.csda.2006.05.021.CrossRef

39.

Bolck, A., Croon, M., & Hagenaars, J. (2004). Estimating latent structure models with categorical variables: One-step versus three-step estimators. Political Analysis, 12(1), 3–27.CrossRef

40.

Vermunt, J. K. (2010). Latent class modeling with covariates: Two improved three-step approaches. Political Analysis, 18(4), 450–469.CrossRef

41.

Muthén, B. O. (2001). Latent variable mixture modeling. New developments and techniques in structural equation modeling. New York: Psychology Press.

42.

Sawatzky, R., Ratner, P. A., Kopec, J. A., & Zumbo, B. D. (2012). Latent variable mixture models: a promising approach for the validation of patient reported outcomes. Quality of Life Research: An International Journal of Quality of Life Aspects of Treatment, Care and Rehabilitation, 21(4), 637–650. https://doi.org/10.1007/s11136-011-9976-6.CrossRef

43.

Lubke, G. H., & Luningham, J. (2017). Fitting latent variable mixture models. Behaviour Research and Therapy, 98, 91–102. https://doi.org/10.1016/j.brat.2017.04.003.CrossRefPubMedPubMedCentral

44.

Tueller, S., & Lubke, G. (2010). Evaluation of structural equation mixture models parameter estimates and correct class assignment. Structural Equation Modeling: A Multidisciplinary Journal, 17(2), 165–192. https://doi.org/10.1080/10705511003659318.CrossRef

45.

Barclay-Goddard, R., Lix, L. M., Tate, R., Weinberg, L., & Mayo, N. E. (2009). Response shift was identified over multiple occasions with a structural equation modeling framework. Journal of Clinical Epidemiology, 62(11), 1181–1188. https://doi.org/10.1016/j.jclinepi.2009.03.014.CrossRefPubMed

46.

Verdam, M. G. E., & Oort, F. J. (2014). Measurement bias detection with Kronecker product restricted models for multivariate longitudinal data: An illustration with health-related quality of life data from thirteen measurement occasions. Frontiers in Psychology, 5, 1022. https://doi.org/10.3389/fpsyg.2014.01022.CrossRefPubMedPubMedCentral

47.

Verdam, M. G. E., & Oort, F. J. (2019). The analysis of multivariate longitudinal data: An instructive application of the longitudinal three-mode model. Multivariate Behavioral Research, 54(4), 457–474. https://doi.org/10.1080/00273171.2018.1520072.CrossRefPubMed

48.

Lowy, A., & Bernhard, J. (2004). Quantitative assessment of changes in patients’ constructs of quality of life: An application of multilevel models. Quality of Life Research: An International Journal of Quality of Life Aspects of Treatment, Care and Rehabilitation, 13(7), 1177–1185. https://doi.org/10.1023/B:QURE.0000037510.17893.d2.CrossRef

49.

Cheng, J., Edwards, L. J., Maldonado-Molina, M. M., Komro, K. A., & Muller, K. E. (2010). Real longitudinal data analysis for real people: Building a good enough mixed model. Statistics in Medicine, 29(4), 504–520. https://doi.org/10.1002/sim.3775.CrossRefPubMedPubMedCentral

50.

Liu, Y., Millsap, R. E., West, S. G., Tein, J.-Y., Tanaka, R., & Grimm, K. J. (2017). Testing measurement invariance in longitudinal data with ordered-categorical measures. Psychological Methods, 22(3), 486–506. https://doi.org/10.1037/met0000075.CrossRefPubMed

51.

Verhagen, J., & Fox, J.-P. (2013). Longitudinal measurement in health-related surveys. A Bayesian joint growth model for multivariate ordinal responses. Statistics in Medicine, 32(17), 2988–3005. https://doi.org/10.1002/sim.5692.CrossRefPubMed

52.

Bishop, J., Geiser, C., & Cole, D. A. (2015). Modeling latent growth with multiple indicators: A comparison of three approaches. Psychological Methods, 20(1), 43–62. https://doi.org/10.1037/met0000018.CrossRefPubMed

53.

Dantan, E., Joly, P., Dartigues, J.-F., & Jacqmin-Gadda, H. (2011). Joint model with latent state for longitudinal and multistate data. Biostatistics (Oxford, England), 12(4), 723–736. https://doi.org/10.1093/biostatistics/kxr003.CrossRef

54.

Zhang, Z., & Sun, J. (2010). Interval censoring. Statistical Methods in Medical Research, 19(1), 53–70. https://doi.org/10.1177/0962280209105023.CrossRefPubMed

55.

Hinds, A. M., Sajobi, T. T., Sebille, V., Sawatzky, R., & Lix, L. M. (2018). A systematic review of the quality of reporting of simulation studies about methods for the analysis of complex longitudinal patient-reported outcomes data. Quality of Life Research: An International Journal of Quality of Life Aspects of Treatment, Care and Rehabilitation, 27(10), 2507–2516. https://doi.org/10.1007/s11136-018-1861-0.CrossRef

56.

de Bock, É., Hardouin, J.-B., Blanchin, M., Le Neel, T., Kubis, G., Bonnaud-Antignac, A., …, & Sébille, V. (2016). Rasch-family models are more valuable than score-based approaches for analysing longitudinal patient-reported outcomes with missing data. Statistical Methods in Medical Research, 25(5), 2067–2087. https://doi.org/10.1177/0962280213515570.CrossRefPubMed

Title: Critical examination of current response shift methods and proposal for advancing new methods
Authors: Véronique Sébille
Lisa M. Lix
Olawale F. Ayilara
Tolulope T. Sajobi
A. Cecile J. W. Janssens
Richard Sawatzky
Mirjam A. G. Sprangers
Mathilde G. E. Verdam
the Response Shift – in Sync Working Group
Publication date: 01-12-2021
Publisher: Springer International Publishing
Published in: Quality of Life Research / Issue 12/2021
Print ISSN: 0962-9343
Electronic ISSN: 1573-2649
DOI: https://doi.org/10.1007/s11136-020-02755-4

At a glance: The STEP trials

Springer Medicine

Critical examination of current response shift methods and proposal for advancing new methods

Abstract

Purpose

Methods

Results

Conclusion

At a glance: The STEP trials

Springer Medicine

Abstract

Purpose

Methods

Results

Conclusion

Please log in to get access to this content

Other articles of this Issue 12/2021

Standard set of patient-reported outcomes for personality disorder

Trends in health-related quality of life of female breast cancer survivors using the Medical Expenditure Panel Survey (MEPS), 2008–2016

Health-related quality of life in children with and without physical–mental multimorbidity

Progress in response-shift research needs diverse approaches

Health-related quality of life in patients with chronic hepatitis C treated with sofosbuvir-based treatment at 1-year post-sustained virological response

The association between the risk of developing type 2 diabetes mellitus and health-related quality of life: baseline results from the Dimini lifestyle intervention