Skip to main content
Key Specialties
Cardiology Diabetology Endocrinology Gastroenterology Geriatrics and Gerontology Gynecology and Obstetrics Hematology Infectious Disease Internal Medicine Nephrology Neurology Oncology Pediatrics Respiratory Medicine Rheumatology Surgery
Congress Coverage
2024 ACC Congress 2023 ESMO Congress 2023 EASD Congress 2023 ERS Congress 2023 ESC Congress
Clinical Collections
Advances in Alzheimer’s

At a glance: The STEP trials

A round-up of the STEP phase 3 clinical trials evaluating semaglutide for weight loss in people with overweight or obesity.
ADVANCED SEARCH
Log in
Top
Quality of Life Research
Published in: Quality of Life Research 7/2018

Open Access 01-07-2018 | Special Section: Test Construction (by invitation only)

Application of validity theory and methodology to patient-reported outcome measures (PROMs): building an argument for validity

Authors: Melanie Hawkins, Gerald R. Elsworth, Richard H. Osborne

Published in: Quality of Life Research | Issue 7/2018

Login to get access

Abstract

Background

Data from subjective patient-reported outcome measures (PROMs) are now being used in the health sector to make or support decisions about individuals, groups and populations. Contemporary validity theorists define validity not as a statistical property of the test but as the extent to which empirical evidence supports the interpretation of test scores for an intended use. However, validity testing theory and methodology are rarely evident in the PROM validation literature. Application of this theory and methodology would provide structure for comprehensive validation planning to support improved PROM development and sound arguments for the validity of PROM score interpretation and use in each new context.

Objective

This paper proposes the application of contemporary validity theory and methodology to PROM validity testing.

Illustrative example

The validity testing principles will be applied to a hypothetical case study with a focus on the interpretation and use of scores from a translated PROM that measures health literacy (the Health Literacy Questionnaire or HLQ).

Discussion

Although robust psychometric properties of a PROM are a pre-condition to its use, a PROM’s validity lies in the sound argument that a network of empirical evidence supports the intended interpretation and use of PROM scores for decision making in a particular context. The health sector is yet to apply contemporary theory and methodology to PROM development and validation. The theoretical and methodological processes in this paper are offered as an advancement of the theory and practice of PROM validity testing in the health sector.
Footnotes
1
There is some exchange in this paper between the terms ‘tool’ and ‘test’. The Standards refers to a ‘test’ and, when referencing the Standards, the authors will also refer to a ‘test’. The Standards is written primarily for educators and psychologists, professions in which testing students and clients, respectively, is undertaken. Patient-reported outcomes measures (PROMs) used in the field of health are not used in the same way as testing for educational grading or for psychological diagnosis. PROMs are primarily used to provide information about healthcare options or effectiveness of treatments.
 
2
We use ‘robust’ to describe the required psychometric properties of a PROM in the same way that it is used more generally in the test development and review literature to indicate (a) that in the development stage, a PROM achieves acceptable benchmarks across a range of relevant statistical tests (e.g. a composite reliability or Cronbach’s alpha of = > 0.8; a single-factor model for each scale in a multi-scale PROM giving satisfactory fit across a range of fit statistics, clear discrimination across these scales etc.) and (b) that these results are replicated (i.e. remain acceptably stable) across a range of different contexts and uses of the PROM.
 
Literature
1.
Nelson, E. C., Eftimovska, E., Lind, C., Hager, A., Wasson, J. H., & Lindblad, S. (2015). Patient reported outcome measures in practice. BMJ, 350, g7818.CrossRefPubMed
2.
Williams, K., Sansoni, J., Morris, D., Grootemaat, P., & Thompson, C. (2016). Patient-reported outcome measures: Literature review. In ACSQHC (ed.). Sydney: Australian Commission on Safety and Quality in Health Care.
3.
Ellwood, P. M. (1988). Shattuck lecture—outcomes management: A technology of patient experience. New England Journal of Medicine, 318(23), 1549–1556.CrossRefPubMed
4.
Marshall, S., Haywood, K., & Fitzpatrick, R. (2006). Impact of patient-reported outcome measures on routine practice: A structured review. Journal of Evaluation in Clinical Practice, 12(5), 559–568.CrossRefPubMed
5.
Zumbo, B. D., & Hubley, A. M. (Eds.). (2017). Understanding and investigating response processes in validation research (Vol. 69). Social Indicators Research Series). Cham: Springer International Publishing.
6.
Zumbo, B. D. (2009). Validity as contextualised and pragmatic explanation, and its implications for validation practice. In R. W. Lissitz (Ed.), The concept of validity: Revisions, new directions, and applications (pp. 65–82). Charlotte, NC: IAP - Information Age Publishing, Inc.
7.
Thompson, C., Sonsoni, J., Morris, D., Capell, J., & Williams, K. (2016). Patient-reported outcomes measures: An environmental scan of the Australian healthcare sector. In ACSQHC (Ed.). Sydney: Australian Commission on Safety and Quality in Health Care.
8.
Lohr, K. N. (2002). Assessing health status and quality-of-life instruments: Attributes and review criteria. Quality of Life Research, 11(3), 193–205.CrossRefPubMed
9.
McClimans, L. (2010). A theoretical framework for patient-reported outcome measures. Theoretical Medicine and Bioethics, 31(3), 225–240.CrossRefPubMed
10.
Zumbo, B. D., & Chan, E. K. (Eds.). (2014). Validity and validation in social, behavioral, and health sciences (Social Indicators Research Series, Vol. 54. Cham: Springer International Publishing.
11.
American Educational Research Association, American Psychological Association, & National Council on Measurement in Education (2014). Standards for educational and psychological testing. Washington, DC: American Educational Research Association.
12.
Kane, M. (2013). The argument-based approach to validation. School Psychology Review, 42(4), 448–457.
13.
Food, U. S., & Administration, D. (2009). Center for Drug Evaluation and Research, Center for Biologics Evaluation and Research. In Center for Devices and Radiological Health (Ed.), Guidance for industry: Patient-reported outcome measures: Use in medical product development to support labeling claims. Federal Register (Vol. 74, pp. 65132–65133). Silver Spring: U.S. Department of Health and Human Services Food and Drug Administration.
14.
Terwee, C. B., Mokkink, L. B., Knol, D. L., Ostelo, R. W., Bouter, L. M., & de Vet, H. C. (2012). Rating the methodological quality in systematic reviews of studies on measurement properties: A scoring system for the COSMIN checklist. Quality of Life Research, 21(4), 651–657.CrossRefPubMed
15.
Reeve, B. B., Wyrwich, K. W., Wu, A. W., Velikova, G., Terwee, C. B., Snyder, C. F., et al. (2013). ISOQOL recommends minimum standards for patient-reported outcome measures used in patient-centered outcomes and comparative effectiveness research. Quality of Life Research, 22(8), 1889–1905.CrossRefPubMed
16.
Castillo-Díaz, M., & Padilla, J.-L. (2013). How cognitive interviewing can provide validity evidence of the response processes to scale items. Social Indicators Research, 114(3), 963–975.CrossRef
17.
Gadermann, A. M., Guhn, M., & Zumbo, B. D. (2011). Investigating the substantive aspect of construct validity for the satisfaction with life scale adapted for children: A focus on cognitive processes. Social Indicators Research, 100(1), 37–60.CrossRef
18.
Cronbach, L. J., & Meehl, P. E. (1955). Construct validity in psychological tests. Psychological Bulletin, 52(4), 281.CrossRefPubMed
19.
Messick, S. (1980). Test validity and the ethics of assessment. American Psychologist, 35(11), 1012.CrossRef
20.
Sireci, S. G. (2007). On validity theory and test validation. Educational Researcher, 36(8), 477–481.CrossRef
21.
Shepard, L. A. (1997). The centrality of test use and consequences for test validity. Educational Measurement: Issues and Practice, 16(2), 5–24.CrossRef
22.
Moss, P. A., Girard, B. J., & Haniford, L. C. (2006). Validity in educational assessment. Review of Research in Education, 30, 109–162.CrossRef
23.
Kane, M. T. (1992). An argument-based approach to validity. Psychological Bulletin, 112(3), 527–535.CrossRef
24.
Hubley, A. M., & Zumbo, B. D. (2011). Validity and the consequences of test interpretation and use. Social Indicators Research, 103(2), 219.CrossRef
25.
Cronbach, L. J. (1971). Test validation. In R. L. Thorndike, W. H. Angoff & E. F. Lindquist (Eds.), Educational measurement (pp. 483–507). Washington, DC: American Council on Education.
26.
Anastasi, A. (1950). The concept of validity in the interpretation of test scores. Educational and Psychological Measurement, 10(1), 67–78.CrossRef
27.
Nelson, E., Hvitfeldt, H., Reid, R., Grossman, D., Lindblad, S., Mastanduno, M., et al. (2012). Using patient-reported information to improve health outcomes and health care value: Case studies from Dartmouth, Karolinska and Group Health. Darmount: The Dartmouth Institute for Health Policy and Clinical Practice and Centre for Population Health.
28.
29.
Moss, P. A. (1998). The role of consequences in validity theory. Educational Measurement: Issues and Practice, 17(2), 6–12.CrossRef
30.
Wild, D., Grove, A., Martin, M., Eremenco, S., McElroy, S., Verjee-Lorenz, A., et al. (2005). Principles of good practice for the translation and cultural adaptation process for patient-reported outcomes (PRO) measures: Report of the ISPOR Task Force for Translation and Cultural Adaptation. Value in Health, 8(2), 94–104.CrossRefPubMed
31.
Buchbinder, R., Batterham, R., Elsworth, G., Dionne, C. E., Irvin, E., & Osborne, R. H. (2011). A validity-driven approach to the understanding of the personal and societal burden of low back pain: Development of a conceptual and measurement model. Arthritis Research & Therapy, 13(5), R152. https://​doi.​org/​10.​1186/​ar3468.CrossRef
32.
Sireci, S. G. (1998). The construct of content validity. Social Indicators Research, 45(1–3), 83–117.CrossRef
33.
Shepard, L. A. (1993). Evaluating test validity. In L. Darling-Hammond (Ed.), Review of research in education (Vol. 19, pp. 405–450), Washington, DC: American Educational Research Asociation.
34.
American Psychological Association, American Educational Research Association, & National Council on Measurement in Education (1954). Technical recommendations for psychological tests and diagnostic techniques (Vol. 51), Washington, DC: American Psychological Association.
35.
Camara, W. J., & Lane, S. (2006). A historical perspective and current views on the standards for educational and psychological testing. Educational Measurement: Issues and Practice, 25(3), 35–41.CrossRef
36.
37.
American Psychological Association, American Educational Research Association, National Council on Measurement in Education, & American Educational Research Association Committee on Test Standards (1966). Standards for educational and psychological tests and manuals, Washington, DC: American Psychological Association.
38.
Moss, P. A. (2007). Reconstructing validity. Educational Researcher, 36(8), 470–476.CrossRef
39.
American Psychological Association, American Educational Research Association, & National Council on Measurement in Education (1974). Standards for educational & psychological tests, Washington, DC: American Psychological Association.
40.
Messick, S. (1995). Standards of validity and the validity of standards in performance asessment. Educational Measurement: Issues and Practice, 14(4), 5–8.CrossRef
41.
Messick, S. (1995). Validity of psychological assessment: Validation of inferences from persons’ responses and performances as scientific inquiry into score meaning. American Psychologist, 50(9), 741.CrossRef
42.
Messick, S. (1989). Meaning and values in test validation: The science and ethics of assessment. Educational Researcher, 18(2), 5–11.CrossRef
43.
American Educational Research Association, American Psychological Association, & National Council on Measurement in Education (1985). Standards for educational and psychological testing, Washington, DC: American Educational Research Association.
44.
American Educational Research Association, American Psychological Association, Joint Committee on Standards for Educational and Psychological Testing (U.S.), & National Council on Measurement in Education (1999). Standards for educational and psychological testing. Washington, DC: American Educational Research Association.
45.
Kane, M. T. (2013). Validating the interpretations and uses of test scores. Journal of Educational Measurement, 50(1), 1–73.CrossRef
46.
Messick, S. (1990). Validity of test interpretation and use. ETS Research Report Series, 1990(1), 1487–1495.CrossRef
47.
Messick, S. (1992). The Interplay of Evidence and consequences in the validation of performance assessments. Princeton: Educational Testing Service.
48.
Litwin, M. S. (1995). How to measure survey reliability and validity (Vol. 7), Thousand Oaks: SAGE.CrossRef
49.
McDowell, I. (2006). Measuring health: A guide to rating scales and questionnaires. New York: Oxford University Press.CrossRef
50.
Kane, M. T. (1990). An argument-based approach to validation. In ACT research report series. Iowa: The American College Testing Program.
51.
Kane, M. (2010). Validity and fairness. Language Testing, 27(2), 177–182.CrossRef
52.
53.
Anthoine, E., Moret, L., Regnault, A., Sébille, V., & Hardouin, J.-B. (2014). Sample size used to validate a scale: A review of publications on newly-developed patient reported outcomes measures. Health and Quality of Life Outcomes, 12(1), 2.CrossRefPubMedCentral
54.
55.
56.
Hawkins, M., Gill, S. D., Batterham, R., Elsworth, G. R., & Osborne, R. H. (2017). The Health Literacy Questionnaire (HLQ) at the patient-clinician interface: A qualitative study of what patients and clinicians mean by their HLQ scores. BMC Health Services Research, 17(1), 309.CrossRefPubMedPubMedCentral
57.
Beauchamp, A., Buchbinder, R., Dodson, S., Batterham, R. W., Elsworth, G. R., McPhee, C., et al. (2015). Distribution of health literacy strengths and weaknesses across socio-demographic groups: A cross-sectional survey using the Health Literacy Questionnaire (HLQ). BMC Public Health, 15, 678.CrossRefPubMedPubMedCentral
58.
59.
Morris, R. L., Soh, S.-E., Hill, K. D., Buchbinder, R., Lowthian, J. A., Redfern, J., et al. (2017). Measurement properties of the Health Literacy Questionnaire (HLQ) among older adults who present to the emergency department after a fall: A Rasch analysis. BMC Health Services Research, 17(1), 605.CrossRefPubMedPubMedCentral
60.
61.
62.
Kolarčik, P., Cepova, E., Geckova, A. M., Elsworth, G. R., Batterham, R. W., & Osborne, R. H. (2017). Structural properties and psychometric improvements of the health literacy questionnaire in a Slovak population. International Journal of Public Health, 62(5), 591–604.CrossRefPubMed
63.
Busija, L., Buchbinder, R., & Osborne, R. (2016). Development and preliminary evaluation of the OsteoArthritis Questionnaire (OA-Quest): A psychometric study. Osteoarthritis and Cartilage, 24(8), 1357–1366.CrossRefPubMed
64.
Batterham, R. W., Buchbinder, R., Beauchamp, A., Dodson, S., Elsworth, G. R., & Osborne, R. H. (2014). The OPtimising HEalth LIterAcy (Ophelia) process: Study protocol for using health literacy profiling and community engagement to create and implement health reform. BMC Public Health, 14(1), 694.CrossRefPubMedPubMedCentral
65.
Friis, K., Lasgaard, M., Osborne, R. H., & Maindal, H. T. (2016). Gaps in understanding health and engagement with healthcare providers across common long-term conditions: A population survey of health literacy in 29 473 Danish citizens. British Medical Journal Open, 6(1), e009627.
66.
67.
Griva, K., Mooppil, N., Khoo, E., Lee, V. Y. W., Kang, A. W. C., & Newman, S. P. (2015). Improving outcomes in patients with coexisting multimorbid conditions—the development and evaluation of the combined diabetes and renal control trial (C-DIRECT): Study protocol. British Medical Journal Open, 5(2), e007253.
68.
Morris, R., Brand, C., Hill, K. D., Ayton, D., Redfern, J., Nyman, S., et al. (2014). RESPOND: A patient-centred programme to prevent secondary falls in older people presenting to the emergency department with a fall—protocol for a mixed methods programme evaluation. Injury Prevention, injuryprev-2014-041453.
69.
Redfern, J., Usherwood, T., Harris, M., Rodgers, A., Hayman, N., Panaretto, K., et al. (2014). A randomised controlled trial of a consumer-focused e-health strategy for cardiovascular risk management in primary care: The Consumer Navigation of Electronic Cardiovascular Tools (CONNECT) study protocol. British Medical Journal Open, 4(2), e004523.
70.
Banbury, A., Parkinson, L., Nancarrow, S., Dart, J., Gray, L., & Buckley, J. (2014). Multi-site videoconferencing for home-based education of older people with chronic conditions: The Telehealth Literacy Project. Journal of Telemedicine and Telecare, 20(7), 353–359.CrossRefPubMed
71.
Faruqi, N., Stocks, N., Spooner, C., el Haddad, N., & Harris, M. F. (2015). Research protocol: Management of obesity in patients with low health literacy in primary health care. BMC Obesity, 2(1), 1.CrossRef
72.
Livingston, P. M., Osborne, R. H., Botti, M., Mihalopoulos, C., McGuigan, S., Heckel, L., et al. (2014). Efficacy and cost-effectiveness of an outcall program to reduce carer burden and depression among carers of cancer patients [PROTECT]: Rationale and design of a randomized controlled trial. BMC Health Services Research, 14(1), 1.CrossRef
73.
Beauchamp, A., Batterham, R. W., Dodson, S., Astbury, B., Elsworth, G. R., McPhee, C., et al. (2017). Systematic development and implementation of interventions to Optimise Health Literacy and Access (Ophelia). BMC Public Health, 17(1), 230.CrossRefPubMedPubMedCentral
74.
75.
76.
Rabin, R., Gudex, C., Selai, C., & Herdman, M. (2014). From translation to version management: A history and review of methods for the cultural adaptation of the EuroQol five-dimensional questionnaire. Value in Health, 17(1), 70–76.CrossRefPubMed
77.
Kolarčik, P., Čepová, E., Gecková, A. M., Tavel, P., & Osborne, R. (2015). Validation of Slovak version of Health Literacy Questionnaire. The European Journal of Public Health, 25(suppl 3), ckv176. 151.
78.
Vamos, S., Yeung, P., Bruckermann, T., Moselen, E. F., Dixon, R., Osborne, R. H., et al. (2016). Exploring health literacy profiles of Texas university students. Health Behavior and Policy Review, 3(3), 209–225.CrossRef
79.
Kolarčik, P., Belak, A., & Osborne, R. H. (2015). The Ophelia (OPtimise HEalth LIteracy and Access) Process. Using health literacy alongside grounded and participatory approaches to develop interventions in partnership with marginalised populations. European Health Psychologist, 17(6), 297–304.
80.
Fornell, C., & Larcker, D. F. (1981). Evaluating structural equation models with unobservable variables and measurement error. Journal of Marketing Research, 1981, 39–50.CrossRef
81.
Farrell, A. M. (2010). Insufficient discriminant validity: A comment on Bove, Pervan, Beatty. and Shiu (2009). Journal of Business Research, 63(3), 324–327.CrossRef
82.
Campbell, D. T., & Fiske, D. W. (1959). Convergent and discriminant validation by the multitrait-multimethod matrix. Psychological Bulletin, 56(2), 81.CrossRefPubMed
83.
Epstein, J., Santo, R. M., & Guillemin, F. (2015). A review of guidelines for cross-cultural adaptation of questionnaires could not bring out a consensus. Journal of Clinical Epidemiology, 68(4), 435–441.CrossRefPubMed
84.
Kuliś, D., Bottomley, A., Velikova, G., Greimel, E., & Koller, M. (2016). EORTC quality of life group translation procedure. (4th edn.). Brussels: EORTC
85.
Enright, M., & Tyson, E. (2011). Validity evidence supporting the interpretation and use of TOEFL iBT scores. (Vol. 4). Princeton, NJ: TOEFL iBT Research Insight:
Metadata
Title
Application of validity theory and methodology to patient-reported outcome measures (PROMs): building an argument for validity
Authors
Melanie Hawkins
Gerald R. Elsworth
Richard H. Osborne
Publication date
01-07-2018
Publisher
Springer International Publishing
Published in
Quality of Life Research / Issue 7/2018
Print ISSN: 0962-9343
Electronic ISSN: 1573-2649
DOI
https://doi.org/10.1007/s11136-018-1815-6

Other articles of this Issue 7/2018

Quality of Life Research 7/2018 Go to the issue

OriginalPaper

Quality of life of patients with gastrointestinal cancers undergoing chemotherapy

Brief Communication

Activity pacing: moving beyond taking breaks and slowing down

Special Section: Test Construction (by invitation only)

The use of latent variable mixture models to identify invariant items in test construction

OriginalPaper

Psychometric evaluation of the Chinese version of the Child Health Utility 9D (CHU9D-CHN): a school-based study in China

Editorial

Introduction to special section: test construction

OriginalPaper

Evaluation of options for presenting health-states from PROMIS® item banks for valuation exercises