Skip to main content
Key Specialties
Cardiology Diabetology Endocrinology Gastroenterology Geriatrics and Gerontology Gynecology and Obstetrics Hematology Infectious Disease Internal Medicine Nephrology Neurology Oncology Pediatrics Respiratory Medicine Rheumatology Surgery
Congress Coverage
2024 ACC Congress 2023 ESMO Congress 2023 EASD Congress 2023 ERS Congress 2023 ESC Congress
Clinical Collections
Advances in Alzheimer’s

Keynote webinar | Spotlight on medication adherence

LIVE: Thursday 27th June 2024, 18:00-19:30 (CEST)
Join our expert panel to discover why you need to understand the drivers of non-adherence in your patients, and how you can optimize medication adherence in your clinics to drastically improve patient outcomes.
ADVANCED SEARCH
Log in
Top
Systematic Reviews
Published in: Systematic Reviews 1/2016

Open Access 01-12-2016 | Methodology

Checklist to operationalize measurement characteristics of patient-reported outcome measures

Authors: David O. Francis, Melissa L. McPheeters, Meaghan Noud, David F. Penson, Irene D. Feurer

Published in: Systematic Reviews | Issue 1/2016

Login to get access

Abstract

Background

The purpose of this study was to advance a checklist of evaluative criteria designed to assess patient-reported outcome (PRO) measures’ developmental measurement properties and applicability, which can be used by systematic reviewers, researchers, and clinicians with a varied range of expertise in psychometric measure development methodology.

Methods

A directed literature search was performed to identify original studies, textbooks, consensus guidelines, and published reports that propose criteria for assessing the quality of PRO measures. Recommendations from these sources were iteratively distilled into a checklist of key attributes. Preliminary items underwent evaluation through 24 cognitive interviews with clinicians and quantitative researchers. Six measurement theory methodological novices independently applied the final checklist to assess six PRO measures encompassing a variety of methods, applications, and clinical constructs. Agreement between novice and expert scores was assessed.

Results

The distillation process yielded an 18-item checklist with six domains: (1) conceptual model, (2) content validity, (3) reliability, (4) construct validity, (5) scoring and interpretation, and (6) respondent burden and presentation. With minimal instruction, good agreement in checklist item ratings was achieved between quantitative researchers with expertise in measurement theory and less experienced clinicians (mean kappa 0.70; range 0.66–0.87).

Conclusions

We present a simplified checklist that can help guide systematic reviewers, researchers, and clinicians with varied measurement theory expertise to evaluate the strengths and weakness of candidate PRO measures’ developmental properties and the appropriateness for specific applications.
Appendix
Available only for authorised users
Literature
1.
Rosser R. A history of the development of health indicators. In: Teeling-Smith G, editor. Measure the social benefits of medicine. London: Office of Health Economics; 1983. p. 50–63.
2.
3.
4.
McDowell I, Newell C. Measuring health: a guide to rating scales and questionnaires. 2nd ed. New York: Oxford University Press; 1996. xix, 523 pp.
5.
Patrick DL, Burke LB, Powers JH, Scott JA, Rock EP, Dawisha S, et al. Patient-reported outcomes to support medical product labeling claims: FDA perspective. Value Health. 2007;10 Suppl 2:S125–37.CrossRefPubMed
6.
Guidance for Industry. Patient-reported outcome measures: use in medical product development to support labeling claims. Rockville, MD: 2009
7.
8.
Reeve BB, Wyrwich KW, Wu AW, Velikova G, Terwee CB, Snyder CF, et al. ISOQOL recommends minimum standards for patient-reported outcome measures used in patient-centered outcomes and comparative effectiveness research. Qual Life Res Int J Qual Life Asp Treat Care Rehab. 2013;22(8):1889–905.CrossRef
9.
Guyatt G, Schunemann H. How can quality of life researchers make their work more useful to health workers and their patients? Qual Life Res Int J Qual Life Asp Treat Care Rehab. 2007;16(7):1097–105.CrossRef
10.
Feeny DH, Eckstrom E, Whitlock EP, Perdue LA. A primer for systematic reviewers on the measurement of functional status and health-related quality of life in older adults (prepared by the Kaiser Permanente Affiliates Evidence-based Practice Center under contract 290-2007-10057-I.) ARHQ Publication No. 13-EHC128-EF. Rockville: Agency for Healthcare Research and Quality; 2013
11.
Mokkink LB, Terwee CB, Patrick DL, Alonso J, Stratford PW, Knol DL, et al. The COSMIN study reached international consensus on taxonomy, terminology, and definitions of measurement properties for health-related patient-reported outcomes. J Clin Epidemiol. 2010;63(7):737–45.CrossRefPubMed
12.
Patrick DL, Erickson P. Health status and health policy: quality of life in health care evaluation and resource allocation. New York: Oxford University Press; 1993. xxv, 478 pp.
13.
Guyatt GH, Feeny DH, Patrick DL. Measuring health-related quality of life. Ann Intern Med. 1993;118(8):622–9.CrossRefPubMed
14.
Patrick DL, Bergner M. Measurement of health status in the 1990s. Annu Rev Public Health. 1990;11:165–83.CrossRefPubMed
15.
Gill TM, Feinstein AR. A critical appraisal of the quality of quality-of-life measurements. JAMA. 1994;272(8):619–26.CrossRefPubMed
16.
Mokkink LB, Terwee CB, Knol DL, Stratford PW, Alonso J, Patrick DL, et al. Protocol of the COSMIN study: COnsensus-based Standards for the selection of health Measurement INstruments. BMC Med Res Methodol. 2006;6:2.CrossRefPubMedPubMedCentral
17.
Weldring T, Smith SM. Patient-reported outcomes (PROs) and patient-reported outcome measures (PROMs). Health Serv Insights. 2013;6:61–8.PubMedPubMedCentral
18.
Guyatt G, Montori V, Devereaux PJ, Schunemann H, Bhandari M. Patients at the center: in our practice, and in our use of language. ACP J Club. 2004;140(1):A11–2.PubMed
19.
Garrow AP, Khan N, Tyson S, Vestbo J, Singh D, Yorke J. The development and first validation of the Manchester Early Morning Symptoms Index (MEMSI) for patients with COPD. Thorax. 2015;70(8):757–63.CrossRefPubMed
20.
Hutchings HA, Cheung WY, Russell IT, Durai D, Alrubaiy L, Williams JG. Psychometric development of the Gastrointestinal Symptom Rating Questionnaire (GSRQ) demonstrated good validity. J Clin Epidemiol. 2015;68(10):1176–83.CrossRefPubMed
21.
Jenkinson C, Coulter A, Wright L. Short form 36 (SF36) health survey questionnaire: normative data for adults of working age. BMJ. 1993;306(6890):1437–40.CrossRefPubMedPubMedCentral
22.
Cuervo J, Castejon N, Khalaf KM, Waweru C, Globe D, Patrick DL. Development of the Incontinence Utility Index: estimating population-based utilities associated with urinary problems from the Incontinence Quality of Life Questionnaire and Neurogenic Module. Health Qual Life Outcomes. 2014;12:147.CrossRefPubMedPubMedCentral
23.
Andrae DA, Patrick DL, Drossman DA, Covington PS. Evaluation of the Irritable Bowel Syndrome Quality of Life (IBS-QOL) questionnaire in diarrheal-predominant irritable bowel syndrome patients. Health Qual Life Outcomes. 2013;11:208.CrossRefPubMedPubMedCentral
24.
Wang Q, Furlong W, Feeny D, Torrance G, Barr R. How robust is the Health Utilities Index Mark 2 utility function? Med Decis Making. 2002;22(4):350–8.CrossRefPubMed
25.
Kopec JA, Schultz SE, Goel V, Ivan WJ. Can the health utilities index measure change? Med Care. 2001;39(6):562–74.CrossRefPubMed
26.
Pouwer F, Snoek FJ, van der Ploeg HM, Ader HJ, Heine RJ. The well-being questionnaire: evidence for a three-factor structure with 12 items (W-BQ12). Psychol Med. 2000;30(2):455–62.CrossRefPubMed
27.
Monk M. Blood pressure awareness and psychological well-being in the health and nutrition examination survey. Clin Invest Med. 1981;4(3–4):183–9.PubMed
28.
29.
Riehm KE, Kwakkenbos L, Carrier ME, Bartlett SJ, Malcarne VL, Mouthon L, et al. Validation of the Self-Efficacy for Managing Chronic Disease (SEMCD) scale: a Scleroderma Patient-centered Intervention Network (SPIN) cohort study. Arthritis care & research. 2016;8(68):1195–200.
30.
Pleil AM, Coyne KS, Reese PR, Jumadilova Z, Rovner ES, Kelleher CJ. The validation of patient-rated global assessments of treatment benefit, satisfaction, and willingness to continue—the BSW. Value Health. 2005;8 Suppl 1:S25–34.CrossRefPubMed
31.
Wegener ST, Castillo RC, Heins SE, Bradford AN, Newell MZ, Pollak AN, et al. The development and validation of the readiness to engage in self-management after acute traumatic injury questionnaire. Rehabil Psychol. 2014;59(2):203–10.CrossRefPubMed
32.
Health USDo, Human Services FDACfDE, Research, Health USDo, Human Services FDACfBE, Research, et al. Guidance for industry: patient-reported outcome measures: use in medical product development to support labeling claims: draft guidance. Health Qual Life Outcomes. 2006;4:79.CrossRef
33.
Johnston BC, Ebrahim S, Carrasco-Labra A, Furukawa TA, Patrick DL, Crawford MW, et al. Minimally important difference estimates and methods: a protocol. BMJ Open. 2015;5(10), e007953.CrossRefPubMedPubMedCentral
34.
Terwee CB, Mokkink LB, Knol DL, Ostelo RW, Bouter LM, de Vet HC. Rating the methodological quality in systematic reviews of studies on measurement properties: a scoring system for the COSMIN checklist. Qual Life Res Int J Qual Life Asp Treat Care Rehab. 2012;21(4):651–7.CrossRef
35.
Scoggins JF, Patrick DL. The use of patient-reported outcomes instruments in registered clinical trials: evidence from ClinicalTrials.gov. Contemp Clin Trials. 2009;30(4):289–92.CrossRefPubMedPubMedCentral
36.
Wyrwich KW, Wolinsky FD. Identifying meaningful intra-individual change standards for health-related quality of life measures. J Eval Clin Pract. 2000;6(1):39–49.CrossRefPubMed
37.
Zheng H, Li W, Harrold L, Ayers DC, Franklin PD. Web-based comparative patient-reported outcome feedback to support quality improvement and comparative effectiveness research in total joint replacement. EGEMS (Wash DC). 2014;2(1):1130.
38.
Black N, Burke L, Forrest CB, Ravens Sieberer UH, Ahmed S, Valderas JM, et al. Patient-reported outcomes: pathways to better health, better services, and better societies. Qual Life Res Int J Qual Life Asp Treat Care Rehab. 2016;25(5):1103–12. doi:10.​1007/​s11136-015-1168-3. Epub 2015 Nov 13.
39.
Basch E, Spertus J, Dudley RA, Wu A, Chuahan C, Cohen P, et al. Methods for developing patient-reported outcome-based performance measures (PRO-PMs). Value Health. 2015;18(4):493–504.CrossRefPubMed
40.
Anastasi A. Psychological testing. New York: Macmillian Publishing Company; 1988.
41.
Feinstein AR. An additional basic science for clinical medicine: IV. The development of clinimetrics. Ann Intern Med. 1983;99(6):843–8.CrossRefPubMed
42.
Terwee CB, Bot SD, de Boer MR, van der Windt DA, Knol DL, Dekker J, et al. Quality criteria were proposed for measurement properties of health status questionnaires. J Clin Epidemiol. 2007;60(1):34–42.CrossRefPubMed
43.
Aaronson N, Alonso J, Burnam A, Lohr KN, Patrick DL, Perrin E, et al. Assessing health status and quality-of-life instruments: attributes and review criteria. Qual Life Res Int J Qual Life Asp Treat Care Rehab. 2002;11(3):193–205.CrossRef
44.
Mokkink LB, Terwee CB, Patrick DL, Alonso J, Stratford PW, Knol DL, et al. The COSMIN checklist for assessing the methodological quality of studies on measurement properties of health status measurement instruments: an international Delphi study. Qual Life Res Int J Qual Life Asp Treat Care Rehab. 2010;19(4):539–49.CrossRef
45.
Mokkink LB, Terwee CB, Stratford PW, Alonso J, Patrick DL, Riphagen I, et al. Evaluation of the methodological quality of systematic reviews of health status measurement instruments. Qual Life Res Int J Qual Life Asp Treat Care Rehab. 2009;18(3):313–33.CrossRef
46.
Mokkink LB, Terwee CB, Knol DL, Stratford PW, Alonso J, Patrick DL, et al. The COSMIN checklist for evaluating the methodological quality of studies on measurement properties: a clarification of its content. BMC Med Res Methodol. 2010;10:22.CrossRefPubMedPubMedCentral
47.
Regnault A, Hamel JF, Patrick DL. Pooling of cross-cultural PRO data in multinational clinical trials: how much can poor measurement affect statistical power? Qual Life Res Int J Qual Life Asp Treat Care Rehab. 2015;24(2):273–7.CrossRef
48.
Mokkink LB, Terwee CB, Gibbons E, Stratford PW, Alonso J, Patrick DL, et al. Inter-rater agreement and reliability of the COSMIN (COnsensus-based Standards for the selection of health status Measurement Instruments) checklist. BMC Med Res Methodol. 2010;10:82.CrossRefPubMedPubMedCentral
49.
Lohr KN, Aaronson NK, Alonso J, Burnam MA, Patrick DL, Perrin EB, et al. Evaluating quality-of-life and health status instruments: development of scientific review criteria. Clin Ther. 1996;18(5):979–92.CrossRefPubMed
50.
American Psychological Association AERA, National Council on Measurement Used in Education. Technical recommendations for psychological tests and diagnostic techniques. Psychol Bull. 1954;51(2, pt 2):1–38.CrossRef
51.
American Psychological Association AERA. Standards for educational and psychological testing. Washington, D.C.: American Educational Research Association; 1999.
52.
Streiner DL, Norman GR. Health measurement scales: a practical guide to their development and use. 4th ed. Oxford: New York: Oxford University Press; 2008. xvii, 431 pp.CrossRef
53.
McDowell I. Measuring health: a guide to rating scales and questionnaires. 3rd ed. Oxford; New York: Oxford University Press; 2006. xvi, 748 pp.CrossRef
54.
Spilker B. Quality of life and pharmacoeconomics in clinical trials. 2nd ed. Philadelphia: Lippincott-Raven; 1996. xlv, 1259 pp.
55.
Cappelleri JC, Jason Lundy J, Hays RD. Overview of classical test theory and item response theory for the quantitative assessment of items in developing patient-reported outcomes measures. Clin Ther. 2014;36(5):648–62.CrossRefPubMedPubMedCentral
56.
Allen MJ, Yen WM. Introduction to measurement theory. Monterey, CA: Brooks/Cole Publishing Company; 1979.
57.
Wright JG, Feinstein AR. A comparative contrast of clinimetric and psychometric methods for constructing indexes and rating scales. J Clin Epidemiol. 1992;45(11):1201–18.CrossRefPubMed
58.
Wilson IB, Cleary PD. Linking clinical variables with health-related quality of life. A conceptual model of patient outcomes. JAMA. 1995;273(1):59–65.CrossRefPubMed
59.
Dahl TH. International classification of functioning, disability and health: an introduction and discussion of its potential impact on rehabilitation services and research. J Rehabil Med. 2002;34(5):201–4.CrossRefPubMed
60.
World Health Organization. International classification of functioning, disability, and health. Geneva: World Health Organization; 2001.
61.
Magasi S, Ryan G, Revicki D, Lenderking W, Hays RD, Brod M, et al. Content validity of patient-reported outcome measures: perspectives from a PROMIS meeting. Qual Life Res Int J Qual Life Asp Treat Care Rehab. 2012;21(5):739–46.CrossRef
62.
Patrick DL, Burke LB, Gwaltney CJ, Leidy NK, Martin ML, Molsen E, et al. Content validity–establishing and reporting the evidence in newly developed patient-reported outcomes (PRO) instruments for medical product evaluation: ISPOR PRO good research practices task force report: part 1—eliciting concepts for a new PRO instrument. Value Health. 2011;14(8):967–77.CrossRefPubMed
63.
Helmstadter GC. Principles of psychological measurement. New York: Appleton; 1964. xx, 248 pp.
64.
Liang MH. Longitudinal construct validity: establishment of clinical meaning in patient evaluative instruments. Med Care. 2000;38(9 Suppl):II84–90.PubMed
65.
Hays RD, Hadorn D. Responsiveness to change: an aspect of validity, not a separate dimension. Qual Life Res Int J Qual Life Asp Treat Care Rehab. 1992;1(1):73–5.CrossRef
66.
Revicki D, Hays RD, Cella D, Sloan J. Recommended methods for determining responsiveness and minimally important differences for patient-reported outcomes. J Clin Epidemiol. 2008;61(2):102–9.CrossRefPubMed
67.
Guyatt G, Walter S, Norman G. Measuring change over time: assessing the usefulness of evaluative instruments. J Chronic Dis. 1987;40(2):171–8.CrossRefPubMed
68.
Jaeschke R, Singer J, Guyatt GH. Measurement of health status. Ascertaining the minimal clinically important difference. Control Clin Trials. 1989;10(4):407–15.CrossRefPubMed
69.
Juniper EF, Guyatt GH, Willan A, Griffith LE. Determining a minimal important change in a disease-specific Quality of Life Questionnaire. J Clin Epidemiol. 1994;47(1):81–7.CrossRefPubMed
70.
Idler EL, Kasl SV, Lemke JH. Self-evaluated health and mortality among the elderly in New Haven, Connecticut, and Iowa and Washington counties, Iowa, 1982–1986. Am J Epidemiol. 1990;131(1):91–103.PubMed
71.
Kroenke K, Monahan PO, Kean J. Pragmatic characteristics of patient-reported outcome measures are important for use in clinical practice. J Clin Epidemiol. 2015;68(9):1085–92.CrossRefPubMed
72.
Guyatt GH, Osoba D, Wu AW, Wyrwich KW, Norman GR, Clinical Significance Consensus Meeting G. Methods to explain the clinical significance of health status measures. Mayo Clin Proc. 2002;77(4):371–83.CrossRefPubMed
73.
Schunemann HJ, Puhan M, Goldstein R, Jaeschke R, Guyatt GH. Measurement properties and interpretability of the Chronic respiratory disease questionnaire (CRQ). COPD. 2005;2(1):81–9.CrossRefPubMed
74.
Ware Jr JE, Keller SD, Hatoum HT, Kong SX. The SF-36 Arthritis-Specific Health Index (ASHI): I. Development and cross-validation of scoring algorithms. Med Care. 1999;37(5 Suppl):MS40–50.PubMed
75.
76.
Brozek JL, Guyatt GH, Schunemann HJ. How a well-grounded minimal important difference can enhance transparency of labelling claims and improve interpretation of a patient reported outcome measure. Health Qual Life Outcomes. 2006;4:69.CrossRefPubMedPubMedCentral
77.
Meyer KB, Clayton KA. Measurement and analysis of patient-reported outcomes. Methods Mol Biol. 2009;473:155–69.CrossRefPubMed
78.
Calvert M, Blazeby J, Altman DG, Revicki DA, Moher D, Brundage MD, et al. Reporting of patient-reported outcomes in randomized trials: the CONSORT PRO extension. JAMA. 2013;309(8):814–22.CrossRefPubMed
79.
Chenok K, Teleki S, SooHoo NF, Huddleston 3rd J, Bozic KJ. Collecting patient-reported outcomes: lessons from the California Joint Replacement Registry. EGEMS (Wash DC). 2015;3(1):1196.
80.
Jahagirdar D, Kroll T, Ritchie K, Wyke S. Patient-reported outcome measures for chronic obstructive pulmonary disease: the exclusion of people with low literacy skills and learning disabilities. The Patient. 2013;6(1):11–21.CrossRefPubMedPubMedCentral
81.
Jacobson B, Johnson A, Grywalski C, Silbergleit A, Jacobson G, Benninger MS. The Voice Handicap Index (VHI): development and validation. Am J Speech-Language Pathology. 1997;6:66–70.CrossRef
82.
Rosen CA, Lee AS, Osborne J, Zullo T, Murry T. Development and validation of the voice handicap index-10. Laryngoscope. 2004;114(9):1549–56.CrossRefPubMed
83.
Ma EP, Yiu EM. Voice activity and participation profile: assessing the impact of voice disorders on daily activities. Journal of speech, language, and hearing research : JSLHR. 2001;44(3):511–24.CrossRefPubMed
84.
Epstein R, Hirani SP, Stygall J, Newman SP. How do individuals cope with voice disorders? Introducing the Voice Disability Coping Questionnaire. J Voice. 2009;23(2):209–17.CrossRefPubMed
85.
Hogikyan ND, Sethuraman G. Validation of an instrument to measure voice-related quality of life (V-RQOL). J Voice. 1999;13(4):557–69.CrossRefPubMed
86.
Urbach DR, Tomlinson GA, Harnish JL, Martino R, Diamant NE. A measure of disease-specific health-related quality of life for achalasia. Am J Gastroenterol. 2005;100(8):1668–76.CrossRefPubMed
87.
Ma EP, Yiu EM. Scaling voice activity limitation and participation restriction in dysphonic individuals. Folia Phoniatr Logop. 2007;59(2):74–82.CrossRefPubMed
88.
Feinstein AR. Benefits and obstacles for development of health status assessment measures in clinical settings. Med Care. 1992;30(5 Suppl):MS50–6.PubMed
89.
Ahmed S, Berzon RA, Revicki DA, Lenderking WR, Moinpour CM, Basch E, et al. The use of patient-reported outcomes (PRO) within comparative effectiveness research: implications for clinical practice and health care policy. Med Care. 2012;50(12):1060–70.CrossRefPubMed
90.
Newton PE, Shaw SD. Standards for talking and thinking about validity. Psychol Methods. 2013;18(3):301–19.CrossRefPubMed
Metadata
Title
Checklist to operationalize measurement characteristics of patient-reported outcome measures
Authors
David O. Francis
Melissa L. McPheeters
Meaghan Noud
David F. Penson
Irene D. Feurer
Publication date
01-12-2016
Publisher
BioMed Central
Published in
Systematic Reviews / Issue 1/2016
Electronic ISSN: 2046-4053
DOI
https://doi.org/10.1186/s13643-016-0307-4

Other articles of this Issue 1/2016

Systematic Reviews 1/2016 Go to the issue

Protocol

Anticipatory and compensatory postural adjustments in people with low back pain: a protocol for a systematic review and meta-analysis

Protocol

The effect of the 2-week wait referral system on the detection of and mortality from colorectal cancer: protocol of a systematic review and meta-analysis

Protocol

Measurement properties of quality of life measurement instruments for infants, children and adolescents with eczema: protocol for a systematic review

Protocol

Effectiveness of eHealth interventions for the promotion of physical activity in older adults: a systematic review protocol

Editorial

Implementing PRISMA-P: recommendations for prospective authors

Research

Searching for qualitative research for inclusion in systematic reviews: a structured methodological review