Skip to main content
Key Specialties
Cardiology Diabetology Endocrinology Gastroenterology Geriatrics and Gerontology Gynecology and Obstetrics Hematology Infectious Disease Internal Medicine Nephrology Neurology Oncology Pediatrics Respiratory Medicine Rheumatology Surgery
Congress Coverage
2024 ACC Congress 2023 ESMO Congress 2023 EASD Congress 2023 ERS Congress 2023 ESC Congress
Clinical Collections
Advances in Alzheimer’s

At a glance: The ONWARDS insulin icodec trials

A round-up of the ONWARDS phase 3 clinical trials evaluating the novel weekly insulin icodec in people with diabetes.
ADVANCED SEARCH
Log in
Top
Clinical Orthopaedics and Related Research®
Published in: Clinical Orthopaedics and Related Research® 11/2013

01-11-2013 | Symposium: ABJS Carl T. Brighton Workshop on Outcome Measures

Statistical Considerations in the Psychometric Validation of Outcome Measures

Authors: Alla Sikorskii, PhD, Philip C. Noble, PhD

Published in: Clinical Orthopaedics and Related Research® | Issue 11/2013

Login to get access

Abstract

Background

The evaluation of the outcomes of total knee arthroplasty requires measurement tools that are valid, reliable, and responsive to change. However, the accuracy of any outcome measurement is determined by the validity and reliability of the instrument used. To ensure this accuracy, it is imperative that each instrument used in orthopaedics is free of biases leading to inaccurate estimates of treatment effects.

Where are we now?

Many patient-derived outcome instruments have been developed and tested through the application of the standard assessments that form the basis of classical test theory: validity, reliability, and responsiveness. These assessments determine if the instrument reliably measures what it is intended to measure, and if it captures differences among groups of patients or changes over time.

Where do we need to go?

Thorough evaluation of the outcome instruments used in orthopaedics is a critical prerequisite for the continued improvement of effective patient care. Additional steps of psychometric testing that are sometimes overlooked include testing for differential item functioning (DIF) and the effects of the mode of administration of the outcome instrument. The use of suitable approaches to test for these potential sources of bias would facilitate the development of more robust outcome assessment in research and clinical practice.

How do we get there?

Testing for DIF, including the effects of mode of administration, may be performed using several analytical approaches. This will allow optimal application of each outcome instrument with respect to patient characteristics, time and mode of the administration, and modification, as necessary.
Literature
1.
Agel J, Rockwood T, Mundt JC, Greist JH, Swiontkowski M. Comparison of interactive voice response and written self-administered patient surveys for clinical research. Orthopedics. 2001;24:1155–1157.PubMed
2.
Beaton D, Hogg-Jonson S, Bombardier C. Evaluating changes in health status: reliability and responsiveness of five generic health status measures in workers with musculoskeletal disorders. J Clin Epidemiol. 1997;50:79–83.PubMedCrossRef
3.
Bellamy N, Buchanan WW, Goldsmith CH, Campbell J, Stitt L. Validation study of WOMAC: a health status instrument for measuring clinically important patient relevant outcomes following total hip or knee arthroplasty in osteoarthritis. J Orthopaedic Rheumatol. 1988;1:95–108.
4.
Benjamini Y, Hochberg Y. Controlling the false discovery rate: a practical and powerful approach to multiple testing. J R Stat Soc B. 1995;57:289–300.
5.
Benjamini Y, Yekutieli D. The control of the false discovery rate in multiple testing under dependency. Ann Stat. 2001;29:1165–1188.CrossRef
6.
Bourne RB, Chesworth BM, Davis AM, Mahomed NN, Charron KD. Patient satisfaction after total knee arthroplasty: who is satisfied and who is not? Clin Orthop Relat Res. 2010;468:57–63.PubMedCrossRef
7.
Chan KS, Orlando M, Ghosh-Dastidar B, Duan N, Sherbourne CD. The interview mode effect on the Center for Epidemiologic Studies Depression (CES-D) scale: an item response theory analysis. Med Care. 2004;42:281–289.PubMedCrossRef
8.
Cohen J. Statistical Power Analysis for the Behavioral Sciences. Hillsdale, NJ, USA: Erlbaum; 1988.
9.
Collins NJ, Roos EM. Patient-reported outcomes for total hip and knee arthroplasty: commonly used instruments and attributes of a ‘good’ measure. Clin Geriatr Med. 2012;28:367–394.PubMedCrossRef
10.
Crane P, Gibbons L, Narasimhalu K, Lai JS, Cella D. Rapid detection of differential item functioning in assessments of health-related quality of life: the functional assessment of cancer therapy. Qual Life Res. 2007;16:101–114.PubMedCrossRef
11.
Cronbach LJ. Coefficient alpha and the internal structure of tests. Psychometrica. 1951;16:297–334.CrossRef
12.
Donovan MA, Drasgow F, Probst TM. Does computerizing paper-and-pencil job attitude scales make a difference? New IRT analyses offer insight. J Appl Psychol. 2000;85:305–313.PubMedCrossRef
13.
Gao W, Stark D, Bennet MI, Siegert RJ, Murray S, Higginson IJ. Using the 12-item General Health Questionnaire to screen psychological distress from survivorshp to end-of ife care: dimensionality and item quality. Psychooncology. 2012;21:954–961.PubMedCrossRef
14.
Goetz C, Ecosse E, Rat AC, Pouchot J, Coste J, Guillemin F. Measurement properties of the osteoarthritis of knee and hip quality of life OAKHQOL questionnaire: an item response theory analysis. Rheumatology (Oxford). 2011;50:500–505.PubMedCrossRef
15.
Guyatt GH, Walter S, Norman G. Measuring change over time: assessing the usefulness of evaluative instruments. J Chronic Dis. 1987;40:171–178.PubMedCrossRef
16.
Hambleton R. Good practices for identifying differential item functioning. Med Care. 2006;44(Suppl 3):S182–S188.PubMedCrossRef
17.
Hansen JI, Neuman JL, Haverkamp BE, Lubinski BR. Comparison of user reaction to two methods of strong interest inventory administration and report feedback. Measurement and Evaluation in Counseling and Development. 1997;30:115–127.
18.
Hart DL, Deutscher D, Crane PK, Wang YC. Differential item functioning was negligible in an adaptive test of functional status for patients with knee impairments who spoke English or Hebrew. Qual Life Res. 2009;18:1067–1083.PubMedCrossRef
19.
Higgins LD, Taylor MK, Park D, Ghodadra N, Marchant M, Pietrobon R, Cook C; International Knee Documentation Committee. Reliability and validity of the International Knee Documentation Committee (IKDC) Subjective Knee Form. Joint Bone Spine. 2007;74:594–599.PubMedCrossRef
20.
Holland PW, Rosenbaum R. Conditional association and unidimensionality in monotone latent variable models. Ann Stat. 1986;14:1523–1543.CrossRef
21.
Impellizzeri FM, Mannion AF, Leunig M, Bizzini M, Naal FD. Comparison of the reliability, responsiveness, and construct validity of 4 different questionnaires for evaluating outcomes after total knee arthroplasty. J Arthroplasty. 2011;26:861–869.PubMedCrossRef
22.
Insall JN, Dorr LD, Scott RD, Scott WN. Rationale of the Knee Society Clinical Scoring System. Clin Orthop Relat Res. 1989;248:13–14.PubMed
23.
Jette AM, McDonough CM, Haley SM, Ni P, Olarsch S, Latham N, Hambleton RK, Felson D, Kim YJ, Hunter D. A computer-adaptive disability instrument for lower extremity osteoarthritis research demonstrated promising breadth, precision, and reliability. J Clin Epidemiol. 2009;62:807–815.PubMedCrossRef
24.
Kosinski M, Bjorner J, Ware J, Batenhorst A, Cady R. The responsiveness of headache impact scales scored using ‘classical’ and ‘modern’ psychometric methods: a re-analysis of three clinical trials. Qual Life Res. 2003;12:903–912.PubMedCrossRef
25.
Lheureux M, Raherison C, Vernejoux JM, Nguyen L, Nocent C, Tunon De Lara M, Taytard A. Quality of life in lunch cancer: does disclosure of the diagnosis have an impact? Lung Cancer. 2004;43:175–182.PubMedCrossRef
26.
Liang M, Fossel A, Larson M. Comparison of five health status instruments for orthopedic evaluation. Med Care. 1990;28:632–642.PubMedCrossRef
27.
Lord F. Application of Item Response Theory to Practical Testing Problems. Hillsdale, NJ, USA: Lawrence Erlbaum Associates; 1980.
28.
Mead AD, Drasgow F. Equivalence of computerized and paper-and-pencil cognitive ability tests: a meta-analysis. Psychol Bull. 1993;114:449–458.CrossRef
29.
McHorney C, Haley S, Ware J. Evaluation of the MOS SF-36 Physical Functioning Scale(PF-10): II. Comparison of relative precision using Likert and Rasch scoring methods. J Clin Epidemiol. 1997;50:451–461.PubMedCrossRef
30.
Noble PC, Scuderi GR, Brekke A, Sikorskii A, Benjamin J, Lonner J, Chadha P, Daylamani D, Scott WN, Bourne RB. Development of a New Knee Society Scoring System. Clin Orthop Relat Res. 2012;470:20–32.PubMedCrossRef
31.
Orlando M, Thissen D. Likelihood-based item-fit indices for dichotomous item response theory models. Appl Psychol Meas. 2002;24:50–64.CrossRef
32.
Orlando M, Thissen D. Further examination of the performance of S-Chi-Square, an item fit index for dichotomus item response theory models. Appl Psychol Meas. 2003;27:289–298.CrossRef
33.
Osborne RH, Elsworth GR, Sprangers MA, Oort FJ, Hopper JL. The value of the Hospital Anxiety and Depression Scale (HADS) for comparing women with early onset breast cancer with population-based reference women. Qual Life Res. 2004;13:191–206.PubMedCrossRef
34.
Petersen M, Groenvold M, Bjorner J, Aaronson N, Conroy T, Cull A, Fayers P, Hjermstad M, Spraingers M, Sullivan M; European Organisation for Research and Treatment of Cancer Quality of Life Group. Use of differential item functioning to assess the equivalence of translations of a questionnaire. Qual Life Res. 2003;12:373–385.PubMedCrossRef
35.
Pollard B, Dixon D, Dieppe P, Johnston M. Measuring the ICF components of impairment, activity limitation and participation restriction: an item analysis using classical test theory and item response theory. Health Qual Life Outcomes. 2009;7:41.PubMedCrossRef
36.
Reeve BB, Hays RD, Bjorner JL, Cook KF, Crane PK, Teresi JA, Thissen D, Revicki DA, Weiss DJ, Hambleton RK, Liu H, Gershon R, Reise SP, Lai JS, Cella D; PROMIS Cooperative Group. Psychometric evaluation and calibration of health-related quality of life item banks. Med Care. 2007;45(Suppl 1):S22–S26.PubMedCrossRef
37.
Roos EM, Toksvig-Larsen S. Knee Injury and Osteoarthritis Outcome Score (KOOS)—validation and comparison to the WOMAC in total knee replacement. Health Qual Life Outcomes. 2003;1:17.PubMedCrossRef
38.
Samejima F. Graded response model. In: van der Linden WJ, Hambleton RK, eds. Handbook of Modern Item Response Theory. Berlin, Germany: Springer; 1997:85–100.CrossRef
39.
Scuderi GR, Bourne RB, Noble PC, Benjamin JB, Lonner JH, Scott WN. The New Knee Society Scoring System. Clin Orthop Relat Res. 2012;470:3–19.PubMedCrossRef
40.
Sikorskii A, Given C, Given B, Jeon S, You M. Differential symptom reporting by mode of administration of the assessment: automated voice response system versus a live telephone interview. Med Care. 2009;47:866–874.PubMedCrossRef
41.
Sloan JA, Berk L, Roscoe J, Fisch MJ, Shaw EG, Wyatt G, Morrow GR, Dueck AC; National Cancer Institute. Integrating patient-reported outcomes into cancer symptom management clinical trials supported by the National Cancer Institute-sponsored clinical trials networks. J Clin Oncol. 2007;25:5070–5077.PubMedCrossRef
42.
Sloan JA, Cella D, Hays RD. Clinical significance of patient-reported questionnaire data: another step toward consensus. J Clin Epidemiol. 2005;58:1217–1219.PubMedCrossRef
43.
Swartz RJ, de Moor K, Cook KF, Fouladi RT, Basen-Engquist K, Eng C, Carmack Taylor CL. Mode effects in the Center for Epidemiologic Studies Depression (CES-D) scale: personal digital assistant vs. paper and pencil administration. Qual Life Res. 2007;16:803–813.PubMedCrossRef
44.
Teresi JA. Different approaches to differential item functioning in health applications: advantages, disadvantages, and some neglected topics. Med Care. 2006;44(Suppl 3):S152–S170.PubMedCrossRef
45.
Teresi JA, Ocepek-Welikson K, Kleinman M, Cook KF, Crane PK, Gibbons LE, Morales LS, Orlando-Edelen M, Cella D. Evaluating measurement equivalence using the item response theory log-likelihood ratio (IRTLR) method to assess differential item functioning (DIF): applications (with illustrations) to measures of physical functioning ability and general distress. Qual Life Res. 2007;16(Suppl 1):43–68.PubMedCrossRef
46.
Teresi JA, Ocepek-Welikson K, Kleinman M, Eimicke JP, Crane PK, Jones RN, Lai JS, Choi SW, Hays RD, Reeve BB, Reise SP, Pilkonis PA, Cella D. Analysis of differential item functioning in the depression item bank from the Patient Reported Outcome Measurement Information System (PROMIS): an item response theory approach. Psychol Sci Q. 2009;51:148–180.PubMed
47.
Thissen D, Steinberg L, Kuang D. Quick and easy implementation of the Benjamimi-Hochberg procedure for controlling the false positive reate in multiple comparisions. J Educ Behav Stat. 2002;27:77–83.CrossRef
48.
Wang D, Jones MH, Khair MM, Miniaci A. Patient-reported outcome measures for the knee. J Knee Surg. 2010;23:137–151.PubMedCrossRef
49.
Weiler K, Christ AM, Woodworth CG, Weiler RL, Weiler JM. Quality of patient-reported outcome data captured using paper and interactive voice response diaries in an allergic rhinitis study: is electronic data capture really better? Ann Allergy Asthma Immunol. 2004;92:335–339.PubMedCrossRef
50.
Wolfe F, Kong SX. Rasch analysis of the Western Ontario MacMaster questionnaire (WOMAC) in 2205 patients with osteoarthritis, rheumatoid arthritis, and fibromyalgia. Ann Rheum Dis. 1999;58:563–568.PubMedCrossRef
Metadata
Title
Statistical Considerations in the Psychometric Validation of Outcome Measures
Authors
Alla Sikorskii, PhD
Philip C. Noble, PhD
Publication date
01-11-2013
Publisher
Springer US
Published in
Clinical Orthopaedics and Related Research® / Issue 11/2013
Print ISSN: 0009-921X
Electronic ISSN: 1528-1132
DOI
https://doi.org/10.1007/s11999-013-3028-1

Other articles of this Issue 11/2013

Clinical Orthopaedics and Related Research® 11/2013 Go to the issue

Clinical Research

Risk Factors for Significant Wound Complications Following Wide Resection of Extremity Soft Tissue Sarcomas

CORR Insights®

CORR Insights®: Trabecular Metal in Total Knee Arthroplasty Associated with Higher Knee Scores: A Randomized Controlled Trial

Clinical Research

Translation and Validation of the Dutch New Knee Society Scoring System©

Symposium: ABJS Carl T. Brighton Workshop on Outcome Measures

Measuring Expectations in Orthopaedic Surgery: A Systematic Review

Reply to the Letter to the Editor

Reply to the Letter to the Editor: Complications of Total Knee Arthroplasty: Standardized List and Definitions of The Knee Society

Clinical Research

Low Wear of a Second-generation Highly Crosslinked Polyethylene Liner: A 5-year Radiostereometric Analysis Study