Top

BMC Medical Research Methodology

Published in:

Open Access 01-12-2018 | Research article

Estimation of an inter-rater intra-class correlation coefficient that overcomes common assumption violations in the assessment of health measurement scales

Authors: Carly A. Bobak, Paul J. Barr, A. James O’Malley

Published in: BMC Medical Research Methodology | Issue 1/2018

Abstract

Background

Intraclass correlation coefficients (ICC) are recommended for the assessment of the reliability of measurement scales. However, the ICC is subject to a variety of statistical assumptions such as normality and stable variance, which are rarely considered in health applications.

Methods

A Bayesian approach using hierarchical regression and variance-function modeling is proposed to estimate the ICC with emphasis on accounting for heterogeneous variances across a measurement scale. As an application, we review the implementation of using an ICC to evaluate the reliability of Observer OPTION⁵, an instrument which used trained raters to evaluate the level of Shared Decision Making between clinicians and patients. The study used two raters to evaluate recordings of 311 clinical encounters across three studies to evaluate the impact of using a Personal Decision Aid over usual care. We particularly focus on deriving an estimate for the ICC when multiple studies are being considered as part of the data.

Results

The results demonstrate that ICC varies substantially across studies and patient-physician encounters within studies. Using the new framework we developed, the study-specific ICCs were estimated to be 0.821, 0.295, and 0.644. If the within- and between-encounter variances were assumed to be the same across studies, the estimated within-study ICC was 0.609. If heteroscedasticity is not properly adjusted for, the within-study ICC estimate was inflated to be as high as 0.640. Finally, if the data were pooled across studies without accounting for the variability between studies then ICC estimates were further inflated by approximately 0.02 while formerly allowing for between study variation in the ICC inflated its estimated value by approximately 0.066 to 0.072 depending on the model.

Conclusion

We demonstrated that misuse of the ICC statistics under common assumption violations leads to misleading and likely inflated estimates of interrater reliability. A statistical analysis that overcomes these violations by expanding the standard statistical model to account for them leads to estimates that are a better reflection of a measurement scale’s reliability while maintaining ease of interpretation. Bayesian methods are particularly well suited to estimating the expanded statistical model.

Fisher RA. On the “probable error” of a coefficient of correlation deduced from a small sample. Metron. 1921; 1:1–32.

Ebel RL. Estimation of the reliability of ratings. Psychometrika. 1951; 16:407–24. https://doi.org/10.1007/bf02288803.CrossRef

Strah KM, Love SM. The in situ carcinomas of the breast. J Am Med Women’s Assoc (1972). 1992; 47:165–8.

Visscher PM, Medland SE, Ferreira MAR, Morley KI, Zhu G, Cornes BK, Montgomery GW, Martin NG. Assumption-free estimation of heritability from genome-wide identity-by-descent sharing between full siblings. PLoS Genet. 2006; 2:41. https://doi.org/10.1371/journal.pgen.0020041.CrossRef

Bradley RA, Schumann DEW. The comparison of the sensitivities of similar experiments: Applications. Int Biom Soc. 1957; 13:496. https://doi.org/10.2307/2527974.

Pellis L, Hal NLWF-v, Burema J, Keijer J. The intraclass correlation coefficient applied for evaluation of data correction, labeling methods, and rectal biopsy sampling in dna microarray experiments. Physiol Genomics. 2003; 16:99–106. https://doi.org/10.1152/physiolgenomics.00111.2003.CrossRefPubMed

Killip S. What is an intracluster correlation coefficient? crucial concepts for primary care researchers. Ann Fam Med. 2004; 2:204–8. https://doi.org/10.1370/afm.141.CrossRefPubMedPubMedCentral

Ukoumunne OC, Davison AC, Gulliford MC, Chinn S. Non-parametric bootstrap confidence intervals for the intraclass correlation coefficient. Stat Med. 2003; 22:3805–21. https://doi.org/10.1002/sim.1643.CrossRefPubMed

Mokkink LB, Terwee CB, Patrick DL, Alonso J, Stratford PW, Knol DL, Bouter LM, de Vet HC. COSMIN checklist manual. Amst Univ Med Cent. 2012.

10.

Mokkink LB, Terwee CB, Patrick DL, Alonso J, Stratford PW, Knol DL, Bouter LM, de Vet HCW. The COSMIN checklist for assessing the methodological quality of studies on measurement properties of health status measurement instruments: an international delphi study. Qual Life Res. 2010; 19:539–49. https://doi.org/10.1007/s11136-010-9606-8.CrossRefPubMedPubMedCentral

11.

McGraw KO, Wong SP. Forming inferences about some intraclass correlation coefficients. Psychol Methods. 1996; 1(1):30–46. https://doi.org/10.1037/1082-989x.1.1.30.CrossRef

12.

Fisher RA. Statistical methods for research workers. In: Breakthroughs in statistics. Springer: 1992. p. 66–70.

13.

Shrout PE, Fleiss JL. Intraclass correlations: uses in assessing rater reliability. Psychol Bull. 1979; 86(2):420.CrossRefPubMed

14.

Koo TK, Li MY. A guideline of selecting and reporting intraclass correlation coefficients for reliability research. J Chiropr Med. 2016; 15(2):155–63. https://doi.org/10.1016/j.jcm.2016.02.012.CrossRefPubMedPubMedCentral

15.

Altaye M, Dormer A, Klar N. Inference procedures for assessing interobserver agreement among multiple raters. Biometrics. 2001; 57(2):584–8.CrossRefPubMed

16.

Ghosh H, Das A. Optimal diallel cross designs for estimation of heritability. Stat Med. 2003; 116:185–96. https://doi.org/10.1016/s0378-3758(02)00180-5.

17.

Shoukri M, Donner A. Efficiency considerations in the analysis of inter-observer agreement. Biostatistics. 2001; 2(3):323–36.CrossRefPubMed

18.

Konishi S. Normalizing and variance stabilizing transformations for intraclass correlations. Ann Inst Stat Math. 1985; 37(1):87–94.CrossRef

19.

Weinberg R, Patel YC. Simulated intraclass correlation coefficients and their z transforms. J Stat Comput Simul. 1981; 13(1):13–26.CrossRef

20.

Wang CS, Yandell BS, Rutledge JJ. Bias of maximum likelihood estimator of intraclass correlation. Theor Appl Genet. 1991; 82:421–4. https://doi.org/10.1007/bf00588594.CrossRefPubMed

21.

Ponzoni R, James J. Possible biases in heritability estimates from intraclass correlation. Theor Appl Genet. 1978; 53(1):25–7.PubMed

22.

Atenafu EG, Hamid JS, To T, Willan AR, Feldman BM, Beyene J. Bias-corrected estimator for intraclass correlation coefficient in the balanced one-way random effects model. BMC Med Res Methodol. 2012;12. https://doi.org/10.1186/1471-2288-12-126.

23.

Olkin I, Pratt JW. Unbiased estimation of certain correlation coefficients. Ann Math Stat. 1958; 29:201–11. https://doi.org/10.1214/aoms/1177706717.CrossRef

24.

Shay LA, Lafata JE. Where is the evidence? a systematic review of shared decision making and patient outcomes. Med Dec Making. 2014; 35(1):114–31. https://doi.org/10.1177/0272989x14551638.CrossRef

25.

Elwyn G, Frosch D, Thomson R, Joseph-Williams N, Lloyd A, Kinnersley P, Cording E, Tomson D, Dodd C, Rollnick S, Edwards A, Barry M. Shared decision making: A model for clinical practice. J Gen Intern Med. 2012; 27(10):1361–7. https://doi.org/10.1007/s11606-012-2077-6.CrossRefPubMedPubMedCentral

26.

Barr PJ, Elwyn G. Measurement challenges in shared decision making: putting the ‘patient’ in patient-reported measures. Health Expect. 2015; 19(5):993–1001. https://doi.org/10.1111/hex.12380.CrossRefPubMedPubMedCentral

27.

Fitzpatrick R.Surveys of patients satisfaction: I–important general considerations. Br Med J. 1991; 302(6781):887.CrossRef

28.

Scholl I, Loon MK-v, Sepucha K, Elwyn G, Légaré F, Härter M, Dirmaier J. Zeitschrift für Evidenz, Fortbildung und Qualität im Gesundheitswesen. 2011; 105(4):313–24. https://doi.org/10.1016/j.zefq.2011.04.012.

29.

Dillon EC, Stults CD, Wilson C, Chuang J, Meehan A, Li M, Elwyn G, Frosch DL, Yu E, Tai-Seale M. An evaluation of two interventions to enhance patient-physician communication using the observer OPTION⁵ measure of shared decision making. Patient Educ Couns. 2017; 100(10):1910–7.CrossRefPubMed

30.

Barr PJ, O’Malley AJ, Tsulukidze M, Gionfriddo MR, Montori V, Elwyn G. The psychometric properties of observer OPTION5, an observer measure of shared decision making. Patient Educ Couns. 2015; 98(8):970–6. https://doi.org/10.1016/j.pec.2015.04.010.CrossRefPubMed

31.

Elwyn G, Lloyd A, May C, van der Weijden T, Stiggelbout A, Edwards A, Frosch DL, Rapley T, Barr P, Walsh T, Grande SW, Montori V, Epstein R. Collaborative deliberation: A model for patient care. Patient Educ Couns. 2014; 97(2):158–64. https://doi.org/10.1016/j.pec.2014.07.027.CrossRefPubMed

32.

Bartlett JW, Keogh RH. Bayesian correction for covariate measurement error: A frequentist evaluation and comparison with regression calibration. Stat Methods Med Res. 2014; 0(0):0962280216667764. https://doi.org/10.1177/0962280216667764.

33.

Gelman A. Prior distributions for variance parameters in hierarchical models. Bayesian Anal. 2006; 1:1–19.CrossRef

34.

R Core Team. R: A Language and Environment for Statistical Computing. Vienna, Austria: R Foundation for Statistical Computing; 2016. https://www.R-project.org/.

35.

Plummer M, et al. JAGS: A program for analysis of Bayesian graphical models using Gibbs sampling. In: Proceedings of the 3rd international workshop on distributed statistical computing, Vol. 124 no. 125.10. Vienna: 2003.

36.

Plummer M. Rjags: Bayesian Graphical Models Using MCMC. 2016. R package version 4-6. https://CRAN.R-project.org/package=rjags.

37.

Hess EP, Knoedler MA, Shah ND, Kline JA, Breslin M, Branda ME, Pencille LJ, Asplin BR, Nestler DM, Sadosty AT, Stiell IG, Ting HH, Montori VM. The chest pain choice decision aid: A randomized trial. Circ Cardiovasc Qual Outcomes. 2012; 5(3):251–9. https://doi.org/10.1161/circoutcomes.111.964791.CrossRefPubMed

38.

Montori VM, Shah ND, Pencille LJ, Branda ME, Houten HKV, Swiglo BA, Kesman RL, Tulledge-Scheitel SM, Jaeger TM, Johnson RE, Bartel GA, Melton LJ, Wermers RA. Use of a decision aid to improve treatment decisions in osteoporosis: The osteoporosis choice randomized trial. Am J Med. 2011; 124(6):549–56. https://doi.org/10.1016/j.amjmed.2011.01.013.CrossRefPubMed

39.

Revicki DA, Leidy NK, Brennan-Diemer F, Sorensen S, Togias A. Integrating patient preferences into health outcomes assessment. Chest. 1998; 114(4):998–1007. https://doi.org/10.1378/chest.114.4.998.CrossRefPubMed

40.

Fried TR, Bradley EH, Towle VR. J Gerontol Ser B Psychol Sci Soc Sci. 2002; 57(6):348–54.

41.

Munters LA, van Vollenhoven RF, Alexanderson H. Patient preference assessment reveals disease aspects not covered by recommended outcomes in polymyositis and dermatomyositis. ISRN Rheumatol. 2011; 2011:1–5. https://doi.org/10.5402/2011/463124.CrossRef

42.

Giere RN. Bayesian statistics and biased procedures. Synthese. 1969; 20(3):371–87.CrossRef

43.

Gill CJ. Why clinicians are natural bayesians. BMJ. 2005; 330(7499):1080–3. https://doi.org/10.1136/bmj.330.7499.1080.CrossRefPubMedPubMedCentral

Title: Estimation of an inter-rater intra-class correlation coefficient that overcomes common assumption violations in the assessment of health measurement scales
Authors: Carly A. Bobak
Paul J. Barr
A. James O’Malley
Publication date: 01-12-2018
Publisher: BioMed Central
Published in: BMC Medical Research Methodology / Issue 1/2018
Electronic ISSN: 1471-2288
DOI: https://doi.org/10.1186/s12874-018-0550-6

At a glance: The ONWARDS insulin icodec trials

Springer Medicine

Estimation of an inter-rater intra-class correlation coefficient that overcomes common assumption violations in the assessment of health measurement scales

Abstract

Background

Methods

Results

Conclusion

At a glance: The ONWARDS insulin icodec trials

Springer Medicine

Abstract

Background

Methods

Results

Conclusion

Please log in to get access to this content

Other articles of this Issue 1/2018

Comparing performance between log-binomial and robust Poisson regression models for estimating risk ratios under model misspecification

Systematic review or scoping review? Guidance for authors when choosing between a systematic or scoping review approach

A multiple imputation method based on weighted quantile regression models for longitudinal censored biomarker data with missing values at early visits

Methodological development of tools to measure how women are treated during facility-based childbirth in four countries: labor observation and community survey

Development of standard clinical endpoints for use in dengue interventional trials: introduction and methodology

Simulation-based power calculations for planning a two-stage individual participant data meta-analysis