Skip to main content
Key Specialties
Cardiology Diabetology Endocrinology Gastroenterology Geriatrics and Gerontology Gynecology and Obstetrics Hematology Infectious Disease Internal Medicine Nephrology Neurology Oncology Pediatrics Respiratory Medicine Rheumatology Surgery
Congress Coverage
2024 ACC Congress 2023 ESMO Congress 2023 EASD Congress 2023 ERS Congress 2023 ESC Congress 2023 EHA Congress 2023 ASCO Annual Meeting
Clinical Collections
Advances in Alzheimer’s

At a glance: The ONWARDS insulin icodec trials

A round-up of the ONWARDS phase 3 clinical trials evaluating the novel weekly insulin icodec in people with diabetes.
ADVANCED SEARCH
Log in
Top
BMC Medical Research Methodology
Published in: BMC Medical Research Methodology 1/2013

Open Access 01-12-2013 | Research article

The reliability of a quality appraisal tool for studies of diagnostic reliability (QAREL)

Authors: Nicholas Lucas, Petra Macaskill, Les Irwig, Robert Moran, Luke Rickards, Robin Turner, Nikolai Bogduk

Published in: BMC Medical Research Methodology | Issue 1/2013

Login to get access

Abstract

Background

The aim of this project was to investigate the reliability of a new 11-item quality appraisal tool for studies of diagnostic reliability (QAREL). The tool was tested on studies reporting the reliability of any physical examination procedure. The reliability of physical examination is a challenging area to study given the complex testing procedures, the range of tests, and lack of procedural standardisation.

Methods

Three reviewers used QAREL to independently rate 29 articles, comprising 30 studies, published during 2007. The articles were identified from a search of relevant databases using the following string: “Reproducibility of results (MeSH) OR reliability (t.w.) AND Physical examination (MeSH) OR physical examination (t.w.).” A total of 415 articles were retrieved and screened for inclusion. The reviewers undertook an independent trial assessment prior to data collection, followed by a general discussion about how to score each item. At no time did the reviewers discuss individual papers. Reliability was assessed for each item using multi-rater kappa (κ).

Results

Multi-rater reliability estimates ranged from κ = 0.27 to 0.92 across all items. Six items were recorded with good reliability (κ > 0.60), three with moderate reliability (κ = 0.41 - 0.60), and two with fair reliability (κ = 0.21 - 0.40). Raters found it difficult to agree about the spectrum of patients included in a study (Item 1) and the correct application and interpretation of the test (Item 10).

Conclusions

In this study, we found that QAREL was a reliable assessment tool for studies of diagnostic reliability when raters agreed upon criteria for the interpretation of each item. Nine out of 11 items had good or moderate reliability, and two items achieved fair reliability. The heterogeneity in the tests included in this study may have resulted in an underestimation of the reliability of these two items. We discuss these and other factors that could affect our results and make recommendations for the use of QAREL.
Literature
1.
Lucas NP, Macaskill PM, Irwig L, Bogduk N: The development of a quality appraisal tool for studies of diagnostic reliability (QAREL). J Clin Epidemiol. 2010, 63: 854-861. 10.1016/j.jclinepi.2009.10.002.CrossRefPubMed
2.
Mokkink LB, Terwee CB, Patrick DL, Alonso J, Stratford PW, Knol DL, Bouter LM, de Vet HCW: The COSMIN checklist for assessing the methodological quality of studies on measurement properties of health status measurement instruments: an international Delphi study. Qual Life Res. 2010, 19: 539-549. 10.1007/s11136-010-9606-8.CrossRefPubMedPubMedCentral
3.
Kottner J, Audige L, Brorson S, Donner A, Gajewski BJ, Hrobjartsson A, et al: Guidelines for Reporting Reliability and Agreement Studies (GRRAS) were proposed. J Clin Epidemiol. 2011, 64: 96-106. 10.1016/j.jclinepi.2010.03.002.CrossRefPubMed
4.
Gemmell H, Miller P: Interexaminer reliability of multidimensional examination regimens used for detecting spinal manipulable lesions: a systematic review. Clin Chiropr. 2005, 8: 199-204. 10.1016/j.clch.2005.09.002.CrossRef
5.
Hestboek L, Leboeuf-Yde C: Are chiropractic tests for the lumbo-pelvic spine reliable and valid? A systematic critical literature review. J Manipulative Physiol Ther. 2000, 23: 258-275.CrossRef
6.
Hollerwoger D: Methodological quality and outcomes of studies addressing manual cervical spine examinations: a review. Man Ther. 2006, 11: 93-98. 10.1016/j.math.2005.12.001.CrossRefPubMed
7.
May S, Littlewook C, Bishop A: Reliability of procedures used in the physical examination of non-specific low back pain: a systematic review. Aust J Physiother. 2006, 52: 91-102. 10.1016/S0004-9514(06)70044-7.CrossRefPubMed
8.
Seffinger MA, Najm WI, Mishra SI, Adams A, Dickerson VM, Murphy LS, et al: Reliability of spinal palpation for diagnosis of back and neck pain: a systematic review of the literature. Spine. 2004, 29: E413-25. 10.1097/01.brs.0000141178.98157.8e.CrossRefPubMed
9.
Stochkendahl MJ, Christensen HW, Hartvigsen J, Vach W, Haas M, Hestbaek L, et al: Manual examination of the spine: a systematic critical literature review of reproducibility. J Manipulative Physiol Ther. 2006, 29: 475-85. 10.1016/j.jmpt.2006.06.011. 485 e1-10CrossRefPubMed
10.
Van Trijffel E, Anderegg Q, Bossuyt PMM, Lucas C: Inter-examiner reliability of passive assessment of intervertebral motion in the cervical and lumbar spine: A systematic review. Man Ther. 2005, 10: 256-269. 10.1016/j.math.2005.04.008.CrossRefPubMed
11.
Bossuyt PM, Reitsma JB, Bruns DE, Gatsonis CA, Glasziou PP, Irwig LM, et al: Towards complete and accurate reporting of studies of diagnostic accuracy: The STARD Initiative. Ann Intern Med. 2003, 138 (1): 40-44. 10.7326/0003-4819-138-1-200301070-00010.CrossRefPubMed
12.
Whiting P, Rutjes AW, Reitsma JB, Bossuyt PM, Kleijnen J: The development of QUADAS: a tool for the quality assessment of studies of diagnostic accuracy included in systematic reviews. BMC Med Res Methodol. 2003, 3: 25-10.1186/1471-2288-3-25.CrossRefPubMedPubMedCentral
13.
Bertilson B, Grunnesjo M, Johansson S-E, et al: Pain drawing in the assessment of neurogenic pain and dysfunction in the neck/shoulder region: Inter-examiner reliability and concordance with clinical examination. Pain Med. 2007, 8: 134-146. 10.1111/j.1526-4637.2006.00145.x.CrossRefPubMed
14.
Bremander AB, Dahl LL, Roos EM: Validity and reliability of functional performace tests in meniscectomized patients with or without knee osteoarthritis. Scand J Med Sci Sports. 2007, 17: 120-127.PubMed
15.
Brushoj C, Langberg H, Larsen K, et al: Reliability of normative values of the foot line test: a technique to assess foot posture. J Ortho Sports Phys Ther. 2007, 37: 703-707. 10.2519/jospt.2007.2525.CrossRef
16.
Bybee RF, Dionne CP: Interrater agreement on assessment, diagnosis, and treatment for neck pain by trained physical therapist students. J Phys Ther Edu. 2007, 21: 39-47.
17.
Cook C, Massa L, Harm-Emandes I, Segneri R, Adcock J, Kennedy C, Figuers C: Interrater reliability and diagnostic accuracy of pelvic girdle pain classification. J Manipulative Physiol Ther. 2007, 30: 252-258. 10.1016/j.jmpt.2007.03.008.CrossRefPubMed
18.
De Jong LD, Nieuwboer A, Aufdemkampe G: The hemiplegic arm: Interrater reliability and concurrent validity of passive range of motion measurements. Disability Rehab. 2007, 29: 1442-1448. 10.1080/09638280601056145.CrossRef
19.
Dionne C, Bybee RF, Tomaka J: Correspondence of diagnosis to initial treatment for neck pain. Physiotherapy. 2007, 93: 62-68. 10.1016/j.physio.2006.11.001.CrossRef
20.
Gladman DD, Inman RD, Cook RJ, van der Heijde D, Landewe RMB, et al: International spondyloarthritis interobserver reliability exercise. The INSPIRE study: I. Assessment of spinal measures. J Rheumatol. 2007, 34: 1733-1739.PubMed
21.
Gladman DD, Inman RD, Cook RJ, Maksymowych WP, Braun J, et al: International spondyloarthritis interobserver reliability exercise. The INSPIRE study: II. Assessment of peripheral joints, enthesitis, and dactylitis. J Rheumatol. 2007, 34: 1740-1745.PubMed
22.
Hacker MR, Funk SM, Manco-Johnson MJ: The Colorado haemophilia paediatric joint physical examination scale: Normal values and interrater reliability. Haemophilia. 2007, 13: 71-78. 10.1111/j.1365-2516.2006.01387.x.CrossRefPubMed
23.
Hickey BW, Milosavljevic S, Bell ML, Milburn PD: Accuracy and reliability of observational motion analysis in identifying shoulder symptoms. Man Ther. 2007, 12: 263-270. 10.1016/j.math.2006.05.005.CrossRefPubMed
24.
Hungerford BA, Gilleard W, Moran M, Emmerson C: Evaluation of the ability of physical therapists to palpate intrapelvic motion with the stork test on the support side. Phys Ther. 2007, 87: 879-887. 10.2522/ptj.20060014.CrossRefPubMed
25.
Kim Y-S, Kim J-M, Ha K-Y, Choy S, Joo M-W, et al: The passive compression test: A new clinical test for superior labral tears of the shoulder. Am J Sports Med. 2007, 35: 1489-1494. 10.1177/0363546507301884.CrossRefPubMed
26.
Kim HW, Ko YJ, Rhee WI, Lee JS, Lim JE, et al: Interexaminer reliability and accuracy of posterior superior iliac spine and iliac crest palpation for spinal level estimations. J Manipulative Physiol Ther. 2007, 30: 386-389. 10.1016/j.jmpt.2007.04.005.CrossRefPubMed
27.
Kryger AI, Lassen CF, Andersen JH: The role of physical examination in studies of musculoskeletal disorders of the elbow. Occup Environ Med. 2007, 64: 776-781. 10.1136/oem.2005.026260.CrossRefPubMedPubMedCentral
28.
Lewis JS, Valentine RE: The pectoralis minor length test: A study of the intra-rater reliability and diagnostic accuracy in subjects with and without shoulder symptoms. BMC Musculoskelet Disord. 2007, 8: 64-10.1186/1471-2474-8-64.CrossRefPubMedPubMedCentral
29.
McCarthy CJ, Gittins M, Roberts C, Oldham JA: The reliability of the clinical tests and questions recommended in international guidelines for low back pain. Spine. 2007, 32: 921-926. 10.1097/01.brs.0000259864.21869.26.CrossRefPubMed
30.
McEwan I, Herrington L, Thom J: The validity of clinical measures of patella position. Man Ther. 2007, 12: 226-230. 10.1016/j.math.2006.06.013.CrossRefPubMed
31.
Myers JB, Oyama S, Wassinger CA, Ricci RD, Abt JP, et al: Reliability, precision, accuracy, and validity of posterior shoulder tightness assessment in overhead athletes. Am J Sports Med. 2007, 35: 1922-1930. 10.1177/0363546507304142.CrossRefPubMed
32.
Neumann PB, Grimmer-Somers KA, Gill VA, Grant RE: Rater reliability of pelvic floor muscle strength. Aust NZ Continence J. 2007, 13: 8-14.
33.
Peeler J, Anderson JE: Reliability of the Thomas test for assessing range of motion about the hip. Phys Ther Sport. 2007, 8: 14-21. 10.1016/j.ptsp.2006.09.023.CrossRef
34.
Rainville J, Noto DJ, Jouve C, Jenis L: Assessment of forearm pronation strength in C6 and C7 radiculopathies. Spine. 2007, 32: 72-75. 10.1097/01.brs.0000251002.39417.2c.CrossRefPubMed
35.
Robinson HS, Brox JI, Robinson R, Bjelland E, Solem S, Telje T: The reliability of selected motion- and pain provocation tests for the sacroiliac joint. Man Ther. 2007, 12: 72-79. 10.1016/j.math.2005.09.004.CrossRefPubMed
36.
Roussel NA, Nijs J, Truijen S, Smeuninx L, Stassijns G: Low back pain: Clinimetric properties of the trendelenburg test, active straight leg raise test, and breathing pattern during active straight leg raising. J Manipulaitve Physiol Ther. 2007, 30: 270-278. 10.1016/j.jmpt.2007.03.001.CrossRef
37.
Savic G, Bergstrom EMK, Frankel HL, Jamous MA, Jones PW: Inter-rater reliability of motor and sensory examinations performed according to American Spinal Injury Association standards. Spinal Cord. 2007, 45: 444-451. 10.1038/sj.sc.3102044.CrossRefPubMed
38.
Schneider M, Homonai R, Moreland B, Delitto A: Interexaminer reliability of the prone leg length analysis procedure. J Manipulative Physio Ther. 2007, 30: 514-521. 10.1016/j.jmpt.2007.07.001.CrossRef
39.
Sedaghat N, Latimer J, Maher C, Wisebey-Roth T: The reproducibility of a clinical grading system of motor control in patients with low back pain. J Manipulative Physiol Ther. 2007, 30: 501-508. 10.1016/j.jmpt.2007.07.008.CrossRefPubMed
40.
Visscher CM, Lobbezoo F, Naeije M: A reliability study of dynamic and static pain tests in temporomandibular disorder patients. J Orofac Pain. 2007, 21: 39-45.PubMed
41.
Fleiss J: Statistical methods for rates and proportions. 2003, Hoboken, N.J.: Wiley-Interscience, 3CrossRef
42.
Mokkink LB, Terwee CB, Gibbons E, Stratford PW, Alonso J, Patrick DL, Knol DL, Bouter LM, de Vet HCW: Inter-rater agreement and reliability of the COSMIN (Consensus-based Standards for the selection of health status Measurement Instruments) Checklist. BMC Med Res Methodol. 2010, 10: 8-10.1186/1471-2288-10-8.CrossRef
Metadata
Title
The reliability of a quality appraisal tool for studies of diagnostic reliability (QAREL)
Authors
Nicholas Lucas
Petra Macaskill
Les Irwig
Robert Moran
Luke Rickards
Robin Turner
Nikolai Bogduk
Publication date
01-12-2013
Publisher
BioMed Central
Published in
BMC Medical Research Methodology / Issue 1/2013
Electronic ISSN: 1471-2288
DOI
https://doi.org/10.1186/1471-2288-13-111

Other articles of this Issue 1/2013

BMC Medical Research Methodology 1/2013 Go to the issue

Research article

Estimating relative intensity using individualized accelerometer cutpoints: the importance of fitness level

Research article

Diagnosing problems with imputation models using the Kolmogorov-Smirnov test: a simulation study

Software

An R package for analyzing and modeling ranking data

Research article

Using case study within a sequential explanatory design to evaluate the impact of specialist and advanced practice roles on clinical outcomes: the SCAPE study

Research article

Correction of the significance level when attempting multiple transformations of an explanatory variable in generalized linear models

Research article

A comparison of Cohen’s Kappa and Gwet’s AC1 when calculating inter-rater reliability coefficients: a study conducted with personality disorder samples