Skip to main content
Key Specialties
Cardiology Diabetology Endocrinology Gastroenterology Geriatrics and Gerontology Gynecology and Obstetrics Hematology Infectious Disease Internal Medicine Nephrology Neurology Oncology Pediatrics Respiratory Medicine Rheumatology Surgery
Congress Coverage
2024 ACC Congress 2023 ESMO Congress 2023 EASD Congress 2023 ERS Congress 2023 ESC Congress 2023 EHA Congress 2023 ASCO Annual Meeting
Clinical Collections
Advances in Alzheimer’s

At a glance: The ONWARDS insulin icodec trials

A round-up of the ONWARDS phase 3 clinical trials evaluating the novel weekly insulin icodec in people with diabetes.
ADVANCED SEARCH
Log in
Top
BMC Medical Research Methodology
Published in: BMC Medical Research Methodology 1/2005

Open Access 01-12-2005 | Research article

No role for quality scores in systematic reviews of diagnostic accuracy studies

Authors: Penny Whiting, Roger Harbord, Jos Kleijnen

Published in: BMC Medical Research Methodology | Issue 1/2005

Login to get access

Abstract

Background

There is a lack of consensus regarding the use of quality scores in diagnostic systematic reviews. The objective of this study was to use different methods of weighting items included in a quality assessment tool for diagnostic accuracy studies (QUADAS) to produce an overall quality score, and to examine the effects of incorporating these into a systematic review.

Methods

We developed five schemes for weighting QUADAS to produce quality scores. We used three methods to investigate the effects of quality scores on test performance. We used a set of 28 studies that assessed the accuracy of ultrasound for the diagnosis of vesico-ureteral reflux in children.

Results

The different methods of weighting individual items from the same quality assessment tool produced different quality scores. The different scoring schemes ranked different studies in different orders; this was especially evident for the intermediate quality studies. Comparing the results of studies stratified as "high" and "low" quality based on quality scores resulted in different conclusions regarding the effects of quality on estimates of diagnostic accuracy depending on the method used to produce the quality score. A similar effect was observed when quality scores were included in meta-regression analysis as continuous variables, although the differences were less apparent.

Conclusion

Quality scores should not be incorporated into diagnostic systematic reviews. Incorporation of the results of the quality assessment into the systematic review should involve investigation of the association of individual quality items with estimates of diagnostic accuracy, rather than using a combined quality score.
Appendix
Available only for authorised users
Literature
1.
2.
Assendelft JJ, Koes BW, van Tulder MW, Bouter LM: Scoring the quality of clinical trials [letter]. JAMA. 2000, 283: 1421-10.1001/jama.283.11.1421.PubMed
3.
ter Riet G, Leffers P, Zeegers M: Scoring the quality of clinical trials [letter]. JAMA. 2000, 283: 1421-10.1001/jama.283.11.1421.PubMed
4.
Berlin JA, Rennie D: Measuring the quality of trials: the quality of quality scales. JAMA. 1999, 282: 1083-5. 10.1001/jama.282.11.1083.CrossRefPubMed
5.
Juni P, Egger M: Scoring the quality of clinical trials [letter]. JAMA. 2000, 283: 1422-3.
6.
Klassen T: Bias against quality scores. 2001, 2002:
7.
Greenland S: Invited commentary: A critical look at some popular meta-analytic methods. American Journal of Epidemiology. 1994, 140: 290-296.PubMed
8.
Greenland S: Quality scores are useless and potentially misleading. American Journal of Epidemiology. 1994, 140: 300-302.
9.
Juni P, Witschi A, Bloch RM, Egger M: The hazards of scoring the quality of clinical trials for meta-analysis. JAMA. 1999, 282: 1054-1060. 10.1001/jama.282.11.1054.CrossRefPubMed
10.
Seffinger MA, Najm WI, Mishra SI, Adams A, Dickerson VM, Murphy LS, Reinsch S: Reliability of spinal palpation for diagnosis of back and neck pain: a systematic review of the literature. Spine. 2004, 29: E413-E425. 10.1097/01.brs.0000141178.98157.8e.CrossRefPubMed
11.
Ejnisman B, Andreoli CV, Soares BG, Fallopa F, Peccin MS, Abdalla RJ, Cohen M: Interventions for tears of the rotator cuff in adults. Cochrane Database Syst Rev. 2004, CD002758-
12.
Macdermid JC, Wessel J: Clinical diagnosis of carpal tunnel syndrome: a systematic review. J Hand Ther. 2004, 17: 309-319.CrossRefPubMed
13.
Warren E, Weatherley-Jones E, Chilcott J, Beverley C: Systematic review and economic evaluation of a long-acting insulin analogue, insulin glargine. Health Technol Assess. 2004, 8: 1-72.CrossRef
14.
Mowatt G, Vale L, Brazzelli M, Hernandez R, Murray A, Scott N, Fraser C, McKenzie L, Gemmell H, Hillis G, Metcalfe M: Systematic review of the effectiveness and cost-effectiveness, and economic evaluation, of myocardial perfusion scintigraphy for the diagnosis and management of angina and myocardial infarction. Health Technol Assess. 2004, 8: iii-207.CrossRefPubMed
15.
Jadad AR, Moore A, Carroll D, Jenkinson C, Reynolds DJ, Gavaghan DJ, McQuay HJ: Assessing the quality of reports of randomised clinical trials: is blinding necessary?. Control Clin Trials. 1996, 17: 1-12. 10.1016/0197-2456(95)00134-4.CrossRefPubMed
16.
Mulrow CD, Linn WD, Gaul MK, Pugh JA: Assessing quality of a diagnostic test evaluation. J Gen Intern Med. 1989, 4: 288-295.CrossRefPubMed
17.
Whiting P, Rutjes AWS, Dinnes J, Reitsma JB, Bossuyt PM, Kleijnen J: A systematic review finds that diagnostic reviews fail to incorporate quality despite available tools. Journal of Clinical Epidemiology. 2005, 58: 1-12. 10.1016/j.jclinepi.2004.04.008.CrossRefPubMed
18.
Whiting P, Rutjes AWS, Dinnes J, Reitsma JB, Bossuyt PMM, Kleijnen J: Development and validation of methods for assessing the quality and reporting of diagnostic accuracy studies. Health Technol Assess. 2004, 8: iii, 1-234.CrossRef
19.
Whiting P, Rutjes AWS, Reitsma JB, Bossuyt PM, Kleijnen J: The development of QUADAS: a tool for the quality assessment of studies of diagnostic accuracy included in systematic reviews. BMC Medical Research Methodology. 2003, 3: 25-10.1186/1471-2288-3-25.CrossRefPubMedPubMedCentral
20.
Whiting P, Rutjes AWS, Reitsma JB, Glas AS, Bossuyt PM, Kleijnen J: Sources of Variation and Bias in Studies of Diagnostic Accuracy: A Systematic Review. Annals of Internal Medicine. 2004, 140: 189-202.CrossRefPubMed
21.
Whiting P, Westwood M, Ginnelly L, Palmer S, Richardson G, Cooper J, Watt I, Glanville J, Sculpher M, Kleijnen J: A systematic review of tests for the diagnosis and evaluation of urinary tract infection (UTI) in children under five years. Health Technology Assessment. 2005
22.
Moses LE, Shapiro D, Littenberg B: Combining independent studies of a diagnostic test into a summary ROC curve: data-analystic approaches and some additional considerations. Stat Med. 1993, 12: 1293-1316.CrossRefPubMed
23.
Bergius AR, Niskanen K, Kekomaki M: Detection of significant vesico-ureteric reflux by ultrasound in infants and children. Z Kinderchir. 1990, 45: 144-5.PubMed
24.
Baronciani D, Bonora G, Andreoli A, Cambie M, Nedbal M, Dellagnola CA: The Value of Ultrasound for Diagnosing the Uropathy in Children with Urinary-Tract Infections. Rivista Italiana di Pediatria-Italian Journal of Pediatrics. 1986, 12: 214-220.
25.
Dura TT, Gonzalez MR, Juste RM, Gonzalez de DJ, Carratala MF, Moya BM, Verdu RJ, Caballero CO: [Usefulness of renal scintigraphy in the assessment of the first febrile urinary infection in children]. An Esp Pediatr. 1997, 47: 378-382.
26.
Evans ED, Meyer JS, Harty MP, Bellah RD: Assessment of increase in renal pelvic size on post-void sonography as a predictor of vesicoureteral reflux. Pediatr Radiol. 1999, 29: 291-4. 10.1007/s002470050591.CrossRefPubMed
27.
Foresman WH, Hulbert WCJ, Rabinowitz R: Does urinary tract ultrasonography at hospitalization for acute pyelonephritis predict vesicoureteral reflux?. J Urol. 2001, 165: 2232-4. 10.1097/00005392-200106001-00004.CrossRefPubMed
28.
Mage K, Zoppardo P, Cohen R, Reinert P, Ponet M: [Imaging and the first urinary infection in children. Respective role of each test during the initial evaluation apropos of 122 cases]. J Radiol. 1989, 70: 279-283.PubMed
29.
Mahant S, Friedman J, MacArthur C: Renal ultrasound findings and vesicoureteral reflux in children hospitalised with urinary tract infection. Arch Dis Child. 2002, 86: 419-420. 10.1136/adc.86.6.419.CrossRefPubMedPubMedCentral
30.
Morin D, Veyrac C, Kotzki PO, Lopez C, Dalla VF, Durand MF, Astruc J, Dumas R: Comparison of ultrasound and dimercaptosuccinic acid scintigraphy changes in acute pyelonephritis. Pediatr Nephrol. 1999, 13: 219-222. 10.1007/s004670050596.CrossRefPubMed
31.
Muensterer OJ: Comprehensive ultrasound versus voiding cysturethrography in the diagnosis of vesicoureteral reflux. Eur J Pediatr. 2002, 161: 435-437. 10.1007/s00431-002-0990-0.CrossRefPubMed
32.
Oostenbrink R, van der Heijden AJ, Moons KG, Moll HA: Prediction of vesico-ureteric reflux in childhood urinary tract infection: a multivariate approach. Acta Paediatr. 2000, 89: 806-10. 10.1080/080352500750043693.CrossRefPubMed
33.
Salih M, Baltaci S, Kilic S, Anafarta K, Beduk Y: Color flow Doppler sonography in the diagnosis of vesicoureteric reflux. Eur Urol. 1994, 26: 93-7.PubMed
34.
Tan SM, Chee T, Tan KP, Cheng HK, Ooi BC: Role of renal ultrasonography (RUS) and micturating cystourethrogram (MCU) in the assessment of vesico-ureteric reflux (VUR) in children and infants with urinary tract infection (UTI). Singapore Med J. 1988, 29: 150-152.PubMed
35.
Verber IG, Strudley MR, Meller ST: 99mTc dimercaptosuccinic acid (DMSA) scan as first investigation of urinary tract infection. Arch Dis Child. 1988, 63: 1320-1325.CrossRefPubMedPubMedCentral
36.
Alzen G, Wildberger JE, Muller-Leisse C, Deutz FJ: [Ultrasound screening of vesico-uretero-renal reflux]. Klin Padiatr. 1994, 206: 178-180.CrossRefPubMed
37.
Berrocal T, Gaya F, Arjonilla A, Lonergan GJ: Vesicoureteral reflux: diagnosis and grading with echo-enhanced cystosonography versus voiding cystourethrography. Radiology. 2001, 221: 359-365.CrossRefPubMed
38.
Berrocal Frutos T, Gaya Moreno F, Gomez Leon N, Jaureguizar Monereo E: Ecocistografia con contraste: una nueva modalidad de imagen para diagnosticar elreflujo vesicoureteral. [Cystosonography with echoenhancer. A new imaging technique for the diagnosis of vesicoureteral reflux]. An Esp Pediatr. 2000, 53: 422-30.CrossRefPubMed
39.
Haberlik A: Detection of low-grade vesicoureteral reflux in children by color Doppler imaging mode. Pediatr Surg Int. 1997, 12: 38-43.CrossRefPubMed
40.
Kessler RM, Altman DH: Real-time sonographic detection of vesicoureteral reflux in children. Am J Roentgenol. 1982, 138: 1033-1036.CrossRef
41.
McEwing RL, Anderson NG, Hellewell S, Mitchell J: Comparison of echo-enhanced ultrasound with fluoroscopic MCU for the detection of vesicoureteral reflux in neonates. Pediatr Radiol. 2002, 32: 853-858. 10.1007/s00247-002-0812-6.CrossRefPubMed
42.
Mentzel HJ, Vogt S, John U, Kaiser WA: Voiding urosonography with ultrasonography contrast medium in children. Pediatr Nephrol. 2002, 17: 272-276. 10.1007/s00467-002-0843-0.CrossRefPubMed
43.
Nakamura M, Wang Y, Shigeta K, Shinozaki T, Taniguchi N, Itoh K: Simultaneous voiding cystourethrography and voiding urosonography: an in vitro and in vivo study. Clin Radiol. 2002, 57: 846-849.CrossRefPubMed
44.
Piaggio G, gl' Innocenti ML, Toma P, Calevo MG, Perfumo F: Cystosonography and voiding cystourethrography in the diagnosis of vesicoureteral reflux. Pediatr Nephrol. 2003, 18: 18-22. 10.1007/s00467-002-0974-3.CrossRefPubMed
45.
Schneider K, Jablonski C, Wiessner M, Kohn M, Fendel H: Screening for vesicoureteral reflux in children using real-time sonography. Pediatr Radiol. 1984, 14: 400-3.CrossRefPubMed
46.
Siamplis D, Vasiou K, Giarmenitis S, Frimas K, Zavras G, Fezoulidis I: Sonographic detection of vesicoureteral reflux with fluid and air cystography. Comparison with VCUG. Rofo Fortschr Geb Rontgenstr Neuen Bildgeb Verfahr. 1996, 165: 166-9.CrossRefPubMed
47.
Valentini AL, Salvaggio E, Manzoni C, Rendeli C, Destito C, Summaria V, Campioni P, Marano P: Contrast-enhanced gray-scale and color Doppler voiding urosonography versus voiding cystourethrography in the diagnosis and grading of vesicoureteral reflux. J Clin Ultrasound. 2001, 29: 65-71. 10.1002/1097-0096(200102)29:2<65::AID-JCU1000>3.0.CO;2-I.CrossRefPubMed
48.
Uhl M, Kromeier J, Zimmerhackl LB, Darge K: Simultaneous voiding cystourethrography and voiding urosonography. Acta Radiol. 2003, 44: 265-268. 10.1034/j.1600-0455.2003.00065.x.PubMed
49.
Von Rohden L, Bosse U, Wiemann D: [Reflux sonography in children with an ultrasound contrast medium in comparison to radiologic voiding cystourethrography]. Paediat Prax. 2004, 49: 49-58.
Metadata
Title
No role for quality scores in systematic reviews of diagnostic accuracy studies
Authors
Penny Whiting
Roger Harbord
Jos Kleijnen
Publication date
01-12-2005
Publisher
BioMed Central
Published in
BMC Medical Research Methodology / Issue 1/2005
Electronic ISSN: 1471-2288
DOI
https://doi.org/10.1186/1471-2288-5-19

Other articles of this Issue 1/2005

BMC Medical Research Methodology 1/2005 Go to the issue

Research article

Structural equation model testing and the quality of natural killer cell activity measurements

Debate

Causal inference based on counterfactuals

Research article

Practical considerations for estimating clinical trial accrual periods: application to a multi-center effectiveness study

Research article

Who wants to join preventive trials? – Experience from the Estonian Postmenopausal Hormone Therapy Trial [ISRCTN35338757]

Research article

A non-parametric framework for estimating threshold limit values

Research article

Systematic reviews of epidemiology in diabetes: finding the evidence