Top

Published in:

Open Access 01-12-2014 | Research article

Incorporating quality assessments of primary studies in the conclusions of diagnostic accuracy reviews: a cross-sectional study

Authors: Eleanor A Ochodo, Wynanda A van Enst, Christiana A Naaktgeboren, Joris AH de Groot, Lotty Hooft, Karel GM Moons, Johannes B Reitsma, Patrick M Bossuyt, Mariska MG Leeflang

Published in: BMC Medical Research Methodology | Issue 1/2014

Abstract

Background

Drawing conclusions from systematic reviews of test accuracy studies without considering the methodological quality (risk of bias) of included studies may lead to unwarranted optimism about the value of the test(s) under study. We sought to identify to what extent the results of quality assessment of included studies are incorporated in the conclusions of diagnostic accuracy reviews.

Methods

We searched MEDLINE and EMBASE for test accuracy reviews published between May and September 2012. We examined the abstracts and main texts of these reviews to see whether and how the results of quality assessment were linked to the accuracy estimates when drawing conclusions.

Results

We included 65 reviews of which 53 contained a meta-analysis. Sixty articles (92%) had formally assessed the methodological quality of included studies, most often using the original QUADAS tool (n = 44, 68%). Quality assessment was mentioned in 28 abstracts (43%); with a majority (n = 21) mentioning it in the methods section. In only 5 abstracts (8%) were results of quality assessment incorporated in the conclusions. Thirteen reviews (20%) presented results of quality assessment in the main text only, without further discussion. Forty-seven reviews (72%) discussed results of quality assessment; the most frequent form was as limitations in assessing quality (n = 28). Only 6 reviews (9%) further linked the results of quality assessment to their conclusions, 3 of which did not conduct a meta-analysis due to limitations in the quality of included studies. In the reviews with a meta-analysis, 19 (36%) incorporated quality in the analysis. Eight reported significant effects of quality on the pooled estimates; in none of them these effects were factored in the conclusions.

Conclusion

While almost all recent diagnostic accuracy reviews evaluate the quality of included studies, very few consider results of quality assessment when drawing conclusions. The practice of reporting systematic reviews of test accuracy should improve if readers not only want to be informed about the limitations in the available evidence, but also on the associated implications for the performance of the evaluated tests.

Available only for authorised users

Agency for Healthcare Research and Quality: Methods Guide for Medical Test Reviews. 2012, Rockville, MD: AHRQ Publication No.12-EC017

Leeflang MM, Deeks JJ, Gatsonis C, Bossuyt PM: Systematic reviews of diagnostic test accuracy. Ann Intern Med. 2008, 149 (12): 889-897. 10.7326/0003-4819-149-12-200812160-00008.CrossRefPubMedPubMedCentral

Higgins JPT, Altman DG, Sterne JAC: Chapter 8: assessing risk of bias in included studies. Cochrane Handbook for Systematic Reviews of Interventions. Version 5.1.0. Edited by: Higgins JPT, Green S. 2011, The Cochrane Collaboration

Reitsma J, Rutjes A, Whiting P, Vlassov VV, Leeflang M, Deeks J: Chapter9: assessing methodological quality. Cochrane Handbook for Systematic Reviews of Diagnostic Test Acuracy. Version 1.0.0. Edited by: Deeks J, Bossuyt P, Gatsonis C. 2009, The Cochrane Collaboration

Whiting P, Rutjes AW, Reitsma JB, Glas AS, Bossuyt PM, Kleijnen J: Sources of variation and bias in studies of diagnostic accuracy: a systematic review. Ann Intern Med. 2004, 140 (3): 189-202. 10.7326/0003-4819-140-3-200402030-00010.CrossRefPubMed

Whiting PF, Rutjes AW, Westwood ME, Mallett S: A systematic review classifies sources of bias and variation in diagnostic test accuracy studies. J Clin Epidemiol. 2013, 66 (10): 1093-1104. 10.1016/j.jclinepi.2013.05.014.CrossRefPubMed

Bossuyt PM, Reitsma JB, Bruns DE, Gatsonis CA, Glasziou PP, Irwig LM, Lijmer JG, Moher D, Rennie D, de Vet HC: Towards complete and accurate reporting of studies of diagnostic accuracy: the STARD initiative. Ann Intern Med. 2003, 138 (1): 40-44. 10.7326/0003-4819-138-1-200301070-00010.CrossRefPubMed

Bossuyt PM, Reitsma JB, Bruns DE, Gatsonis CA, Glasziou PP, Irwig LM, Moher D, Rennie D, de Vet HC, Lijmer JG: The STARD statement for reporting studies of diagnostic accuracy: explanation and elaboration. Ann Intern Med. 2003, 138 (1): W1-W12. 10.7326/0003-4819-138-1-200301070-00012-w1.CrossRefPubMed

Lijmer JG, Mol BW, Heisterkamp S, Bonsel GJ, Prins MH, van der Meulen JH, Bossuyt PM: Empirical evidence of design-related bias in studies of diagnostic tests. JAMA. 1999, 282: 1061-1066. 10.1001/jama.282.11.1061.CrossRefPubMed

10.

Whiting P, Rutjes AW, Reitsma JB, Bossuyt PM, Kleijnen J: The development of QUADAS: a tool for the quality assessment of studies of diagnostic accuracy included in systematic reviews. BMC Med Res Methodol. 2003, 3: 25-10.1186/1471-2288-3-25.CrossRefPubMedPubMedCentral

11.

Whiting PF, Rutjes AW, Westwood ME, Mallett S, Deeks JJ, Reitsma JB, Leeflang MM, Sterne JA, Bossuyt PM: QUADAS-2: a revised tool for the quality assessment of diagnostic accuracy studies. Ann Intern Med. 2011, 155: 529-536. 10.7326/0003-4819-155-8-201110180-00009.CrossRefPubMed

12.

Willis BH, Quigley M: Uptake of newer methodological developments and the deployment of meta-analysis in diagnostic test research: a systematic review. BMC Med Res Methodol. 2011, 11: 27-10.1186/1471-2288-11-27.CrossRefPubMedPubMedCentral

13.

Dahabreh IJ, Chung M, Kitsios GD, Terasawa T, Raman G, Tatsioni A, Tobar A, Lau J, Trikalinos TA, Schimd CH: Methods Research Report. Comprehensive Overview of Methods and Reporting of Meta-Analyses of Test Accuracy. 2012, Agency for Healthcare Research and Quality: AHRQ, Rockville, MD

14.

Moja LP, Telaro E, D’Amico R, Moschetti I, Coe L, Liberati A: Assessment of methodological quality of primary studies by systematic reviews: results of the metaquality cross sectional study. BMJ. 2005, 330 (7499): 1053-10.1136/bmj.38414.515938.8F.CrossRefPubMedPubMedCentral

15.

Hopewell S, Boutron I, Altman DG, Ravaud P: Incorporation of assessments of risk of bias of primary studies in systematic reviews of randomised trials: a cross-sectional study. BMJ Open. 2013, 3 (8): e003342-CrossRefPubMedPubMedCentral

16.

Zhelev Z, Garside R, Hyde C: A qualitative study into the difficulties experienced by healthcare decision makers when reading a Cochrane diagnostic test accuracy review. Syst Rev. 2013, 2: 32-10.1186/2046-4053-2-32.CrossRefPubMedPubMedCentral

17.

Beller EM, Glasziou PP, Altman DG, Hopewell S, Bastian H, Chalmers I, Gotzsche PC, Lasseson T, Tovey D: PRISMA for abstracts: reporting systematic reviews in journal and conference abstracts. PLoS Med. 2013, 10 (4): e1001419-10.1371/journal.pmed.1001419.CrossRefPubMedPubMedCentral

18.

Boutron I, Dutton S, Ravaud P, Altman DG: Reporting and interpretation of randomized controlled trials with statistically nonsignificant results for primary outcomes. JAMA. 2010, 303 (20): 2058-2064. 10.1001/jama.2010.651.CrossRefPubMed

19.

Yavchitz A, Boutron I, Bafeta A, Marroun I, Charles P, Mantz J, Ravaud P: Misrepresentation of randomized controlled trials in press releases and news coverage: a cohort study. PLoS Med. 2012, 9 (9): e1001308-10.1371/journal.pmed.1001308.CrossRefPubMedPubMedCentral

20.

Underwood M, Arbyn M, Parry-Smith WP, Bellis-Ayres S, Todd R, Redman CWE: Accuracy of colposcopic directed punch biopsies: a systematic review and meta-analysis. BJOG. 2012, 119: 1293-1301. 10.1111/j.1471-0528.2012.03444.x.CrossRefPubMed

21.

Atluri S, Singh V, Datta S, Geffert S, Sehgal N, Falco FJE: Diagnostic accuracy of thoracic facet joint nerve blocks: an update of the assessment of evidence. Pain Physician. 2012, 15: e483-e496.PubMed

22.

Mejare IA, Axelsson S, Davidson T, Frisk F, Hakeberg M, Kvist T, Norlund A, Petersson A, Portenier I, Sandberg H, Tranaeus S, Bergenholtz G: Diagnosis of the condition of the dental pulp: a systematic review. Int Endod J. 2012, 45 (7): 597-613. 10.1111/j.1365-2591.2012.02016.x.CrossRefPubMed

23.

Tijssen M, van Cingel R, Willemsen L, de Visser E: Diagnostics of femoroacetabular impingement and labral pathology of the hip: a systematic review of the accuracy and validity of physical tests. Arthroscopy. 2012, 28 (6): 860-871. 10.1016/j.arthro.2011.12.004.CrossRefPubMed

24.

Cook C, Mabry L, Reima M, Hegedus EJ: Best tests/clinical findings for screening and diagnosis of patellofemoral pain syndrome: a systematic review. Physiotherapy. 2012, 98: 93-100. 10.1016/j.physio.2011.09.001.CrossRefPubMed

25.

Chang K, Lu W, Wang J, Zhang K, Jia S, Li F, Deng S, Chen M: Rapid and effective diagnosis of tuberculosis and rifampicin resistance with Xpert MTB/RIF assay: a meta-analysis. J Infect. 2012, 64 (6): 580-588. 10.1016/j.jinf.2012.02.012.CrossRefPubMed

26.

Chen J, Yang R, Lu Y, Xia Y, Zhou H: Diagnostic accuracy of endoscopic ultrasound-guided fine-needle aspiration for solid pancreatic lesion: a systematic review. J Cancer Res Clin Oncol. 2012, 138 (9): 1433-1441. 10.1007/s00432-012-1268-1.CrossRefPubMed

27.

van Teeffelen AS, Van Der Heijden J, Oei SG, Porath MM, Willekes C, Opmeer B, Mol BW: Accuracy of imaging parameters in the prediction of lethal pulmonary hypoplasia secondary to mid-trimester prelabor rupture of fetal membranes: a systematic review and meta-analysis. Ultrasound Obstet Gynecol. 2012, 39 (5): 495-499. 10.1002/uog.10047.CrossRefPubMed

28.

Wu L, Dai ZY, Qian YH, Shi Y, Liu FJ, Yang C: Diagnostic value of serum human epididymis protein 4 (HE4) in ovarian carcinoma: a systematic review and meta-analysis. Int J Gynecol Cancer. 2012, 22 (7): 1106-1112. 10.1097/IGC.0b013e318263efa2.CrossRefPubMed

29.

Lin CY, Chen JH, Liang JA, Lin CC, Jeng LB, Kao CH: 18 F-FDG PET or PET/CT for detecting extrahepatic metastases or recurrent hepatocellular carcinoma: a systematic review and meta-analysis. Eur J Radiol. 2012, 81 (9): 2417-2422. 10.1016/j.ejrad.2011.08.004.CrossRefPubMed

30.

Wu L-M, Xu J-R, Ye Y-Q, Lu Q, Hu J-N: The clinical value of diffusion-weighted imaging in combination with T2-weighted imaging in diagnosing prostate carcinoma: a systematic review and meta-analysis. Am J Roentgenol. 2012, 199 (1): 103-110. 10.2214/AJR.11.7634.CrossRef

31.

Quatman CE, Quatman-Yates CC, Schmitt LC, Paterno MV: The clinical utility and diagnostic performance of MRI for identification and classification of knee osteochondritis dissecans. J Bone Joint Surg Am. 2012, 94: 1036-1044.CrossRefPubMedPubMedCentral

32.

Smith TO, Drew B, Toms AP, Jerosch-Herold C, Chojnowski AJ: Diagnostic accuracy of magnetic resonance imaging and magnetic resonance arthrography for triangular fibrocartilaginous complex injury: a systematic review and meta-analysis. J Bone Joint Surg Am. 2012, 94 (9): 824-832.CrossRefPubMed

33.

Ochodo EA, de Haan MC, Reitsma JB, Hooft L, Bossuyt PM, Leeflang MM: Overinterpretation and misreporting of diagnostic accuracy studies: evidence of “spin”. Radiology. 2013, 267 (2): 581-588. 10.1148/radiol.12120527.CrossRefPubMed

34.

Whiting P, Rutjes AW, Dinnes J, Reitsma JB, Bossuyt PM, Kleijnen J: A systematic review finds that diagnostic reviews fail to incorporate quality despite available tools. J Clin Epidemiol. 2005, 58 (1): 1-12. 10.1016/j.jclinepi.2004.04.008.CrossRefPubMed

35.

Macaskill P, Gatsonis C, Deeks JJ, Harbord R, Takwoingi Y: Chapter 10: analysing and presenting results. Cochrane Handbook for Systematic Reviews of Diagnostic Test Accuracy. Version 1.0. 1. Edited by: Deeks JJ, Bossuyt PM, Gatsonis C. 2010, The Cochrane Collaboration

36.

Moher D, Liberati A, Tetzlaff J, Altman DG: Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. PLoS Med. 2009, 6 (7): e1000097-10.1371/journal.pmed.1000097.CrossRefPubMedPubMedCentral

37.

Shea BJ, Grimshaw JM, Wells GA, Boers M, Andersson N, Hamel C, Porter AC, Tugwell P, Moher D, Bouter LM: Development of AMSTAR: a measurement tool to assess the methodological quality of systematic reviews. BMC Med Res Methodol. 2007, 7: 10-10.1186/1471-2288-7-10.CrossRefPubMedPubMedCentral

38.

Brozek JL, Akl EA, Jaeschke R, Lang DM, Bossuyt P, Glasziou P, Helfand M, Ueffing E, Alonso-Coello P, Meerpohl J, Phillips B, Harvath AR, Bousquet J, Guyatt GH, Schünemann HJ: Grading quality of evidence and strength of recommendations in clinical practice guidelines: Part 2 of 3. The GRADE approach to grading quality of evidence about diagnostic tests and strategies. Allergy. 2009, 64 (8): 1109-1116. 10.1111/j.1398-9995.2009.02083.x.CrossRefPubMed

39.

Guyatt GH, Oxman AD, Vist GE, Kunz R, Falck-Ytter Y, Alonso-Coello P, Schünemann HJ: GRADE: an emerging consensus on rating quality of evidence and strength of recommendations. BMJ. 2008, 336 (7650): 924-926. 10.1136/bmj.39489.470347.AD.CrossRefPubMedPubMedCentral

40.

Juni P, Witschi A, Bloch R, Egger M: The hazards of scoring the quality of clinical trials for meta-analysis. JAMA. 1999, 282 (11): 1054-1060. 10.1001/jama.282.11.1054.CrossRefPubMed

41.

Whiting P, Harbord R, Kleijnen J: No role for quality scores in systematic reviews of diagnostic accuracy studies. BMC Med Res Methodol. 2005, 5: 19-10.1186/1471-2288-5-19.CrossRefPubMedPubMedCentral

The pre-publication history for this paper can be accessed here:http://www.biomedcentral.com/1471-2288/14/33/prepub

Title: Incorporating quality assessments of primary studies in the conclusions of diagnostic accuracy reviews: a cross-sectional study
Authors: Eleanor A Ochodo
Wynanda A van Enst
Christiana A Naaktgeboren
Joris AH de Groot
Lotty Hooft
Karel GM Moons
Johannes B Reitsma
Patrick M Bossuyt
Mariska MG Leeflang
Publication date: 01-12-2014
Publisher: BioMed Central
Published in: BMC Medical Research Methodology / Issue 1/2014
Electronic ISSN: 1471-2288
DOI: https://doi.org/10.1186/1471-2288-14-33

At a glance: The ONWARDS insulin icodec trials

Springer Medicine

Incorporating quality assessments of primary studies in the conclusions of diagnostic accuracy reviews: a cross-sectional study

Abstract

Background

Methods

Results

Conclusion

At a glance: The ONWARDS insulin icodec trials

Springer Medicine

Abstract

Background

Methods

Results

Conclusion

Please log in to get access to this content

Other articles of this Issue 1/2014

A methodological systematic review of what’s wrong with meta-ethnography reporting

Accuracy of the Berger-Exner test for detecting third-order selection bias in randomised controlled trials: a simulation-based investigation

Joint modelling rationale for chained equations

External validation of multivariable prediction models: a systematic review of methodological conduct and reporting

Comparison of methods for imputing limited-range variables: a simulation study

Mediation analysis of the relationship between institutional research activity and patient survival