Top

Systematic Reviews

Published in:

Open Access 01-12-2021 | Methodology

Application of weighting methods for presenting risk-of-bias assessments in systematic reviews of diagnostic test accuracy studies

Authors: Yasaman Vali, Mariska M. G. Leeflang, Patrick M. M. Bossuyt

Published in: Systematic Reviews | Issue 1/2021

Abstract

Background

An assessment of the validity of individual diagnostic accuracy studies in systematic reviews is necessary to guide the analysis and the interpretation of results. Such an assessment is performed for each included study and typically reported at the study level. As studies may differ in sample size and disease prevalence, with larger studies contributing more to the meta-analysis, such a study-level report does not always reflect the risk of bias in the total body of evidence. We aimed to develop improved methods of presenting the risk of bias in the available evidence on diagnostic accuracy of medical tests in systematic reviews, reflecting the relative contribution of the study to the body of evidence in the review.

Methods

We applied alternative methods to represent evaluations with the Quality Assessment of Diagnostic Accuracy Studies tool (QUADAS-2), weighting studies according to their relative contribution to the total sample size or their relative effective sample size. We used these methods in four existing systematic reviews of diagnostic accuracy studies, including 9, 13, 22, and 32 studies, respectively.

Results

The risk-of-bias summaries for each domain of the QUADAS-2 checklist changed in all four sets of studies after replacing unit weights for the studies with relative sample sizes or with the relative effective sample size. As an example, the risk of bias was high in the patient selection domain in 31% of the studies in one review, unclear in 23% and low in 46% of studies. Weighting studies according to the relative sample size changed the corresponding proportions to 4%, 4%, and 92%, respectively. The difference between the two weighting methods was small and more noticeable when the reviews included a smaller number of studies with wider range of sample size.

Conclusions

We present an alternative way of presenting the results of risk-of-bias assessments in systematic reviews of diagnostic accuracy studies. Weighting studies according to their relative sample size or their relative effective sample size can be used as more informative summaries of the risk of bias in the total body of available evidence.

Systematic review registrations

Not applicable

Burns PB, Rohrich RJ, Chung KC. The levels of evidence and their role in evidence-based medicine. Plast Reconstr Surg. 2011;128(1):305–10. https://doi.org/10.1097/PRS.0b013e318219c171.CrossRefPubMedPubMedCentral

Higgins JP, Thomas J, Chandler J, Cumpston M, Li T, Page MJ, et al. Cochrane handbook for systematic reviews of interventions: John Wiley & Sons; 2019. https://doi.org/10.1002/9781119536604.CrossRef

Pussegoda K, Turner L, Garritty C, Mayhew A, Skidmore B, Stevens A, et al. Systematic review adherence to methodological or reporting quality. Syst Rev. 2017;6(1):1–14.CrossRef

Whiting P, Rutjes AW, Reitsma JB, Bossuyt PM, Kleijnen J. The development of QUADAS: a tool for the quality assessment of studies of diagnostic accuracy included in systematic reviews. BMC Med Res Methodol. 2003;3(1):25. https://doi.org/10.1186/1471-2288-3-25.CrossRefPubMedPubMedCentral

Clarke M. The Cochrane Collaboration and systematic reviews. Br J Surg. 2007;94(4):391–2. https://doi.org/10.1002/bjs.5812.CrossRefPubMed

Viswanathan M, Patnode CD, Berkman ND, Bass EB, Chang S, Hartling L, et al. Assessing the risk of bias in systematic reviews of health care interventions. Methods guide for effectiveness and comparative effectiveness reviews: Agency for Healthcare Research and Quality (US); 2017.

Macaskill P, Gatsonis C, Deeks J, Harbord R, Takwoingi Y. Cochrane handbook for systematic reviews of diagnostic test accuracy. Version 09 0. London: The Cochrane Collaboration; 2010.

Whiting PF, Rutjes AW, Westwood ME, Mallett S, Deeks JJ, Reitsma JB, et al. QUADAS-2: a revised tool for the quality assessment of diagnostic accuracy studies. Ann Intern Med. 2011;155(8):529–36. https://doi.org/10.7326/0003-4819-155-8-201110180-00009.CrossRefPubMed

Leeflang MM, Deeks JJ, Gatsonis C, Bossuyt PM. Systematic reviews of diagnostic test accuracy. Ann Intern Med. 2008;149(12):889–97. https://doi.org/10.7326/0003-4819-149-12-200812160-00008.CrossRefPubMedPubMedCentral

10.

Ochodo EA, Van Enst WA, Naaktgeboren CA, De Groot JA, Hooft L, Moons KG, et al. Incorporating quality assessments of primary studies in the conclusions of diagnostic accuracy reviews: a cross-sectional study. BMC Med Res Methodol. 2014;14(1):33. https://doi.org/10.1186/1471-2288-14-33.CrossRefPubMedPubMedCentral

11.

Hultcrantz M, Rind D, Akl EA, Treweek S, Mustafa RA, Iorio A, et al. The GRADE Working Group clarifies the construct of certainty of evidence. J Clin Epidemiol. 2017;87:4–13. https://doi.org/10.1016/j.jclinepi.2017.05.006.CrossRefPubMedPubMedCentral

12.

Sterne J, Savović J, Page M, Elbers R, Blencowe N, Boutron I, et al. RoB 2: A revised Cochrane risk-of-bias tool for randomized trials. BMJ. 2019;366:l48981.

13.

Lee CH, Cook S, Lee JS, Han B. Comparison of two meta-analysis methods: inverse-variance-weighted average and weighted sum of Z-scores. Genom Inform. 2016;14(4):173–80. https://doi.org/10.5808/GI.2016.14.4.173.CrossRef

14.

Marín-Martínez F, Sánchez-Meca J. Weighting by inverse variance or by sample size in random-effects meta-analysis. Educ Psychol Meas. 2010;70(1):56–73. https://doi.org/10.1177/0013164409344534.CrossRef

15.

Sánchez-Meca J, Marin-Martinez F. Weighting by inverse variance or by sample size in meta-analysis: a simulation study. Educ Psychol Meas. 1998;58(2):211–20. https://doi.org/10.1177/0013164498058002005.CrossRef

16.

Deeks JJ, Macaskill P, Irwig L. The performance of tests of publication bias and other sample size effects in systematic reviews of diagnostic test accuracy was assessed. J Clin Epidemiol. 2005;58(9):882–93. https://doi.org/10.1016/j.jclinepi.2005.01.016.CrossRefPubMed

17.

Gatsonis C, Paliwal P. Meta-analysis of diagnostic and screening test accuracy evaluations: methodologic primer. Am J Roentgenol. 2006;187(2):271–81. https://doi.org/10.2214/AJR.06.0226.CrossRef

18.

McGuinness LA, Higgins JP. Risk-of-bias VISualization (robvis): an R package and Shiny web app for visualizing risk-of-bias assessments. Res Synth Methods. 2020.

19.

Lee J, Vali Y, Boursier J, Duffin K, Verheij J, Brosnan MJ, et al. Accuracy of cytokeratin 18 (M30 and M65) in detecting non-alcoholic steatohepatitis and fibrosis: a systematic review and meta-analysis. PLoS ONE. 2020:1–19.

20.

Vali Y, Lee J, Boursier J, Spijker R, Löffler J, Verheij J, et al. Enhanced liver fibrosis test for the non-invasive diagnosis of fibrosis in patients with NAFLD: a systematic review and meta-analysis. J Hepatol. 2020;73(2):252–62. https://doi.org/10.1016/j.jhep.2020.03.036.CrossRefPubMed

21.

Lombardi G, Crescioli G, Cavedo E, Lucenteforte E, Casazza G, Bellatorre AG, et al. Structural magnetic resonance imaging for the early diagnosis of dementia due to Alzheimer’s disease in people with mild cognitive impairment. Cochrane Database Syst Rev. 2020;3.

22.

Mattioni A, Cenciarelli S, Eusebi P, Brazzelli M, Mazzoli T, Del Sette M, et al. Transcranial Doppler sonography for detecting stenosis or occlusion of intracranial arteries in people with acute ischaemic stroke. Cochrane Database Syst Rev. 2020;2.

23.

McClish DK. Combining and comparing area estimates across studies or strata. Med Decis Mak. 1992;12(4):274–9. https://doi.org/10.1177/0272989X9201200405.CrossRef

24.

Harbord RM, Whiting P, Sterne JA, Egger M, Deeks JJ, Shang A, et al. An empirical comparison of methods for meta-analysis of diagnostic accuracy showed hierarchical models are necessary. J Clin Epidemiol. 2008;61(11):1095–103. https://doi.org/10.1016/j.jclinepi.2007.09.013.CrossRefPubMed

25.

Rutter CM, Gatsonis CA. A hierarchical regression approach to meta-analysis of diagnostic test accuracy evaluations. Stat Med. 2001;20(19):2865–84. https://doi.org/10.1002/sim.942.CrossRefPubMed

26.

Whiting P, Savović J, Higgins JP, Caldwell DM, Reeves BC, Shea B, et al. ROBIS: a new tool to assess risk of bias in systematic reviews was developed. J Clin Epidemiol. 2016;69:225–34. https://doi.org/10.1016/j.jclinepi.2015.06.005.CrossRefPubMedPubMedCentral

27.

Leeflang M, Reitsma J, Scholten R, Rutjes A, Di Nisio M, Deeks J, et al. Impact of adjustment for quality on results of metaanalyses of diagnostic accuracy. Clin Chem. 2007;53(2):164–72. https://doi.org/10.1373/clinchem.2006.076398.CrossRefPubMed

28.

Burke DL, Ensor J, Snell KI, van der Windt D, Riley RD. Guidance for deriving and presenting percentage study weights in meta-analysis of test accuracy studies. Res Synth Methods. 2018;9(2):163–78. https://doi.org/10.1002/jrsm.1283.CrossRefPubMed

29.

La Torre G, Chiaradia G, Gianfagna F, Boccia S, De Laurentis A, Ricciardi W. Quality assessment in meta-analyses. Ital J Public Health. 2006;3(2).

30.

Whiting P, Harbord R, Kleijnen J. Scoring the quality of diagnostic accuracy studies: an example using QUADAS; 2004.

31.

Greenland S, O’rourke K. On the bias produced by quality scores in meta-analysis, and a hierarchical view of proposed solutions. Biostatistics. 2001;2(4):463–71. https://doi.org/10.1093/biostatistics/2.4.463.CrossRefPubMed

32.

Jüni P, Altman DG, Egger M. Assessing the quality of controlled clinical trials. Bmj. 2001;323(7303):42–6. https://doi.org/10.1136/bmj.323.7303.42.CrossRefPubMedPubMedCentral

Title: Application of weighting methods for presenting risk-of-bias assessments in systematic reviews of diagnostic test accuracy studies
Authors: Yasaman Vali
Mariska M. G. Leeflang
Patrick M. M. Bossuyt
Publication date: 01-12-2021
Publisher: BioMed Central
Published in: Systematic Reviews / Issue 1/2021
Electronic ISSN: 2046-4053
DOI: https://doi.org/10.1186/s13643-021-01744-z

At a glance: The STEP trials

Springer Medicine

Application of weighting methods for presenting risk-of-bias assessments in systematic reviews of diagnostic test accuracy studies

Abstract

Background

Methods

Results

Conclusions

Systematic review registrations

At a glance: The STEP trials

Springer Medicine

Abstract

Background

Methods

Results

Conclusions

Systematic review registrations

Please log in to get access to this content

Other articles of this Issue 1/2021

Non-primary progressive language impairment in neurodegenerative conditions: protocol for a scoping review

Effectiveness of task-shifting for the detection of diabetic retinopathy in low- and middle-income countries: a rapid review protocol

Theories of change for e-health interventions targeting HIV/STIs and sexual risk, substance use and mental ill health amongst men who have sex with men: systematic review and synthesis

School-based interventions targeting double burden of malnutrition and educational outcomes of adolescents in low- and middle-income countries: protocol for a systematic review

Anti-vascular endothelial growth factor therapy for age-related macular degeneration: a systematic review and network meta-analysis

Efficacy of adipose tissue-derived stem cells in locomotion recovery after spinal cord injury: a systematic review and meta-analysis on animal studies