Skip to main content
Key Specialties
Cardiology Diabetology Endocrinology Gastroenterology Geriatrics and Gerontology Gynecology and Obstetrics Hematology Infectious Disease Internal Medicine Nephrology Neurology Oncology Pediatrics Respiratory Medicine Rheumatology Surgery
Congress Coverage
2024 ACC Congress 2023 ESMO Congress 2023 EASD Congress 2023 ERS Congress 2023 ESC Congress
Clinical Collections
Advances in Alzheimer’s

At a glance: The STEP trials

A round-up of the STEP phase 3 clinical trials evaluating semaglutide for weight loss in people with overweight or obesity.
ADVANCED SEARCH
Log in
Top
BMC Medical Research Methodology
Published in: BMC Medical Research Methodology 1/2011

Open Access 01-12-2011 | Research article

The assessment of the quality of reporting of meta-analyses in diagnostic research: a systematic review

Authors: Brian H Willis, Muireann Quigley

Published in: BMC Medical Research Methodology | Issue 1/2011

Login to get access

Abstract

Background

Over the last decade there have been a number of guidelines published, aimed at improving the quality of reporting in published studies and reviews. In systematic reviews this may be measured by their compliance with the PRISMA statement. This review aims to evaluate the quality of reporting in published meta-analyses of diagnostic tests, using the PRISMA statement and establish whether there has been a measurable improvement over time.

Methods

Eight databases were searched for reviews published prior to 31st December 2008. Studies were selected if they evaluated a diagnostic test, measured performance, searched two or more databases, stated the search terms and inclusion criteria, and used a statistical method to summarise a test's performance. Data were extracted on the review characteristics and items of the PRISMA statement. To measure the change in the quality of reporting over time, PRISMA items for two periods of equal duration were compared.

Results

Compliance with the PRISMA statement was generally poor: none of the reviews completely adhered to all 27 checklist items. Of the 236 meta-analyses included following selection: only 2(1%) reported the study protocol; 59(25%) reported the searches used; 76(32%) reported the results of a risk of bias assessment; and 82(35%) reported the abstract as a structured summary. Only 11 studies were published before 2000. Thus, the impact of QUOROM on the quality of reporting was not evaluated. However, the periods 2001-2004 and 2005-2008 (covering 93% of studies) were compared using relative risks (RR). There was an increase in the proportion of reviews reporting on five PRISMA items: eligibility criteria (RR 1.13, 95% CI 1.00 - 1.27); risk of bias across studies (methods) (RR 1.81, 95% CI 1.34 - 2.44); study selection results (RR 1.48, 95% CI 1.05 - 2.09); results of individual studies (RR 1.37, 95% CI 1.09 - 1.72); risk of bias across studies (results) (RR 1.65, 95% CI 1.20 - 2.25).

Conclusion

Although there has been an improvement in the quality of meta-analyses in diagnostic research, there are still many deficiencies in the reporting which future reviewers need to address if readers are to trust the validity of the reported findings.
Appendix
Available only for authorised users
Literature
1.
Buntinx F, Aertgeerts B, Macaskill P: Guidelines for conducting systematic reviews of studies evaluating the accuracy of diagnostic tests. The evidence base for clinical diagnosis: theory and methods of diagnostic research. Edited by: Knottnerus JA, Buntix F. 2009, London: BMJ books, 180-212. 2
2.
3.
Reitsma JB, Glas AS, Rutjes AW, Scholten RJ, Bossuyt PM, Zwinderman AH: Bivariate analysis of sensitivity and specificity produces informative summary measures in diagnostic reviews. J Clin Epidemiol. 2005, 58: 982-90. 10.1016/j.jclinepi.2005.02.022.CrossRefPubMed
4.
Rutter CM, Gatsonis CA: A hierarchical regression approach to meta-analysis of diagnostic test accuracy evaluations. Stat Med. 2001, 20: 2865-2884. 10.1002/sim.942.CrossRefPubMed
5.
Song F, Eastwood AJ, Gilbody S, Duley L, Sutton AJ: Publication and related biases. Health Technol Assess. 2000, 4 (10):
6.
Egger M, Dickersin K, Smith GD: Chapter 3, Problems and limitations in conducting systematic reviews. Systematic reviews in healthcare, BMJ books. Edited by: Egger M, Smith GD, Altman DG. 2001, London, 43-68. 2CrossRef
7.
Higgins JPT, Thompson SG: Quantifying heterogeneity in a meta-analysis. Stat Med. 2002, 21 (11): 1539-1559. 10.1002/sim.1186.CrossRefPubMed
8.
Thompson SG: Why and how sources of heterogeneity should be investigated. Systematic reviews in healthcare, BMJ books. Edited by: Egger M, Smith GD, Altman DG. 2001, 157-175. 2CrossRef
9.
Deville W, Butinx F, Bouter LM, Montori VM, de Vet HCW, van der Windt DAWM, Bezemer PR: Conducting systematic reviews of diagnostic studies: didactic guidelines. BMC Medical Research Methodology. 2002, 2: 9-10.1186/1471-2288-2-9.CrossRefPubMedPubMedCentral
10.
Bossuyt PM, Reitsma JB, Bruns DE, Gatsonis CA, Glasziou PP, Irwig LM, Lijmer JG, Moher D, Rennie D, de Vet HC: Standards for Reporting of Diagnostic Accuracy. Towards complete and accurate reporting of studies of diagnostic accuracy: the STARD initiative. BMJ. 2003, 326 (7379): 41-4. 10.1136/bmj.326.7379.41.CrossRefPubMedPubMedCentral
11.
Reitsma JB, Rutjes AWS, Whiting P, Vlassov VV, Leeflang MMG, Deeks JJ: Chapter 9: Assessing methodological quality. Cochrane Handbook for Systematic Reviews of Diagnostic Test Accuracy Version 1.0.0. The Cochrane Collaboration. Edited by: Deeks JJ, Bossuyt PM, Gatsonis C. 2009, Available from: http://​srdta.​cochrane.​org/​
12.
Whiting P, Rutjes AWS, Dinnes J, Reitsma JB, Bossuyt PMM, Kleijnen J: Development and validation of methods for assessing the quality of diagnostic accuracy studies. Health Technol Assess. 2004, 8 (25):
13.
Moher D, Cook DJ, Eastwood S, Olkin I, Rennie D, Stroup DF: Improving the quality of reports of meta-analyses of randomised controlled trials: the QUOROM statement. Quality of Reporting of Meta-analyses. Lancet. 1999, 354: 1896-900. 10.1016/S0140-6736(99)04149-5.CrossRefPubMed
14.
Liberati A, Altman DG, Tetzlaff J, Mulrow C, Gotzsche PC, Ioannidis JPA, Clarke M, Devereaux PJ, Kleijnen J, Moher D: The PRISMA statement for reporting systematic reviews and meta-analyses of studies that evaluate healthcare interventions: explanation and elaboration. PLOS Medicine. 2009, 6 (7): e1000100-10.1371/journal.pmed.1000100.CrossRefPubMedPubMedCentral
15.
Tatsioni A, Zarin DA, Aronson N, Samson DJ, Flamm CR, Schmid C, Lau J: Challenges in Systematic Reviews of Diagnostic Technologies. Ann Intern Med. 2005, 142: 1048-1055.CrossRefPubMed
16.
Willis and Quigley: Uptake of newer methodological developments and the deployment of meta-analysis in diagnostic test research: a systematic review. BMC Medical Research Methodology. 2011, 11: 27-10.1186/1471-2288-11-27.CrossRefPubMed
17.
Golder S, McIntosh HM, Duffy S, Glanville J: Centre for Reviews and Dissemination and UK Cochrane Centre Search Filters Design Group. Developing efficient search strategies to identify reports of adverse effects in MEDLINE and EMBASE. Health Information and Libraries Journal. 2006, 23: 3-12.CrossRefPubMed
18.
Whiting P, Westwood M, Burke M, Sterne J, Glanville J: Systematic reviews of test accuracy should search a range of databases to identify primary studies Journal of Clinical Epidemiology. 2008, 61: 357-364.
19.
Morris JA, Gardner MJ: Epidemiological Studies. Statistics with confidence. Edited by: Altman D, Machin D, Bryant TN, Gardner M. 2001, London: BMJ books, 57-67. 2
20.
Hopewell S, Dutton S, Yu LM, Chan AW, Altman DG: The quality of reports of randomised trials in 2000 and 2006: comparative study of articles indexed in PubMed. BMJ. 2010, 340: c723-10.1136/bmj.c723.CrossRefPubMedPubMedCentral
21.
Armitage P, Berry G, Matthews JNS: Statistical methods in medical research. 2002, Oxford, UK: Blackwell, 134-137. 4CrossRef
22.
Cohen J: A Coefficient of Agreement for Nominal Scales. Educ Psychol Meas. 1960, 20: 37-46. 10.1177/001316446002000104.CrossRef
23.
Fleiss JL, Levin B, Paik MC: Statistical methods for rates and proportions. 2003, New Jersey: John Wiley & Sons, 598-622. 3CrossRef
24.
Landis JR, Koch GG: The Measurement of Observer Agreement for Categorical Data. Biometrics. 1977, 33 (1): 159-174. 10.2307/2529310.CrossRefPubMed
25.
Stevens A, Milne R, Burls A: Health technology assessment: history and demand. J Pub Hlth Med. 2003, 25: 98-101.CrossRef
26.
Schmid C, Chung M, Chew P, Lau J: [O13] Survey of diagnostic test meta-analyses. 12th Cochrane Colloquium. 2004, Ottawa
27.
Devereaux PJ, Choi PT, El-Dika S, Bhandari M, Montori VM, Schunemann HJ, Garg AX, Busse JW, Heels-Ansdell D, Ghali WA, Manns BJ, Guyatt GH: An observational study found that authors of randomised controlled trials frequently use concealment of randomization and blinding, despite the failure to report these methods. J Clin Epidemiol. 2004, 57: 1232-6. 10.1016/j.jclinepi.2004.03.017.CrossRefPubMed
28.
Daniels S, Kumar A, Scott C, Swann S, Djulbegovic B: Bad reporting does not mean bad methods for randomised trials: observational study of randomised controlled trials performed by the Radiation Therapy Oncology Group. BMJ. 2004, 328: 22-4. 10.1136/bmj.328.7430.22.CrossRefPubMedPubMedCentral
29.
Chan AW, Hrobjartsson A, Jorgensen KJ, Gotzsche PC, Altman DG: Discrepancies in sample size calculations and data analyses reported in randomised trials: comparison of publications with protocols. BMJ. 2008, 337: a2299-10.1136/bmj.a2299.CrossRefPubMedPubMedCentral
30.
Pildal J, Chan AW, Hrobjartsson A, Forfang E, Altman DG, Gotzsche PC: Comparison of descriptions of allocation concealment in trial protocols and the published reports: cohort study. BMJ. 2005, 330: 1049-10.1136/bmj.38414.422650.8F.CrossRefPubMedPubMedCentral
31.
Moses LE, Shapiro D, Littenberg B: Combining independent studies of a diagnostic test into a summary ROC curve: data-analytic approaches and some additional considerations. Stat Med. 1993, 12 (14): 1293-316. 10.1002/sim.4780121403.CrossRefPubMed
32.
Littenberg B, Moses LE: Estimating diagnostic accuracy from multiple conflicting reports: a new meta-analytic method. Med decis Making. 1993, 13: 313-321. 10.1177/0272989X9301300408.CrossRefPubMed
33.
Van Houwelingen HC, Zwinderman KH, Stijnen T: A bivariate approach to meta-analysis. Statist Med. 1993, 12: 2273-2284. 10.1002/sim.4780122405.CrossRef
34.
Chalmers I, Altman D: Foreword. Systematic Reviews. Edited by: Chalmers I, Altman D. 1995, London: BMJ Publishing Group
35.
Green S, Higgins JPT, Alderson P, Clarke M, Mulrow CD, Oxman AD: Chapter 1: Introduction. Cochrane Handbook for Systematic Reviews of Interventions Version 5.0.1 (updated September 2008). The Cochrane Collaboration. Edited by: Higgins JPT, Green S. 2008, Available from http://​www.​cochrane-handbook.​org
Metadata
Title
The assessment of the quality of reporting of meta-analyses in diagnostic research: a systematic review
Authors
Brian H Willis
Muireann Quigley
Publication date
01-12-2011
Publisher
BioMed Central
Published in
BMC Medical Research Methodology / Issue 1/2011
Electronic ISSN: 1471-2288
DOI
https://doi.org/10.1186/1471-2288-11-163

Other articles of this Issue 1/2011

BMC Medical Research Methodology 1/2011 Go to the issue

Research article

Psychometric properties and longitudinal validation of the self-reporting questionnaire (SRQ-20) in a Rwandan community setting: a validation study

Research article

Potential risk factors associated with human encephalitis: application of canonical correlation analysis

Research article

Estimating the mode of inheritance in genetic association studies of qualitative traits based on the degree of dominance index

Research article

A poisson regression approach for modelling spatial autocorrelation between geographically referenced observations

Research article

Being a quantitative interviewer: qualitatively exploring interviewers' experiences in a longitudinal cohort study

Study protocol

Study protocol: the empirical investigation of methods to correct for measurement error in biobanks with dietary assessment