Top

BMC Medical Research Methodology

Published in:

Open Access 01-12-2006 | Research article

Does updating improve the methodological and reporting quality of systematic reviews?

Authors: Beverley Shea, Maarten Boers, Jeremy M Grimshaw, Candyce Hamel, Lex M Bouter

Published in: BMC Medical Research Methodology | Issue 1/2006

Abstract

Background

Systematic reviews (SRs) must be of high quality. The purpose of our research was to compare the methodological and reporting quality of original versus updated Cochrane SRs to determine whether updating had improved these two quality dimensions.

Methods

We identifed updated Cochrane SRs published in issue 4, 2002 of the Cochrane Library. We assessed the updated and original versions of the SRs using two instruments: the 10 item enhanced Overview Quality Assessment Questionnaire (OQAQ), and an 18-item reporting quality checklist and flow chart based upon the Quality of Reporting of Meta-analyses (QUOROM) statement. At least two reviewers extracted data and assessed quality. We calculated the percentage (with a 95% confidence interval) of 'yes' answers to each question. We calculated mean differences in percentage, 95% confidence intervals and p-values for each of the individual items and the overall methodological quality score of the updated and pre-updated versions using OQAQ.

Results

We assessed 53 SRs. There was no significant improvement in the global quality score of the OQAQ (mean difference 0.11 (-0.28; 0.70 p = 0.52)). Updated reviews showed a significant improvement of 18.9 (7.2; 30.6 p < .01) on the OQAQ item assessing whether the conclusions drawn by the author(s) were supported by the data and/or analysis presented in the SR. The QUOROM statement showed that the quality of reporting of Cochrane reviews improved in some areas with updating. Improvements were seen on the items relating to data sources reported in the abstract, with a significant difference of 17.0 (9.8; 28.7 p = 0.01), review methods, reported in the abstract 35 (24.1; 49.1 p = 0.00), searching methods 18.9 (9.7; 31.6 p = 0.01), and data abstraction 18.9 (11.7; 30.9 p = 0.00).

Conclusion

The overall quality of Cochrane SRs is fair-to-good. Although reporting quality improved on certain individual items there was no overall improvement seen with updating and methodological quality remained unchanged. Further improvement of quality of reporting is possible. There is room for improvement of methodological quality as well. Authors updating reviews should address identified methodological or reporting weaknesses. We recommend to give full attention to both quality domains when updating SRs.

Available only for authorised users

Sacks HS, Berrier J, Reitman D, Ancona-Berk VA, Chalmers TC: Meta-analyses of randomized controlled trials. N Engl J Med. 1987, 316: 450-5.CrossRefPubMed

Sacks HS, Reitman D, Pagano D, Kupelnick B: Meta-analysis: an update. Mount Sinai Journal of Medicine. 1996, 63: 216-24.PubMed

Shea B: Assessing the quality of reporting meta-analyses of randomized controlled trials. MSc thesis. 1999, University of Ottawa, Department of Epidemiology and Community Medicine

Assendelft WJJ, Koes BW, Knipschild PG, Bouter LM: The relationship between methodological quality and conclusions in reviews of spinal manipulation. JAMA. 1995, 274: 1942-1948. 10.1001/jama.274.24.1942.CrossRefPubMed

Jadad AR, McQuay HJ: Meta-analyses to evaluate analogies interventions: a systematic qualitative review of their methodology. J Clin Epidemiol. 1996, 49: 235-43. 10.1016/0895-4356(95)00062-3.CrossRefPubMed

The Cochrane Collaboration. [http://www.cochrane.org/]

Jadad AR, Cook DJ, Jones A, Klassen TP, Tugwell P, Moher M, Moher D: Methodology and reports of systematic reviews and meta-analyses: a comparison of Cochrane reviews with articles published in paper-based journals. JAMA. 1998, 280: 278-280. 10.1001/jama.280.3.278.CrossRefPubMed

Jadad AR, Moher M, Brownman GP, Booker L, Sigouin C, Fuentes M, Stevens R: Systematic reviews and meta-analyses on treatment of asthma: critical evaluation. BMJ. 2000, 320: 537-540. 10.1136/bmj.320.7234.537.CrossRefPubMedPubMedCentral

Shea B, Moher D, Graham I, Pham B, Tugwell P: A comparison of the quality of reporting of Cochrane review and systematic reviews published in paper-based journals. Evaluations and the Health Professions. 2002, 25: 116-129. 10.1177/0163278702025001008.CrossRef

10.

The Cochrane Library. 2002, Chichester, UK: John Wiley & Sons, Ltd, 4

11.

Clarke M, Oxman AD, editors: Cochrane Reviewers' Handbook. The Cochrane Library. 2002, UK: Update Softeware, 2

12.

Shea B, Dube C, Moher D: Assessing the quality of reports of systematic reviews: the QUOROM statement compared to other tools. Systematic reviews in health care: meta-analysis in context. Edited by: Egger M, Smith GD, Altman DG. 2001, London, UK: BMJ Publishing Group, 122-39.CrossRef

13.

Oxman AD, Guyatt GH: Validation of an index of the quality of review articles. J Clin Epidemiol. 1991, 44: 1271-8. 10.1016/0895-4356(91)90160-B.CrossRefPubMed

14.

Oxman AD: Checklists for review articles. BMJ. 1994, 309: 648-651.CrossRefPubMedPubMedCentral

15.

Moher D, Cook DJ, Eastwood S, Olkin I, Rennie D, Stroup DF: Improving the quality of reports of meta-analyses of randomized controlled trials: the QUOROM statement. Quality of reporting of meta-analyses. Lancet. 1999, 354: 1896-900. 10.1016/S0140-6736(99)04149-5.CrossRefPubMed

16.

Moja LP, Telaro E, D'Amico R, Moschetti I, Coe L, Liberati A, on behalf of the Metaquality Study Group: Assessment of methodological quality of primary studies by systematic review: results of the metaquality cross sectional study. BMJ. 2005, 330: 1053-1058. 10.1136/bmj.38414.515938.8F.CrossRefPubMedPubMedCentral

17.

The Cochrane Library. 2005, Chichester, UK: John Wiley & Sons, Ltd, 4

The pre-publication history for this paper can be accessed here:http://www.biomedcentral.com/1471-2288/6/27/prepub

Title: Does updating improve the methodological and reporting quality of systematic reviews?
Authors: Beverley Shea
Maarten Boers
Jeremy M Grimshaw
Candyce Hamel
Lex M Bouter
Publication date: 01-12-2006
Publisher: BioMed Central
Published in: BMC Medical Research Methodology / Issue 1/2006
Electronic ISSN: 1471-2288
DOI: https://doi.org/10.1186/1471-2288-6-27

Keynote webinar | Spotlight on medication adherence

Springer Medicine

Does updating improve the methodological and reporting quality of systematic reviews?

Abstract

Background

Methods

Results

Conclusion

Keynote webinar | Spotlight on medication adherence

Springer Medicine

Abstract

Background

Methods

Results

Conclusion

Please log in to get access to this content

Other articles of this Issue 1/2006

How good is probabilistic record linkage to reconstruct reproductive histories? Results from the Aberdeen children of the 1950s study

Dealing with missing data in a multi-question depression scale: a comparison of imputation methods

Protocol of the COSMIN study: COnsensus-based Standards for the selection of health Measurement INstruments

Patient acceptability of larval therapy for leg ulcer treatment: a randomised survey to inform the sample size calculation of a randomised trial

UVB phototherapy in an outpatient setting or at home: a pragmatic randomised single-blind trial designed to settle the discussion. The PLUTO study

Identification of candidate categories of the International Classification of Functioning Disability and Health (ICF) for a Generic ICF Core Setbased on regression modelling