Skip to main content
Key Specialties
Cardiology Diabetology Endocrinology Gastroenterology Geriatrics and Gerontology Gynecology and Obstetrics Hematology Infectious Disease Internal Medicine Nephrology Neurology Oncology Pediatrics Respiratory Medicine Rheumatology Surgery
Congress Coverage
2024 ACC Congress 2023 ESMO Congress 2023 EASD Congress 2023 ERS Congress 2023 ESC Congress
Clinical Collections
Advances in Alzheimer’s

At a glance: The STEP trials

A round-up of the STEP phase 3 clinical trials evaluating semaglutide for weight loss in people with overweight or obesity.
ADVANCED SEARCH
Log in
Top
BMC Medical Research Methodology
Published in: BMC Medical Research Methodology 1/2016

Open Access 01-12-2016 | Debate

Resuming the discussion of AMSTAR: What can (should) be made better?

Authors: Uta Wegewitz, Beate Weikert, Alba Fishta, Anja Jacobs, Dawid Pieper

Published in: BMC Medical Research Methodology | Issue 1/2016

Login to get access

Abstract

Background

Evidence syntheses, and in particular systematic reviews (SRs), have become one of the cornerstones of evidence-based health care. The Assessment of Multiple Systematic Reviews (AMSTAR) tool has become the most widely used tool for investigating the methodological quality of SRs and is currently undergoing revision. The objective of this paper is to present insights, challenges and potential solutions from the point of view of a group of assessors, while referring to earlier methodological discussions and debates with respect to AMSTAR.

Discussion

One major drawback of AMSTAR is that it relies heavily on reporting quality rather than on methodological quality. This can be found in several items. Furthermore, it should be acknowledged that there are now new methods and procedures that did not exist when AMSTAR was developed. For example, the note to item 1 should now refer to the International Prospective Register of Ongoing Systematic Reviews (PROSPERO). Furthermore, item 3 should consider the definition of hand-searching, as the process of reviewing conference proceedings using the search function (e.g. in Microsoft Word or in a PDF file) does not meet the definition set out by the Cochrane Collaboration. Moreover, methods for assessing the quality of the body of evidence have evolved since AMSTAR was developed and should be incorporated into a revised AMSTAR tool.

Summary

Potential solutions are presented for each AMSTAR item with the aim of allowing a more thorough assessment of SRs. As the AMSTAR tool is currently undergoing further development, our paper hopes to add to preceding discussions and papers regarding this tool and stimulate further discussion.
Literature
1.
2.
Pieper D, et al. Overviews of reviews often have limited rigor: a systematic review. J Clin Epidemiol. 2012;65(12):1267–73.CrossRefPubMed
3.
Oxman AD, Guyatt GH. Validation of an index of the quality of review articles. J Clin Epidemiol. 1991;44(11):1271–8.CrossRefPubMed
4.
Oxman AD, et al. Agreement among reviewers of review articles. J Clin Epidemiol. 1991;44(1):91–8.CrossRefPubMed
5.
6.
Pieper D, et al. Systematic review found AMSTAR, but not R(evised)-AMSTAR, to have good measurement properties. J Clin Epidemiol. 2015;68(5):574–83.CrossRefPubMed
7.
Johnson BT, et al. Methodological quality of meta-analyses on the blood pressure response to exercise: a review. J Hypertens. 2014;32(4):706–23.CrossRefPubMed
8.
Huedo-Medina TB, et al. Methodologic quality of meta-analyses and systematic reviews on the Mediterranean diet and cardiovascular disease outcomes: a review. Am J Clin Nutr. 2016;103(3):841–50.CrossRefPubMed
9.
Berkhof M, et al. Effective training strategies for teaching communication skills to physicians: an overview of systematic reviews. Patient Educ Couns. 2011;84(2):152–62.CrossRefPubMed
10.
Kelley GA, Kelley KS. Effects of exercise in the treatment of overweight and obese children and adolescents: a systematic review of meta-analyses. J Obes. 2013;2013:783103.CrossRefPubMedPubMedCentral
11.
Aziz T, et al. Methodological quality and descriptive characteristics of prosthodontic-related systematic reviews. J Oral Rehabil. 2013;40(4):263–78.CrossRefPubMed
12.
Faggion Jr CM. Critical appraisal of AMSTAR: challenges, limitations, and potential solutions from the perspective of an assessor. BMC Med Res Methodol. 2015;15:63.CrossRefPubMedPubMedCentral
13.
Burda BU, Holmer HK, Norris SL. Limitations of A Measurement Tool to Assess Systematic Reviews (AMSTAR) and suggestions for improvement. Syst Rev. 2016;5(1):58.CrossRefPubMedPubMedCentral
14.
15.
16.
17.
18.
Kirkham JJ, Altman DG, Williamson PR. Bias due to changes in specified outcomes during the systematic review process. PLoS One. 2010;5(3).
19.
Page MJ, et al. Bias due to selective inclusion and reporting of outcomes and analyses in systematic reviews of randomised trials of healthcare interventions. Cochrane Database Syst Rev. 2014;10:MR000035.
20.
Silagy CA, Middleton P, Hopewell S. Publishing protocols of systematic reviews: comparing what was done to what was planned. JAMA. 2002;287(21):2831–4.CrossRefPubMed
21.
Booth A, et al. An international registry of systematic-review protocols. Lancet. 2011;377(9760):108–9.CrossRefPubMed
22.
23.
24.
Buscemi N, et al. Single data extraction generated more errors than double data extraction in systematic reviews. J Clin Epidemiol. 2006;59(7):697–703.CrossRefPubMed
25.
26.
Morrison A, et al. The effect of English-language restriction on systematic review-based meta-analyses: a systematic review of empirical studies. Int J Technol Assess Health Care. 2012;28(2):138–44.CrossRefPubMed
27.
Pham B, et al. Language of publication restrictions in systematic reviews gave different results depending on whether the intervention was conventional or complementary. J Clin Epidemiol. 2005;58(8):769–76.CrossRefPubMed
28.
Xue J, et al. Significant discrepancies were found in pooled estimates of searching with Chinese indexes versus searching with English indexes. J Clin Epidemiol. 2015;70:246–53.CrossRefPubMed
29.
Higgins, JPTD, JJ. Chapter 7: Selecting studies and collecting data, in Cochrane Handbook for Systematic Reviews of Interventions Version 5.1.0, J.P.T.G. Higgins, S., Editor. 2011, The Cochrane Collaboration.
30.
Hartling L, et al. Testing the Newcastle Ottawa Scale showed low reliability between individual reviewers. J Clin Epidemiol. 2013;66(9):982–93.CrossRefPubMed
31.
Stang A. Critical evaluation of the Newcastle-Ottawa scale for the assessment of the quality of nonrandomized studies in meta-analyses. Eur J Epidemiol. 2010;25(9):603–5.CrossRefPubMed
32.
Schünemann HJ, et al. Interpreting Results and Drawing Conclusions, in Cochrane Handbook for Systematic Reviews of Interventions. 2008, John Wiley & Sons, Ltd. p. 359–387
33.
34.
Berkman ND, et al. Grading the Strength of a Body of Evidence When Assessing Health Care Interventions for the Effective Health Care Program of the Agency for Healthcare Research and Quality: An Update. 2008
35.
Guyatt G, et al. GRADE guidelines: 1. Introduction-GRADE evidence profiles and summary of findings tables. J Clin Epidemiol. 2011;64(4):383–94.CrossRefPubMed
36.
Fu R, et al. Conducting Quantitative Synthesis When Comparing Medical Interventions: AHRQ and the Effective Health Care Program. 2008
37.
Kemp CA. Qigong as a therapeutic intervention with older adults. J Holist Nurs. 2004;22(4):351–73.CrossRefPubMed
38.
Ng BH, Tsang HW. Psychophysiological outcomes of health qigong for chronic conditions: a systematic review. Psychophysiology. 2009;46(2):257–69.CrossRefPubMed
39.
Guyatt G, et al. GRADE guidelines: 11. Making an overall rating of confidence in effect estimates for a single outcome and for all outcomes. J Clin Epidemiol. 2013;66(2):151–7.CrossRefPubMed
40.
Guyatt GH, et al. GRADE guidelines: 5. Rating the quality of evidence--publication bias. J Clin Epidemiol. 2011;64(12):1277–82.CrossRefPubMed
41.
Deeks JJ, Higgins JPT, Altman DG. Analysing Data and Undertaking Meta-Analyses, in Cochrane Handbook for Systematic Reviews of Interventions. 2008, John Wiley & Sons, Ltd. p. 243–296
42.
Kliner M, Garner P. When trial authors write Cochrane Reviews: competing interests need to be better managed. Cochrane Database Syst Rev. 2014;9:ED000089.
43.
44.
Moher D, et al. Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. Ann Intern Med. 2009;151(4):264–9. W64.CrossRefPubMed
Metadata
Title
Resuming the discussion of AMSTAR: What can (should) be made better?
Authors
Uta Wegewitz
Beate Weikert
Alba Fishta
Anja Jacobs
Dawid Pieper
Publication date
01-12-2016
Publisher
BioMed Central
Published in
BMC Medical Research Methodology / Issue 1/2016
Electronic ISSN: 1471-2288
DOI
https://doi.org/10.1186/s12874-016-0183-6

Other articles of this Issue 1/2016

BMC Medical Research Methodology 1/2016 Go to the issue

Research article

Time to publication among completed diagnostic accuracy studies: associated with reported accuracy estimates

Research article

Meta-analyses including non-randomized studies of therapeutic interventions: a methodological review

Research article

The effect of a monetary incentive for administrative assistants on the survey response rate: a randomized controlled trial

Research Article

Confidence intervals for the between-study variance in random-effects meta-analysis using generalised heterogeneity statistics: should we use unequal tails?

Research article

Inter-rater reliability of the QuIS as an assessment of the quality of staff-inpatient interactions

Research article

Mechanisms and direction of allocation bias in randomised clinical trials