Skip to main content
Key Specialties
Cardiology Diabetology Endocrinology Gastroenterology Geriatrics and Gerontology Gynecology and Obstetrics Hematology Infectious Disease Internal Medicine Nephrology Neurology Oncology Pediatrics Respiratory Medicine Rheumatology Surgery
Congress Coverage
2024 ACC Congress 2023 ESMO Congress 2023 EASD Congress 2023 ERS Congress 2023 ESC Congress
Clinical Collections
Advances in Alzheimer’s

Keynote webinar | Spotlight on medication adherence

LIVE: Thursday 27th June 2024, 18:00-19:30 (CEST)
Join our expert panel to discover why you need to understand the drivers of non-adherence in your patients, and how you can optimize medication adherence in your clinics to drastically improve patient outcomes.
ADVANCED SEARCH
Log in
Top
BMC Medical Research Methodology
Published in: BMC Medical Research Methodology 1/2021

Open Access 01-12-2021 | Acupuncture | Research

Low methodological quality of systematic reviews on acupuncture: a cross-sectional study

Authors: Leonard Ho, Fiona Y. T. Ke, Charlene H. L. Wong, Irene X. Y. Wu, Andy K. L. Cheung, Chen Mao, Vincent C. H. Chung

Published in: BMC Medical Research Methodology | Issue 1/2021

Login to get access

Abstract

Background

While well-conducted systematic reviews (SRs) can provide the best evidence on the potential effectiveness of acupuncture, limitations on the methodological rigour of SRs may impact the trustworthiness of their conclusions. This cross-sectional study aimed to evaluate the methodological quality of a representative sample of SRs on acupuncture effectiveness.

Methods

Cochrane Database of Systematic Reviews, MEDLINE, and EMBASE were searched for SRs focusing on the treatment effect of manual acupuncture or electro-acupuncture published during January 2018 and March 2020. Eligible SRs must contain at least one meta-analysis and be published in English language. Two independent reviewers extracted the bibliographical characteristics of the included SRs with a pre-designed questionnaire and appraised the methodological quality of the studies with the validated AMSTAR 2 (A MeaSurement Tool to Assess systematic Reviews 2). The associations between bibliographical characteristics and methodological quality ratings were explored using Kruskal-Wallis rank tests and Spearman’s rank correlation coefficients.

Results

A total of 106 SRs were appraised. Only one (0.9%) SR was of high overall methodological quality, zero (0%) was of moderate-quality, six (5.7%) and 99 (93.4%) were of low-quality and critically low-quality respectively. Among appraised SRs, only ten (9.4%) provided an a priori protocol, four (3.8%) conducted a comprehensive literature search, five (4.7%) provided a list of excluded studies, and six (5.7%) performed meta-analysis appropriately. Cochrane SRs, updated SRs, and SRs that did not search non-English databases had relatively higher overall quality.

Conclusions

Methodological quality of SRs on acupuncture is unsatisfactory. Future reviewers should improve critical methodological aspects of publishing protocols, performing comprehensive search, providing a list of excluded studies with justifications for exclusion, and conducting appropriate meta-analyses. These recommendations can be implemented via enhancing the technical competency of reviewers in SR methodology through established education approaches as well as quality gatekeeping by journal editors and reviewers. Finally, for evidence users, skills in SR critical appraisal remain to be essential as relevant evidence may not be available in pre-appraised formats.
Appendix
Available only for authorised users
Literature
1.
World Health Organization. WHO traditional medicine strategy: 2014–2023. Geneva: World Health Organization; 2013.
2.
Chung VCH, Ma PHX, Wang HHX, Wang JJ, Hong LC, Wei X, et al. Integrating traditional Chinese medicine Services in Community Health Centers: insights into utilization patterns in the Pearl River region of China. Evid Based Complement Alternat Med. 2013;2013:426360.PubMedPubMedCentral
3.
Chen FP, Kung YY, Chen TJ, Hwang SJ. Demographics and patterns of acupuncture use in the Chinese population: the Taiwan experience. J Altern Complement Med. 2006;12(4):379–87.PubMedCrossRef
4.
Wu MY, Lee YC, Lin CL, Huang MC, Sun MF, Yen HR. Trends in use of acupuncture among adults in Taiwan from 2002 to 2011: a nationwide population-based study. PLoS One. 2018;13(4):e0195490.PubMedPubMedCentralCrossRef
5.
6.
Trinczek K. Reimbursement for acupuncture treatments in the German statutory health insurance system. Forsch Komplementmed. 2015;22(2):118–23.PubMed
7.
Jacobsen R, Fønnebø VM, Foss N, Kristoffersen AE. Use of complementary and alternative medicine within Norwegian hospitals. BMC Complement Altern Med. 2015;15(1):275.PubMedPubMedCentralCrossRef
8.
9.
10.
Liu Y, Zhang R, Huang J, Zhao X, Liu D, Sun W, et al. Reporting quality of systematic reviews/Meta-analyses of acupuncture. PLoS One. 2014;9(11):e113172.PubMedPubMedCentralCrossRef
11.
12.
Shea BJ, Reeves BC, Wells G, Thuku M, Hamel C, Moran J, et al. AMSTAR 2: a critical appraisal tool for systematic reviews that include randomised or non-randomised studies of healthcare interventions, or both. BMJ. 2017;358:j4008.PubMedPubMedCentralCrossRef
13.
Shea BJ, Grimshaw JM, Wells GA, Boers M, Andersson N, Hamel C, et al. Development of AMSTAR: a measurement tool to assess the methodological quality of systematic reviews. BMC Med Res Methodol. 2007;7(1):10.PubMedPubMedCentralCrossRef
14.
Turner EH, Matthews AM, Linardatos E, Tell RA, Rosenthal R. Selective publication of antidepressant trials and its influence on apparent efficacy. N Engl J Med. 2008;358(3):252–60.PubMedCrossRef
15.
Krnic Martinic M, Pieper D, Glatt A, Puljak L. Definition of a systematic review used in overviews of systematic reviews, meta-epidemiological studies and textbooks. BMC Med Res Methodol. 2019;19(1):203.PubMedPubMedCentralCrossRef
16.
Zhang J, Shang H, Gao X, Ernst E. Acupuncture-related adverse events: a systematic review of the Chinese literature. Bull World Health Organ. 2010;88(12):915–21c.PubMedPubMedCentralCrossRef
17.
World Health Organization. Acupuncture: review and analysis of reports on controlled clinical trials. Geneva: World Health Organization; 2003.
18.
19.
Lau CH, Wu X, Chung VC, Liu X, Hui EP, Cramer H, et al. Acupuncture and related therapies for symptom Management in Palliative Cancer Care: systematic review and Meta-analysis. Medicine (Baltimore). 2016;95(9):e2901.CrossRef
20.
21.
22.
Lunny C, Brennan SE, McDonald S, McKenzie JE. Toward a comprehensive evidence map of overview of systematic review methods: paper 1-purpose, eligibility, search and data extraction. Syst Rev. 2017;6(1):231.PubMedPubMedCentralCrossRef
23.
Wu XY, Lam VCK, Yu YF, Ho RST, Feng Y, Wong CHL, et al. Epidemiological characteristics and methodological quality of meta-analyses on diabetes mellitus treatment: a systematic review. Eur J Endocrinol. 2016;175(5):353.PubMedCrossRef
24.
Wu XY, Du XJ, Ho RS, Lee CC, Yip BH, Wong MC, et al. Characteristics and methodological quality of Meta-analyses on hypertension treatments-a cross-sectional study. J Clin Hypertens (Greenwich). 2017;19(2):137–42.CrossRef
25.
Ho RST, Wu X, Yuan J, Liu S, Lai X, Wong SYS, et al. Methodological quality of meta-analyses on treatments for chronic obstructive pulmonary disease: a cross-sectional study using the AMSTAR (assessing the methodological quality of systematic reviews) tool. NPJ Prim Care Respir Med. 2015;25(1):14102.PubMedPubMedCentralCrossRef
26.
Tsoi AKN, Ho LTF, Wu IXY, Wong CHL, Ho RST, Lim JYY, et al. Methodological quality of systematic reviews on treatments for osteoporosis: a cross-sectional study. Bone. 2020;139:115541.PubMedCrossRef
27.
Saha S, Saint S, Christakis DA. Impact factor: a valid measure of journal quality? J Med Libr Assoc. 2003;91(1):42–6.PubMedPubMedCentral
28.
Lei L, Sun Y. Should highly cited items be excluded in impact factor calculation? The effect of review articles on journal impact factor. Scientometrics. 2020;122(3):1697–706.CrossRef
29.
Wang J, Shapira P. Is there a relationship between research sponsorship and publication impact? An analysis of funding acknowledgments in nanotechnology papers. PLoS One. 2015;10(2):e0117727.PubMedPubMedCentralCrossRef
30.
Mackinnon S, Drozdowska BA, Hamilton M, Noel-Storr AH, McShane R, Quinn T. Are methodological quality and completeness of reporting associated with citation-based measures of publication impact? A secondary analysis of a systematic review of dementia biomarker studies. BMJ Open. 2018;8(3):e020331.PubMedPubMedCentralCrossRef
31.
Lorenz RC, Matthias K, Pieper D, Wegewitz U, Morche J, Nocon M, et al. A psychometric study found AMSTAR 2 to be a valid and moderately reliable appraisal tool. J Clin Epidemiol. 2019;114:133–40.PubMedCrossRef
32.
Leclercq V, Beaudart C, Tirelli E, Bruyère O. Psychometric measurements of AMSTAR 2 in a sample of meta-analyses indexed in PsycINFO. J Clin Epidemiol. 2020;119:144–5.PubMedCrossRef
33.
Lorenz RC, Matthias K, Pieper D, Wegewitz U, Morche J, Nocon M, et al. AMSTAR 2 overall confidence rating: lacking discriminating capacity or requirement of high methodological quality? J Clin Epidemiol. 2020;119:142–4.PubMedCrossRef
34.
35.
Neimann Rasmussen L, Montgomery P. The prevalence of and factors associated with inclusion of non-English language studies in Campbell systematic reviews: a survey and meta-epidemiological study. Syst Rev. 2018;7(1):129.PubMedPubMedCentralCrossRef
36.
Wu IXY, Deng Y, Wang H, Chen Y, Wong CHL, Chung VCH. Methodological quality of systematic reviews and Meta-analysis on asthma treatments. A cross-sectional study. Ann Am Thorac Soc. 2020;17(8):949–57.PubMedCrossRef
37.
Wu IX, Wang H, Zhu L, Chen Y, Wong CH, Mao C, et al. Methodological quality of systematic reviews on interventions for osteoarthritis: a cross-sectional study. Ther Adv Musculoskelet Dis. 2020;12:1759720x20959967.PubMedPubMedCentralCrossRef
38.
Silagy CA, Middleton P, Hopewell S. Publishing protocols of systematic reviews: comparing what was done to what was planned. JAMA. 2002;287(21):2831–4.PubMedCrossRef
39.
The PLoS Medicine Editors. Best practice in systematic reviews: the importance of protocols and registration. PLoS Med. 2011;8(2):e1001009.PubMedCentralCrossRef
40.
41.
Hart B, Lundh A, Bero L. Effect of reporting bias on meta-analyses of drug trials: reanalysis of meta-analyses. BMJ. 2012;344:d7202.PubMedCrossRef
42.
Jefferson T, Jones MA, Doshi P, Del Mar CB, Hama R, Thompson MJ, et al. Neuraminidase inhibitors for preventing and treating influenza in healthy adults and children. Cochrane Database Syst Rev. 2014;2014(4):Cd008965.PubMedCentral
43.
McAuley L, Pham B, Tugwell P, Moher D. Does the inclusion of grey literature influence estimates of intervention effectiveness reported in meta-analyses? Lancet. 2000;356(9237):1228–31.PubMedCrossRef
44.
Morrison A, Polisena J, Husereau D, Moulton K, Clark M, Fiander M, et al. The effect of English-language restriction on systematic review-based meta-analyses: a systematic review of empirical studies. Int J Technol Assess Health Care. 2012;28(2):138–44.PubMedCrossRef
45.
Wu XY, Tang JL, Mao C, Yuan JQ, Qin Y, Chung VC. Systematic Reviews and Meta-Analyses of Traditional Chinese Medicine Must Search Chinese Databases to Reduce Language Bias. Evid Based Complement Alternat Med. 2013;2013:812179.PubMedPubMedCentral
46.
Faggion CM. Critical appraisal of AMSTAR: challenges, limitations, and potential solutions from the perspective of an assessor. BMC Med Res Methodol. 2015;15(1):63.PubMedPubMedCentralCrossRef
47.
Wegewitz U, Weikert B, Fishta A, Jacobs A, Pieper D. Resuming the discussion of AMSTAR: what can (should) be made better? BMC Med Res Methodol. 2016;16(1):111.PubMedPubMedCentralCrossRef
48.
49.
Borenstein M, Hedges LV, Higgins JP, Rothstein HR. A basic introduction to fixed-effect and random-effects models for meta-analysis. Res Synth Methods. 2010;1(2):97–111.PubMedCrossRef
50.
Page MJ, McKenzie JE, Bossuyt PM, Boutron I, Hoffmann TC, Mulrow CD, et al. The PRISMA 2020 statement: an updated guideline for reporting systematic reviews. BMJ. 2021;372:n71.PubMedPubMedCentralCrossRef
51.
Dicenso A, Bayley L, Haynes RB. Accessing pre-appraised evidence: fine-tuning the 5S model into a 6S model. Evid Based Nurs. 2009;12(4):99–101.PubMedCrossRef
52.
Birch S, Lee MS, Alraek T, Kim TH. Overview of treatment guidelines and clinical practical guidelines that recommend the use of acupuncture: a bibliometric analysis. J Altern Complement Med. 2018;24(8):752–69.PubMedCrossRef
53.
Guo Y, Zhao H, Wang F, Li S-n, Sun Y-x, Han M-j, et al. Recommendations for acupuncture in clinical practice guidelines of the national guideline clearinghouse. Chin J Integr Med. 2017;23(11):864–70.PubMedCrossRef
54.
55.
McGowan BS, Reed JB, Yatcilla JK. Graduate student confidence following a for-credit systematic review course pilot. J Med Libr Assoc. 2021;109(2):323–9.PubMedPubMedCentralCrossRef
56.
Tsujimoto H, Kataoka Y, Sato Y, Banno M, Tsujino-Tsujimoto E, Sumi Y, et al. A model six-month workshop for developing systematic review protocols at teaching hospitals: action research and scholarly productivity. BMC Med Educ. 2021;21(1):98.PubMedPubMedCentralCrossRef
57.
Li T, Saldanha IJ, Vedula SS, Yu T, Rosman L, Twose C, et al. Learning by doing-teaching systematic review methods in 8 weeks. Res Synth Methods. 2014;5(3):254–63.PubMedCrossRef
Metadata
Title
Low methodological quality of systematic reviews on acupuncture: a cross-sectional study
Authors
Leonard Ho
Fiona Y. T. Ke
Charlene H. L. Wong
Irene X. Y. Wu
Andy K. L. Cheung
Chen Mao
Vincent C. H. Chung
Publication date
01-12-2021
Publisher
BioMed Central
Keyword
Acupuncture
Published in
BMC Medical Research Methodology / Issue 1/2021
Electronic ISSN: 1471-2288
DOI
https://doi.org/10.1186/s12874-021-01437-0

Other articles of this Issue 1/2021

BMC Medical Research Methodology 1/2021 Go to the issue

Research

A scoping review of Q-methodology in healthcare research

Research article

Improving postal survey response using behavioural science: a nested randomised control trial

Research article

A framework for exploring non-response patterns over time in health surveys

Research

Analysis of multivariate longitudinal substance use outcomes using multivariate mixed cumulative logit model

Research

Predicting clinical events using Bayesian multivariate linear mixed models with application to scleroderma

Research article

Sampling strategies to evaluate the prognostic value of a new biomarker on a time-to-event end-point