http://orcid.org/0000-0002-9836-4800
Figures
Abstract
Background
Meta-analysis is often regarded as one of the best sources of evidence for clinical nurses due to its rigorous design and scientific reflection of the true results of nursing interventions. The quality of a meta-analysis is critical to the work of clinical decision-makers. Therefore, the objective of this study was to use the JBI guidelines to summarize the quality of RCT-based meta-analyses of reports published in domestic nursing professional journals, with a view to standardizing the research process and reporting methods.
Methods
We performed a comprehensive literature search for RCT-based meta-analyses published in Chinese professional nursing journals, from their inception to December 31, 2015, using bibliographic databases (e.g. CNKI, WanFang Database). March 1, 2017, supplementary search 2016 literature. Candidate reviews were assessed for inclusion by two independent reviewers using pre-specified eligibility criteria. We evaluated the quality of reporting of the included meta-analyses using the systematic review literature reporting specification of JBI. Analyses were performed using Excel and STATA 12.0 software.
Results
Three hundred and twenty-two meta-analyses were included. According to the JBI guidelines, the overall quality of the meta-analysis report was poor. The quality of core journal reports and the implementation of retrieval were better than those of non-core journals. The nature of the authors and the availability of funding support had no significant impact on the quality of the meta-analyses. Multi-unit and multi-author collaboration can help improve the quality of meta-analyses with significant impact.
Conclusion
The understanding and implementation of systematic evaluation and meta-analyses in domestic nursing professional journals is worthy of recognition, and there is more work that can be done to improve the quality of these reports. Systematic review / Meta-analysis (SR / MA) makers should include the findings of this study. Multi-institutional and multi-author collaborations appear to improve research capacity and provide more reliable evidence support for clinicians.
Citation: Hou Y, Tian J, Zhang J, Yun R, Zhang Z, Chen K-H, et al. (2017) Quality of meta-analysis in nursing fields: An exploration based on the JBI guidelines. PLoS ONE 12(5): e0177648. https://doi.org/10.1371/journal.pone.0177648
Editor: Joel Joseph Gagnier, University of Michigan, UNITED STATES
Received: December 19, 2016; Accepted: April 30, 2017; Published: May 23, 2017
Copyright: © 2017 Hou et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Data Availability: All relevant data are within the paper and its Supporting Information files.
Funding: The authors received no specific funding for this work.
Competing interests: The authors have declared that no competing interests exist.
Introduction
Evidence-based nursing is one branch of evidence-based medicine, in which caregivers conscientiously and judiciously combine scientific findings with clinical experience and patient desires in the clinical care decision-making process[1]. As the most important part of evidence-based nursing, clinical care evidence can be divided into four levels, of which, systematic reviews/ meta-analyses (SR/ MA) represent the first-level evidence, where the research design is more rigorous and can reflect the actual results of nursing intervention and are often regarded as one of the best evidence sources by clinicians[2–3].
Systematic review is a new method of literature synthesis. Systematic reviews systematically evaluate a series of published and unpublished studies on a specific problem, using rigorous principles and methods of document evaluation, selecting documents that meet quality standards and conducting qualitative or quantitative synthesis, to draw the best conclusions. Systematic evaluation can be qualitative or quantitative (i.e., a process involving meta-analysis)[4]. Although SR/ MA are one of the best sources of clinical evidence, only high-quality SR/ MA can provide a scientific basis for decisions made by clinicians, nurses and other decision makers. Poor quality SR/ MA may mislead decision-makers. Therefore, effective quality assessment is an important part of the proper use of SR/ MA, as well as caution with the use of any findings.
The Joanna Briggs Institute (JBI) is a non-profit global organization that was founded in 1996. It is the second evidence-based care center following the establishment of the Evidence-Based Nursing Center at York University. It is also the largest international organization to promote evidence-based care in the world. As with other international organizations, JBI is committed to standardizing the process of systematic reviews to enhance both quality and reliability among collaborating agencies. JBI has developed a set of theories, methods and rigorous procedures for evaluating and synthesizing different types of evidence[5] and provides an overview of JBI systems that must be reported. A review of the JBI system, which focuses on the qualitative studies of intervention outcomes, should include 18 parts with 40 items, along with corresponding flowcharts[6].
Given the usefulness of the JBI system and the importance of meta-analysis to decision-makers, the main aim of this study is to use the JBI guidelines to summarize the quality of the RCT-based meta-analyses published in domestic nursing professional journals, with a view to standardizing the research process and reporting methods. The secondary aim is to explore whether the quality of meta-analyses published in core journals and non-core journals differs, and identify factors that influence the quality of meta-analyses.
Materials and methods
Eligibility criteria
Before we searched the literature, nursing professional journals were identified through State Administration of Press, Publication, Radio, Film and Television of The People’s Republic of China. CNKI and Wanfang database were searched for the literatures of SR/ MA in these journals. The search period included the beginning of each journal to December 31, 2015. March 1, 2017 supplementary seach 2016 literature. Repetitive literature, review, and methodological literature, systematic reviews of etiology/ diagnostics/ methodological studies, protocol, foreign abstracts or translations of systematic or meta-analyses, non-interventional quantitative systematic review, non-RCT-based meta-analyses were excluded. Only RCT-based meta-analyses in the intervention class were included.
Data abstraction
Two reviewers independently screened the articles based on the inclusion and exclusion criteria. Articles that did not meet the inclusion criteria in title, abstract, full text, and author information were excluded. Microsoft Office Excel 2007 software was used to establish the information extraction table, and the data was extracted independently by two reviewers. Data was collected from the included meta-analyses for study characteristics (e.g. primary author and publication year, journal type, agency, number of authors, number of outcomes, instruments used for quality assessment, funding, trial registries), retrieval information (e.g. number of bibliographic databases searched, additional search methods such as grey literature, hand searching, citation mapping) and reports on the contents for each of the various items of JBI guidelines. Prior to data extraction and quality evaluation, the use of each evaluation scale was assessed, and potential problems in data extraction and content consistency were evaluated before data extraction. The formal quality assessment process was independently completed and cross-checked by two investigators and resolved in the case of differences by a third party.
Data synthesis and quality appraisal
Two dimensions of meta-analysis reporting and retrieval conduct were explored. For the evaluation of retrieved documents, each entry was evaluated with "yes" or "no", and the number of "yes" was counted. We used the JBI guidelines to evaluate each included review to examine the by-item quality of report. The JBI guidelines contain a total of 40 items, and the response options for each domain were “compliance”, “partial compliance”, “nonconformity”. Each included study was evaluated individually, and we counted each sufficiently reported item (answer “yes” / “compliance”). The odds ratio (OR) was used as the statistical effect and the 95% confidence interval (95% CI) was calculated, differences in the quality of meta-analyses in core and non-core journals and the factors that influence the quality of the meta-analyses were analyzed using STATA 12.0 software.
Results
Selection and meta-analysis samples
The detailed study retrieval steps according to the PRISMA statement was shown in Fig 1. In total, 936 SR/ MA were retrieved from 26 Chinese nursing journals. After reading the title and abstract, 314 studies were excluded. Following full-text review, 150 non-quantitative studies, 106 non-RCTs and 44 non-interventional studies were excluded, and a total of 322 meta-analyses were finally included.
Characteristics of meta-analysis
Table 1 summarizes the characteristics of the included meta-analyses. The included 322 meta-analyses were from 20 of the 26 professional nursing journals in the country, including 15 core journals and 5 non-core journals. 296 articles (91.9%) were published in core journals, and 26 articles (8.1%) were published in non-core journals. Meta-analyses that met the requirements of this study were published after 2004, published in 2010 before the number of small and basically the same, after 2010, the number of published gradually increased5and faster. 83 (25.8%) of the studies were funded, but only one study was registered; 255 (79.2%) of the studies had ≥3 authors; 133 (41.3%) of the authors were from hospitals, 95 (29.5%) of the authors were from universities, 93 (28.9%) of the authors were multidisciplinary collaborations between schools and hospitals, and just 1 (0.3%) of the authors was from another institution. Of the included RCTs, the quality of the included RCTs was evaluated for 308 studies, 196 (60.9%) were evaluated using the Cochrane Handbook and 19 (5.9%) using the Jadad scale, and 16 (5%) were evaluated using the JBI standard.
Retrieval implementation of meta-analysis
Table 2 summarizes the retrieval implementation of the included meta-analyses. In retrieving journals from the databases, the search implementation was better in core journals than non-core journals and this difference was statistically significant (OR = 1.60, 95%CI: 1.18~2.17, P = 0.002). Among them, Pubmed / medline and Cochrane Library database retrieval status was significantly improved and the differences were statistically significant (Pubmed / Medline: P = 0.006 Cochrane Library: P = 0.006). There was no statistically significant difference in other database searches.
Reporting of meta-analysis: Compliance with the JBI guideline
Table 3 summarizes the compliance of the included meta-analysis reports with the JBI guidelines. None of the literature fully complied with the JBI guidelines. The overall completeness of the published articles in the core journals was higher than that of those published in the non-core journals and this difference was statistically significant (OR = 1.37, 95%CI:1.10~1.70, P = 0.005). Among these reports, the completion rate of item 11, item 19 and item 33 was obviously improved and the difference was statistically significant (item11: P = 0.042; item 19: P = 0.035; item 33: P = 0.017).
Table 4 summarizes the compliance of the included meta-analysis reports with the JBI guidelines of each item, stratified by published year. Overall, the completeness of the items is not relevant to the year of publication. Of the 40 items, seven items (item 3, item 4, item 16, item 26, item 31, item 38, item 40) that were included in the reports were fully compliant with the JBI guidelines. All reports were completely inconsistent in the following items with the JBI guidelines: item 2, item 5, Inclusion criteria、Invention、types of study in item 7, item 8, item 10, item 15, item 17, item 39.
Variables associated with JBI guidelines, stratified by funding status
Overall, there was no significant difference in the quality of the reports with or without funding (OR = 1.13, 95%CI: 0.98~1.30; P = 0.098), but the reporting completeness of item 18 was significantly increased in studies that were funded (item 18: OR = 1.78, 95%CI: 1.02~3.12, P = 0.043).
Variables associated with JBI guidelines, stratified by author institution status
The overall quality of the JBI report from studies with multi-agency cooperation (Hospital and University) was higher than those from hospitals alone and the difference was statistically significant. The items with statistically significant differences were: item 18, item 20, item 22, item 28, item 32, item 33, item 34 and item 35. Furthermore, the overall quality of the JBI report in studies from universities was higher than those from hospitals. The items that had statistically significant differences were: item 18, item 19, Item20, item 23, item 27, Item28, Item33 and item 35. There was no statistically significant difference in other items. (Table 5)
Variables associated with JBI guidelines, stratified by the number of authors
Overall, the JBI report quality was superior when ≥3 authors were involved, compared to when there were <3 authors and this difference was statistically significant. The completion rate of item 1, item 22, item 34 increased and the difference was statistically significant when ≥3 authors were involved.(Table 6)
Variables associated with JBI guidelines, stratified by the number of author units
Overall, the JBI report quality was greater when ≥2 author units were involved, compared to when <2 author units were involved and this difference was statistically significant (OR = 1.27, 95%CI: 1.11~1.46, P = 0.001). Multi-unit cooperation led to a significant increase in the reporting completeness of item 22 and 34, with a statistically significant difference (item 22: OR = 2.27, 95%CI: 1.10~4.71, P = 0.027; item 34: OR = 2.12, 95%CI: 1.21~3.68, P = 0.008).
In summary, the quality of meta-analyses published in core journals was better than that of non-core journals. The implementation of literature searches was better in the core journals than the non-core journals. The nature of the authors also affects the quality of the article. Overall quality of the JBI report from studies with multi-agency cooperation (Hospital and University) was higher than those from hospitals alone. Furthermore, the overall quality of the JBI report from studies with universities was superior to those from hospitals. The presence or absence of funding had no significant impact on the quality of the meta-analysis reports. When the number of authors was ≥3, the quality of the meta-analyses was better than when the number of authors <3, and multi-unit cooperation (≥2) was better than a single unit.
Discussion
RCT-based meta-analyses published in Chinese nursing journals were retrieved from a variety of sources. Since 2010, there has been a rapid increase in the number of research articles. Ninety-nine meta-analyses based on RCT were published in 2016, accounting for 30.7% of the total. From the number of authors and the author’s unit, more and more studies tended towards multi-institutional and multi-disciplinary cooperation. This trend is conducive to diversification and sustainable development.
The overall quality of the reports based on the JBI guideline in the literature was poor, and none of the studies were fully compliant with the JBI guidelines. However, the quality of articles published in core journals was significantly higher than those in non-core journals, which indicates that the efforts of domestic core journals in understanding and implementing SR / MA are worthy of recognition, but also have a lot of room for improvement. Multidisciplinary collaboration and multi-researcher have significantly improved the quality of meta-analyses, which has implications for future research and encourages multi-disciplinary collaboration to improve research quality.
The quality of the 322 included meta-analyses was variable. The major problems were: 1) the inclusion of multiple items in the abstract that did not meet the JBI guidelines (item 5, inclusion criteria, invention and types of study in item 7, 10, 12, 15), which may be related to the Chinese journals and their abstract requirements. There was a significant lack of registration number information. The registration of systematic reviews can reduce the risk of too many systematic reviews of the same topic[7,8], but also improve the transparency and credibility of any updates. 2) In the text, 91.9% of the studies did not report a complete search strategy, and the development of a reasonable and detailed search strategy can improve the recall rate and precision of the literature to ensure that the quality of the systematic evaluation, but also that the search results can be reproduced. 82.9% of the studies did not provide a PRISMA flow chart; 3) In the discussion section, only 38.5% of the studies described in detail the implications of the findings for future studies; only 0.62% of the studies address conflicts of interest, and potential stakeholder involvement could affect research design, implementation, and positive reporting of outcomes. Ignoring conflicts of interest may exaggerate the interpretation of the results, with unpredictable consequences[9,10].
In addition, the included meta-analyses also had certain positive merits: 1) 71.7% of the studies described in detail the methodological quality of the articles and improved the reliability of the results; 2) 66.5% of the studies reported the significance of research to clinical practice.
Multidisciplinary collaboration significantly improved the quality of meta-analysis reporting. The number of reports with ≥ 3 authors was significantly higher than those with <3 authors, which was in accordance with the minimum number of participants in the systematic review. The quality of the JBI report from studies with multi-agency cooperation (Hospital and University) was higher than those from hospitals alone and the difference was statistically significant. Furthermore, the overall quality of the JBI report in studies from universities was higher than those from hospitals. This may be related to the nature of the work of the authors, the degree of emphasis on scientific research, the direction of the research, but also shows that multi-agency, cross-professional cooperation is a current and future research trend.
Limitations of this study
1) The inclusion and screening of the literature is strongly subjective, and we still cannot completely rule out missing and false detection; 2) Although the quality evaluation of this study was carried out by two independent assessors and the evaluators were trained and pre-evaluated before the evaluation, the influence of subjective factors cannot be eliminated and may affect the objectivity of the evaluation.
Conclusions
Evaluation of 322 RCT-based meta-analyses in the domestic nursing field showed that overall reporting quality was poor based on the JBI guidelines. There were 34 of 40 items with varying degrees of missing information. In recent years, the number of meta-analyses has been increasing rapidly. In order to provide high-quality evidence support for clinicians and decision-makers, according to the shortcomings found in this study, future studies should have: 1) Strict compliance with the JBI guidelines for report writing, especially the design of the methodology should strictly abide by the relevant requirements of the JBI items to improve the strength of evidence; 2) Fellows engaged in SR / MA trained in epidemiology, statistics, and computer science to improve the ability of researchers to conduct evidence-based research, to control potential biases and to ensure reproducible and reliable results. 3) An increase in the number of multi-unit co-authors to improve the quality of the study. 4) Improving the quality of original studies, which greatly influence the quality of the evidence from the SR / MA.
Author Contributions
- Conceptualization: YYH JHT ZGZ.
- Data curation: RRY BW.
- Formal analysis: YYH JZ.
- Methodology: JHT KHC.
- Project administration: CYZ.
- Software: YYH JZ JHT.
- Supervision: CYZ ZGZ.
- Writing – original draft: YYH.
- Writing – review & editing: YYH JZ JHT.
References
- 1. Hu Yan, Xing Weijie. Concept and Steps of evidence—based nursing[J]. Shanghai Nursing. 2015,15(1): 89–93.
- 2. Hu Yan. “Evidence—based Nursing” and Clinical Nursing Practice[J]. Nurs J Chin PLA, 2002,19:3–4.
- 3. Paul M, Leibovici L. Systematic review or meta-analysis? Their place in the evidence hierarchy[J]. Clinical Microbiology Infection.2014,20:97–100. pmid:24354996
- 4.
Li You-Ping, Yang Ke-Hu. Evidence—Based Medicine[M]. Beijing: People's Medical Publishing House. 2014,110–111.
- 5. Pearson A, Wiechula R, Court A, Lockwood C. The JBI model of evidence-based healthcare[J]. Int J Evid Based Healthc, 2005, 3(8): 207–215. pmid:21631749
- 6.
Sun Feng. Explanation of Medical Research Report Norms[M]. Beijing: Peking University Medical Press. 2015, 265.
- 7. Bagshaw SM, McAlister FA, Manns BJ, Ghali WA. Acetylcysteine in the prevention of contrast-induced nephropathy: a case study of the pitfalls in the evolution of evidence. Arch Intern Med, 2006, 116(2): 161–166.
- 8. Biondi-Zoccai GG, Lotrionte M, Abbate A, Testa L, Remigi E, Burzotta F, et al. Compliance with QUOROM and quality of reporting of overlapping meta analyses on the role of acetylcysteine in the prevention of contrast associated nephropathy: Case study. BMJ, 2006, 322(7535), 202–209.
- 9. An Ni, Xu Jun-Feng, Ge Long, Liang Li, Shi Xin-Tong, Zhou Wei-Wen, et al. Reporting quality assessment of systematic reviews or meta-analysis of interventions published in Chinese Journal of Evidence-Based Pediatrics. Chin J Evid Based Pediatr. 2013, 8(2), 110–115.
- 10. Xu Jun-Feng, An Ni, Zhou Wei-Wen, Shi Xin-Tong, Liu Yin-Chun, Liang Li, et al. Methodological Quality Assessment of Systematic Reviews or Meta-Analyses of Intervention Published in the Chinese Journal of Evidence-Based Medicine. Chin J Evid Based Pediatr. 2013, 13(5): 605–611.