Skip to main content
Top
Published in: BMC Medical Research Methodology 1/2019

Open Access 01-12-2019 | Research article

Methodological steps used by authors of systematic reviews and meta-analyses of clinical trials: a cross-sectional study

Authors: Hoang Thi Nam Giang, Ali Mahmoud Ahmed, Reem Yousry Fala, Mohamed Magdy Khattab, Mona Hassan Ahmed Othman, Sara Attia Mahmoud Abdelrahman, Le Phuong Thao, Ahmed Elsaid Abd Elsamie Gabl, Samar Ahmed Elrashedy, Peter N. Lee, Kenji Hirayama, Hosni Salem, Nguyen Tien Huy

Published in: BMC Medical Research Methodology | Issue 1/2019

Login to get access

Abstract

Background

The quality of systematic reviews and meta-analyses (SR/MAs) depends on the extent of the methods used. We investigated the methodological steps used by authors of SR/MAs of clinical trials via an author survey.

Methods

We conducted an email-based cross-sectional study by contacting corresponding authors of SR/MAs that were published in 2015 and 2016 and retrieved through the PubMed database. The 27-item questionnaire was developed to study the methodological steps used by authors when conducting a SR/MA and the demographic characteristics of the respondent. Besides the demographic characteristics, methodological questions regarding the source, extraction and synthesis of data were included.

Results

From 10,292 emails sent, 384 authors responded and were included in the final analysis. Manual searches were carried out by 69.2% of authors, while 87.3% do updated searches, 49.2% search grey literature, 74.9% use the Cochrane tool for risk of bias assessment, 69.8% assign more than two reviewers for data extraction, 20.5% use digital software to extract data from graphs, 57.9% use raw data in the meta-analysis, and 43.8% meta-analyze both adjusted and non-adjusted data. There was a positive correlation of years of experience in conducting of SR/MAs with both searching grey literature (P = 0.0003) and use of adjusted and non-adjusted data (P = 0.006).

Conclusions

Many authors still do not carry out many of the vital methodological steps to be taken when performing any SR/MA. The experience of the authors in SR/MAs is highly correlated with use of the recommended tips for SR/MA conduct. The optimal methodological approach for researchers conducting a SR/MA should be standardized.
Appendix
Available only for authorised users
Literature
1.
2.
go back to reference Yuan Y, Hunt RH. Systematic reviews: the good, the bad, and the ugly. Am J Gastroenterol. 2009;104(5):1086–92.CrossRef Yuan Y, Hunt RH. Systematic reviews: the good, the bad, and the ugly. Am J Gastroenterol. 2009;104(5):1086–92.CrossRef
3.
go back to reference Zoccali C. Moderator's view: Meta-analysis: the best knowledge but not always shining gold. Nephrol Dial Transplant. 2016;31(6):886–9.CrossRef Zoccali C. Moderator's view: Meta-analysis: the best knowledge but not always shining gold. Nephrol Dial Transplant. 2016;31(6):886–9.CrossRef
4.
go back to reference Hung BT, Long NP, Hung le P, Luan NT, Anh NH, Nghi TD, Hieu MV, Trang NT, Rafidinarivo HF, Anh NK et al. Research trends in evidence-based medicine: a joinpoint regression analysis of more than 50 years of publication data. PloS one. 2015;10(4):e0121054.CrossRef Hung BT, Long NP, Hung le P, Luan NT, Anh NH, Nghi TD, Hieu MV, Trang NT, Rafidinarivo HF, Anh NK et al. Research trends in evidence-based medicine: a joinpoint regression analysis of more than 50 years of publication data. PloS one. 2015;10(4):e0121054.CrossRef
5.
go back to reference Esterhuizen TM, Thabane L. Con: Meta-analysis: some key limitations and potential solutions. Nephrol Dial Transplant. 2016;31(6):882–5. Esterhuizen TM, Thabane L. Con: Meta-analysis: some key limitations and potential solutions. Nephrol Dial Transplant. 2016;31(6):882–5.
6.
go back to reference Sackett DL, Rosenberg WMC, Gray JAM, Haynes RB, Richardson WS. Evidence based medicine: what it is and what it isn't. BMJ. 1996;312(7023):71.CrossRef Sackett DL, Rosenberg WMC, Gray JAM, Haynes RB, Richardson WS. Evidence based medicine: what it is and what it isn't. BMJ. 1996;312(7023):71.CrossRef
7.
go back to reference Juni P, Witschi A, Bloch R, Egger M. The hazards of scoring the quality of clinical trials for meta-analysis. Jama. 1999;282(11):1054–60.CrossRef Juni P, Witschi A, Bloch R, Egger M. The hazards of scoring the quality of clinical trials for meta-analysis. Jama. 1999;282(11):1054–60.CrossRef
8.
go back to reference Balk EM, Bonis PA, Moskowitz H, Schmid CH, Ioannidis JP, Wang C, Lau J. Correlation of quality measures with estimates of treatment effect in meta-analyses of randomized controlled trials. Jama. 2002;287(22):2973–82.CrossRef Balk EM, Bonis PA, Moskowitz H, Schmid CH, Ioannidis JP, Wang C, Lau J. Correlation of quality measures with estimates of treatment effect in meta-analyses of randomized controlled trials. Jama. 2002;287(22):2973–82.CrossRef
9.
go back to reference Moyer A, Finney JW. Rating methodological quality: toward improved assessment and investigation. Account Res. 2005;12(4):299–313.CrossRef Moyer A, Finney JW. Rating methodological quality: toward improved assessment and investigation. Account Res. 2005;12(4):299–313.CrossRef
10.
go back to reference Moayyedi P. Meta-analysis: Can We Mix Apples and Oranges? Am J Gastroenterol. 2004;99:2297.CrossRef Moayyedi P. Meta-analysis: Can We Mix Apples and Oranges? Am J Gastroenterol. 2004;99:2297.CrossRef
11.
go back to reference Jahan N, Naveed S, Zeshan M, Tahir MA. How to Conduct a Systematic Review: A Narrative Literature Review. Cureus. 2016;8(11):e864. Jahan N, Naveed S, Zeshan M, Tahir MA. How to Conduct a Systematic Review: A Narrative Literature Review. Cureus. 2016;8(11):e864.
12.
go back to reference Henderson LK, Craig JC, Willis NS, Tovey D, Webster AC. How to write a Cochrane systematic review. Nephrology (Carlton, Vic). 2010;15(6):617–24.CrossRef Henderson LK, Craig JC, Willis NS, Tovey D, Webster AC. How to write a Cochrane systematic review. Nephrology (Carlton, Vic). 2010;15(6):617–24.CrossRef
13.
go back to reference Shea BJ, Grimshaw JM, Wells GA, Boers M, Andersson N, Hamel C, Porter AC, Tugwell P, Moher D, Bouter LM. Development of AMSTAR: a measurement tool to assess the methodological quality of systematic reviews. BMC Med Res Methodol. 2007;7(1):10. Shea BJ, Grimshaw JM, Wells GA, Boers M, Andersson N, Hamel C, Porter AC, Tugwell P, Moher D, Bouter LM. Development of AMSTAR: a measurement tool to assess the methodological quality of systematic reviews. BMC Med Res Methodol. 2007;7(1):10.
14.
go back to reference Moher D, Liberati A, Tetzlaff J, Altman DG, Group P. Preferred reporting items for systematic reviews and meta-analyses: the PRISMA Statement. Open medicine : a peer-reviewed, independent. Open-access Journal. 2009;3(3):e123–30. Moher D, Liberati A, Tetzlaff J, Altman DG, Group P. Preferred reporting items for systematic reviews and meta-analyses: the PRISMA Statement. Open medicine : a peer-reviewed, independent. Open-access Journal. 2009;3(3):e123–30.
15.
go back to reference Shea BJ, Reeves BC, Wells G, Thuku M, Hamel C, Moran J, Moher D, Tugwell P, Welch V, Kristjansson E. AMSTAR 2: a critical appraisal tool for systematic reviews that include randomised or non-randomised studies of healthcare interventions, or both. BmJ. 2017;358:j4008. Shea BJ, Reeves BC, Wells G, Thuku M, Hamel C, Moran J, Moher D, Tugwell P, Welch V, Kristjansson E. AMSTAR 2: a critical appraisal tool for systematic reviews that include randomised or non-randomised studies of healthcare interventions, or both. BmJ. 2017;358:j4008.
16.
go back to reference Whiting P, Savović J, Higgins JP, Caldwell DM, Reeves BC, Shea B, Davies P, Kleijnen J, Churchill R. ROBIS: a new tool to assess risk of bias in systematic reviews was developed. J Clin Epidemiol. 2016;69:225–34.CrossRef Whiting P, Savović J, Higgins JP, Caldwell DM, Reeves BC, Shea B, Davies P, Kleijnen J, Churchill R. ROBIS: a new tool to assess risk of bias in systematic reviews was developed. J Clin Epidemiol. 2016;69:225–34.CrossRef
17.
go back to reference Popay J, Roberts H, Sowden A, Petticrew M, Arai L, Rodgers M, Britten N, Roen K, Duffy S. Guidance on the conduct of narrative synthesis in systematic reviews. A product from the ESRC methods programme Version. 2006;1:b92. Popay J, Roberts H, Sowden A, Petticrew M, Arai L, Rodgers M, Britten N, Roen K, Duffy S. Guidance on the conduct of narrative synthesis in systematic reviews. A product from the ESRC methods programme Version. 2006;1:b92.
18.
go back to reference Mokkink LB, Terwee CB, Stratford PW, Alonso J, Patrick DL, Riphagen I, Knol DL, Bouter LM, de Vet HC. Evaluation of the methodological quality of systematic reviews of health status measurement instruments. Qual Life Res. 2009;18(3):313–33.CrossRef Mokkink LB, Terwee CB, Stratford PW, Alonso J, Patrick DL, Riphagen I, Knol DL, Bouter LM, de Vet HC. Evaluation of the methodological quality of systematic reviews of health status measurement instruments. Qual Life Res. 2009;18(3):313–33.CrossRef
19.
go back to reference Andrews D, Nonnecke B, Preece J. Electronic Survey Methodology: A Case Study in Reaching Hard-to-Involve Internet Users. Int J Human–Computer Interact. 2003;16(2):185–210.CrossRef Andrews D, Nonnecke B, Preece J. Electronic Survey Methodology: A Case Study in Reaching Hard-to-Involve Internet Users. Int J Human–Computer Interact. 2003;16(2):185–210.CrossRef
20.
go back to reference Cook C, Heath F, Thompson RL. A Meta-Analysis of Response Rates in Web- or Internet-Based Surveys. Educ Psychol Meas. 2000;60(6):821–36.CrossRef Cook C, Heath F, Thompson RL. A Meta-Analysis of Response Rates in Web- or Internet-Based Surveys. Educ Psychol Meas. 2000;60(6):821–36.CrossRef
22.
go back to reference World Medical Association Declaration of Helsinki: ethical principles for medical research involving human subjects. Jama. 2013;310(20):2191–4. World Medical Association Declaration of Helsinki: ethical principles for medical research involving human subjects. Jama. 2013;310(20):2191–4.
23.
go back to reference Moher D, Cook DJ, Eastwood S, Olkin I, Rennie D, Stroup DF: Improving the quality of reports of meta-analyses of randomised controlled trials: the QUOROM statement. Quality of Reporting of Meta-analyses. Lancet. 1999;354(9193):1896–900.CrossRef Moher D, Cook DJ, Eastwood S, Olkin I, Rennie D, Stroup DF: Improving the quality of reports of meta-analyses of randomised controlled trials: the QUOROM statement. Quality of Reporting of Meta-analyses. Lancet. 1999;354(9193):1896–900.CrossRef
24.
go back to reference Dosenovic S, Kadic AJ, Vucic K, Markovina N, Pieper D, Puljak L. Comparison of methodological quality rating of systematic reviews on neuropathic pain using AMSTAR and R-AMSTAR. BMC Med Res Methodol. 2018;18(1):37. Dosenovic S, Kadic AJ, Vucic K, Markovina N, Pieper D, Puljak L. Comparison of methodological quality rating of systematic reviews on neuropathic pain using AMSTAR and R-AMSTAR. BMC Med Res Methodol. 2018;18(1):37.
25.
go back to reference Sharma S, Oremus M. PRISMA and AMSTAR show systematic reviews on health literacy and cancer screening are of good quality. J Clin Epidemiol. 2018;99:123–31.CrossRef Sharma S, Oremus M. PRISMA and AMSTAR show systematic reviews on health literacy and cancer screening are of good quality. J Clin Epidemiol. 2018;99:123–31.CrossRef
26.
go back to reference Wu X, Sun H, Zhou X, Wang J, Li J. Quality assessment of systematic reviews on total hip or knee arthroplasty using mod-AMSTAR. BMC Med Res Methodol. 2018;18(1):30. Wu X, Sun H, Zhou X, Wang J, Li J. Quality assessment of systematic reviews on total hip or knee arthroplasty using mod-AMSTAR. BMC Med Res Methodol. 2018;18(1):30.
27.
go back to reference Shea B, Bouter LM, Grimshaw JM, Francis D, Ortiz Z, Wells GA, Tugwell PS, Boers M. Scope for improvement in the quality of reporting of systematic reviews. From the Cochrane Musculoskeletal Group. J Rheumatol. 2006;33(1):9–15. Shea B, Bouter LM, Grimshaw JM, Francis D, Ortiz Z, Wells GA, Tugwell PS, Boers M. Scope for improvement in the quality of reporting of systematic reviews. From the Cochrane Musculoskeletal Group. J Rheumatol. 2006;33(1):9–15.
28.
go back to reference Moseley AM, Elkins MR, Herbert RD, Maher CG, Sherrington C. Cochrane reviews used more rigorous methods than non-Cochrane reviews: survey of systematic reviews in physiotherapy. J Clin Epidemiol. 2009;62(10):1021–30.CrossRef Moseley AM, Elkins MR, Herbert RD, Maher CG, Sherrington C. Cochrane reviews used more rigorous methods than non-Cochrane reviews: survey of systematic reviews in physiotherapy. J Clin Epidemiol. 2009;62(10):1021–30.CrossRef
29.
go back to reference Egger M, Juni P, Bartlett C, Holenstein F, Sterne J. How important are comprehensive literature searches and the assessment of trial quality in systematic reviews? Empirical study. Health Technol Assess (Winchester, England). 2003;7(1):76. Egger M, Juni P, Bartlett C, Holenstein F, Sterne J. How important are comprehensive literature searches and the assessment of trial quality in systematic reviews? Empirical study. Health Technol Assess (Winchester, England). 2003;7(1):76.
30.
go back to reference Royle P, Waugh N. Literature searching for clinical and cost-effectiveness studies used in health technology assessment reports carried out for the National Institute for Clinical Excellence appraisal system. Health Technol Assess (Winchester, England). 2003;7(34):iii, ix–x, 1-51. Royle P, Waugh N. Literature searching for clinical and cost-effectiveness studies used in health technology assessment reports carried out for the National Institute for Clinical Excellence appraisal system. Health Technol Assess (Winchester, England). 2003;7(34):iii, ix–x, 1-51.
31.
go back to reference Chapman AL, Morgan LC, Gartlehner G. Semi-automating the manual literature search for systematic reviews increases efficiency. Health Info Libr J. 2010;27(1):22-27.CrossRef Chapman AL, Morgan LC, Gartlehner G. Semi-automating the manual literature search for systematic reviews increases efficiency. Health Info Libr J. 2010;27(1):22-27.CrossRef
32.
go back to reference Koffel JB. Use of Recommended Search Strategies in Systematic Reviews and the Impact of Librarian Involvement: A Cross-Sectional Survey of Recent Authors. PloS One 2015;10(5):e0125931.CrossRef Koffel JB. Use of Recommended Search Strategies in Systematic Reviews and the Impact of Librarian Involvement: A Cross-Sectional Survey of Recent Authors. PloS One 2015;10(5):e0125931.CrossRef
33.
go back to reference Charrois TL. Systematic Reviews: What Do You Need to Know to Get Started? Can J Hosp Pharm. 2015;68(2):144–8. Charrois TL. Systematic Reviews: What Do You Need to Know to Get Started? Can J Hosp Pharm. 2015;68(2):144–8.
34.
go back to reference Haynes RB. Clinical epidemiology: how to do clinical practice research: Lippincott williams & wilkins; 2012. Haynes RB. Clinical epidemiology: how to do clinical practice research: Lippincott williams & wilkins; 2012.
35.
go back to reference Pildal J, Hrobjartsson A, Jorgensen KJ, Hilden J, Altman DG, Gotzsche PC. Impact of allocation concealment on conclusions drawn from meta-analyses of randomized trials. Int J Epidemiol. 2007;36(4):847–57.CrossRef Pildal J, Hrobjartsson A, Jorgensen KJ, Hilden J, Altman DG, Gotzsche PC. Impact of allocation concealment on conclusions drawn from meta-analyses of randomized trials. Int J Epidemiol. 2007;36(4):847–57.CrossRef
36.
go back to reference Lajeunesse MJ, Koricheva J, Gurevitch J, Mengersen K. Recovering missing or partial data from studies: a survey of conversions and imputations for meta-analysis. In: Handbook of meta-analysis in ecology and evolution; 2013. p. 195–206.CrossRef Lajeunesse MJ, Koricheva J, Gurevitch J, Mengersen K. Recovering missing or partial data from studies: a survey of conversions and imputations for meta-analysis. In: Handbook of meta-analysis in ecology and evolution; 2013. p. 195–206.CrossRef
37.
go back to reference Dosenovic S, Jelicic Kadic A, Vucic K, Markovina N, Pieper D, Puljak L. Comparison of methodological quality rating of systematic reviews on neuropathic pain using AMSTAR and R-AMSTAR. BMC Med Res Methodol. 2018;18:37. Dosenovic S, Jelicic Kadic A, Vucic K, Markovina N, Pieper D, Puljak L. Comparison of methodological quality rating of systematic reviews on neuropathic pain using AMSTAR and R-AMSTAR. BMC Med Res Methodol. 2018;18:37.
38.
go back to reference Noyes J, Lewin S. Chapter 5: Extracting qualitative evidence. In: Noyes J, Booth A, Hannes K, et al., eds. Supplementary Guidance for Inclusion of Qualitative Research in Cochrane Systematic Reviews of Interventions. Version 1 ed; 2011. Noyes J, Lewin S. Chapter 5: Extracting qualitative evidence. In: Noyes J, Booth A, Hannes K, et al., eds. Supplementary Guidance for Inclusion of Qualitative Research in Cochrane Systematic Reviews of Interventions. Version 1 ed; 2011.
39.
go back to reference Jones AP, Remmington T, Williamson PR, Ashby D, Smyth RL. High prevalence but low impact of data extraction and reporting errors were found in Cochrane systematic reviews. J Clin Epidemiol. 2005;58(7):741–2.CrossRef Jones AP, Remmington T, Williamson PR, Ashby D, Smyth RL. High prevalence but low impact of data extraction and reporting errors were found in Cochrane systematic reviews. J Clin Epidemiol. 2005;58(7):741–2.CrossRef
40.
go back to reference Zein H, Tran VL-H, Azmy A, et al. How to Extract Data from Graphs using Plot Digitizer or Getdata Graph Digitizer. 2015. Zein H, Tran VL-H, Azmy A, et al. How to Extract Data from Graphs using Plot Digitizer or Getdata Graph Digitizer. 2015.
41.
go back to reference Jelicic Kadic A, Vucic K, Dosenovic S, Sapunar D, Puljak L. Extracting data from figures with software was faster, with higher interrater reliability than manual extraction. J Clin Epidemiol. 2016;74:119–23.CrossRef Jelicic Kadic A, Vucic K, Dosenovic S, Sapunar D, Puljak L. Extracting data from figures with software was faster, with higher interrater reliability than manual extraction. J Clin Epidemiol. 2016;74:119–23.CrossRef
42.
go back to reference J. LM, R. FM. Variable reporting and quantitative reviews: a comparison of three meta-analytical techniques. Ecol Lett. 2003;6(5):448–54. J. LM, R. FM. Variable reporting and quantitative reviews: a comparison of three meta-analytical techniques. Ecol Lett. 2003;6(5):448–54.
43.
go back to reference Del Re AC. compute. es: Compute effect sizes. R package version 2013: 0.2-. Del Re AC. compute. es: Compute effect sizes. R package version 2013: 0.2-.
44.
go back to reference Lajeunesse MJ, Koricheva J, Gurevitch J, Mengersen K. Recovering missing or partial data from studies: a survey of conversions and imputations for meta-analysis. In: Handbook of Meta-analysis in Ecology and Evolution; 2013. p. 195–206.CrossRef Lajeunesse MJ, Koricheva J, Gurevitch J, Mengersen K. Recovering missing or partial data from studies: a survey of conversions and imputations for meta-analysis. In: Handbook of Meta-analysis in Ecology and Evolution; 2013. p. 195–206.CrossRef
45.
go back to reference Rubin DB, Schenker N: Multiple imputation in health-care databases: an overview and some applications. Stat Med. 1991;10(4):585–98.CrossRef Rubin DB, Schenker N: Multiple imputation in health-care databases: an overview and some applications. Stat Med. 1991;10(4):585–98.CrossRef
46.
go back to reference Bracken MB. Statistical methods for analysis of effects of treatment in overviews of randomized trials. Effective care of the newborn infant 1992: 13–20. Bracken MB. Statistical methods for analysis of effects of treatment in overviews of randomized trials. Effective care of the newborn infant 1992: 13–20.
47.
go back to reference Schroll JB, Bero L, Gøtzsche PC. Searching for unpublished data for Cochrane reviews: cross sectional study. BMJ. 2013, 346. Schroll JB, Bero L, Gøtzsche PC. Searching for unpublished data for Cochrane reviews: cross sectional study. BMJ. 2013, 346.
Metadata
Title
Methodological steps used by authors of systematic reviews and meta-analyses of clinical trials: a cross-sectional study
Authors
Hoang Thi Nam Giang
Ali Mahmoud Ahmed
Reem Yousry Fala
Mohamed Magdy Khattab
Mona Hassan Ahmed Othman
Sara Attia Mahmoud Abdelrahman
Le Phuong Thao
Ahmed Elsaid Abd Elsamie Gabl
Samar Ahmed Elrashedy
Peter N. Lee
Kenji Hirayama
Hosni Salem
Nguyen Tien Huy
Publication date
01-12-2019
Publisher
BioMed Central
Published in
BMC Medical Research Methodology / Issue 1/2019
Electronic ISSN: 1471-2288
DOI
https://doi.org/10.1186/s12874-019-0780-2

Other articles of this Issue 1/2019

BMC Medical Research Methodology 1/2019 Go to the issue