Skip to main content
Key Specialties
Cardiology Diabetology Endocrinology Gastroenterology Geriatrics and Gerontology Gynecology and Obstetrics Hematology Infectious Disease Internal Medicine Nephrology Neurology Oncology Pediatrics Respiratory Medicine Rheumatology Surgery
Congress Coverage
2024 ACC Congress 2023 ESMO Congress 2023 EASD Congress 2023 ERS Congress 2023 ESC Congress
Clinical Collections
Advances in Alzheimer’s

At a glance: The STEP trials

A round-up of the STEP phase 3 clinical trials evaluating semaglutide for weight loss in people with overweight or obesity.
ADVANCED SEARCH
Log in
Top
BMC Medicine
Published in: BMC Medicine 1/2016

Open Access 01-12-2016 | Research article

Wasted research when systematic reviews fail to provide a complete and up-to-date evidence synthesis: the example of lung cancer

Authors: Perrine Créquit, Ludovic Trinquart, Amélie Yavchitz, Philippe Ravaud

Published in: BMC Medicine | Issue 1/2016

Login to get access

Abstract

Background

Multiple treatments are frequently available for a given condition, and clinicians and patients need a comprehensive, up-to-date synthesis of evidence for all competing treatments. We aimed to quantify the waste of research related to the failure of systematic reviews to provide a complete and up-to-date evidence synthesis over time.

Methods

We performed a series of systematic overviews and networks of randomized trials assessing the gap between evidence covered by systematic reviews and available trials of second-line treatments for advanced non-small cell lung cancer. We searched the Cochrane Database of Systematic Reviews, Database of Abstracts of Reviews of Effects, MEDLINE, EMBASE, and other resources sequentially by year from 2009 to March 2, 2015. We sequentially compared the amount of evidence missing from systematic reviews to the randomized evidence available for inclusion each year. We constructed cumulative networks of randomized evidence over time and evaluated the proportion of trials, patients, treatments, and treatment comparisons not covered by systematic reviews on December 31 each year from 2009 to 2015.

Results

We identified 77 trials (28,636 patients) assessing 47 treatments with 54 comparisons and 29 systematic reviews (13 published after 2013). From 2009 to 2015, the evidence covered by existing systematic reviews was consistently incomplete: 45 % to 70 % of trials; 30 % to 58 % of patients; 40 % to 66 % of treatments; and 38 % to 71 % of comparisons were missing. In the cumulative networks of randomized evidence, 10 % to 17 % of treatment comparisons were partially covered by systematic reviews and 55 % to 85 % were partially or not covered.

Conclusions

We illustrate how systematic reviews of a given condition provide a fragmented, out-of-date panorama of the evidence for all treatments. This waste of research might be reduced by the development of live cumulative network meta-analyses.
Appendix
Available only for authorised users
Literature
1.
Institute of Medicine. Initial national priorities for comparative effectiveness research. Washington, DC: The National Academies Press; 2009.
2.
3.
Jefferson T, Doshi P, Thompson M, Heneghan C. Ensuring safe and effective drugs: who can do what it takes? BMJ. 2011;342:c7258.CrossRefPubMed
4.
Liberati A. Need to realign patient-oriented and commercial and academic research. Lancet. 2011;378(9805):1777–8.CrossRefPubMed
5.
Ioannidis JP. Integration of evidence from multiple meta-analyses: a primer on umbrella reviews, treatment networks and multiple treatments meta-analyses. CMAJ. 2009;181(8):488–93.CrossRefPubMedPubMedCentral
6.
Naci H, Ioannidis JP. How good is “evidence” from clinical studies of drug effects and why might such evidence fail in the prediction of the clinical utility of drugs? Annu Rev Pharmacol Toxicol. 2015;55:169–89.CrossRefPubMed
7.
Haidich AB, Pilalas D, Contopoulos-Ioannidis DG, Ioannidis JP. Most meta-analyses of drug interventions have narrow scopes and many focus on specific agents. J Clin Epidemiol. 2013;66(4):371–8.CrossRefPubMed
8.
Moher D, Tsertsvadze A, Tricco AC, Eccles M, Grimshaw J, Sampson M, et al. When and how to update systematic reviews. Cochrane Database Syst Rev. 2008;1:MR000023.PubMed
9.
Shojania KG, Sampson M, Ansari MT, Ji J, Doucette S, Moher D. How quickly do systematic reviews go out of date? A survival analysis. Ann Intern Med. 2007;147(4):224–33.CrossRefPubMed
10.
Ioannidis JP, Karassa FB. The need to consider the wider agenda in systematic reviews and meta-analyses: breadth, timing, and depth of the evidence. BMJ. 2010;341:c4875.CrossRefPubMed
11.
Global Burden of Disease Cancer Collaboration, Fitzmaurice C, Dicker D, Pain A, Hamavid H, Moradi-Lakeh M, et al. The global burden of cancer 2013. JAMA Oncol. 2015;1(4):505–27.CrossRefPubMedCentral
12.
Chan AW. Out of sight but not out of mind: how to search for unpublished clinical trial evidence. BMJ. 2012;344:d8013.CrossRefPubMed
13.
Shea BJ, Grimshaw JM, Wells GA, Boers M, Andersson N, Hamel C, et al. Development of AMSTAR: a measurement tool to assess the methodological quality of systematic reviews. BMC Med Res Methodol. 2007;7:10.CrossRefPubMedPubMedCentral
14.
Shea BJ, Bouter LM, Peterson J, Boers M, Andersson N, Ortiz Z, et al. External validation of a measurement tool to assess systematic reviews (AMSTAR). PLoS One. 2007;2(12):e1350.CrossRefPubMedPubMedCentral
15.
Shea BJ, Hamel C, Wells GA, Bouter LM, Kristjansson E, Grimshaw J, et al. AMSTAR is a reliable and valid measurement tool to assess the methodological quality of systematic reviews. J Clin Epidemiol. 2009;62(10):1013–20.CrossRefPubMed
16.
Chandler J, Churchill R, Higgins J, Lasserson T, Tovey D. Methodological standards for the conduct of new Cochrane Intervention Reviews, Methodological expectations of Cochrane intervention reviews (MECIR). London: Cochrane Editorial Unit; 2013.
17.
Al-Saleh K, Quinton C, Ellis PM. Role of pemetrexed in advanced non-small-cell lung cancer: meta-analysis of randomized controlled trials, with histology subgroup analysis. Curr Oncol. 2012;19(1):e9–e15.PubMedPubMedCentral
18.
Barlesi F, Jacot W, Astoul P, Pujol JL. Second-line treatment for advanced non-small cell lung cancer: a systematic review. Lung Cancer. 2006;51(2):159–72.CrossRefPubMed
19.
Bonfill X, Serra C, Sacristan M, Nogue M, Losa F, Montesinos J. Second-line chemotherapy for non-small cell lung cancer. Cochrane Database Syst Rev. 2002;2:CD002804.PubMed
20.
Cui J, Cai X, Zhu M, Liu T, Zhao N. The efficacy of bevacizumab compared with other targeted drugs for patients with advanced NSCLC: a meta-analysis from 30 randomized controlled clinical trials. PLoS One. 2013;8(4):e62038.CrossRefPubMedPubMedCentral
21.
Gao G, Ren S, Li A, He Y, Chen X, Li W, et al. A meta-analysis of comparing EGFR-TKI with chemotherapy as the second-line treatment of NSCLC patients with wild-type EGFR. J Clin Oncol. 2013;31(15):suppl 1.
22.
Hawkins N, Scott DA, Woods BS, Thatcher N. No study left behind: a network meta-analysis in non-small-cell lung cancer demonstrating the importance of considering all relevant data. Value Health. 2009;12(6):996–1003.CrossRefPubMed
23.
Jiang J, Huang L, Liang X, Zhou X, Huang R, Chu Z, et al. Gefitinib versus docetaxel in previously treated advanced non-small-cell lung cancer: a meta-analysis of randomized controlled trials. Acta Oncol. 2011;50(4):582–8.CrossRefPubMed
24.
Jin Y, Sun Y, Shi X, Zhao J, Shi L, Hong W, et al. Meta-analysis to assess the efficacy and toxicity of docetaxel-based doublet compared with docetaxel alone for patients with advanced NSCLC who failed first-line treatment. Clin Ther. 2014;36(12):1980–90.CrossRefPubMed
25.
Lee JK, Hahn S, Kim DW, Suh KJ, Keam B, Kim TM, et al. Epidermal growth factor receptor tyrosine kinase inhibitors vs conventional chemotherapy in non-small cell lung cancer harboring wild-type epidermal growth factor receptor: a meta-analysis. JAMA. 2014;311(14):1430–7.CrossRefPubMed
26.
Li N, Yang L, Ou W, Zhang L, Zhang SL, Wang SY. Meta-analysis of EGFR tyrosine kinase inhibitors compared with chemotherapy as second-line treatment in pretreated advanced non-small cell lung cancer. PLoS One. 2014;9(7):e102777.CrossRefPubMedPubMedCentral
27.
Li X, Wang H, Lin W, Xu Q. Efficacy of combining targeted therapy with pemetrexed or docetaxel as second-line treatment in patients with advanced non-small-cell lung cancer: a meta-analysis of 14 randomized controlled trials. Curr Med Res Opin. 2014;30(11):2295–304.CrossRefPubMed
28.
Liang W, Wu X, Hong S, Zhang Y, Kang S, Fang W, et al. Multi-targeted antiangiogenic tyrosine kinase inhibitors in advanced non-small cell lung cancer: meta-analyses of 20 randomized controlled trials and subgroup analyses. PLoS One. 2014;9(10):e109757.CrossRefPubMedPubMedCentral
29.
Perez-Moreno MA, Galvan-Banqueri M, Flores-Moreno S, Villalba-Moreno A, Cotrina-Luque J, Bautista-Paloma FJ. Systematic review of efficacy and safety of pemetrexed in non-small-cell-lung cancer. Int J Clin Pharm. 2014;36(3):476–87.CrossRefPubMed
30.
Popat S, Mellemgaard A, Fahrbach K, Martin A, Rizzo M, Kaiser R, et al. Nintedanib plus docetaxel as second-line therapy in patients with non-small-cell lung cancer: a network meta-analysis. Future Oncol. 2015;11(3):409–20.CrossRefPubMed
31.
Qi WX, Shen Z, Lin F, Sun YJ, Min DL, Tang LN, et al. Comparison of the efficacy and safety of EFGR tyrosine kinase inhibitor monotherapy with standard second-line chemotherapy in previously treated advanced non-small-cell lung cancer: a systematic review and meta-analysis. Asian Pac J Cancer Prev. 2012;13(10):5177–82.CrossRefPubMed
32.
Qi WX, Shen Z, Yao Y. Meta-analysis of docetaxel-based doublet versus docetaxel alone as second-line treatment for advanced non-small-cell lung cancer. Cancer Chemother Pharmacol. 2012;69(1):99–106.CrossRefPubMed
33.
Qi WX, Tang LN, He AN, Shen Z, Yao Y. The role of vandetanib in the second-line treatment for advanced non-small-cell-lung cancer: a meta-analysis of four randomized controlled trials. Lung. 2011;189(6):437–43.CrossRefPubMed
34.
Qi WX, Tang LN, He AN, Shen Z, Yao Y. Effectiveness and safety of pemetrexed-based doublet versus pemetrexed alone as second-line treatment for advanced non-small-cell lung cancer: a systematic review and meta-analysis. J Cancer Res Clin Oncol. 2012;138(5):745–51.CrossRefPubMed
35.
Qi WX, Wang Q, Jiang YL, Sun YJ, Tang LN, He AN, et al. Overall survival benefits for combining targeted therapy as second-line treatment for advanced non-small-cell-lung cancer: a meta-analysis of published data. PLoS One. 2013;8(2):e55637.CrossRefPubMedPubMedCentral
36.
Sun CT, Xu X, Sheng W, Wang XW, Wen SL, Han JQ. A meta-analysis of pemetrexed-based doublet compared with pemetrexed alone for the second-line treatment of advanced non-small-cell lung cancer. Bratisl Lek Listy. 2014;115(4):233–7.PubMed
37.
Sun L, Ma JT, Zhang SL, Zou HW, Han CB. Efficacy and safety of chemotherapy or tyrosine kinase inhibitors combined with bevacizumab versus chemotherapy or tyrosine kinase inhibitors alone in the treatment of non-small cell lung cancer: a systematic review and meta-analysis. Med Oncol. 2015;32(2):473.CrossRefPubMed
38.
Tao L, Zhuo W, Yang F, Zhu B. Vandetanib for advanced non-small cell lung cancer: a meta-analysis. Zhongguo Fei Ai Za Zhi. 2012;15(3):172–8.PubMed
39.
Tassinari D, Scarpi E, Sartori S, Drudi F, Castellani C, Carloni F, et al. Noninferiority trials in second-line treatments of nonsmall cell lung cancer: a systematic review of literature with meta-analysis of phase III randomized clinical trials. Am J Clin Oncol. 2012;35(6):593–9.CrossRefPubMed
40.
Tassinari D, Scarpi E, Sartori S, Tamburini E, Santelmo C, Tombesi P, et al. Second-line treatments in non-small cell lung cancer. A systematic review of literature and metaanalysis of randomized clinical trials. Chest. 2009;135(6):1596–609.CrossRefPubMed
41.
Vale CL, Burdett S, Fisher DJ, Navani N, Parmar MK, Copas AJ, et al. Should tyrosine kinase inhibitors be considered for advanced non-small-cell lung cancer patients with wild type EGFR? Two systematic reviews and meta-analyses of randomized trials. Clin Lung Cancer. 2015;16(3):173–82. e174.CrossRefPubMedPubMedCentral
42.
Wong KM, Victor C, Eng L, Verma S. The use of epidermal growth factor receptor tyrosine kinase inhibitors in treatment of advanced EGFR wild-type non-small cell lung cancer: a meta-analysis study. J Thorac Oncol. 2013;8(suppl 2):S1184.
43.
Xiao BK, Yang JY, Dong JX, Ji ZS, Si HY, Wang WL, et al. Meta-analysis of seven randomized control trials to assess the efficacy and toxicity of combining EGFR-TKI with chemotherapy for patients with advanced NSCLC who failed first-line treatment. Asian Pac J Cancer Prev. 2015;16(7):2915–21.CrossRefPubMed
44.
Yang X, Yang K, Kuang K. The efficacy and safety of EGFR inhibitor monotherapy in non-small cell lung cancer: a systematic review. Curr Oncol Rep. 2014;16(6):390.CrossRefPubMed
45.
Zhao N, Zhang XC, Yan HH, Yang JJ, Wu YL. Efficacy of epidermal growth factor receptor inhibitors versus chemotherapy as second-line treatment in advanced non-small-cell lung cancer with wild-type EGFR: a meta-analysis of randomized controlled clinical trials. Lung Cancer. 2014;85(1):66–73.CrossRefPubMed
46.
Chalmers I, Bracken MB, Djulbegovic B, Garattini S, Grant J, Gulmezoglu AM, et al. How to increase value and reduce waste when research priorities are set. Lancet. 2014;383(9912):156–65.CrossRefPubMed
47.
DeSantis SM, Zhu H. A Bayesian mixed-treatment comparison meta-analysis of treatments for alcohol dependence and implications for planning future trials. Med Decis Making. 2014;34(7):899–910.CrossRefPubMedPubMedCentral
48.
Nikolakopoulou A, Mavridis D, Salanti G. Using conditional power of network meta-analysis (NMA) to inform the design of future clinical trials. Biom J. 2014;56(6):973–90.CrossRefPubMed
49.
50.
51.
Salanti G, Kavvoura FK, Ioannidis JP. Exploring the geometry of treatment networks. Ann Intern Med. 2008;148(7):544–53.CrossRefPubMed
52.
Hutton B, Salanti G, Caldwell DM, Chaimani A, Schmid CH, Cameron C, et al. The PRISMA extension statement for reporting of systematic reviews incorporating network meta-analyses of health care interventions: checklist and explanations. Ann Intern Med. 2015;162(11):777–84.CrossRefPubMed
53.
Bafeta A, Trinquart L, Seror R, Ravaud P. Analysis of the systematic reviews process in reports of network meta-analyses: methodological systematic review. BMJ. 2013;347:f3675.CrossRefPubMedPubMedCentral
54.
Caldwell DM, Ades AE, Higgins JP. Simultaneous comparison of multiple treatments: combining direct and indirect evidence. BMJ. 2005;331(7521):897–900.CrossRefPubMedPubMedCentral
55.
56.
Mills EJ, Kanters S, Thorlund K, Chaimani A, Veroniki AA, Ioannidis JP. The effects of excluding treatments from network meta-analyses: survey. BMJ. 2013;347:f5195.CrossRefPubMedPubMedCentral
57.
Ioannidis JP, Chang CQ, Lam TK, Schully SD, Khoury MJ. The geometric increase in meta-analyses from China in the genomic era. PLoS One. 2013;8(6):e65602.CrossRefPubMedPubMedCentral
58.
59.
Helfer B, Prosser A, Samara MT, Geddes JR, Cipriani A, Davis JM, et al. Recent meta-analyses neglect previous systematic reviews and meta-analyses about the same topic: a systematic examination. BMC Med. 2015;13(1):82.CrossRefPubMedPubMedCentral
60.
61.
Elliott JH, Turner T, Clavisi O, Thomas J, Higgins JP, Mavergames C, et al. Living systematic reviews: an emerging opportunity to narrow the evidence-practice gap. PLoS Med. 2014;11(2):e1001603.CrossRefPubMedPubMedCentral
62.
63.
Marshall IJ, Kuiper J, Wallace BC. Automating risk of bias assessment for clinical trials. IEEE J Biomed Health Inform. 2015;19(4):1406–12.CrossRefPubMed
64.
Badgett RG, Vindhyal M, Stirnaman JT, Gibson CM, Halaby R. A living systematic review of nebulized hypertonic saline for acute bronchiolitis in infants. JAMA Pediatr. 2015;169(8):788–9.CrossRefPubMed
65.
Synnot A, Gruen RL, Menon D, Steyerberg EW, Buki A, Peul W, et al. A new approach to evidence synthesis in traumatic brain injury: living systematic reviews. J Neurotrauma. 2015. Epub ahead of print. doi:10.​1089/​neu.​2015.​4124.
66.
Brazinova A, Rehorcikova V, Taylor MS, Buckova V, Majdan M, Psota M, et al. Epidemiology of traumatic brain injury in Europe: a living systematic review. J Neurotrauma. 2015. Epub ahead of print. doi:10.​1089/​neu.​2015.​4126
67.
Cnossen MC, Scholten AC, Lingsma H, Synnot A, Tavender E, Gantner D, et al. Adherence to guidelines in adult patients with traumatic brain injury: A living systematic review. J Neurotrauma. 2015. Epub ahead of print. doi:10.​1089/​neu.​2015.​4121
68.
Ters P, Badgett RG. A living meta-analysis of colchicine for pericarditis. Ann Pharmacother. 2014;48(10):1398–9.CrossRefPubMed
69.
70.
Gartlehner G, Flamm M. Is the Cochrane Collaboration prepared for the era of patient-centred outcomes research? Cochrane Database Syst Rev. 2013;3:ED000054.PubMed
71.
Naci H, O’Connor AB. Assessing comparative effectiveness of new drugs before approval using prospective network meta-analyses. J Clin Epidemiol. 2013;66(8):812–6.CrossRefPubMed
72.
Puhan MA, Schünemann HJ, Murad MH, Li T, Brignardello-Petersen R, Singh JA, et al. A GRADE Working Group approach for rating the quality of treatment effect estimates from network meta-analysis. BMJ. 2014;349:g5630.CrossRefPubMed
73.
Li T, Rouse B, Shi Q, Dickersin K. Working from all angles. The Inaugural REWARD/EQUATOR Conference: increasing value and reducing waste in biomedical research conference, 28–30 September 2015, Edinburgh, UK.
74.
Haas DM, Caldwell DM, Kirkpatrick P, McIntosh JJ, Welton NJ. Tocolytic therapy for preterm delivery: systematic review and network meta-analysis. BMJ. 2012;345:e6226.CrossRefPubMedPubMedCentral
75.
Smalheiser NR, Lin C, Jia L, Jiang Y, Cohen AM, Yu C, et al. Design and implementation of Metta, a metasearch engine for biomedical literature retrieval intended for systematic reviewers. Health Inf Sci Syst. 2014;2:1.CrossRefPubMedPubMedCentral
76.
77.
Elliott J, Sim I, Thomas J, Owens N, Dooley G, Riis J, et al. #CochraneTech: technology and the future of systematic reviews. Cochrane Database Syst Rev. 2014;9:ED000091.PubMed
78.
79.
Wallace BC, Trikalinos TA, Lau J, Brodley C, Schmid CH. Semi-automated screening of biomedical citations for systematic reviews. BMC Bioinformatics. 2010;11:55.CrossRefPubMedPubMedCentral
80.
O’Mara-Eves A, Thomas J, McNaught J, Miwa M, Ananiadou S. Using text mining for study identification in systematic reviews: a systematic review of current approaches. Syst Rev. 2015;4:5.CrossRefPubMedPubMedCentral
81.
Miwa M, Thomas J, O'Mara-Eves A, Ananiadou S. Reducing systematic review workload through certainty-based screening. J Biomed Inform. 2014;51:242–53.CrossRefPubMedPubMedCentral
82.
Rathbone J, Hoffmann T, Glasziou P. Faster title and abstract screening? Evaluating Abstrackr, a semi-automated online screening program for systematic reviewers. Syst Rev. 2015;4:80.CrossRefPubMedPubMedCentral
83.
84.
Cepeda MS, Lobanov V, Berlin JA. From ClinicalTrials.gov trial registry to an analysis-ready database of clinical trial results. Clin Trials. 2013;10(2):347–8.CrossRefPubMed
85.
Tasneem A, Aberle L, Ananth H, Chakraborty S, Chiswell K, McCourt BJ, et al. The database for aggregate analysis of ClinicalTrials.gov (AACT) and subsequent regrouping by clinical specialty. PLoS One. 2012;7(3):e33677.CrossRefPubMedPubMedCentral
86.
Maruani A, Boutron I, Baron G, Ravaud P. Impact of sending email reminders of the legal requirement for posting results on ClinicalTrials.gov: cohort embedded pragmatic randomized controlled trial. BMJ. 2014;349:g5579.CrossRefPubMedPubMedCentral
87.
88.
89.
Nikolakopoulou A, Mavridis D, Salanti G. Prospective use of network metaanalysis using formal statistical monitoring. 36th Annual Conference of the International Society for Clinical Biostatistics, 23–27 August 2015, Utrecht, The Netherlands.
Metadata
Title
Wasted research when systematic reviews fail to provide a complete and up-to-date evidence synthesis: the example of lung cancer
Authors
Perrine Créquit
Ludovic Trinquart
Amélie Yavchitz
Philippe Ravaud
Publication date
01-12-2016
Publisher
BioMed Central
Published in
BMC Medicine / Issue 1/2016
Electronic ISSN: 1741-7015
DOI
https://doi.org/10.1186/s12916-016-0555-0

Other articles of this Issue 1/2016

BMC Medicine 1/2016 Go to the issue

Research article

Burden of respiratory tract infections at post mortem in Zambian children

Research article

A novel approach to identifying patterns of human invasion-inhibitory antibodies guides the design of malaria vaccines incorporating polymorphic antigens

Research article

Meta-analysis of the prognostic and clinical value of tumor-associated macrophages in adult classical Hodgkin lymphoma

Correspondence

TIME Impact – a new user-friendly tuberculosis (TB) model to inform TB policy decisions

Commentary

Living cumulative network meta-analysis to reduce waste in research: A paradigmatic shift for systematic reviews?

Research article

Ethnic variations in asthma hospital admission, readmission and death: a retrospective, national cohort study of 4.62 million people in Scotland