Skip to main content
Key Specialties
Cardiology Diabetology Endocrinology Gastroenterology Geriatrics and Gerontology Gynecology and Obstetrics Hematology Infectious Disease Internal Medicine Nephrology Neurology Oncology Pediatrics Respiratory Medicine Rheumatology Surgery
Congress Coverage
2024 ACC Congress 2023 ESMO Congress 2023 EASD Congress 2023 ERS Congress 2023 ESC Congress 2023 EHA Congress 2023 ASCO Annual Meeting
Clinical Collections
Advances in Alzheimer’s

At a glance: The ONWARDS insulin icodec trials

A round-up of the ONWARDS phase 3 clinical trials evaluating the novel weekly insulin icodec in people with diabetes.
ADVANCED SEARCH
Log in
Top
BMC Medicine
Published in: BMC Medicine 1/2020

Open Access 01-12-2020 | Research article

Statistical analyses and quality of individual participant data network meta-analyses were suboptimal: a cross-sectional study

Authors: Ya Gao, Shuzhen Shi, Muyang Li, Xinyue Luo, Ming Liu, Kelu Yang, Junhua Zhang, Fujian Song, Jinhui Tian

Published in: BMC Medicine | Issue 1/2020

Login to get access

Abstract

Background

Network meta-analyses using individual participant data (IPD-NMAs) have been increasingly used to compare the effects of multiple interventions. Although there have been many studies on statistical methods for IPD-NMAs, it is unclear whether there are statistical defects in published IPD-NMAs and whether the reporting of statistical analyses has improved. This study aimed to investigate statistical methods used and assess the reporting and methodological quality of IPD-NMAs.

Methods

We searched four bibliographic databases to identify published IPD-NMAs. The methodological quality was assessed using AMSTAR-2 and reporting quality assessed based on PRISMA-IPD and PRISMA-NMA. We performed stratified analyses and correlation analyses to explore the factors that might affect quality.

Results

We identified 21 IPD-NMAs. Only 23.8% of the included IPD-NMAs reported statistical techniques used for missing participant data, 42.9% assessed the consistency, and none assessed the transitivity. None of the included IPD-NMAs reported sources of funding for trials included, only 9.5% stated pre-registration of protocols, and 28.6% assessed the risk of bias in individual studies. For reporting quality, compliance rates were lower than 50.0% for more than half of the items. Less than 15.0% of the IPD-NMAs reported data integrity, presented the network geometry, or clarified risk of bias across studies. IPD-NMAs with statistical or epidemiological authors often better assessed the inconsistency (P = 0.017). IPD-NMAs with a priori protocol were associated with higher reporting quality in terms of search (P = 0.046), data collection process (P = 0.031), and syntheses of results (P = 0.006).

Conclusions

The reporting of statistical methods and compliance rates of methodological and reporting items of IPD-NMAs were suboptimal. Authors of future IPD-NMAs should address the identified flaws and strictly adhere to methodological and reporting guidelines.
Appendix
Available only for authorised users
Literature
1.
Stewart LA, Clarke M, Rovers M, Riley RD, Simmonds M, Stewart G, Tierney JF. Preferred reporting items for systematic review and meta-analyses of individual participant data: the PRISMA-IPD statement. JAMA. 2015;313(16):1657–65.PubMedCrossRef
2.
Simmonds M, Stewart G, Stewart L. A decade of individual participant data meta-analyses: a review of current practice. Contemp Clin Trials. 2015;45(Pt A):76–83.PubMedCrossRef
3.
Veroniki AA, Straus SE, Soobiah C, Elliott MJ, Tricco AC. A scoping review of indirect comparison methods and applications using individual patient data. BMC Med Res Methodol. 2016;16:47.PubMedCrossRef
4.
Schuit E, Li AH, Ioannidis JPA. How often can meta-analyses of individual-level data individualize treatment? A meta-epidemiologic study. Int J Epidemiol. 2019;48(2):596–608.PubMedCrossRef
5.
Stewart LA, Tierney JF. To IPD or not to IPD? Advantages and disadvantages of systematic reviews using individual patient data. Eval Health Prof. 2002;25(1):76–97.CrossRef
6.
Tsujimoto Y, Fujii T, Onishi A, Omae K, Luo Y, Imai H, Takahashi S, Itaya T, Pinson C, Nevitt SJ, et al. No consistent evidence of data availability bias existed in recent individual participant data meta-analyses: a meta-epidemiological study. J Clin Epidemiol. 2020;118:107–14.e5.PubMedCrossRef
7.
Debray TPA, Schuit E, Efthimiou O, Reitsma JB, Ioannidis JPA, Salanti G, Moons KGM. An overview of methods for network meta-analysis using individual participant data: when do benefits arise? Stat Methods Med Res. 2018;27(5):1351–64.PubMedCrossRef
8.
Tierney JF, Vale C, Riley R, Smith CT, Stewart L, Clarke M, Rovers M. Individual participant data (IPD) meta-analyses of randomised controlled trials: guidance on their use. PLoS Med. 2015;12(7):e1001855.PubMedPubMedCentralCrossRef
9.
Li T, Puhan MA, Vedula SS, Singh S, Dickersin K. Network meta-analysis-highly attractive but more methodological research is needed. BMC Med. 2011;9:79.PubMedCrossRef
10.
Mills EJ, Ioannidis JP, Thorlund K, Schunemann HJ, Puhan MA, Guyatt GH. How to use an article reporting a multiple treatment comparison meta-analysis. JAMA. 2012;308(12):1246–53.PubMedCrossRef
11.
Caldwell DM, Ades AE, Higgins JP. Simultaneous comparison of multiple treatments: combining direct and indirect evidence. BMJ (Clinical research ed). 2005;331(7521):897–900.CrossRef
12.
Li L, Tian J, Tian H, Moher D, Liang F, Jiang T, Yao L, Yang K. Network meta-analyses could be improved by searching more sources and by involving a librarian. J Clin Epidemiol. 2014;67(9):1001–7.PubMedCrossRef
13.
Thom HH, Capkun G, Cerulli A, Nixon RM, Howard LS. Network meta-analysis combining individual patient and aggregate data from a mixture of study designs with an application to pulmonary arterial hypertension. BMC Med Res Methodol. 2015;15:34.PubMedCrossRef
14.
Donegan S, Welton NJ, Tudur Smith C, D'Alessandro U, Dias S. Network meta-analysis including treatment by covariate interactions: consistency can vary across covariate values. Res Synth Methods. 2017;8(4):485–95.PubMedCrossRef
15.
Donegan S, Williamson P, D'Alessandro U, Garner P, Smith CT. Combining individual patient data and aggregate data in mixed treatment comparison meta-analysis: individual patient data may be beneficial if only for a subset of trials. Stat Med. 2013;32(6):914–30.PubMedCrossRef
16.
Saramago P, Sutton AJ, Cooper NJ, Manca A. Mixed treatment comparisons using aggregate and individual participant level data. Stat Med. 2012;31(28):3516–36.PubMedCrossRef
17.
Berlin JA, Santanna J, Schmid CH, Szczech LA, Feldman HI. Individual patient- versus group-level data meta-regressions for the investigation of treatment effect modifiers: ecological bias rears its ugly head. Stat Med. 2002;21(3):371–87.PubMedCrossRef
18.
Riley RD, Lambert PC, Abo-Zaid G. Meta-analysis of individual participant data: rationale, conduct, and reporting. BMJ (Clinical research ed). 2010;340:c221.CrossRef
19.
Veroniki AA, Straus SE, Ashoor HM, Hamid JS, Hemmelgarn BR, Holroyd-Leduc J, Majumdar SR, McAuley G, Tricco AC. Comparative safety and effectiveness of cognitive enhancers for Alzheimer’s dementia: protocol for a systematic review and individual patient data network meta-analysis. BMJ Open. 2016;6(1):e010251.PubMedCrossRef
20.
Simmonds MC, Higgins JPT, Stewart LA, Tierney JF, Clarke MJ, Thompson SG. Meta-analysis of individual patient data from randomized trials: a review of methods used in practice. Clin Trials. 2005;2(3):209–17.PubMedCrossRef
21.
Zarin W, Veroniki AA, Nincic V, Vafaei A, Reynen E, Motiwala SS, Antony J, Sullivan SM, Rios P, Daly C, et al. Characteristics and knowledge synthesis approach for 456 network meta-analyses: a scoping review. BMC Med. 2017;15(1):3.PubMedPubMedCentralCrossRef
22.
Tonin FS, Borba HH, Leonart LP, Mendes AM, Steimbach LM, Pontarolo R, Fernandez-Llimos F. Methodological quality assessment of network meta-analysis of drug interventions: implications from a systematic review. Int J Epidemiol. 2019;48(2):620–32.PubMedCrossRef
23.
Petropoulou M, Nikolakopoulou A, Veroniki AA, Rios P, Vafaei A, Zarin W, Giannatsi M, Sullivan S, Tricco AC, Chaimani A, et al. Bibliographic study showed improving statistical methodology of network meta-analyses published between 1999 and 2015. J Clin Epidemiol. 2017;82:20–8.PubMedCrossRef
24.
Gao Y, Ge L, Ma X, Shen X, Liu M, Tian J. Improvement needed in the network geometry and inconsistency of Cochrane network meta-analyses: a cross-sectional survey. J Clin Epidemiol. 2019;113:214–27.PubMedCrossRef
25.
Chaimani A, Vasiliadis HS, Pandis N, Schmid CH, Welton NJ, Salanti G. Effects of study precision and risk of bias in networks of interventions: a network meta-epidemiological study. Int J Epidemiol. 2013;42(4):1120–31.PubMedCrossRef
26.
Jansen JP. Network meta-analysis of individual and aggregate level data. Res Synth Methods. 2012;3(2):177–90.PubMedCrossRef
27.
Veroniki AA, Ashoor HM, Le SPC, Rios P, Stewart LA, Clarke M, Mavridis D, Straus SE, Tricco AC. Retrieval of individual patient data depended on study characteristics: a randomized controlled trial. J Clin Epidemiol. 2019;113:176–88.PubMedCrossRef
28.
Freeman SC, Fisher D, Tierney JF, Carpenter JR. A framework for identifying treatment-covariate interactions in individual participant data network meta-analysis. Res Synth Methods. 2018;9(3):393–407.PubMed
29.
Riley RD, Ensor J, Jackson D, Burke DL. Deriving percentage study weights in multi-parameter meta-analysis models: with application to meta-regression, network meta-analysis and one-stage individual participant data models. Stat Methods Med Res. 2018;27(10):2885–905.PubMedCrossRef
30.
Signorovitch JE, Wu EQ, Yu AP, Gerrits CM, Kantor E, Bao Y, Gupta SR, Mulani PM. Comparative effectiveness without head-to-head trials: a method for matching-adjusted indirect comparisons applied to psoriasis treatment with adalimumab or etanercept. PharmacoEconomics. 2010;28(10):935–45.PubMedCrossRef
31.
Bucher HC, Guyatt GH, Griffith LE, Walter SD. The results of direct and indirect treatment comparisons in meta-analysis of randomized controlled trials. J Clin Epidemiol. 1997;50(6):683–91.PubMedCrossRef
32.
Caro JJ, Ishak KJ. No head-to-head trial? Simulate the missing arms. PharmacoEconomics. 2010;28(10):957–67.PubMedCrossRef
33.
Hutton B, Salanti G, Caldwell DM, Chaimani A, Schmid CH, Cameron C, Ioannidis JP, Straus S, Thorlund K, Jansen JP, et al. The PRISMA extension statement for reporting of systematic reviews incorporating network meta-analyses of health care interventions: checklist and explanations. Ann Intern Med. 2015;162(11):777–84.PubMedCrossRef
34.
Ge L, Tian JH, Li YN, Pan JX, Li G, Wei D, Xing X, Pan B, Chen YL, Song FJ, et al. Association between prospective registration and overall reporting and methodological quality of systematic reviews: a meta-epidemiological study. J Clin Epidemiol. 2018;93:45–55.PubMedCrossRef
35.
Shea BJ, Reeves BC, Wells G, Thuku M, Hamel C, Moran J, Moher D, Tugwell P, Welch V, Kristjansson E, et al. AMSTAR 2: a critical appraisal tool for systematic reviews that include randomised or non-randomised studies of healthcare interventions, or both. BMJ (Clinical research ed). 2017;358:j4008.CrossRef
36.
Villain B, Dechartres A, Boyer P, Ravaud P. Feasibility of individual patient data meta-analyses in orthopaedic surgery. BMC Med. 2015;13:131.PubMedCrossRef
37.
Jaspers GJ, Degraeuwe PL. A failed attempt to conduct an individual patient data meta-analysis. Syst Rev. 2014;3:97.
38.
Jiang Q, Liu Q, Chen F, Zeng X, Song F, Lu Z, Cao S. Reporting quality and statistical analysis of published dose-response meta-analyses was suboptimal: a cross-sectional literature survey. J Clin Epidemiol. 2019;115:133–40.PubMedCrossRef
39.
Song F, Loke YK, Walsh T, Glenny AM, Eastwood AJ, Altman DG. Methodological problems in the use of indirect comparisons for evaluating healthcare interventions: survey of published systematic reviews. BMJ (Clinical research ed). 2009;338:b1147.CrossRef
40.
Hingorani AD, Windt DA, Riley RD, Abrams K, Moons KG, Steyerberg EW, Schroter S, Sauerbrei W, Altman DG, Hemingway H. Prognosis research strategy (PROGRESS) 4: stratified medicine research. BMJ (Clinical research ed). 2013;346:e5793.
41.
Bafeta A, Trinquart L, Seror R, Ravaud P. Reporting of results from network meta-analyses: methodological systematic review. BMJ (Clinical research ed). 2014;348:g1741.
42.
Habtewold TD, Alemu SM, Mohammed SH, Endalamaw A, Mohammed MA, Teferra AA, Tura AK, Asefa NG, Tegegne BS. Biomedical and public health reviews and meta-analyses in Ethiopia had poor methodological quality: overview of evidence from 1970 to 2018. J Clin Epidemiol. 2019;109:90–8.PubMedCrossRef
43.
Yan P, Yao L, Li H, Zhang M, Xun Y, Li M, Cai H, Lu C, Hu L, Guo T, et al. The methodological quality of robotic surgical meta-analyses needed to be improved: a cross-sectional study. J Clin Epidemiol. 2019;109:20–9.PubMedCrossRef
44.
Riado Minguez D, Kowalski M, Vallve Odena M, Longin Pontzen D, Jelicic Kadic A, Jeric M, Dosenovic S, Jakus D, Vrdoljak M, Poklepovic Pericic T, et al. Methodological and reporting quality of systematic reviews published in the highest ranking journals in the field of pain. Anesth Analg. 2017;125(4):1348–54.PubMedCrossRef
45.
Silagy CA, Middleton P, Hopewell S. Publishing protocols of systematic reviews: comparing what was done to what was planned. JAMA. 2002;287(21):2831–4.PubMedCrossRef
46.
Koensgen N, Rombey T, Allers K, Mathes T, Hoffmann F, Pieper D. Comparison of non-Cochrane systematic reviews and their published protocols: differences occurred frequently but were seldom explained. J Clin Epidemiol. 2019;110:34–41.PubMedCrossRef
47.
Page MJ, McKenzie JE, Forbes A. Many scenarios exist for selective inclusion and reporting of results in randomized trials and systematic reviews. J Clin Epidemiol. 2013;66(5):524–37.PubMedCrossRef
48.
Allers K, Hoffmann F, Mathes T, Pieper D. Systematic reviews with published protocols compared to those without: more effort, older search. J Clin Epidemiol. 2018;95:102–10.PubMedCrossRef
Metadata
Title
Statistical analyses and quality of individual participant data network meta-analyses were suboptimal: a cross-sectional study
Authors
Ya Gao
Shuzhen Shi
Muyang Li
Xinyue Luo
Ming Liu
Kelu Yang
Junhua Zhang
Fujian Song
Jinhui Tian
Publication date
01-12-2020
Publisher
BioMed Central
Published in
BMC Medicine / Issue 1/2020
Electronic ISSN: 1741-7015
DOI
https://doi.org/10.1186/s12916-020-01591-0

Other articles of this Issue 1/2020

BMC Medicine 1/2020 Go to the issue

Research article

Cerebrospinal fluid and plasma biomarkers in individuals at risk for genetic prion disease

Research article

Vegetarian and vegan diets and risks of total and site-specific fractures: results from the prospective EPIC-Oxford study

Commentary

Raising awareness of antimicrobial resistance: comment on ‘Reducing expectations for antibiotics in primary care: a randomised experiment to test the response to fear based messages about antimicrobial resistance’

Research article

Impact of intermittent preventive treatment of malaria in pregnancy with dihydroartemisinin-piperaquine versus sulfadoxine-pyrimethamine on the incidence of malaria in infancy: a randomized controlled trial

Research article

Hyperthyroidism is associated with breast cancer risk and mammographic and genetic risk predictors

Research article

The duration of chemoprophylaxis against malaria after treatment with artesunate-amodiaquine and artemether-lumefantrine and the effects of pfmdr1 86Y and pfcrt 76T: a meta-analysis of individual patient data