Skip to main content
Key Specialties
Cardiology Diabetology Endocrinology Gastroenterology Geriatrics and Gerontology Gynecology and Obstetrics Hematology Infectious Disease Internal Medicine Nephrology Neurology Oncology Pediatrics Respiratory Medicine Rheumatology Surgery
Congress Coverage
2024 ACC Congress 2023 ESMO Congress 2023 EASD Congress 2023 ERS Congress 2023 ESC Congress
Clinical Collections
Advances in Alzheimer’s

Keynote webinar | Spotlight on medication adherence

LIVE: Thursday 27th June 2024, 18:00-19:30 (CEST)
Join our expert panel to discover why you need to understand the drivers of non-adherence in your patients, and how you can optimize medication adherence in your clinics to drastically improve patient outcomes.
ADVANCED SEARCH
Log in
Top
BMC Medical Research Methodology
Published in: BMC Medical Research Methodology 1/2016

Open Access 01-12-2016 | Research article

Meta-analyses including non-randomized studies of therapeutic interventions: a methodological review

Authors: Timor Faber, Philippe Ravaud, Carolina Riveros, Elodie Perrodeau, Agnes Dechartres

Published in: BMC Medical Research Methodology | Issue 1/2016

Login to get access

Abstract

Background

There is an increasing number of meta-analyses including data from non-randomized studies for therapeutic evaluation. We aimed to systematically assess the methods used in meta-analyses including non-randomized studies evaluating therapeutic interventions.

Methods

For this methodological review, we searched MEDLINE via PubMed, from January 1, 2013 to December 31, 2013 for meta-analyses including at least one non-randomized study evaluating therapeutic interventions. Etiological assessments and meta-analyses with no comparison group were excluded. Two reviewers independently assessed the general characteristics and key methodological components of the systematic review process and meta-analysis methods.

Results

One hundred eighty eight meta-analyses were selected: 119 included both randomized controlled trials (RCTs) and non-randomized studies of interventions (NRSI) and 69 only NRSI. Half of the meta-analyses (n = 92, 49 %) evaluated non-pharmacological interventions. “Grey literature” was searched for 72 meta-analyses (38 %). An assessment of methodological quality or risk of bias was reported in 135 meta-analyses (72 %) but this assessment considered the risk of confounding bias in only 33 meta-analyses (18 %). In 130 meta-analyses (69 %), the design of each NRSI was not clearly specified. In 131 (70 %), whether crude or adjusted estimates of treatment effect for NRSI were combined was unclear or not reported. Heterogeneity across studies was assessed in 182 meta-analyses (97 %) and further explored in 157 (84 %). Reporting bias was assessed in 127 (68 %).

Conclusions

Some key methodological components of the systematic review process—search for grey literature, description of the type of NRSI included, assessment of risk of confounding bias and reporting of whether crude or adjusted estimates were combined—are not adequately carried out or reported in meta-analyses including NRSI.
Appendix
Available only for authorised users
Literature
1.
Castillo RC, Scharfstein DO, MacKenzie EJ. Observational studies in the era of randomized trials: finding the balance. J Bone Joint Surg Am. 2012;94 Suppl 1:112–7.CrossRefPubMed
2.
Chou R, Helfand M. Challenges in systematic reviews that assess treatment harms. Ann Intern Med. 2005;142(12 Pt 2):1090–9.CrossRefPubMed
3.
Grootendorst DC, Jager KJ, Zoccali C, Dekker FW. Observational studies are complementary to randomized controlled trials. Nephron. 2010;114(3):c173–177.PubMed
4.
Yang W, Zilov A, Soewondo P, Bech OM, Sekkal F, Home PD. Observational studies: going beyond the boundaries of randomized controlled trials. Diabetes Res Clin Pract. 2010;88 Suppl 1:S3–9.CrossRefPubMed
5.
6.
7.
Benson K, Hartz AJ. A comparison of observational studies and randomized, controlled trials. N Engl J Med. 2000;342(25):1878–86.CrossRefPubMed
8.
Goulart BH, Ramsey SD, Parvathaneni U. Observational study designs for comparative effectiveness research: an alternative approach to close evidence gaps in head-and-neck cancer. Int J Radiat Oncol Biol Phys. 2014;88(1):106–14.CrossRefPubMed
9.
Silverman SL. From randomized controlled trials to observational studies. Am J Med. 2009;122(2):114–20.CrossRefPubMed
10.
Hannan EL. Randomized clinical trials and observational studies: guidelines for assessing respective strengths and limitations. JACC Cardiovasc Interv. 2008;1(3):211–7.CrossRefPubMed
11.
Alemayehu D, Cappelleri JC. Revisiting issues, drawbacks and opportunities with observational studies in comparative effectiveness research. J Eval Clin Pract. 2013;19(4):579–83.CrossRefPubMed
12.
Garabedian LF, Chu P, Toh S, Zaslavsky AM, Soumerai SB. Potential bias of instrumental variable analyses for observational comparative effectiveness research. Ann Intern Med. 2014;161(2):131–8.CrossRefPubMed
13.
Marko NF, Weil RJ. The role of observational investigations in comparative effectiveness research. Value Health. 2010;13(8):989–97.CrossRefPubMed
14.
Roche N, Reddel H, Martin R, Brusselle G, Papi A, Thomas M, et al. Quality standards for real-world research. Focus on observational database studies of comparative effectiveness. Ann Am Thorac Soc. 2014;11 Suppl 2:S99–104.CrossRefPubMed
15.
Shrier I, Boivin JF, Steele RJ, Platt RW, Furlan A, Kakuma R, et al. Should meta-analyses of interventions include observational studies in addition to randomized controlled trials? A critical examination of underlying principles. Am J Epidemiol. 2007;166(10):1203–9.CrossRefPubMed
16.
Norris SL, Atkins D, Bruening W, Fox S, Johnson E, Kane R, et al. Observational studies in systematic [corrected] reviews of comparative effectiveness: AHRQ and the Effective Health Care Program. J Clin Epidemiol. 2011;64(11):1178–86.CrossRefPubMed
17.
Holve E, Pittman P. A first look at the volume and cost of comparative effectiveness research in the United States [monograph]. Washington, DC: Academy Health; 2009.
18.
Abraham NS, Byrne CJ, Young JM, Solomon MJ. Meta-analysis of well-designed nonrandomized comparative studies of surgical procedures is as good as randomized controlled trials. Journal of Clinical Epidemiology. 2010;63(3):238–45.CrossRefPubMed
19.
Concato J, Shah N, Horwitz RI. Randomized, controlled trials, observational studies, and the hierarchy of research designs. N Engl J Med. 2000;342(25):1887–92.CrossRefPubMedPubMedCentral
20.
Golder S, Loke YK, Bland M. Meta-analyses of adverse effects data derived from randomised controlled trials as compared to observational studies: methodological overview. PLoS Med. 2011;8(5):e1001026.CrossRefPubMedPubMedCentral
21.
Ioannidis JP, Haidich AB, Pappa M, Pantazis N, Kokori SI, Tektonidou MG, et al. Comparison of evidence of treatment effects in randomized and nonrandomized studies. Jama. 2001;286(7):821–30.CrossRefPubMed
22.
Vandenbroucke JP. When are observational studies as credible as randomised trials? Lancet. 2004;363(9422):1728–31.CrossRefPubMed
23.
Zhang Z, Ni H, Xu X. Observational studies using propensity score analysis underestimated the effect sizes in critical care medicine. J Clin Epidemiol. 2014;67(8):932–9.CrossRefPubMed
24.
Higgins JPT, Green S, editors: Cochrane Handbook for Systematic Reviews of Interventions Version 5.1.0 [updated March 2011]. The Cochrane Collaboration 2011, Available from www.cochrane-handbook.org
25.
Brugha TS, Matthews R, Morgan Z, Hill T, Alonso J, Jones DR. Methodology and reporting of systematic reviews and meta-analyses of observational studies in psychiatric epidemiology: systematic review. Br J Psychiatry. 2012;200(6):446–53.CrossRefPubMed
26.
Golder S, Loke Y, McIntosh HM. Room for improvement? A survey of the methods used in systematic reviews of adverse effects BMC Med Res Methodol. 2006;6:3.PubMed
27.
28.
Zorzela L, Golder S, Liu Y, Pilkington K, Hartling L, Joffe A, et al. Quality of reporting in systematic reviews of adverse events: systematic review. Bmj. 2014;348:f7668.CrossRefPubMedPubMedCentral
29.
Stroup DF, Berlin JA, Morton SC, Olkin I, Williamson GD, Rennie D, et al. Meta-analysis of observational studies in epidemiology: a proposal for reporting. Meta-analysis Of Observational Studies in Epidemiology (MOOSE) group. Jama. 2000;283(15):2008–12.CrossRefPubMed
30.
Liberati A, Altman DG, Tetzlaff J, Mulrow C, Gotzsche PC, Ioannidis JPA, et al. The PRISMA statement for reporting systematic reviews and meta-analyses of studies that evaluate health care interventions: explanation and elaboration. Journal of Clinical Epidemiology. 2009;62(10):e1–34.CrossRefPubMed
31.
Moher D, Liberati A, Tetzlaff J, Altman DG, Group P. Preferred reporting items for systematic reviews and meta-analyses: the PRISMA Statement. Open Med. 2009;3(3):e123–130.PubMedPubMedCentral
32.
Shea BJ, Grimshaw JM, Wells GA, Boers M, Andersson N, Hamel C, et al. Development of AMSTAR: a measurement tool to assess the methodological quality of systematic reviews. BMC Med Res Methodol. 2007;7:10.CrossRefPubMedPubMedCentral
33.
Delgado-Rodriguez M, Ruiz-Canela M, De Irala-Estevez J, Llorca J, Martinez-Gonzalez A. Participation of epidemiologists and/or biostatisticians and methodological quality of published controlled clinical trials. J Epidemiol Community Health. 2001;55(8):569–72.CrossRefPubMedPubMedCentral
34.
Vale CL, Tierney JF, Burdett S. Can trial quality be reliably assessed from published reports of cancer trials: evaluation of risk of bias assessments in systematic reviews. Bmj. 2013;346:f1798.CrossRefPubMed
35.
Murad MH, Montori VM, Ioannidis JP, Jaeschke R, Devereaux PJ, Prasad K, et al. How to read a systematic review and meta-analysis and apply the results to patient care: users' guides to the medical literature. Jama. 2014;312(2):171–9.CrossRefPubMed
36.
Dal-Re R, Ioannidis JP, Bracken MB, Buffler PA, Chan AW, Franco EL, et al. Making prospective registration of observational research a reality. Sci Transl Med. 2014;6(224):224cm221.CrossRef
37.
Baudart M, Ravaud P, Baron G, Dechartres A, Haneef R, Boutron I. Public availability of results of observational studies evaluating an intervention registered at ClinicalTrials.gov. BMC Med. 2016;14(1):7.CrossRefPubMedPubMedCentral
38.
39.
Terrin N, Schmid CH, Lau J. In an empirical evaluation of the funnel plot, researchers could not visually identify publication bias. J Clin Epidemiol. 2005;58(9):894–901.CrossRefPubMed
40.
Hopewell S, Boutron I, Altman DG, Ravaud P. Incorporation of assessments of risk of bias of primary studies in systematic reviews of randomised trials: a cross-sectional study. BMJ Open. 2013;3(8):e003342.CrossRefPubMedPubMedCentral
41.
Higgins JP, Altman DG, Gotzsche PC, Juni P, Moher D, Oxman AD, et al. The Cochrane Collaboration’s tool for assessing risk of bias in randomised trials. Bmj. 2011;343:d5928.CrossRefPubMedPubMedCentral
42.
Jadad AR, Moore RA, Carroll D, Jenkinson C, Reynolds DJ, Gavaghan DJ, et al. Assessing the quality of reports of randomized clinical trials: is blinding necessary? Control Clin Trials. 1996;17(1):1–12.CrossRefPubMed
43.
Atkins D, Best D, Briss PA, Eccles M, Falck-Ytter Y, Flottorp S, et al. Grading quality of evidence and strength of recommendations. Bmj. 2004;328(7454):1490.CrossRefPubMed
44.
Metadata
Title
Meta-analyses including non-randomized studies of therapeutic interventions: a methodological review
Authors
Timor Faber
Philippe Ravaud
Carolina Riveros
Elodie Perrodeau
Agnes Dechartres
Publication date
01-12-2016
Publisher
BioMed Central
Published in
BMC Medical Research Methodology / Issue 1/2016
Electronic ISSN: 1471-2288
DOI
https://doi.org/10.1186/s12874-016-0136-0

Other articles of this Issue 1/2016

BMC Medical Research Methodology 1/2016 Go to the issue

Research article

Joint models for longitudinal and time-to-event data: a review of reporting quality with a view to meta-analysis

Research article

A scoping review of indirect comparison methods and applications using individual patient data

Research Article

Developing a system that can automatically detect health changes using transfer times of older adults

Research article

Bias and precision of methods for estimating the difference in restricted mean survival time from an individual patient data meta-analysis

Research article

Utilizing distributional analytics and electronic records to assess timeliness of inpatient blood glucose monitoring in non-critical care wards

Research article

Design and development of the Australian and New Zealand (ANZ) myeloma and related diseases registry