Top

Published in:

Open Access 01-12-2021 | Research article

Validity of observational evidence on putative risk and protective factors: appraisal of 3744 meta-analyses on 57 topics

Authors: Perrine Janiaud, Arnav Agarwal, Ioanna Tzoulaki, Evropi Theodoratou, Konstantinos K. Tsilidis, Evangelos Evangelou, John P. A. Ioannidis

Published in: BMC Medicine | Issue 1/2021

Abstract

Background

The validity of observational studies and their meta-analyses is contested. Here, we aimed to appraise thousands of meta-analyses of observational studies using a pre-specified set of quantitative criteria that assess the significance, amount, consistency, and bias of the evidence. We also aimed to compare results from meta-analyses of observational studies against meta-analyses of randomized controlled trials (RCTs) and Mendelian randomization (MR) studies.

Methods

We retrieved from PubMed (last update, November 19, 2020) umbrella reviews including meta-analyses of observational studies assessing putative risk or protective factors, regardless of the nature of the exposure and health outcome. We extracted information on 7 quantitative criteria that reflect the level of statistical support, the amount of data, the consistency across different studies, and hints pointing to potential bias. These criteria were level of statistical significance (pre-categorized according to 10⁻⁶, 0.001, and 0.05 p-value thresholds), sample size, statistical significance for the largest study, 95% prediction intervals, between-study heterogeneity, and the results of tests for small study effects and for excess significance.

Results

3744 associations (in 57 umbrella reviews) assessed by a median number of 7 (interquartile range 4 to 11) observational studies were eligible. Most associations were statistically significant at P < 0.05 (61.1%, 2289/3744). Only 2.6% of associations had P < 10⁻⁶, ≥1000 cases (or ≥20,000 participants for continuous factors), P < 0.05 in the largest study, 95% prediction interval excluding the null, and no large between-study heterogeneity, small study effects, or excess significance. Across the 57 topics, large heterogeneity was observed in the proportion of associations fulfilling various quantitative criteria. The quantitative criteria were mostly independent from one another. Across 62 associations assessed in both RCTs and in observational studies, 37.1% had effect estimates in opposite directions and 43.5% had effect estimates differing beyond chance in the two designs. Across 94 comparisons assessed in both MR and observational studies, such discrepancies occurred in 30.8% and 54.7%, respectively.

Conclusions

Acknowledging that no gold-standard exists to judge whether an observational association is genuine, statistically significant results are common in observational studies, but they are rarely convincing or corroborated by randomized evidence.

Available only for authorised users

Ioannidis JP, Haidich AB, Pappa M, Pantazis N, Kokori SI, Tektonidou MG, et al. Comparison of evidence of treatment effects in randomized and nonrandomized studies. JAMA. 2001;286(7):821–30. https://doi.org/10.1001/jama.286.7.821.CrossRefPubMed

Anglemyer A, Horvath HT, Bero L. Healthcare outcomes assessed with observational study designs compared with those assessed in randomized trials. Cochrane Database Syst Rev. 2014;4:MR000034.

Solmi M, Correll CU, Carvalho AF, Ioannidis JPA. The role of meta-analyses and umbrella reviews in assessing the harms of psychotropic medications: beyond qualitative synthesis. Epidemiol Psychiatr Sci. 2018;16:1–6.

Papatheodorou S. Umbrella reviews: what they are and why we need them. Eur J Epidemiol. 2019;34(6):543–6. https://doi.org/10.1007/s10654-019-00505-6.CrossRefPubMed

Bellou V, Belbasis L, Tzoulaki I, Evangelou E, Ioannidis JPA. Environmental risk factors and Parkinson’s disease: an umbrella review of meta-analyses. Parkinsonism Relat Disord. 2016;23:1–9. https://doi.org/10.1016/j.parkreldis.2015.12.008.CrossRefPubMed

Li X, Meng X, Timofeeva M, Tzoulaki I, Tsilidis KK, Ioannidis JP, et al. Serum uric acid levels and multiple health outcomes: umbrella review of evidence from observational studies, randomised controlled trials, and Mendelian randomisation studies. BMJ. 2017;357:j2376.CrossRef

Lawlor DA, Harbord RM, Sterne JAC, Timpson N, Davey SG. Mendelian randomization: using genes as instruments for making causal inferences in epidemiology. Stat Med. 2008;27(8):1133–63. https://doi.org/10.1002/sim.3034.CrossRefPubMed

Janiaud P, Agarwal A. Umbrella review of umbrella reviews. 2018 [cited 2021 May 26]; Available from: https://osf.io/g2hd7.

WHO | Risk factors [Internet]. WHO. [cited 2018 Jan 11]. Available from: http://www.who.int/topics/risk_factors/en/.

10.

Shea BJ, Grimshaw JM, Wells GA, Boers M, Andersson N, Hamel C, et al. Development of AMSTAR: a measurement tool to assess the methodological quality of systematic reviews. BMC Med Res Methodol. 2007;7(1):10. https://doi.org/10.1186/1471-2288-7-10.CrossRefPubMedPubMedCentral

11.

Rücker G, Schwarzer G, Carpenter J, Olkin I. Why add anything to nothing? The arcsine difference as a measure of treatment effect in meta-analysis with zero cells. Stat Med. 2009;28(5):721–38. https://doi.org/10.1002/sim.3511.CrossRefPubMed

12.

Higgins JPT, Thompson SG. Quantifying heterogeneity in a meta-analysis. Stat Med. 2002;21(11):1539–58. https://doi.org/10.1002/sim.1186.CrossRefPubMed

13.

Cohen J. A coefficient of agreement for nominal scales. Educ Psychol Meas. 1960;20(1):37–46. https://doi.org/10.1177/001316446002000104.CrossRef

14.

McHugh ML. Interrater reliability: the kappa statistic. Biochem Med. 2012;22(3):276–82.CrossRef

15.

Baldwin JR, Reuben A, Newbury JB, Danese A. Agreement between prospective and retrospective measures of childhood maltreatment: a systematic review and meta-analysis. JAMA Psychiatry. 2019;76(6):584–93. https://doi.org/10.1001/jamapsychiatry.2019.0097.CrossRefPubMedPubMedCentral

16.

Higgins JPT, Thompson SG, Spiegelhalter DJ. A re-evaluation of random-effects meta-analysis. J R Stat Soc Ser A Stat Soc. 2009;172(1):137–59. https://doi.org/10.1111/j.1467-985X.2008.00552.x.CrossRefPubMedPubMedCentral

17.

Riley RD, Higgins JPT, Deeks JJ. Interpretation of random effects meta-analyses. BMJ. 2011;342(feb10 2):d549. https://doi.org/10.1136/bmj.d549.CrossRefPubMed

18.

Sterne JAC, Sutton AJ, Ioannidis JPA, Terrin N, Jones DR, Lau J, et al. Recommendations for examining and interpreting funnel plot asymmetry in meta-analyses of randomised controlled trials. BMJ. 2011;343(jul22 1):d4002. https://doi.org/10.1136/bmj.d4002.CrossRefPubMed

19.

Schulz KF, Chalmers I, Hayes RJ, Altman DG. Empirical evidence of bias. Dimensions of methodological quality associated with estimates of treatment effects in controlled trials. JAMA. 1995;273(5):408–12. https://doi.org/10.1001/jama.1995.03520290060030.CrossRefPubMed

20.

Egger M, Davey Smith G, Schneider M, Minder C. Bias in meta-analysis detected by a simple, graphical test. BMJ. 1997;315(7109):629–34. https://doi.org/10.1136/bmj.315.7109.629.CrossRefPubMedPubMedCentral

21.

Ioannidis JPA. Clarifications on the application and interpretation of the test for excess significance and its extensions. J Math Psychol. 2013;57(5):184–7. https://doi.org/10.1016/j.jmp.2013.03.002.CrossRef

22.

Ioannidis JPA. Excess significance bias in the literature on brain volume abnormalities. Arch Gen Psychiatry. 2011;68(8):773–80. https://doi.org/10.1001/archgenpsychiatry.2011.28.CrossRefPubMed

23.

Ioannidis JPA, Trikalinos TA. An exploratory test for an excess of significant findings. Clin Trials Lond Engl. 2007;4(3):245–53. https://doi.org/10.1177/1740774507079441.CrossRef

24.

Benjamin DJ, Berger JO, Johannesson M, Nosek BA, Wagenmakers E-J, Berk R, et al. Redefine statistical significance. Nat Hum Behav. 2018;2(1):6–10. https://doi.org/10.1038/s41562-017-0189-z.CrossRefPubMed

25.

Hoaglin DC. Misunderstandings about Q and ‘Cochran’s Q test’ in meta-analysis. Stat Med. 2016;35(4):485–95. https://doi.org/10.1002/sim.6632.CrossRefPubMed

26.

Chinn S. A simple method for converting an odds ratio to effect size for use in meta-analysis. Stat Med. 2000;19(22):3127–31. https://doi.org/10.1002/1097-0258(20001130)19:22<3127::AID-SIM784>3.0.CO;2-M.CrossRefPubMed

27.

Belbasis L, Köhler CA, Stefanis N, Stubbs B, van Os J, Vieta E, et al. Risk factors and peripheral biomarkers for schizophrenia spectrum disorders: an umbrella review of meta-analyses. Acta Psychiatr Scand. 2018;137(2):88–97. https://doi.org/10.1111/acps.12847.CrossRefPubMed

28.

Belbasis L, Bellou V, Evangelou E, Ioannidis JPA, Tzoulaki I. Environmental risk factors and multiple sclerosis: an umbrella review of systematic reviews and meta-analyses. Lancet Neurol. 2015;14(3):263–73. https://doi.org/10.1016/S1474-4422(14)70267-4.CrossRefPubMed

29.

Belbasis L, Bellou V, Evangelou E. Environmental risk factors and amyotrophic lateral sclerosis: an umbrella review and critical assessment of current evidence from systematic reviews and meta-analyses of observational studies. Neuroepidemiology. 2016;46(2):96–105. https://doi.org/10.1159/000443146.CrossRefPubMed

30.

Belbasis L, Savvidou MD, Kanu C, Evangelou E, Tzoulaki I. Birth weight in relation to health and disease in later life: an umbrella review of systematic reviews and meta-analyses. BMC Med. 2016;14(1):147. https://doi.org/10.1186/s12916-016-0692-5.CrossRefPubMedPubMedCentral

31.

Belbasis L, Stefanaki I, Stratigos AJ, Evangelou E. Non-genetic risk factors for cutaneous melanoma and keratinocyte skin cancers: an umbrella review of meta-analyses. J Dermatol Sci. 2016;84(3):330–9. https://doi.org/10.1016/j.jdermsci.2016.09.003.CrossRefPubMed

32.

Bellou V, Belbasis L, Tzoulaki I, Middleton LT, Ioannidis JPA, Evangelou E. Systematic evaluation of the associations between environmental risk factors and dementia: an umbrella review of systematic reviews and meta-analyses. Alzheimers Dement J Alzheimers Assoc. 2017;13(4):406–18. https://doi.org/10.1016/j.jalz.2016.07.152.CrossRef

33.

Bellou V, Belbasis L, Tzoulaki I, Evangelou E. Risk factors for type 2 diabetes mellitus: an exposure-wide umbrella review of meta-analyses. PLoS One. 2018;13(3):e0194127. https://doi.org/10.1371/journal.pone.0194127.CrossRefPubMedPubMedCentral

34.

Bortolato B, Köhler CA, Evangelou E, León-Caballero J, Solmi M, Stubbs B, et al. Systematic assessment of environmental risk factors for bipolar disorder: an umbrella review of systematic reviews and meta-analyses. Bipolar Disord. 2017;19(2):84–96. https://doi.org/10.1111/bdi.12490.CrossRefPubMed

35.

Giannakou K, Evangelou E, Papatheodorou SI. Genetic and non-genetic risk factors for pre-eclampsia: an umbrella review of systematic reviews and metaanalyses of observational studies. Ultrasound Obstet Gynecol Off J Int Soc Ultrasound Obstet Gynecol. 2018;51(6):720–30.

36.

He Y, Li X, Gasevic D, Brunt E, McLachlan F, Millenson M, et al. Statins and multiple noncardiovascular outcomes: umbrella review of meta-analyses of observational studies and randomized controlled trials. Ann Intern Med. 2018;169(8):543–53. https://doi.org/10.7326/M18-0808.CrossRefPubMed

37.

Kalliala I, Markozannes G, Gunter MJ, Paraskevaidis E, Gabra H, Mitra A, et al. Obesity and gynaecological and obstetric conditions: umbrella review of the literature. BMJ. 2017;359:j4511.CrossRef

38.

Köhler CA, Evangelou E, Stubbs B, Solmi M, Veronese N, Belbasis L, et al. Mapping risk factors for depression across the lifespan: an umbrella review of evidence from meta-analyses and Mendelian randomization studies. J Psychiatr Res. 2018;103:189–207. https://doi.org/10.1016/j.jpsychires.2018.05.020.CrossRefPubMed

39.

Kyrgiou M, Kalliala I, Markozannes G, Gunter MJ, Paraskevaidis E, Gabra H, et al. Adiposity and cancer at major anatomical sites: umbrella review of the literature. BMJ. 2017;356:j477.CrossRef

40.

Machado MO, Veronese N, Sanches M, Stubbs B, Koyanagi A, Thompson T, et al. The association of depression and all-cause and cause-specific mortality: an umbrella review of systematic reviews and meta-analyses. BMC Med. 2018;16(1):112. https://doi.org/10.1186/s12916-018-1101-z.CrossRefPubMedPubMedCentral

41.

Markozannes G, Tzoulaki I, Karli D, Evangelou E, Ntzani E, Gunter MJ, et al. Diet, body size, physical activity and risk of prostate cancer: an umbrella review of the evidence. Eur J Cancer Oxf Engl 1990. 2016;69:61–9.

42.

Radua J, Ramella-Cravaro V, Ioannidis JPA, Reichenberg A, Phiphopthatsanee N, Amir T, et al. What causes psychosis? An umbrella review of risk and protective factors. World Psychiatry Off J World Psychiatr Assoc WPA. 2018;17(1):49–66.

43.

de Rezende LFM, de Sá TH, Markozannes G, Rey-López JP, Lee I-M, Tsilidis KK, et al. Physical activity and cancer: an umbrella review of the literature including 22 major anatomical sites and 770 000 cancer cases. Br J Sports Med. 2018;52(13):826–33.

44.

Solmi M, Köhler CA, Stubbs B, Koyanagi A, Bortolato B, Monaco F, et al. Environmental risk factors and nonpharmacological and nonsurgical interventions for obesity: an umbrella review of meta-analyses of cohort studies and randomized controlled trials. Eur J Clin Invest. 2018;20:e12982.CrossRef

45.

Theodoratou E, Tzoulaki I, Zgaga L, Ioannidis JPA. Vitamin D and multiple health outcomes: umbrella review of systematic reviews and meta-analyses of observational studies and randomised trials. BMJ. 2014;348(apr01 2):g2035. https://doi.org/10.1136/bmj.g2035.CrossRefPubMedPubMedCentral

46.

Tsilidis KK, Kasimis JC, Lopez DS, Ntzani EE, Ioannidis JPA. Type 2 diabetes and cancer: umbrella review of meta-analyses of observational studies. BMJ. 2015;350(jan02 1):g7607. https://doi.org/10.1136/bmj.g7607.CrossRefPubMed

47.

Veronese N, Demurtas J, Celotto S, Caruso MG, Maggi S, Bolzetta F, et al. Is chocolate consumption associated with health outcomes? An umbrella review of systematic reviews and meta-analyses. Clin Nutr. 2019;38(3):1101–8.

48.

Veronese N, Solmi M, Caruso MG, Giannelli G, Osella AR, Evangelou E, et al. Dietary fiber and health outcomes: an umbrella review of systematic reviews and meta-analyses. Am J Clin Nutr. 2018;107(3):436–44. https://doi.org/10.1093/ajcn/nqx082.CrossRefPubMed

49.

Zhou C, Zhong X, Song Y, Shi J, Wu Z, Guo Z, et al. Prognostic biomarkers for gastric cancer: an umbrella review of the evidence. Front Oncol. 2019;9:1321. https://doi.org/10.3389/fonc.2019.01321.CrossRefPubMedPubMedCentral

50.

Zhao L-G, Li Z-Y, Feng G-S, Ji X-W, Tan Y-T, Li H-L, et al. Coffee drinking and cancer risk: an umbrella review of meta-analyses of observational studies. BMC Cancer. 2020;20(1):101. https://doi.org/10.1186/s12885-020-6561-9.CrossRefPubMedPubMedCentral

51.

Yu H, Zhong X, Gao P, Shi J, Wu Z, Guo Z, et al. The potential effect of metformin on cancer: an umbrella review. Front Endocrinol. 2019;10:617. https://doi.org/10.3389/fendo.2019.00617.CrossRef

52.

Veronese N, Demurtas J, Pesolillo G, Celotto S, Barnini T, Calusi G, et al. Magnesium and health outcomes: an umbrella review of systematic reviews and meta-analyses of observational and intervention studies. Eur J Nutr. 2020;59(1):263–72. https://doi.org/10.1007/s00394-019-01905-w.CrossRefPubMed

53.

Smith L, Luchini C, Demurtas J, Soysal P, Stubbs B, Hamer M, et al. Telomere length and health outcomes: an umbrella review of systematic reviews and meta-analyses of observational studies. Ageing Res Rev. 2019;51:1–10. https://doi.org/10.1016/j.arr.2019.02.003.CrossRefPubMed

54.

Raglan O, Kalliala I, Markozannes G, Cividini S, Gunter MJ, Nautiyal J, et al. Risk factors for endometrial cancer: an umbrella review of the literature. Int J Cancer. 2019;145(7):1719–30.CrossRef

55.

Papola D, Ostuzzi G, Gastaldon C, Morgano GP, Dragioti E, Carvalho AF, et al. Antipsychotic use and risk of life-threatening medical events: umbrella review of observational studies. Acta Psychiatr Scand. 2019;140(3):227–43. https://doi.org/10.1111/acps.13066.CrossRefPubMed

56.

Kim JY, Steingroever J, Lee KH, Oh J, Choi MJ, Lee J, et al. Clinical interventions and all-cause mortality of patients with chronic kidney disease: an umbrella systematic review of meta-analyses. J Clin Med. 2020;9(2):394.

57.

Kim JY, Son MJ, Son CY, Radua J, Eisenhut M, Gressier F, et al. Environmental risk factors and biomarkers for autism spectrum disorder: an umbrella review of the evidence. Lancet Psychiatry. 2019;6(7):590–600. https://doi.org/10.1016/S2215-0366(19)30181-6.CrossRefPubMed

58.

Grabovac I, Veronese N, Stefanac S, Haider S, Jackson SE, Koyanagi A, et al. Human immunodeficiency virus infection and diverse physical health outcomes: an umbrella review of meta-analyses of observational studies. Clin Infect Dis Off Publ Infect Dis Soc Am. 2020;70(9):1809–15. https://doi.org/10.1093/cid/ciz539.CrossRef

59.

Giannakou K, Evangelou E, Yiallouros P, Christophi CA, Middleton N, Papatheodorou E, et al. Risk factors for gestational diabetes: an umbrella review of meta-analyses of observational studies. PLoS One. 2019;14(4):e0215372. https://doi.org/10.1371/journal.pone.0215372.CrossRefPubMedPubMedCentral

60.

Fullana MA, Tortella-Feliu M, de la Cruz LF, Chamorro J, Pérez-Vigil A, Ioannidis JPA, et al. Risk and protective factors for anxiety and obsessive-compulsive disorders: an umbrella review of systematic reviews and meta-analyses. Psychol Med. 2020;50(8):1300–15. https://doi.org/10.1017/S0033291719001247.CrossRefPubMed

61.

Dragioti E, Solmi M, Favaro A, Fusar-Poli P, Dazzan P, Thompson T, et al. Association of antidepressant use with adverse health outcomes: a systematic umbrella review. JAMA Psychiatry. 2019;76(12):1241–55.

62.

Bellou V, Belbasis L, Konstantinidis AK, Evangelou E. Elucidating the risk factors for chronic obstructive pulmonary disease: an umbrella review of meta-analyses. Int J Tuberc Lung Dis Off J Int Union Tuberc Lung Dis. 2019;23(1):58–66.CrossRef

63.

Belbasis L, Dosis V, Evangelou E. Elucidating the environmental risk factors for rheumatic diseases: an umbrella review of meta-analyses. Int J Rheum Dis. 2018;21(8):1514–24. https://doi.org/10.1111/1756-185X.13356.CrossRefPubMed

64.

Kim JH, Kim JY, Lee J, Jeong GH, Lee E, Lee S, et al. Environmental risk factors, protective factors, and peripheral biomarkers for ADHD: an umbrella review. Lancet Psychiatry. 2020;7(11):955–70. https://doi.org/10.1016/S2215-0366(20)30312-6.CrossRefPubMed

65.

Solmi M, Dragioti E, Arango C, Radua J, Ostinelli E, Kilic O, et al. Risk and protective factors for mental disorders with onset in childhood/adolescence: an umbrella review of published meta-analyses of observational longitudinal studies. Neurosci Biobehav Rev. 2021;120:565–73.

66.

Valenzuela PL, Santos-Lozano A, Morales JS, López-Ortiz S, Pinto-Fraga J, Castillo-García A, et al. Physical activity, sports and risk of atrial fibrillation: umbrella review of meta-analyses. Eur J Prev Cardiol. 2020;16:2047487320923183.CrossRef

67.

Belbasis L, Mavrogiannis MC, Emfietzoglou M, Evangelou E. Environmental factors, serum biomarkers and risk of atrial fibrillation: an exposure-wide umbrella review of meta-analyses. Eur J Epidemiol. 2020;35(3):223–39. https://doi.org/10.1007/s10654-020-00618-3.CrossRefPubMed

68.

Solmi M, Veronese N, Galvano D, Favaro A, Ostinelli EG, Noventa V, et al. Factors associated with loneliness: an umbrella review of observational studies. J Affect Disord. 2020;271:131–8. https://doi.org/10.1016/j.jad.2020.03.075.CrossRefPubMed

69.

Veronese N, Demurtas J, Thompson T, Solmi M, Pesolillo G, Celotto S, et al. Effect of low-dose aspirin on health outcomes: an umbrella review of systematic reviews and meta-analyses. Br J Clin Pharmacol. 2020;86(8):1465–75. https://doi.org/10.1111/bcp.14310.CrossRefPubMed

70.

Song Y, Zhong X, Gao P, Zhou C, Shi J, Wu Z, et al. Aspirin and its potential preventive role in cancer: an umbrella review. Front Endocrinol. 2020;11:3. https://doi.org/10.3389/fendo.2020.00003.CrossRef

71.

Zhang X, Gill D, He Y, Yang T, Li X, Monori G, et al. Non-genetic biomarkers and colorectal cancer risk: umbrella review and evidence triangulation. Cancer Med. 2020;9(13):4823–35. https://doi.org/10.1002/cam4.3051.CrossRefPubMedPubMedCentral

72.

Xu W, He Y, Wang Y, Li X, Young J, Ioannidis JPA, et al. Risk factors and risk prediction models for colorectal cancer metastasis and recurrence: an umbrella review of systematic reviews and meta-analyses of observational studies. BMC Med. 2020;18(1):172. https://doi.org/10.1186/s12916-020-01618-6.CrossRefPubMedPubMedCentral

73.

Markozannes G, Koutsioumpa C, Cividini S, Monori G, Tsilidis KK, Kretsavos N, et al. Global assessment of C-reactive protein and health-related outcomes: an umbrella review of evidence from observational studies and Mendelian randomization studies. Eur J Epidemiol. 2021;36(1):11–36.

74.

Biffi A, Cantarutti A, Rea F, Locatelli A, Zanini R, Corrao G. Use of antidepressants during pregnancy and neonatal outcomes: an umbrella review of meta-analyses of observational studies. J Psychiatr Res. 2020;124:99–108. https://doi.org/10.1016/j.jpsychires.2020.02.023.CrossRefPubMed

75.

Solmi M, Radua J, Stubbs B, Ricca V, Moretti D, Busatta D, et al. Risk factors for eating disorders: an umbrella review of published meta-analyses. Braz J Psychiatry. 2021;43(3):314–23.

76.

Lee KH, Seong HJ, Kim G, Jeong GH, Kim JY, Park H, et al. Consumption of fish and ω-3 fatty acids and cancer risk: an umbrella review of meta-analyses of observational studies. Adv Nutr Bethesda Md. 2020;11(5):1134–49. https://doi.org/10.1093/advances/nmaa055.CrossRef

77.

Demurtas J, Celotto S, Beaudart C, Sanchez-Rodriguez D, Balci C, Soysal P, et al. The efficacy and safety of influenza vaccination in older people: an umbrella review of evidence from meta-analyses of both observational and randomized controlled studies. Ageing Res Rev. 2020;62:101118. https://doi.org/10.1016/j.arr.2020.101118.CrossRefPubMed

78.

Soysal P, Hurst C, Demurtas J, Firth J, Howden R, Yang L, et al. Handgrip strength and health outcomes: Umbrella review of systematic reviews with metaanalyses of observational studies. J Sport Health Sci. 2021;10(3):290–5.

79.

Carvalho AF, Solmi M, Sanches M, Machado MO, Stubbs B, Ajnakina O, et al. Evidence-based umbrella review of 162 peripheral biomarkers for major mental disorders. Transl Psychiatry. 2020;10(1):152. https://doi.org/10.1038/s41398-020-0835-5.CrossRefPubMedPubMedCentral

80.

Wang Y, Zhong X, Zhou L, Lu J, Jiang B, Liu C, et al. Prognostic biomarkers for pancreatic ductal adenocarcinoma: an umbrella review. Front Oncol. 2020;10:1466. https://doi.org/10.3389/fonc.2020.01466.CrossRefPubMedPubMedCentral

81.

Kim TL, Jeong GH, Yang JW, Lee KH, Kronbichler A, van der Vliet HJ, et al. Tea consumption and risk of cancer: an umbrella review and meta-analysis of observational studies. Adv Nutr Bethesda Md. 2020;11(6):1437–52. https://doi.org/10.1093/advances/nmaa077.CrossRef

82.

Janiaud P, Agarwal A. Umbrella review of umbrella reviews. 2018 [cited 2021 May 26]; Available from: https://osf.io/xj5cf/

83.

Ioannidis JPA, Trikalinos TA. The appropriateness of asymmetry tests for publication bias in meta-analyses: a large survey. CMAJ. 2007;176(8):1091–6. https://doi.org/10.1503/cmaj.060410.CrossRefPubMedPubMedCentral

84.

Cornell JE, Mulrow CD, Localio R, Stack CB, Meibohm AR, Guallar E, et al. Random-effects meta-analysis of inconsistent effects: a time for change. Ann Intern Med. 2014;160(4):267–70. https://doi.org/10.7326/M13-2886.CrossRefPubMed

85.

Terrin N, Schmid CH, Lau J, Olkin I. Adjusting for publication bias in the presence of heterogeneity. Stat Med. 2003;22(13):2113–26. https://doi.org/10.1002/sim.1461.CrossRefPubMed

86.

Patel CJ, Ji J, Sundquist J, Ioannidis JPA, Sundquist K. Systematic assessment of pharmaceutical prescriptions in association with cancer risk: a method to conduct a population-wide medication-wide longitudinal study. Sci Rep. 2016;6(1):31308. https://doi.org/10.1038/srep31308.CrossRefPubMedPubMedCentral

87.

Schuemie MJ, Ryan PB, Hripcsak G, Madigan D, Suchard MA. Improving reproducibility by using high-throughput observational studies with empirical calibration. Philos Transact A Math Phys Eng Sci. 2018;13:376(2128).

88.

Schuemie MJ, Cepeda MS, Suchard MA, Yang J, Tian Y, Schuler A, et al. How confident are we about observational findings in healthcare: a benchmark study. Harv Data Sci Rev. 2020; [cited 2021 Feb 2]; Available from: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7755157/.

89.

Djulbegovic B, Glasziou P, Klocksieben FA, Reljic T, VanDenBergh M, Mhaskar R, et al. Larger effect sizes in nonrandomized studies are associated with higher rates of EMA licensing approval. J Clin Epidemiol. 2018;98:24–32. https://doi.org/10.1016/j.jclinepi.2018.01.011.CrossRefPubMed

90.

Razavi M, Glasziou P, Klocksieben FA, Ioannidis JPA, Chalmers I, Djulbegovic B. US Food and Drug Administration approvals of drugs and devices based on nonrandomized clinical trials: a systematic review and meta-analysis. JAMA Netw Open. 2019;2(9):e1911111. https://doi.org/10.1001/jamanetworkopen.2019.11111.CrossRefPubMedPubMedCentral

91.

Sacks H, Chalmers TC, Smith H. Randomized versus historical controls for clinical trials. Am J Med. 1982;72(2):233–40. https://doi.org/10.1016/0002-9343(82)90815-4.CrossRefPubMed

92.

Colditz GA, Miller JN, Mosteller F. How study design affects outcomes in comparisons of therapy. I: Medical. Stat Med. 1989;8(4):441–54. https://doi.org/10.1002/sim.4780080408.CrossRefPubMed

93.

Benson K, Hartz AJ. A comparison of observational studies and randomized, controlled trials. N Engl J Med. 2000;342(25):1878–86. https://doi.org/10.1056/NEJM200006223422506.CrossRefPubMed

94.

Concato J, Shah N, Horwitz RI. Randomized, controlled trials, observational studies, and the hierarchy of research designs. N Engl J Med. 2000;342(25):1887–92. https://doi.org/10.1056/NEJM200006223422507.CrossRefPubMedPubMedCentral

95.

Hemkens LG, Contopoulos-Ioannidis DG, Ioannidis JPA. Agreement of treatment effects for mortality from routinely collected data and subsequent randomized trials: meta-epidemiological survey. BMJ. 2016;352:i493.CrossRef

96.

Guyatt G, Oxman AD, Akl EA, Kunz R, Vist G, Brozek J, et al. GRADE guidelines: 1. Introduction-GRADE evidence profiles and summary of findings tables. J Clin Epidemiol. 2011;64(4):383–94. https://doi.org/10.1016/j.jclinepi.2010.04.026.CrossRefPubMed

97.

Whiting P, Savović J, Higgins JPT, Caldwell DM, Reeves BC, Shea B, et al. ROBIS: a new tool to assess risk of bias in systematic reviews was developed. J Clin Epidemiol. 2016;69:225–34. https://doi.org/10.1016/j.jclinepi.2015.06.005.CrossRefPubMedPubMedCentral

98.

Ioannidis JPA. The mass production of redundant, misleading, and conflicted systematic reviews and meta-analyses. Milbank Q. 2016;94(3):485–514. https://doi.org/10.1111/1468-0009.12210.CrossRefPubMedPubMedCentral

Title: Validity of observational evidence on putative risk and protective factors: appraisal of 3744 meta-analyses on 57 topics
Authors: Perrine Janiaud
Arnav Agarwal
Ioanna Tzoulaki
Evropi Theodoratou
Konstantinos K. Tsilidis
Evangelos Evangelou
John P. A. Ioannidis
Publication date: 01-12-2021
Publisher: BioMed Central
Published in: BMC Medicine / Issue 1/2021
Electronic ISSN: 1741-7015
DOI: https://doi.org/10.1186/s12916-021-02020-6

At a glance: The STEP trials

Springer Medicine

Validity of observational evidence on putative risk and protective factors: appraisal of 3744 meta-analyses on 57 topics

Abstract

Background

Methods

Results

Conclusions

At a glance: The STEP trials

Springer Medicine

Abstract

Background

Methods

Results

Conclusions

Please log in to get access to this content

Other articles of this Issue 1/2021

Serum transferrin as a biomarker of hepatocyte nuclear factor 4 alpha activity and hepatocyte function in liver diseases

Identification of NOTCH4 mutation as a response biomarker for immune checkpoint inhibitor therapy

Maternal lithium use and the risk of adverse pregnancy and neonatal outcomes: a Swedish population-based cohort study

Effects of the Norfolk diabetes prevention lifestyle intervention (NDPS) on glycaemic control in screen-detected type 2 diabetes: a randomised controlled trial

Sequelae, persistent symptomatology and outcomes after COVID-19 hospitalization: the ANCOHVID multicentre 6-month follow-up study

A systematic assessment of the association between frequently prescribed medicines and the risk of common cancers: a series of nested case-control studies