Top

Pediatric Surgery International

Published in:

01-03-2021 | Original Article

Meta-analyses in paediatric surgery are often fragile: implications and consequences

Authors: Arne Schröder, Oliver J. Muensterer, Christina Oetzmann von Sochaczewski

Published in: Pediatric Surgery International | Issue 3/2021

Abstract

Purpose

Meta-analyses occupy the highest level of evidence and thereby guide clinical decision-making. Recently, randomised-controlled trials were evaluated for the robustness of their findings by calculating the fragility index. The fragility index is the number of events that needs to be added to one treatment arm until the statistical significance collapses. We, therefore, aimed to evaluate the robustness of paediatric surgical meta-analyses.

Methods

We searched MEDLINE for paediatric surgical meta-analyses in the last decade. All meta-analyses on a paediatric surgical condition were eligible for analysis if they based their conclusion on a statistically significant meta-analysis.

Results

We screened 303 records and conducted a full-text evaluation of 60 manuscripts. Of them, 39 were included in our analysis that conducted 79 individual meta-analyses with significant results. Median fragility index was 5 (Q25–Q75% 2–11). Median fragility in relation to included patients was 0.77% (Q25–Q75% 0.29–1.87%).

Conclusion

Paediatric surgical meta-analyses are often fragile. In almost 60% of results, the statistical significance depends on less than 1% of the included population. However, as the fragility index is just a transformation of the P value, it basically conveys the same information in a different format. It therefore should be avoided.

Walsh M, Srinathan SK, McAuley DF et al (2014) The statistical significance of randomized controlled trial results is frequently fragile: a case for a Fragility Index. J Clin Epidemiol 67:622–628. https://doi.org/10.1016/j.jclinepi.2013.10.019CrossRefPubMed

Ridgeon EE, Young PJ, Bellomo R et al (2016) The Fragility Index in multicenter randomized controlled critical care trials. Crit Care Med 44:1278–1284. https://doi.org/10.1097/CCM.0000000000001670CrossRefPubMed

Tignanelli CJ, Napolitano LM (2019) The Fragility Index in randomized clinical trials as a means of optimizing patient care. JAMA Surg 154:74–79. https://doi.org/10.1001/jamasurg.2018.4318CrossRefPubMed

Mazzinari G, Ball L, Serpa Neto A et al (2018) The fragility of statistically significant findings in randomised controlled anaesthesiology trials: systematic review of the medical literature. Brit J Anaesth 120:935–941. https://doi.org/10.1016/j.bja.2018.01.012CrossRefPubMed

Robinson T, Al-Shahwani N, Easterbrook B, VanHouwelingen L (2020) The fragility of statistically significant findings from randomized controlled trials in pediatric appendicitis: a systematic review. J Pediatr Surg 55:800–804. https://doi.org/10.1016/j.jpedsurg.2020.01.030CrossRefPubMed

Moss RL, Henry MCW, Dimmitt RA et al (2001) The role of prospective randomized clinical trials in pediatric surgery: state of the art? J Pediatr Surg 36:1182–1186. https://doi.org/10.1053/jpsu.2001.25749CrossRefPubMed

Curry JI, Reeves B, Stringer MD (2003) Randomized controlled trials in pediatric surgery: could we do better? J Pediatr Surg 38:556–559. https://doi.org/10.1053/jpsu.2003.50121CrossRefPubMed

Atal I, Porcher R, Boutron I, Ravaud P (2019) The statistical significance of meta-analyses is frequently fragile: definition of a fragility index for meta-analyses. J Clin Epidemiol 111:32–40. https://doi.org/10.1016/j.jclinepi.2019.03.012CrossRefPubMed

Atal I (2018) The Fragility Index of meta-analyses. Paris

10.

Ahmed W, Fowler RA, McCredie VA (2016) Does sample size matter when interpreting the Fragility Index? Crit Care Med 44:e1142–e1143. https://doi.org/10.1097/CCM.0000000000001976CrossRefPubMed

11.

R Core Team (2019) R: a language and environment for statistical computing. R Foundation for Statistical Computing, Vienna

12.

Ioannidis JPA (2016) The mass production of redundant, misleading, and conflicted systematic reviews and meta-analyses: mass production of systematic reviews and meta-analyses. Milbank Q 94:485–514. https://doi.org/10.1111/1468-0009.12210CrossRefPubMedPubMedCentral

13.

Salim A, Mullassery D, Losty PD (2017) Quality of systematic reviews and meta-analyses published in pediatric surgery. J Pediatr Surg 52:1732–1735. https://doi.org/10.1016/j.jpedsurg.2017.07.022CrossRefPubMed

14.

Hughes D, Judge C, Murphy R et al (2020) Association of blood pressure lowering with incident dementia or cognitive impairment: a systematic review and meta-analysis. JAMA 323:1934–1944. https://doi.org/10.1001/jama.2020.4249CrossRefPubMed

15.

Gudivada KK, Kumar A, Shariff M et al (2020) Antioxidant micronutrient supplementation in critically ill adults: a systematic review with meta-analysis and trial sequential analysis. Clin Nutr. https://doi.org/10.1016/j.clnu.2020.06.033CrossRefPubMed

16.

Xing A, Chu H, Lin L (2020) Fragility index of network meta-analysis with application to smoking cessation data. J Clin Epidemiol 127:29–39. https://doi.org/10.1016/j.jclinepi.2020.07.003CrossRefPubMed

17.

Carter RE, McKie PM, Storlie CB (2017) The Fragility Index: a P-value in sheep’s clothing? Eur Heart J 38:346–348. https://doi.org/10.1093/eurheartj/ehw495CrossRefPubMed

18.

Greenland S, Senn SJ, Rothman KJ et al (2016) Statistical tests, P values, confidence intervals, and power: a guide to misinterpretations. Eur J Epidemiol 31:337–350. https://doi.org/10.1007/s10654-016-0149-3CrossRefPubMedPubMedCentral

19.

Rafi Z, Greenland S (2020) Semantic and cognitive tools to aid statistical science: replace confidence and significance by compatibility and surprise. BMC Med Res Methodol 20:244. https://doi.org/10.1186/s12874-020-01105-9CrossRefPubMedPubMedCentral

20.

Reito A, Raittio L, Helminen O (2019) Fragility Index, power, strength and robustness of findings in sports medicine and arthroscopic surgery: a secondary analysis of data from a study on use of the Fragility Index in sports surgery. PeerJ 7:e6813. https://doi.org/10.7717/peerj.6813CrossRefPubMedPubMedCentral

21.

Wasserstein RL, Lazar NA (2016) The ASA statement on p-values: context, process, and purpose. Am Stat 70:129–133. https://doi.org/10.1080/00031305.2016.1154108CrossRef

22.

Niforatos JD, Zheutlin AR, Chaitoff A, Pescatore RM (2020) The fragility index of practice changing clinical trials is low and highly correlated with P-values. J Clin Epidemiol 119:140–142. https://doi.org/10.1016/j.jclinepi.2019.09.029CrossRefPubMed

23.

Acuna SA, Sue-Chue-Lam C, Dossa F (2019) The Fragility Index—P values reimagined flaws all. JAMA Surg 154:674. https://doi.org/10.1001/jamasurg.2019.0567CrossRefPubMed

24.

Condon TM, Sexton RW, Wells AJ, To M-S (2020) The weakness of fragility index exposed in an analysis of the traumatic brain injury management guidelines: a meta-epidemiological and simulation study. PLoS ONE 15:e0237879. https://doi.org/10.1371/journal.pone.0237879CrossRefPubMedPubMedCentral

25.

Gelijns AC, Ascheim DD, Parides MK et al (2009) Randomized trials in surgery. Surgery 145:581–587. https://doi.org/10.1016/j.surg.2009.04.003CrossRefPubMedPubMedCentral

26.

Potter GE (2020) Dismantling the Fragility Index: a demonstration of statistical reasoning. Stat Med 39:3720–3731. https://doi.org/10.1002/sim.8689CrossRefPubMed

27.

Porco TC, Lietman TM (2018) A Fragility Index: handle with Care. Ophthalmology 125:649. https://doi.org/10.1016/j.ophtha.2018.03.002CrossRefPubMed

28.

Berman NG, Parker RA (2002) Meta-analysis: neither quick nor easy. BMC Med Res Methodol 2:10. https://doi.org/10.1186/1471-2288-2-10CrossRefPubMedPubMedCentral

29.

Rao G, Lopez-Jimenez F, Boyd J et al (2017) Methodological standards for meta-analyses and qualitative systematic reviews of cardiac prevention and treatment studies: a scientific statement from the American Heart Association. Circulation 136:e172–e194. https://doi.org/10.1161/CIR.0000000000000523CrossRefPubMed

30.

Shea BJ, Hamel C, Wells GA et al (2009) AMSTAR is a reliable and valid measurement tool to assess the methodological quality of systematic reviews. J Clin Epidemiol 62:1013–1020. https://doi.org/10.1016/j.jclinepi.2008.10.009CrossRefPubMed

31.

Whiting P, Savović J, Higgins JPT et al (2016) ROBIS: A new tool to assess risk of bias in systematic reviews was developed. J Clin Epidemiol 69:225–234. https://doi.org/10.1016/j.jclinepi.2015.06.005CrossRefPubMedPubMedCentral

32.

Shea BJ, Reeves BC, Wells G et al (2017) AMSTAR 2: a critical appraisal tool for systematic reviews that include randomised or non-randomised studies of healthcare interventions, or both. BMJ 358:j4008. https://doi.org/10.1136/bmj.j4008CrossRefPubMedPubMedCentral

33.

Pieper D, Puljak L, González-Lorenzo M, Minozzi S (2019) Minor differences were found between AMSTAR 2 and ROBIS in the assessment of systematic reviews including both randomized and nonrandomized studies. J Clin Epidemiol 108:26–33. https://doi.org/10.1016/j.jclinepi.2018.12.004CrossRefPubMed

34.

De Santis KK, Kaplan I (2020) Assessing the quality of systematic reviews in healthcare using AMSTAR and AMSTAR2: a comparison of scores on both scales. Z Psychol 228:36–42. https://doi.org/10.1027/2151-2604/a000397CrossRef

35.

Schröder A, Muensterer OJ, Oetzmann von Sochaczewski C (2020) Meta-analyses in paediatric surgery are often fragile: implications and consequences. https://doi.org/10.5281/zenodo.4275919CrossRef

Title: Meta-analyses in paediatric surgery are often fragile: implications and consequences
Authors: Arne Schröder
Oliver J. Muensterer
Christina Oetzmann von Sochaczewski
Publication date: 01-03-2021
Publisher: Springer Berlin Heidelberg
Published in: Pediatric Surgery International / Issue 3/2021
Print ISSN: 0179-0358
Electronic ISSN: 1437-9813
DOI: https://doi.org/10.1007/s00383-020-04827-5

At a glance: The ONWARDS insulin icodec trials

Springer Medicine

Meta-analyses in paediatric surgery are often fragile: implications and consequences

Abstract

Purpose

Methods

Results

Conclusion

At a glance: The ONWARDS insulin icodec trials

Springer Medicine

Abstract

Purpose

Methods

Results

Conclusion

Please log in to get access to this content

Other articles of this Issue 3/2021

Discrepancy between the survival rate and neuropsychological development in postsurgical extremely low-birth-weight infants: a retrospective study over two decades at a single institution

Telemedicine in the pediatric surgery in Germany during the COVID-19 pandemic

Doppler ultrasound assessment of splanchnic perfusion and heart rate for the detection of necrotizing enterocolitis

The impact of early and successful orchidopexy on hormonal follow-up for 208 boys with bilateral non-syndromic cryptorchidism

The 33rd international symposium on pediatric surgical research

Nasogastric decompression after intestinal surgery in children: a systematic review and meta-analysis