Published in:
22-08-2021 | Original Article
Randomized controlled trials published on patients with hernias have a high percentage of unreproducible statistics
Published in: Hernia | Issue 3/2022
Login to get accessAbstract
Introduction
Surgeons rely on randomized controlled trials (RCT) to compare the effectiveness of treatments. RCTs require careful planning and substantial effort to complete. Because of the careful study design, statistics performed are often easy to reproduce such as Chi-squared or t-test. Issues such as statistical discordance, or reporting statistical results that cannot be reproduced, should be uncommon.
Methods
RCTs pertaining to hernias were identified in PubMed using the search terms “hernia” and “randomized controlled trial.” Studies were selected using a random number generator. Studies were included if the primary outcome could be reproduced using the data and statistical test reported in the manuscript. Discordance between the obtained p-value from our analysis and the published p-value was assessed. Primary outcome was the number of studies that reported p-values that crossed the level of statistical significance (p-value = 0.05) but on reproduction analysis did not.
Results
Of the 100 included RCTs, five reported p-values that crossed the “p = 0.05” threshold that our team was unable to reproduce using the statistical test reported in the manuscript. An additional three studies reported p-values that crossed the “p = 0.05” threshold that our team was unable to reproduce using the appropriate statistical test (i.e., Fisher’s exact test when all expected cell counts < 5). All eight studies published p-values < 0.05, whereas, our re-analysis demonstrated p ≥ 0.05.
Conclusion
Eight percent of the RCTs analyzed in this study reported p-values < 0.05 that on reproduction analysis was ≥ 0.05. The next steps should be to determine reasons for discordance and how to prevent this from happening.