Skip to main content
Key Specialties
Cardiology Diabetology Endocrinology Gastroenterology Geriatrics and Gerontology Gynecology and Obstetrics Hematology Infectious Disease Internal Medicine Nephrology Neurology Oncology Pediatrics Respiratory Medicine Rheumatology Surgery
Congress Coverage
2024 ACC Congress 2023 ESMO Congress 2023 EASD Congress 2023 ERS Congress 2023 ESC Congress
Clinical Collections
Advances in Alzheimer’s

Keynote webinar | Spotlight on medication adherence

LIVE: Thursday 27th June 2024, 18:00-19:30 (CEST)
Join our expert panel to discover why you need to understand the drivers of non-adherence in your patients, and how you can optimize medication adherence in your clinics to drastically improve patient outcomes.
ADVANCED SEARCH
Log in
Top
BMC Medical Research Methodology
Published in: BMC Medical Research Methodology 1/2018

Open Access 01-12-2018 | Research article

Bonferroni-Holm and permutation tests to compare health data: methodological and applicative issues

Authors: Massimiliano Giacalone, Zirilli Agata, Paolo Carmelo Cozzucoli, Angela Alibrandi

Published in: BMC Medical Research Methodology | Issue 1/2018

Login to get access

Abstract

Background

Statistical methodology is a powerful tool in the health research; however, there is wide accord that statistical methodologies are not usually used properly. In particular when multiple comparisons are needed, it is necessary to check the rate of false positive results and the potential inflation of type I errors. In this case, permutation testing methods are useful to check the simultaneous significance level and identify the most significant factors.

Methods

In this paper an application of permutation tests, in the medical context of Inflammatory Bowel Diseases, is performed. The main goal is to assess the existence of significant differences between Crohn’s Disease (CD) and Ulcerative Colitis (UC). The Sequentially Rejective Multiple Test (Bonferroni-Holm procedure) is used to find which of the partial tests are effectively significant and solve the problem of the multiplicity control.

Results

Applying Non-Parametric Combination (NPC) Test for partial and combined tests we conclude that Crohn’s Disease patients and Ulcerative Colitis patients differ between them for most examined variables. UC patients compared with the CD patients, have a higher diagnosis age, not show smoking status, proportion of patients treated with immunosuppressants or with biological drugs is lower than the CD patients, even if the duration of such therapies is longer. CD patients have a higher rate of re-hospitalization. Diabetes is more present in the sub-population of UC patients. Analyzing the Charlson score we can highlight that UC patients have a more severe clinical situation than CD patients. Finally, CD patients are more frequently subject to surgery compared to UC. Appling of the Bonferroni Holm procedure, which provided adjusted p-values, we note that only nine of the examined variables are statistically significant: Smoking habit, Immunosuppressive therapy, Surgery, Biological Drug, Diabetes, Adverse Events, Re-hospitalization, Gender and Duration of Immunosoppressive Therapy. Therefore, we can conclude that these are the specific variables that can discriminate effectively the Crohn’s Disease and Ulcerative Colitis groups.

Conclusions

We identified significant variables that discriminate the two groups, satisfying the multiplicity problem, in fact we can affirm that Smoking habit, Immunosuppressive therapy, Surgery, Biological Drug, Diabetes, Adverse Events, Hospitalization, Gender and Duration of Immunosoppressive Therapy are the effectively significant variables.
Literature
1.
Aickin M, Gensler H. Adjusting for multiple testing when reporting research results: the Bonferroni vs. Holm methods. Am J Public Health 1996;86(5): 726–728.
2.
Alibrandi A, Giacalone M, Zirilli A, Moleti M. NPC to assess effects of maternal iodine nutrition and thyroid status on children cognitive development. In Proceedings of Compstat 2016, 22nd International Conference on Computatìonal Statistics 2016. ISBN/EAN: 978–90–73592-36-0.
3.
Antolini L, Bolzan M, Salmaso L. Metodi non parametrici per la verifica di ipotesi in indagini multicentriche. Statistica. 2007;62(3):523–33.
4.
Arboretti Giancristofaro R, Marozzi M, Salmaso L. Repeated measures designs: a permutation approach for testing for active effects. Far East J Theoret Stat. 2005;16(2):303–25.
5.
Arboretti Giancristofaro R, Brombin C. Overview of NonParametric combination-based permutation tests for multivariate multi-sample problems. Statistica. 2014;74(3):233–46.
6.
Basso D, Chiarandini M, Salmaso L. Synchronized permutation tests in I×J designs. J Stat Plan Inference. 2007;137(8):2564–78.CrossRef
7.
Bonnini S, Pesarin F, Salmaso L. Statistical Analysis in biomedical studies: an application of NPC Test to a clinical trial on a respiratory drug. In Congresso Nazionale della Società Italiana di Biometria. Società Italiana di Biometria; 2003. p. 107-10.
8.
Bonnini S, Corain L, Munaò F, Salmaso L. Neurocognitive effects in welders exposed to Aluminium: an application of the NPC test and NPC ranking methods. JISS. 2006;15(2):191–208.CrossRef
9.
10.
Callegaro A, Pesarin R, Salmaso L. Test di permutazione per il confronto di curve di sopravvivenza. Statistica Applicata. 2003;15(2):241–61.
11.
12.
Corain L, Salmaso L. Multivariate and multistrata nonparametric tests: the NPC method. J Modern Appl Stat Methods. 2004;3(2):443–61.CrossRef
13.
Corrao G, Tragnone A, Caprilli R, Trallori G, Papi C, Andreoli A, Di Paolo M, Riegler G, 2Rigo GP, Ferraù O, Mansi C, Ingrosso M, Valpiani D. Risk of inflammatory bowel disease attributable to smoking, oral contraception and breastfeeding in Italy: a nationwide case-control study. Int J Epidemiol. 1998;27(3):397–404.
14.
Di Castelnuovo A, Mazzaro D, Pesarin R, Salmaso L. Test di permutazione multidimensionali in problemi d'inferenza isotonica: un'applicazione alla genetica. Statistica. 2000;60(4):691–700.
15.
Edward V, Loftus JR. Clinical epidemiology of inflammatory bowel disease: incidence, prevalence and environmental influences. Gastroenterology. 2004;126:1504–17.CrossRef
16.
Finos L., Pesarin R, Salmaso L., Solari A. Nonparametric iterated combined tests for genetic differentiation. In Atti XLIH Riunione Scientica SIS 2004; CLEUP, Padova.
17.
Finos L, Salmaso L. Weighted methods controlling the multiplicity when the number of variables is much higher than the number of observations. J Nonparametr Stat. 2006;18(2):245–61.CrossRef
18.
Floreani A, Caroli D, Variola A. A 35-year follow-up of a large cohort of patients with primary biliary cirrhosis seen at a single Centre. Liver Int. 2011;31:361–8.CrossRefPubMed
19.
Floreani A, Cazzagon N, Franceschet I, Canesso F, Salmaso L, Baldo V. Metabolic syndrome associated with primary biliary cirrhosis. J Clin Gastroenterol. 2015;49:57–60.CrossRefPubMed
20.
Friedrich S, Brunner E, Pauly M. Permuting longitudinal data in spite of the dependencies. J Multivar Anal. 2017;153:255–65.CrossRef
21.
Fries W, Viola A, Manetti N, Frankovic I, Pugliese D, Monterubbianesi R, Samperi L. Disease patterns in late-onset ulcerative colitis: results from the IG-IBD “AGED study”. Dig Liver Dis. 2017a;49(1):17–23.CrossRefPubMed
22.
Fries W, Viola A, Manetti N, Frankovic I, Pugliese D, Monterubbianesi R, Scalisi G, Aratari A, Cantoro L, Cappello M, Samperi L, Saibeni S, Casella G, Mocci G, Rea M, Furfaro F, Contaldo A, Magarotto A, Calella F, Manguso F, Inserra G, Privitera AC, Principi M, Castiglione F, Caprioli F, Ardizzone S, Danese S, Papi C, Bossa F, Kohn A, Armuzzi A, D’Incà R, Annese V, Alibrandi A, Bonovas S, Fiorino G, Italian Group for the study of Inflammatory Bowel Disease (IG-IBD). Disease patterns in late-onset ulcerative colitis: results from the IG-IBD "AGED study". Digestive Liver Diseas. 2017b;49(1):17–23.CrossRef
23.
24.
Giacalone M., Zirilli A., Alibrandi A.. The use of permutation tests on large-sized datasets. In Proceedings of the 48th Scientific Meeting of the Italian Statistical Society; Università degli Studi di Salerno, Monica Pratesi and Cira Perna Editors;2016. ISBN: 9788861970618.
25.
Giacalone M, Zirilli A, Moleti M, Alibrandi A. Does the iodized salt therapy of pregnant mothers increase the children IQ? Empirical evidence of a statistical study based on permutation tests. Qual Quant. 2018;52:1423–35.CrossRef
26.
Holm S. A simple sequentially rejective multiple test procedure. Scand J Stat. 1979;6(2):65–70.
27.
Janssen A. Studentized permutation tests for non-iid hypotheses and the generalized Behrens-fisher problem. Stat Probabil Lett. 1997;36(1):9–21.CrossRef
28.
29.
Kobashi G, Hata A, Uchida K, Ishige T, Abukawa D, Tajiri H, Uchiyama K, Hirota Y, Nagai M, T. J. P. I. B. D. Research Group. A case-control study to detect genetic and acquired risk factors for pediatric inflammatory bowel disease. Int J Epidemiol. 2015;44(1):232.CrossRef
30.
Ludbrook J, Dudley H. Why permutation tests are superior to t and F tests in biomedicai research. Am Stat. 1998;52(2):127–32.
31.
Opdyke JD. Bootstraps, permutation tests and sampling orders of magnitude faster using SAS, Computational Statistics-WIREs 2013;5(5):390–405.
32.
Pajouheshnia R, Pestman WR, Teerenstra S, Groenwold RHH. A computational approach to compare regression modelling strategies in prediction research. BMC Med Res Methodol. 2016;16:1.CrossRef
33.
34.
Pesarin F. Multivariate Permutation Test. Chichester: Wiley and Sons; 2001.
35.
Pesarin F, Salmaso L. Permutation Tests For Univariate And Multivariate Ordered Categorical Data. Aust J Statistica. 2006;35(2):315–4.
36.
Pesarin F, Salmaso L. Permutation Tests for Complex Data. Theory, Applications and Software (a). Chichester: Wiley and Sons; 2010.CrossRef
37.
Pesarin F, Salmaso L. Finite-sample consistency of combination-based permutation tests with application to repeated measures designs. J Nonparametr Stat. 2010b;22:669–84.CrossRef
38.
Pesarin F, Salmaso L. The permutation testing approach: a review. Statistica. 2010;70(4):481–509.
39.
40.
Pesarin F. Permutation tests: multivariate. Wiley StatsRef: Statistics Reference Online; 2014. p. 1–15.
41.
Racioppi M, Salmaso L, Brombin C, Arboretti R, D'Agostino D, Colombo R, Serretta V, Brausi M, Casetta G, Gontero P, Hurle R, Tenaglia R, Altieri V, Bartoletti R, Maffezzini M, Siracusano S, Morgia G, Bassi PF. The clinical use of statistical permutation test methodology: a tool for identifying predective variables of outcome. Urol Int. 2015;94(3):262–9.CrossRefPubMed
42.
Ranzi T, Bodini P, Zambelli A, Politi P, Lupinacci G, Campanini MC, Dal Lago AL, Lisciandrano D, Bianchi PA. Epidemiological aspects of inflammatory bowel disease in a north Italian population: a 4 year prospective study. Eur J Gastroenterol Hepatol. 1996;8:657–61.PubMed
43.
Reiss PT, Lei H, Maarten M. Fast function-on-scalar regression with penalized basis expansions. Int J Biostat. 2010;6(1)
44.
Rezzani A. Big Data. Architettura, tecnologie e metodi per l'utilizzo di grandi basi di dati; Apogeo education, Maggioli Editore, Milano; 2013.
45.
Roy SN. On heuristic method of test construction and its use in multivariate analysis. Ann Math Stat. 1953;24:220–8.CrossRef
46.
47.
Rubin DB. Evaluations of the optimal discovery procedure for multiple testing. Int J Biostat. 2016;12(1):21–9.CrossRefPubMed
48.
Salmaso L. Permutation tests in screening two-level factorial experiments. Adv App Stat. 2005;5(1):91–110.
49.
Seibold H, Zeileis A, Hothron T. Model-based recursive partitioning for subgroup analyses. Int J Biostat. 2016;12(1):45–3.CrossRefPubMed
50.
Sturino J, Zorych I, Mallick B, Pokusaeva K, Chang YY, Carroll RJ, Bliznuyk N. Statistical methods for comparative phenomics using high-throughput phenotype microarrays. Int J Biostat. 2010;6(1).
51.
Tragnone A, Corrao G, Miglio F, Caprilli R, Lanfranchi GA. Incidence of inflammatory bowel disease in Italy: a nationwide population-based study. Int J Epidemiol. 1996;25:1044–52.CrossRefPubMed
52.
Ueno F, Nakayama Y, Hagiwara E, Kurimoto S, Hibi T. Impact of inflammatory bowel disease on Japanese patients’ quality of life: results of a patient questionnaire survey. J Gastroenterol. 2017;52(5):555–67.CrossRefPubMed
53.
Zirilli A, Alibrandi A. A permutation solution to compare two hepatocellular carcinoma markers. JP J Biostat. 2011;5:2,109–19.
54.
Zirilli A, Alibrandi A. The alteration of t,t-muconic acid and s-phenilmercapturic acid levels due to benzene exposure: an application of NPC test. JP J of Biostat. 2012;7(2):91–104.
Metadata
Title
Bonferroni-Holm and permutation tests to compare health data: methodological and applicative issues
Authors
Massimiliano Giacalone
Zirilli Agata
Paolo Carmelo Cozzucoli
Angela Alibrandi
Publication date
01-12-2018
Publisher
BioMed Central
Published in
BMC Medical Research Methodology / Issue 1/2018
Electronic ISSN: 1471-2288
DOI
https://doi.org/10.1186/s12874-018-0540-8

Other articles of this Issue 1/2018

BMC Medical Research Methodology 1/2018 Go to the issue

Research article

Comparison of methodological quality rating of systematic reviews on neuropathic pain using AMSTAR and R-AMSTAR

Research article

Study filters for non-randomized studies of interventions consistently lacked sensitivity upon external validation

Debate

Getting messier with TIDieR: embracing context and complexity in intervention reporting

Research article

Rapid response in health technology assessment: a Delphi study for a Brazilian guideline

Research article

Lasso regularization for left-censored Gaussian outcome and high-dimensional predictors

Research article

Quality assessment of systematic reviews on total hip or knee arthroplasty using mod-AMSTAR