Exploratory studies, confirmatory trials, and Bonferroni correction

Medical research is primarily performed using samples of humans, laboratory animals, or cells, but the studied phenomena are rarely limited to what can be observed in samples of these. On the contrary, the aim is almost always to learn about the population from which the sample was drawn. This leads to generalisation problems. Sampling variability makes the results from sample studies uncertain, and non-random sampling may induce bias. However, the uncertainty can, under certain conditions, be quantified. Quantification and reduction of uncertainty are thus essential components of successful scientific research. Statistical inference is a crucial port of modern empirical science.

A fundamental principle in statistical inference is that a hypothesis cannot be generated and confirmed using the same sample. Doing so induces selection bias in the confirmation testing and invalidates standard p-values and confidence intervals. The aim of studying a sample can therefore be either to generate (or re-generate) a new hypothesis or to confirm an old one. The first type of studies are known as exploratory, the second ones confirmatory. While the results from exploratory studies may be more uncertain, perhaps even in an unknown degree, the results data dependent, and relying on subjective assumptions, confirmatory studies are designed to give an objective result with a specified uncertainty level. This typically requires a prospective study with detailed pre-specification of endpoints and statistical analysis, randomised and concealed allocation to study groups, and masking of the study groups. In practice, this can be achieved only with an experimental study design, i.e. a clinical trial. Furthermore, to provide results with objectively specified uncertainty, the pre-specified statistical analysis needs to account for all possible multiplicity problems (simultaneous inference), which may have consequences for the experimental design, not least regarding the sample size.

For practical reasons, multiplicity is usually impossible to correct adequately for in observational studies (1). Nevertheless, many exploratory studies include Bonferroni corrections, typically without a clear and rational motivation. One example is the publication by Lenes et al. (2), which includes Bonferroni correction in order "to avoid false positive results". The consequence of the correction is, in this study, that the significance level for the Bonferroni corrected tests is lowered from 0.05 to 0.008. Not surprisingly, "[a]fter Bonferroni correction, no significant associations were found".

It is possible that multiplicity correction has a place in exploratory studies, but it is vital to understand that just performing a Bonferroni correction does not make an exploratory study confirmatory. Moreover, not providing a clear and rational motivation for the use of the multiplicity correction may make the critical reader question why a Bonferroni correction was implemented, whether the purpose just was to eliminate inopportune significant findings.

References

1. Bender R, Lange S. Adjusting for multiple testing--when and how? J Clin Epidemiol. 2001 Apr;54(4):343-9. doi: 10.1016/s0895-4356(00)00314-0. PMID: 11297884.

2. Lenes A, Klasen M, Bohorquez-Mendoza G, Gecht J, Sopka S, Vogt L. Psychological determinants of successful practical teaching: personality traits, self-efficacy, and subjective perception in a hands-on clinical skills course. BMC Med Educ. 2026 Jul 2;26(1):1069. doi: 10.1186/s12909-026-09788-2. PMID: 42393695.