Nobody Puts Bonferroni in a Corner
Mårten Schultzberg · 2026
Read Nobody Puts Bonferroni in a Corner by Mårten Schultzberg (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-09256.txt.
Introduction
Schultzberg defends Bonferroni correction for online experimentation, a choice better than its reputation suggests. First, it is the simplest broadly usable method controlling family-wise error that also gives simultaneous confidence intervals for every metric. Guardrail metrics use intersection-union logic and cannot inflate false positives, so the family counts success metrics only. It is uniquely tractable for pre-experiment sample size calculations. From simulations and 1,296 Spotify experiments, power loss versus fancier methods depends on family specification and how many metrics are truly non-null.
Catalog record
| CREATOR | Mårten Schultzberg |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 51152 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.09256
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Nobody Puts Bonferroni in a Corner Mårten Schultzberg Abstract We argue that Bonferroni correction is a better choice for online experimentation than it is commonly given credit for. The case rests on four considerations. First, it is the simplest broadly implementable FWER-controlling method that produces unconditional simultaneous confidence intervals for every metric. Second, in a well-specified decision framework, guardrail and quality metrics use intersection-union logic and cannot inflate the false positive rate, so the Bonferroni denominator is the number of success metrics only, not the total metric count. Third, it is uniquely tractable for pre-experiment sample size calculations. Fourth, we contextualise the power cost empirically. Drawing on a simulation study and an empirical analysis of 1,296 experiments run on Spotify’s experimentation platform, Confidence, we show that the power loss relative to more sophisticated FWER methods depends on both how the correction family is specified and how many metrics are truly non-null. When guardrail metrics are incorrectly included in the family, Holm and Hommel are nearly indistinguishable from Bonferroni. When the family is correctly restricted to success metrics only, they gain roughly 4–5 percentage points in ship rate (the fraction of experiments where the treatment is deployed). When few metrics are truly non-null, the gap narrows to near zero regardless of method. 1 Introduction Most experimenters know that testing many hypotheses simultaneously inflates the false positive rate. Corrections exist to address this, but they carry a reputation for being costly and inconvenient ( Perneger 1998 ) , so they are often skipped. When they are applied, practitioners sometimes reach for methods that seem more powerful on paper, without considering whether those methods actually support the inferences they want to make. We argue that this reputation is largely undeserved ( VanderWeele and Mathur 2018 ) , and that Bonferroni correction specifically is a better choice for online experimentation than it is commonly given credit for. The organizing claim is simple. Bonferroni is the simplest broadly implementable FWER-controlling method that remains coherent across the full experimentation workflow: confidence intervals (CIs), decision logic, sample size planning, and sequential monitoring. The case rests on four considerations. First, it is the simplest such method that produces unconditional simultaneous confidence intervals for every metric. Second, in a well-specified decision framework, the correction denominator is smaller than it appears: guardrail and quality metrics use intersection-union logic and cannot inflate the false positive rate, so the Bonferroni denominator is the number of success metrics only , not the total metric count. Third, it is uniquely tractable for pre-experiment sample size calculations. Fourth, the power loss relative to more sophisticated FWER methods depends on both how the correction family is specified and how many metrics are truly non-null: when guardrail metrics are incorrectly included in the family, Holm and Hommel are nearly indistinguishable from Bonferroni; when the family is correctly restricted to success metrics only, they gain roughly 4–5 percentage points in ship rate (the fraction of experiments where the treatment is deployed); and when few metrics are truly non-null, the gap narrows to near zero regardless of method. Bonferroni also composes cleanly with group sequential testing across multiple metrics; this is discussed in Appendix C. The first three of these (CI coherence, the decision-framework denominator, and sample size tractability) constitute the paper’s core contribution. The fourth contextualizes the cost. We support this argument with a simulation study and an empirical analysis of 1,296 experiments (1,840 treatment-control comparisons) run on Spotify’s experimentation platform, Confidence. 2 Background: The Multiple Testing Problem When an experiment tests m hypotheses simultaneously at significance level \alpha , the probability of at least one false positive (the family-wise error rate, or FWER) inflates well beyond \alpha . Under independence, \text{FWER}=1-(1-\alpha)^{m} . For m=8 metrics at \alpha=0.05 , this is approximately 34%. A high false positive rate undermines practitioner trust in the experimentation platform. Practitioners who recognize it begin discounting experiment readouts and spending time validating results through alternative means, slowing the platform’s influence on decisions. If the elevated rate goes unrecognized, teams ship ineffective changes, compounding the problem over time. Several correction methods exist to control FWER or the related false discovery rate (FDR): 1. Bonferroni : Test each hypothesis at \alpha/m . Requires no assumptions about the joint distribution of test statistics; controls FWER exactly under independence and conservatively under positive dependence. 2. Holm : Step-down procedure, uniformly more powerful than Bonferroni while still controlling FWER. 3. Hochberg / Hommel : Step-up procedures, more powerful still, with Hommel being the most powerful of the closed-testing family under certain conditions. 4. Benjamini–Hochberg (BH) : Controls FDR rather than FWER. More powerful than all of the above, but answers a different question. 5. Benjamini–Yekutieli (BY) : FDR control without positive dependency assumptions. Much more conservative than BH. The narrative in both academic and industry literature tends to cast Bonferroni as the naive baseline that everyone has moved past, with BH or Hommel positioned as the modern choice. We argue this framing does not transfer cleanly to the product experimentation setting, and the FDR versus FWER distinction is where the mismatch is sharpest. FDR control was developed for genomics, a setting where thousands of hypotheses are tested simultaneously and many true effects are expected among them. In that context, permitting a controlled proportion of false discoveries among significant results is sensible: the goal is discovery at scale, and the false positives are diluted across a large portfolio. Product experimentation is structurally the opposite. Berman and Van den Bulte 2022 , analyzing nearly 5,000 effects from Optimizely experiments, estimate that roughly 70% of tested effects are true nulls and that correctly run single-metric tests at 5% significance already produce false discovery rates of 18–25%. This is not a consequence of statistical malpractice. It is a consequence of the base rate. Most product changes simply do not move metrics. In that environment, FDR control’s permissiveness compounds the problem rather than addressing it. FDR methods explicitly allow a higher proportion of false discoveries among significant results. Applied to a setting where false discoveries are already endemic due to a high null fraction, they lower a bar that was already producing too many false positives. A higher shipping rate under BH is not evidence that more true effects are being captured. It is evidence that the threshold has been relaxed. From a decision-theoretic perspective, the choice depends on cost structure and the prior probability of true effects. Under an additive loss with cost c_{1} per false positive and c_{2} per false negative, stringent thresholds are favoured when c_{1}/c_{2} is large, that is, when false positives are individually costly relative to missed effects. FWER control is designed for exactly this regime: it bounds the probability of any false positive, which is appropriate when each false positive causes a specific, consequential error rather than being diluted across a large portfolio of discoveries. FDR control was designed for the opposite regime, where many true effects are expected and some false discoveries among them are tolerable. The high null fraction in product experimentation undermines the case for FDR on both grounds. When roughly 70% of tested effects are null, FDR’s permissiveness produces a lower positive predictive value: a larger share of its additional rejections are false