Exact statistical tests using integer programming: Leveraging an overlooked approach for maximizing power for differences between binomial proportions

Stef Baas · 2025

Read Exact statistical tests using integer programming: Leveraging an overlooked approach for maximizing power for differences between binomial proportions by Stef Baas (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-13689.txt.

Introduction

Stef Baas and three co-authors build exact tests for comparing two rates, such as success rates in medical trials, with the most power math allows. Common tests are either too loose or too strict, which wastes power. Their work came out in twenty twenty-five.

The team uses whole number programming to draw the best decision edge for a test. Their test lifts average power as high as it can go while holding false alarm rates in check across the full null range. Number trials showed steady power gains over the classic exact test and over strong rivals. For large samples where the build gets heavy, a mid value variant is the best looser option, and a web tool aids use.

Catalog record

CREATORStef Baas
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE61 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-13689

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Exact statistical tests using integer programming: Leveraging an overlooked approach for maximizing power for differences between binomial proportions Stef Baas Abstract Traditional hypothesis testing methods for differences in binomial proportions can either be too liberal (Wald test) or overly conservative (Fisher’s exact test), especially in small samples. Regulators favour conservative approaches for robust type I error control, though excessive conservatism may significantly reduce statistical power. We offer fundamental theoretical contributions that extend an approach proposed in 1969, resulting in the derivation of a family of exact tests designed to maximize a specific type of power. We establish theoretical guarantees for controlling type I error despite the discretization of the null parameter space. This theoretical advancement is supported by a comprehensive series of experiments to empirically quantify the power advantages compared to traditional hypothesis tests. The approach determines the rejection region through a binary decision for each outcome dataset and uses integer programming to find an optimal decision boundary that maximizes power subject to type I error constraints. Our analysis provides new theoretical properties and insights into this approach’s comparative advantages. When optimized for average power over all possible parameter configurations under the alternative, the method exhibits remarkable robustness, performing optimally or near-optimally across specific alternatives while maintaining exact type I error control. The method can be further customized for particular prior beliefs by using a weighted average. The findings highlight both the method’s practical utility and how techniques from combinatorial optimization can enhance statistical methodology. Keywords Average power, Berger and Boos’ procedure, Boschloo’s test, Fisher’s exact test, Knapsack problem, Maximin Power. 1 Introduction Statistical tests for comparing response proportions in a randomized trial are among the first concepts presented in statistics courses. They are also frequently encountered in practice, as binary outcome measures are very common in confirmatory two-arm superiority clinical trials, with a frequency of use of at least 28% reported in Thompson et al. 2025 . One of the most commonly used exact tests to compare response proportions is Fisher’s exact test ( Fisher 1934 ) . Fisher’s exact test is a conditional test ensuring type I error control for each potential total sum of treatment successes. This property makes Fisher’s exact test a preferred option from a regulatory perspective, particularly when testing in small samples or close to the parameter boundaries (i.e., for proportions close to 0 or 1). Using a query, we found that around 25\%-30\% (about 2000 analyses) of two-sample binary or categorical data analyses in clinical trials publicly reported on https://clinicaltrials.gov used Fisher’s exact test (see Appendix A ). This value can be seen as a lower bound for the amount of studies that can benefit from the alternative tests presented in this paper. However, Fisher’s exact test is also often viewed as (too) conservative(see, e.g., Section 3.5.6 of Agresti 2002 ) and consequently can be expected to be substantially less powerful than other tests. Another commonly used test for this problem is the asymptotic Z-test (or Wald test) which can yield higher power but also be (too) liberal (i.e., leading to type I error rate inflation), especially if using the Z-unpooled test in small samples ( Eberhardt and Fligner 1977 ) . Many tests exist for the two-sample binomial proportions hypothesis testing problem, however, which test has highest power depends on the configuration of success probabilities and treatment group sizes; no test is guaranteed to have highest power out-of-the-box. Mehrotra et al. 2003 show that a poor choice of test statistic in this setting can actually lead to a test with lower unconditional power than Fisher’s exact test. In this paper, we propose an integer programming-based approach to construct unconditional exact one-sided two-sample tests for a difference in binomial proportions that ensure a maximum power measure of interest. The underlying optimization problem can be classified as a knapsack optimization problem, where the value of items included in the knapsack is maximized, given that the combined weight of all included items does not exceed a threshold. In our case, items packed correspond to datasets for which the test yields a rejection, the value of the knapsack equals power, and the weight of the knapsack equals type I error. This integer programming approach to construct binomial tests is not new, and was, to the best of our knowledge, first proposed in Paroush 1969 , who considered six formulations of the knapsack problem but did not consider evaluation of the corresponding type I error and power under these formulations. The integer programming approach was also more recently considered for the two-sided test on proportions in Keer 2023 . We propose one-sided knapsack-based tests that maximize: (1) the average power, where the power is averaged over the parameter space under the alternative hypothesis and has an analytic expression, (2) a weighted average power, which could reflect prior beliefs of the experimenter or which tailors the test to an area covering a specific alternative hypothesis and (3) the minimum power over a set of parameters, noting that this maximin approach is different from the maximin test considered in Keer 2023 , who maximized the minimum deviation of the type I error rate from the significance level. Our theoretical contributions to the development of this optimization-based approach to construct exact tests are of crucial importance to facilitating wider uptake in practice. First, we propose a method to construct a (non-trivial) p-value based on this test and substantiate the approach by theoretical results from the literature. Similarly to the approach sketched in Section 4.3.2.2 of Keer 2023 , our strategy is to explicitly enforce the nesting of rejection regions when iterating over different possible p-values. Second, for the one-sided case we make use of an analytic expression to compute average power exactly, and furthermore enforce Barnard’s convexity condition. Lastly, we provide a detailed analysis of the type I error rate and power profiles for the approach, considering both balanced and unbalanced treatment group size configurations, and comparing the approach with state-of-the-art tests in the literature such as Berger and Boos’ modification of the pooled-z test and Boschloo’s test (which were not considered in Keer 2023 ). Our paper presents comparisons for these tests for multiple treatment group configurations and up to a trial size of 100 participants and furthermore shows the performance of the knapsack-based approach for a clinical trial application involving 280 participants. The current paper is organized as follows. Section 2 describes the data model, hypotheses, and theoretical properties for the one-sided two-sample binomial testing problem. Section 3 describes the proposed average power knapsack-based test, other (comparator) knapsack-based tests, and comparator unconditional exact tests. Section 4 provides a detailed type I error and power comparison of the considered tests. Section 5 considers a case study based on the Merck Research Laboratories Trial, also considered in Mehrotra et al. 2003 , where, next to the one-sided p-values under the considered tests, we also provide a detailed type I error and power comparison. Section 6 provides a discussion and concludes the paper. 2 One-sided two-sample binomial exact testing problem In this section, we formalize the one-sided two-sample binomial testing problem which formalizes the problem of comparing response proportions mentioned in Section 1 . Let the tuple of unknown success probabilities

Read the complete text