Unified Conformalized Multiple Testing with Full Data Efficiency
Yuyang Huo, Xiaoyang Wu, Changliang Zou, Haojie Ren · 2025
Read Unified Conformalized Multiple Testing with Full Data Efficiency by Yuyang Huo, Xiaoyang Wu, Changliang Zou, Haojie Ren (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-12085.txt.
Introduction
Yuyang Huo, Xiaoyang Wu, Changliang Zou, and Haojie Ren rebuild conformal multiple testing so that no data are wasted, in this 2025 paper. Conformal methods make predictions with error control and no model assumptions, prized where thousands of hypotheses compete. The tax was efficiency: existing procedures split the data, using one part to build scores and another to calibrate, and each setting needed its own recipe.
Their unified framework feeds everything, confirmed nulls, confirmed alternatives, and unlabeled cases, into both jobs. Scores are built from all of it, and p-values are calibrated by a full permutation strategy. Power rises twice over: sharper scores from more data, and a larger calibration set, while the false discovery rate stays controlled. The framework doubles as a design principle, even choosing automatically among candidate conformal procedures without extra splitting. Simulations across scenarios showed gains in efficiency and adaptability.
Catalog record
| CREATOR | Yuyang Huo, Xiaoyang Wu, Changliang Zou, Haojie Ren |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 117 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2508-12085
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Unified Conformalized Multiple Testing with Full Data Efficiency Yuyang Huo, Xiaoyang Wu, Changliang Zou, Haojie Ren Abstract Conformalized multiple testing offers a model-free way to control predictive uncertainty in decision-making. Existing methods typically use only part of the available data to build score functions tailored to specific settings. We propose a unified framework that puts data utilization at the center: it uses all available data—null, alternative, and unlabeled—to construct scores and calibrate p-values through a full permutation strategy. This unified use of all available data significantly improves power by enhancing non-conformity score quality and maximizing calibration set size while rigorously controlling the false discovery rate. Crucially, our framework provides a systematic design principle for conformal testing and enables automatic selection of the best conformal procedure among candidates without extra data splitting. Extensive numerical experiments demonstrate that our enhanced methods deliver superior efficiency and adaptability across diverse scenarios. Keywords: Conformal inference, Exchangeability, False discovery rate, Permutation, Outlier detection 1 Introduction In recent years, conformalized multiple testing has attracted much attention in statistics and machine learning society. It provides uncertainty quantification for identifying multiple individuals with unobserved labels of interest when implementing a black-box model, such as in the scenarios of outlier detection ( Bates et al. 2023 ) and sample selection ( Jin and Candès 2023 ) . For example, detecting reliable data for large language models training with statistical guarantees is essential for accurately evaluating benchmark performance ( Dekoninck et al. 2024 ) ; and in drug discovery, researchers often utilize deep learning models to identify potential drugs from a large candidate pool, and the false errors are hoped to be controlled ( Dara et al. 2022 ) . Suppose we observe independent and identically distributed (i.i.d.) data pairs (X,Y) , where X\in\mathcal{X} is a covariate and Y\in\{0,1\} is the binary response. Generally, conformalized multiple testing involves three types of datasets: a negative (null) labeled dataset {\mathcal{D}}_{0}=\{(X_{i},Y_{i})\}_{i=1}^{n_{0}} with all Y_{i}=0 , a positive (non-null/alternative) labeled dataset {\mathcal{D}}_{1}=\{(X_{i},Y_{i})\}_{i=n_{0}+1}^{n_{0}+n_{1}} with all Y_{i}=1 (this dataset is optional) and a test/unlabeled dataset {\mathcal{D}}_{u}=\{X_{j}\}_{j=n+1}^{n+m} with unobserved responses, where n=n_{0}+n_{1} . Let the null labeled samples be indexed by {\mathcal{L}}_{0} , the non-null labeled samples by {\mathcal{L}}_{1} , and the unlabeled samples by {\mathcal{U}} . Each test point X_{j} , j\in{\mathcal{U}} is then associated with a hypothesis: H_{0,j}:Y_{j}=0\quad\text{v.s.}\quad H_{1,j}:Y_{j}=1. (1) A rejection set {\mathcal{R}} is then determined from {\mathcal{U}} such that the false discovery rate (FDR) ( Benjamini and Hochberg 1995 ) is controlled at a given level \alpha\in(0,1) : \operatorname{FDR}={\mathbb{E}}\left[\frac{\sum_{j\in{\mathcal{U}}}{\mathbb{I}}\{j\in{\mathcal{R}},Y_{j}=0\}}{1\vee|{\mathcal{R}}|}\right]\leq\alpha. A conformalized testing procedure generally consists of the following three key steps. • Score construction : A non-conformity score function S:\mathcal{X}\to\mathbb{R} is constructed based on a predictive model trained on labeled data ( Vovk et al. 2005 ) . A large value of S(X_{i}) indicates evidence against the null hypothesis. For example, S(X_{i})=\widehat{\Pr}(Y_{i}=1\mid X_{i}) is the predicted probability from a machine learning method such as random forest. • P-value computation : Given a calibration set {\mathcal{D}}_{c}\subseteq{\mathcal{D}}_{0} with index set {\mathcal{C}} , the conformal p-value ( Bates et al. 2023 ) for each test point j\in{\mathcal{U}} is computed by comparing the score S(X_{j}) with those from {\mathcal{C}} as p_{j}=\frac{\sum_{i\in{\mathcal{C}}\cup\{j\}}{\mathbb{I}}\{S(X_{j})\leq S(X_{i})\}}{|{\mathcal{C}}|+1},\quad j\in{\mathcal{U}}. (2) • Testing procedure : With the conformal p-values \{p_{j}\}_{j\in{\mathcal{U}}} , one applies some appropriate multiple testing rules, such as the well-known Benjamini-Hochberg (BH) procedure ( Benjamini and Hochberg 1995 ) or its variants, to determine the rejection set {\mathcal{R}} . The key principle of conformalized multiple testing is to ensure exchangeability among null scores. This guarantees that p-values under the null hypothesis are super-uniform and exhibit a well-structured dependence. Together, these properties facilitate finite-sample FDR control in a distribution-free manner. Numerous methods have been developed along this principle for various settings. Below, we highlight three representative examples. Example 1 (Basic conformal p-value) As pioneers of conformalized multiple testing, Bates et al. 2023 considered a novelty detection setting without non-null samples ( {\mathcal{D}}_{1} ). To ensure exchangeability, the null data {\mathcal{D}}_{0} is divided into a training set {\mathcal{D}}_{t} and a calibration set {\mathcal{D}}_{c} . A one-class classifier is trained on {\mathcal{D}}_{t} to produce a score function S . Conformal p-values are computed via ( 2 ) and then the BH procedure is applied. Example 2 (AdaDetect) Marandon et al. 2024 improved upon Bates et al. 2023 by incorporating the test data {\mathcal{D}}_{u} into score construction. Similarly, they split {\mathcal{D}}_{0} into {\mathcal{D}}_{t} and {\mathcal{D}}_{c} . However, instead of a one-class classifier, a binary classifier is trained to distinguish between {\mathcal{D}}_{t} and the combined dataset {\mathcal{D}}_{c}\cup{\mathcal{D}}_{u} , using its output as the score function S . The p-value computation and testing procedure remain the same with Example 1. Example 3 (Integrative conformal p-value) Liang et al. 2024b enhanced the quality of score functions by assuming access to both {\mathcal{D}}_{0} and {\mathcal{D}}_{1} . After splitting both datasets into training and calibration sets, they train separate one-class classifiers s_{0} and s_{1} on the training sets of {\mathcal{D}}_{0} and {\mathcal{D}}_{1} , respectively. For each test point j\in{\mathcal{U}} , test-specific score functions S^{(j)} are constructed by integrating s_{0} and s_{1} . Due to the complicated dependence structure of integrative p-values, a conditional calibration procedure ( Fithian and Lei 2022 ) is used as testing rule. The above examples illustrate how existing conformalized multiple testing approaches utilize subsets of the data, leading to varied strategies for score function construction, p-value computation, and testing procedures. As each method is tailored to its specific setting, direct connections between them are generally unclear. This naturally raises a central question: whether it is possible to establish a unified and flexible framework that encompasses those existing methods while also enabling principled development of novel procedures that effectively balance the computation and the use of data for a specific setting? 1.1 Preview of contributions In this paper, we propose a unified framework that systematically incorporates existing conformalized multiple testing approaches, with a focus on how they utilize available data. By adopting a full permutation strategy, our framework imposes no restrictions on the form of the score function and ensures finite-sample, distribution-free FDR control. Building on this foundation, we introduce general design principles adaptable to diverse practical scenarios, leading to two key applications: Enhanced approaches by fully utilizing data information. Current methods typically use only a fraction of the available data for score construction and calibration (p-value computation). We instead use all three datasets— {\mathcal{D}}_{0} , {\mathcal{D}}_{1} , and {\mathcal{D}}_{u} —to build the score and keep the entire {\mathcal{D}}_{0} for calibration. This straightforward change increases power by improving the score and enlarging the calibration set. Adaptive selection of testing approaches. Our framework also enables a data-driven selection strategy, automatically picking the most powerful procedure from several candidates. Reusing the same data for selecting model and conducting tests would normally inflate the FDR; we avoid this by embedding the selection step into the score construction. The resulting procedure keeps finite-sample FDR control without extra data splitting. 1.2 Related works Conformal inference. Conformal inference originally exploits the exchangeability of data to produce distribution-free prediction intervals ( Vovk et al. 2005 ; Lei et al. 2018 ) . Among many others, recent developments include advanced score construction ( Romano et al. 2019 ; Chernozhukov et al. 2021 ) , valid cross validation