Everywhere Valid Bounds on False Discovery Proportions in Conformal Inference
Ziang Song, Ying Jin, Emmanuel J. Candès · 2026
Read Everywhere Valid Bounds on False Discovery Proportions in Conformal Inference by Ziang Song, Ying Jin, Emmanuel J. Candès (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-20726.txt.
Introduction
Ziang Song, Ying Jin, and Emmanuel Candes establish bounds on false discovery shares in conformal inference that hold everywhere. Selecting test samples by p-values from this method is common in outlier spotting and candidate choice, but existing methods usually control only the expected share of false finds. Their finite-sample upper bounds that need no data model hold at once over all rejection thresholds. That permits arbitrary post-hoc threshold choice without breaking statistical guarantees, via a high-probability envelope build.
Catalog record
| CREATOR | Ziang Song, Ying Jin, Emmanuel J. Candès |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 79858 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.20726
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Everywhere Valid Bounds on False Discovery Proportions in Conformal Inference Ziang Song, Ying Jin, Emmanuel J. Candès Abstract Modern applications of conformal inference to multiple testing problems—such as outlier detection and candidate selection—often involve selecting test samples whose conformal p-values fall below a threshold. The quality of such methods is often measured by the false discovery proportion (FDP), defined as the fraction of incorrect selections. Existing approaches typically control the expected value of the FDP, using methods such as the Benjamini–Hochberg procedure. This approach fails to provide high-probability bounds on the realized false discovery proportion and invalidates statistical guarantees if the rejection threshold is selected after inspecting the data. This paper establishes finite-sample, distribution-free upper bounds on the FDP that hold simultaneously over all possible rejection thresholds, enabling arbitrary post hoc selection of the threshold. Simultaneous validity is achieved by constructing a high-probability envelope for the empirical distribution function of ‘null’ conformal p-values by sampling from their joint distribution. Furthermore, our framework allows practitioners to modulate the envelope’s shape, thereby producing tight bounds in rejection regions of primary interest. We use this flexible approach to derive simultaneous FDP upper bounds for both outlier detection and conformal selection. We demonstrate through synthetic and real-data experiments that the resulting bounds are both valid and substantially less conservative than those derived from existing approaches. 1 Introduction Conformal p-values have become a fundamental tool for distribution-free hypothesis testing ( 30 ; 2 ) . Their appeal is that they can be constructed using arbitrary predictive models while offering finite-sample guarantees. These properties have enabled broad applications ranging from outlier detection ( 2 ; 24 ) to scientific discovery ( 17 ; 16 ; 18 ) , and decision-making in which practitioners wish to select a subset of test samples obeying a user-specified criterion. A common strategy in these applications is to select test samples whose p-values (or scores) fall below a threshold. Such a selection rule can be viewed as a binary classifier that predicts whether each test sample is of interest. Thresholding decision rules naturally induce a family of binary decisions; as the threshold varies, the resulting decisions trace out curves of binary error metrics across the test samples, analogous to precision-recall curves in classification ( 5 ; 28 ) . The reliability of such selections is naturally quantified by the False Discovery Proportion (FDP) ( 3 ) , defined as the ratio of false discoveries to the total number of selections. In classification where we would like to single out samples with a label of interest, the FDP would be the fraction of predicted positives that are incorrect. This is simply one minus the precision. Controlling the FDP/precision is often crucial because each selected unit typically triggers costly follow-up actions, such as experimental validation, manual inspection, or downstream scientific investigation. We illustrate this through two motivating examples. Outlier Detection In outlier detection ( 2 ; 24 ) , one has access to a calibration set \mathcal{D}_{\textrm{calib}}=\{X_{i}\}_{i=1}^{n} where each X_{i} is independently and identically distributed (i.i.d.) from an unknown distribution \mathcal{P}_{0} . Given test data \mathcal{D}_{\textrm{test}}=\{X_{n+j}\}_{j=1}^{m} , the goal is to detect which points are likely to be outliers, i.e., drawn from a distribution different from \mathcal{P}_{0} . This task arises in fraud detection, anomaly detection in manufacturing, and the detection of unusual patterns in scientific data. The conformal inference framework provides a principled approach by calculating p-values \{p_{j}\}_{j=1}^{m} that quantify how well each test sample conforms to the calibration data; smaller values indicate stronger evidence of being an outlier. Common procedures ( 2 ; 24 ) use the thresholding rule \mathcal{R}(t)=\{j:p_{j}\leq t\} in which t\in[0,1] is a data-driven threshold computed via the Benjamini-Hochberg (BH) procedure ( 3 ) . Let \mathcal{H}_{0}=\{j:X_{n+j}\sim\mathcal{P}_{0}\} denote the index set of inliers so that a false discovery means a selection from \mathcal{H}_{0} . The FDP is then \displaystyle\textnormal{FDP}(t)=\frac{|\mathcal{R}(t)\cap\mathcal{H}_{0}|}{\max\{1,|\mathcal{R}(t)|\}}. (1) The numerator above |\mathcal{R}(t)\cap\mathcal{H}_{0}| is the number of inliers incorrectly selected as outliers. Maintaining a low FDP is often desired since the identified outliers often call for follow-up investigations, and a low FDP indicates that these costly investigations are rarely wasted ( 3 ; 9 ) . Drug Discovery Conformal p-values also arise in drug discovery pipelines ( 17 ; 16 ) , where early-stage screening aims to identify promising drug candidates from a large library. Machine learning predictions are useful in an initial screening stage to shortlist candidates; in later, more costly stages, only those shortlisted are carefully investigated to confirm promising cases. In this problem, the calibration set is \mathcal{D}_{\textrm{calib}}=\{(X_{i},Y_{i})\}_{i=1}^{n} where X_{i} is the biochemical feature of the i th drug, and Y_{i} is the experimentally evaluated property, such as the binding affinity to a disease target. For new test drugs \{X_{n+j}\}_{j=1}^{m} with unknown properties \{Y_{n+j}\}_{j=1}^{m} , the goal is to find those Y_{n+j} ’s that exceed a pre-specified threshold c_{n+j} . The conformal selection framework 17 constructs conformal p-values to make such selections. Here, a smaller p-value p_{j} provides strong evidence for a large outcome. The selection set is given by \mathcal{R}(t)=\{j:p_{j}\leq t\} , where t\in\mathbb{R} is determined by the BH procedure at a desired nominal level ( 3 ) . A false discovery is here a selected drug whose response Y_{n+j} is no larger than c_{n+j} , and thus the FDP is as in ( 1 ) with \mathcal{H}_{0}=\{j:Y_{n+j}\leq c_{n+j}\} . Controlling the FDP ensures a high precision or ‘hit rate’, which is critical to prioritizing experimental validation efforts. Limitations of existing methods Existing methods mainly focus on controlling the false discovery rate (FDR), which is the expected value of the FDP averaged over the randomness in the data. For a pre-specified error rate \alpha\in(0,1) , these methods determine a data-dependent threshold t(\alpha) such that \mathbb{E}[\textnormal{FDP}(t(\alpha))]\leq\alpha . These approaches suffer from two major limitations: • What’s the realized FDP? The realized FDP on a particular dataset may substantially exceed its expected value, even when the FDR is controlled ( 20 ) . FDR control only guarantees that the FDP is small on average over repeated realizations of the data, and thus provides no guarantee for the dataset at hand. However, practitioners often desire guarantees for the data at hand . They might want to know that with 90% or 95% chance, the FDP is below 20%. It may be of little comfort to know that over independent realizations of the data, the FDR may be at most 10% since this does not say anything about their study. • Fixed- \alpha guarantees only. These methods require the nominal level \alpha to be fixed in advance, which limits their flexibility. A practitioner may wish to choose \alpha adaptively after inspecting the data. For example, if \alpha=0.05 yields only one discovery, they may want to increase it to 0.1 to include more discoveries. Such post hoc adjustments, however, invalidate FDR control, see Figure 7 in the Appendix for a concrete illustration. In this work, we address these limitations by proposing a framework to construct simultaneously valid, high-probability upper bounds on the FDP that remain valid for all thresholds t\in[0,1] . This enables flexible, data-driven exploration while maintaining rigorous statistical guarantees. Given a confidence level \delta\in(0,1) , we provide a data-driven function \widehat{\text{FDP}}(t) obeying \mathbb{P}\Big(\textnormal{FDP}(t)\leq\widehat{\textnormal{FDP}}(t),~\forall t\in[0,1]\Big)\geq 1-\delta. That