A Leakage Bound for Confidence Sets after Black-Box Selection
Sayantan Banerjee · 2026
Read A Leakage Bound for Confidence Sets after Black-Box Selection by Sayantan Banerjee (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-26706.txt.
Introduction
Banerjee bounds the leakage of confidence sets after black-box selection, where the reported object is chosen by a preliminary search rather than fixed in advance. Classical selective inference works best when the search forms an explicit selection event, but many searches are opaque. The note shows selected-target noncoverage stays within nominal noncoverage plus a distance measure comparing the data's marginal law with its law given the selected object. At once, a mutual-information bound follows, and sample splitting emerges as the zero-leakage case.
Catalog record
| CREATOR | Sayantan Banerjee |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 28920 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.26706
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
A Leakage Bound for Confidence Sets after Black-Box Selection Sayantan Banerjee Abstract In many analyses the object reported at the end is not fixed in advance, but is chosen after a preliminary search over variables, subgroups, transformations, models or contrasts. Classical selective-inference methods are most effective when this search can be written as an explicit selection event. This note treats the less structured case in which the selection rule is a black box and inference is required for the target indexed by the selected object. We show that, for any fixed-target confidence procedure, selected-target noncoverage is bounded by the nominal fixed-target noncoverage plus the average total variation distance between the marginal law of the inferential data and its conditional law given the selected object. A mutual-information bound follows immediately. The result recovers sample splitting as the zero-leakage case and gives explicit guarantees for noisy screening through a Gaussian information bound. Thus the inferential cost of black-box selection is quantified by the information that the selected object carries about the inferential sample. Keywords: Black-box selection; confidence coverage; information leakage; post-selection inference; selected target; selective inference. 1 Introduction Classical confidence statements are usually justified for a parameter, contrast or hypothesis specified before the data are examined. In many current analyses this order is reversed. A preliminary screening stage first selects variables, subgroups, outcomes, transformations, models or contrasts, and formal inference is then reported only for the selected object. The screening stage may be an algorithmic search, a machine-learning pipeline, proprietary software, or a human analyst interacting with the data. If the selection step is ignored, the resulting confidence intervals and tests need not have their nominal interpretation. The effect of data-dependent selection on inference is well known. Leeb and Pötscher, 2005 showed that model selection can have a substantial impact on subsequent inference, even asymptotically. Berk et al., 2013 proposed universally valid post-selection inference (PoSI) by protecting against all possible selected submodels. The PoSI approach protects against a large class of possible selected models by widening intervals; the present note instead keeps the fixed-target procedure and charges the dependence between the selected object and the inferential sample. A complementary line of work develops selective inference conditional on the event that a specified procedure has selected a given model or hypothesis; see, among others, Fithian et al., 2014 , Lee et al., 2016 , Taylor and Tibshirani, 2015 and Tibshirani et al., 2016 . Randomization has also been used to smooth selection events and improve selective inference ( Tian and Taylor, 2018 ) . These approaches are most effective when the selection mechanism is mathematically available. The present note is motivated by screening rules that are not available in a tractable mathematical form. The rule may be stochastic, proprietary, adaptive over several analyst interactions, or based on an external black-box system. In such cases it may be unrealistic to condition on the exact selection event, or even to write that event down. The question is then not how to adjust for a particular known selection rule, but what can be guaranteed when the rule is treated only through its output. Let D denote the data used for final inference. A screening rule returns a selected object S^{\star} in a class \mathcal{S} . For each fixed s\in\mathcal{S} , let \theta_{s}(P) be the corresponding population target, and let C_{s}(D) be a confidence set computed from D . The selected target is \theta_{S^{\star}}(P) . Thus, after subgroup screening, the target is the population quantity for the selected subgroup; after variable screening, it may be the population projection parameter in the selected model. The target is random through the selected object. The main result is a finite-sample coverage inequality. Suppose that, before screening, the confidence procedure has noncoverage probability at most \alpha for every fixed object that could be selected. After screening, the target itself is random because it is indexed by the selected object. We prove that \text{selected-target noncoverage}\leq\alpha+\text{information leakage}. The leakage term is the total variation distance, averaged over the screening output, between the original law of the inferential data and its conditional law after the selected object is known. A mutual-information bound gives a simpler, coarser version. Thus sample splitting is the zero-leakage case, while noisy, randomized or information-limited screening gives intermediate guarantees. A related line of work studies validity under adaptive data analysis. Dwork et al., 2015 showed how a holdout sample can be reused while controlling overfitting, using ideas from algorithmic stability and privacy. Bassily et al., 2016 developed stability-based guarantees for adaptive queries, while Russo and Zou, 2016 and Xu and Raginsky, 2017 gave mutual-information bounds for bias and generalization error. These papers show that the information an adaptive procedure extracts from the data is a useful measure of its inferential cost. The distinction here is the object being controlled. Russo–Zou-type results control the bias of an adaptively chosen statistic, whereas the present result controls the probability of a selected-target noncoverage event. This event depends simultaneously on the selected object, the target attached to that object, and the confidence set reported after selection. The contribution of the paper is a finite-sample selected-target coverage bound for settings in which the selection rule is unavailable as a mathematical object. The bound identifies the exact quantity that selection adds to the usual fixed-target error: the change in the law of the inferential data after the selected object is known. This formulation is useful because it does not require the geometry or probability of a selection event. It only requires control of the joint law of the selected object and the data used for inference. In this sense, sample splitting, noisy screening and compressed screening are different ways of controlling the same leakage term. This perspective gives a design rule for black-box analyses. If the selected object is formed from data independent of the inferential sample, the leakage term vanishes. If the selected object is formed from a noisy or compressed summary, the leakage can be bounded in terms of the information carried by that summary. If the same inferential data are reused without restriction, the bound gives no nominal protection. The result therefore complements selective-inference methods that condition on a known selection event, while covering cases in which that event is not available in a tractable form. The rest of the paper is organized as follows. Section 2 introduces selected targets and leakage. Section 3 proves the main leakage bound and a finite sharpness example. Section 4 gives a noisy-screening consequence and points to a finite-message variant in Appendix B . Section 5 gives a small numerical illustration. Section 6 discusses interpretation and limitations. Proofs are given in Appendix A . 2 Selected targets and leakage Let D denote the data used to construct the final confidence set. In the simplest case, D=(Z_{1},\ldots,Z_{m}), where Z_{1},\ldots,Z_{m} are independent observations from an unknown distribution P\in\mathcal{P} on a measurable space \mathcal{Z} . The results below use only the law of D under P , and therefore also apply when D is one fold or one holdout part of a larger dataset. A screening rule returns a random element S^{\star}\in\mathcal{S}, where \mathcal{S} is a measurable selection space. An element s\in\mathcal{S} may represent a selected subgroup, model, variable set,