Inference conditional on selection: a review

Anna Neufeld, Ronan Perry, Daniela Witten · 2026

Read Inference conditional on selection: a review by Anna Neufeld, Ronan Perry, Daniela Witten (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-09779.txt.

Introduction

Neufeld, Perry, and Witten review selective inference, for questions that are a function of the data. Guesses and values are often picked from the data rather than set in advance, and standard methods then fail to deliver nominal range cover. Three cases need picked fixes: a winner, a tree-region mean, and a gap in means between a pair of clusters. The authors argue that limited cover matters for science here, then review methods providing them, showing them in tests and on single-cell RNA data.

Catalog record

CREATORAnna Neufeld, Ronan Perry, Daniela Witten
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE101194
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.09779

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Inference conditional on selection: a review Anna Neufeld, Ronan Perry, Daniela Witten Abstract In this article, we review selective inference , a set of techniques for inference when the statistical question asked is a function of the data . This setting often arises in contemporary scientific workflows, where hypotheses and parameters may be selected from the data, rather than specified in advance. In this setting, classical inferential techniques do not achieve “classical” guarantees, such as nominal coverage of confidence intervals. We consider three examples for which selective inference solutions are required: inference on a “winner”, inference on the mean of a region in a regression tree, and inference on the difference in means between a pair of clusters. We argue that conditional guarantees are of scientific interest in such settings. We then review and draw connections between several approaches that provide such guarantees. Finally, we illustrate these approaches in simulation and through an application to single-cell RNA sequencing data. 1 Introduction In classical statistics, models, hypotheses, and parameters are specified before the data are observed. By contrast, modern scientific practice often involves exploring the data before selecting models, hypotheses, or parameters that appear promising. When traditional statistical methods such as t-tests or Wald intervals are applied without accounting for this data-driven selection, they fail to provide guarantees such as Type 1 error control or nominal coverage. This issue has been partially blamed for the so-called replication crisis in science ( 6 ) . The practice of naively using the same data both to select a statistical question, and to answer it, is sometimes referred to as double dipping . Concerns about double dipping date back at least to the 1940s, when 37 noted, “we constantly take hints from the data regarding the choice of hypotheses to be tested from the same data–although we know that the degree of confidence to be placed in the test is affected by that practice.” Decades later, 10 called this practice “a quiet scandal in the statistical community.” Despite such warnings, double dipping remains common in modern scientific practice, as recently documented in the fields of neuroscience ( 38 ) , genomics ( 43 ) , and ecology ( 11 ) . It may be that the statistical community has only itself to blame for the pervasive nature of double dipping: as noted in Chapter 1 of 41 , introductory statistics textbooks routinely teach classical p-values for linear model coefficients while simultaneously encouraging iterative model selection (i.e., variable selection via stepwise regression), without emphasizing that such p-values are invalid after this iterative process. Of course, one could avoid double dipping by simply restricting attention to pre-specified hypotheses. But as 84 remarked, “to concentrate on confirmation, to the exclusion or submergence of exploration, is an obvious mistake. Where does new knowledge come from? How can an undetected criminal be put on trial?” The field of selective inference aims to provide tools for valid inference on parameters that were selected based on data exploration. In this paper, we review recent advances in the field. Much of the foundational work in selective inference relates to the problem of inference after variable selection in linear regression ( 46 ; 8 ; 44 ; 80 ; 79 ) . Recently, selective inference solutions have been developed for a much broader set of problems, such as inference after changepoint detection ( 33 ; 34 ) , inference after outlier or anomaly detection ( 16 ; 60 ) , and inference after detecting “regions” of interest ( 7 ; 61 ) . To illustrate the breadth of settings in which a need for selective inference arises, we frame this article around the following three motivating examples. Example 1: Inference on a winner Suppose we observe realizations Y_{1},\ldots,Y_{n} corresponding to n different candidates , with Y_{k}\overset{\mathrm{ind.}}{\sim}\mathrm{N}(\mu_{k},\sigma^{2}) . The parameter of interest is \mu_{\hat{k}} , where \hat{k}=\mathop{\mathrm{arg\,max}}_{k}Y_{k} . As we are most likely to select an index k precisely when Y_{k} is unusually large, it follows that Y_{\hat{k}} is biased for \mu_{\hat{k}} . Therefore, the classical 1-\alpha confidence interval for \mu_{k} corresponding to some fixed index k , which takes the form \left[Y_{k}\pm z_{1-\alpha/2}\sigma\right], (1) does not attain 1-\alpha coverage when k is replaced with \hat{k} . This issue is often referred to as the winner’s curse , and has been considered by a number of authors ( 90 ; 26 ; 2 ; 91 ; 92 ) . Example 2: Inference on a regression tree Regression tree algorithms such as CART ( 9 ) partition observations \left\{(X_{i},Y_{i})\right\}_{i=1}^{n} into regions in covariate space with similar Y_{i} values. We consider inference on \mu_{\hat{R}}=\frac{1}{|i:X_{i}\in{\hat{R}}|}\sum_{i:X_{i}\in{\hat{R}}}\mathbb{E}[Y_{i}] , where {\hat{R}} is one such region. Assuming that Y_{i}\overset{\mathrm{ind.}}{\sim}\mathrm{N}(\mu_{i},\sigma^{2}) , the classical 1-\alpha confidence interval for \mu_{R} corresponding to some fixed region R takes the form \left[\left(\frac{1}{|i:X_{i}\in R|}\sum_{i:X_{i}\in R}Y_{i}\right)\pm z_{1-\alpha/2}\sigma/\sqrt{|i:X_{i}\in R|}\right]. (2) This interval does not attain 1-\alpha coverage when R is replaced with \hat{R} . This issue has been considered by 3 ; 85 ; 55 and 5 . Example 3: Inference after clustering The analysis of single-cell RNA sequencing data often centers around identifying genes that are differentially expressed across cell types. Since the cell types are not known a priori , scientists (i) estimate the cell types by clustering the rows of X , a matrix of expression counts for n cells and p genes; and (ii) test the expression counts of the j th gene, X_{j} , for association with the estimated cluster labels. Assuming that X_{ij}\overset{\mathrm{ind.}}{\sim}\mathrm{N}(\mu_{ij},\sigma^{2}) , the classical 1-\alpha confidence interval for a difference in means of feature j across two pre-specified groups A and B , denoted \frac{1}{|i\in A|}\sum_{i\in A}\mathbb{E}[X_{ij}]-\frac{1}{|i\in B|}\sum_{i\in B}\mathbb{E}[X_{ij}] , takes the form \left[\left(\frac{1}{|i\in A|}\sum_{i\in A}X_{ij}-\frac{1}{|i\in B|}\sum_{i\in B}X_{ij}\right)\pm z_{1-\alpha/2}\sigma\sqrt{1/|i\in A|+1/|i\in B|}\right]. (3) This interval does not attain 1-\alpha coverage when the pre-specified groups A and B are replaced with clusters estimated from the data. This issue has been described as one of the “grand challenges” in single cell data science ( 43 ) . In what follows, we highlight key considerations that arise in selective inference, as well as pros and cons of various selective inference frameworks, through the lens of these three examples. The rest of this article is organized as follows. In Section 2 , we discuss conditional versus unconditional inferential guarantees, and argue using a simulation related to Example 1 that conditional guarantees are more scientifically meaningful. We therefore focus the remainder of the review on methods that achieve conditional guarantees. In Sections 3 and 4 , we review several distinct approaches to achieve conditional guarantees, and demonstrate that they can all be cast as special cases of a “unifying recipe”. A simulation study related to Example 1 is in Section 5 , and an application to single-cell RNA sequencing data related to Example 1 is in Section 6 . We conclude with a discussion of challenges and future directions in Section 7 . The Appendix contains details related to empirical results. 2 Unconditional and conditional coverage Suppose that we wish to construct a confidence interval around a parameter selected from the data, as in Examples 1 - 1 . What inferential guarantee should we demand of this interval? To formalize this question, suppose that our data is

Read the complete text