Know When to Abstain: Optimal Selective Classification with Likelihood Ratios
Alvin Heng & Harold Soh · 2025
Read Know When to Abstain: Optimal Selective Classification with Likelihood Ratios by Alvin Heng & Harold Soh (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-15008.txt.
Introduction
Alvin Heng and Harold Soh rework, in a 2025 paper, how a model may pass on a call it finds too unsure. Such passes lift trust, since a wrong call can cost more than no call. The pair ground the pass rule in a classic statistics result, the Neyman-Pearson lemma. That lemma says the best test rests on a ratio of likelihoods. Seen that way, many past pass rules turn out to be forms of one test, which ties them in one frame.
The hard case is shift: test data drawn from a new mix, not the training mix. That case is real-world common but little studied for pass rules. The team tests their ratio-based rules on sight and language tasks, with plain and vision-language models. Their rules beat past baselines in a steady way. Code is shared in public.
Catalog record
| CREATOR | Alvin Heng & Harold Soh |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 39 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-15008
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Know When to Abstain: Optimal Selective Classification with Likelihood Ratios Alvin Heng & Harold Soh Abstract Selective classification enhances the reliability of predictive models by allowing them to abstain from making uncertain predictions. In this work, we revisit the design of optimal selection functions through the lens of the Neyman–Pearson lemma, a classical result in statistics that characterizes the optimal rejection rule as a likelihood ratio test. We show that this perspective not only unifies the behavior of several post-hoc selection baselines, but also motivates new approaches to selective classification which we propose here. A central focus of our work is the setting of covariate shift, where the input distribution at test time differs from that at training. This realistic and challenging scenario remains relatively underexplored in the context of selective classification. We evaluate our proposed methods across a range of vision and language tasks, including both supervised learning and vision-language models. Our experiments demonstrate that our Neyman–Pearson-informed methods consistently outperform existing baselines, indicating that likelihood ratio-based selection offers a robust mechanism for improving selective classification under covariate shifts. Our code is publicly available at https://github.com/clear-nus/sc-likelihood-ratios . 1 Introduction Machine learning models are inherently fallible and can make erroneous predictions. Unlike humans, who can abstain from answering when uncertain, e.g., by saying “I don’t know”, predictive models typically produce a prediction for every input regardless of confidence. Selective classification aims to address this limitation by enabling models to abstain on uncertain inputs, thereby improving overall performance and robustness, for instance by deferring ambiguous cases to human experts. A wide range of methods have been proposed to determine whether a model should accept or reject an input. Common post-hoc approaches rely on heuristic confidence estimates, such as the maximum softmax probability [ 1 , 2 ] , logit margins [ 3 ] , or Monte Carlo dropout [ 1 , 4 ] . Other techniques assess a sample’s proximity to the training distribution [ 5 , 6 ] , under the assumption that samples farther from the data manifold are more likely to be misclassified. A separate line of work trains models with explicit abstention mechanisms, such as a rejection logit or head [ 7 , 8 , 9 ] . In this work we focus on post-hoc methods, which are model-agnostic and do not require specialized training. Despite the rich literature, two important gaps remain. First, while foundational results (such as [ 10 , 1 ] ) provide theoretical underpinnings for selective classification, there is a lack of general, principled guidance for designing effective selector functions in the context of modern deep networks. Second, most evaluations are conducted in the i.i.d. setting, where test data is assumed to follow the training distribution. This setup fails to reflect challenges in real-world scenarios, where distribution shifts, especially covariate shifts, are common, reducing the practical relevance of existing results. To address these challenges, we propose a new perspective rooted in the Neyman–Pearson lemma, a classical result from statistics that defines the optimal hypothesis test in terms of a likelihood ratio. We show that existing selectors can be interpreted as approximations to this test, and we use this insight to derive two new selectors, \Delta -MDS and \Delta -KNN, as well as a simple linear combination strategy. We evaluate our methods on a comprehensive suite of vision and language benchmarks under covariate shift, where the input distribution changes while the label space remains fixed. We focus on covariate shift for two key reasons: first, it is underexplored relative to semantic shifts [ 11 ] (which is well studied in the context of out-of-distribution detection [ 2 , 12 , 5 , 13 ] ); second, it is increasingly relevant in modern applications such as vision-language models (VLMs), where the label set is large and variable, rendering most practical shifts in deployment covariate in nature. Our results demonstrate that the proposed selectors outperform existing baselines and provide robust performance across distribution shifts, including on powerful VLMs like CLIP. In summary, our key contributions are: 1. We introduce for the first time a Neyman–Pearson-based framework for defining optimality in selective classification via likelihood ratio tests. 2. We unify several existing selector methods and propose two new selectors and a linear combination approach under this framework. 3. We conduct a thorough evaluation under covariate shift across vision and language tasks and demonstrate superior performance across both VLMs and traditional supervised models. 2 Background Selective Classification. Consider a standard classification problem with input space \mathcal{X}\subseteq\mathbb{R}^{d} , label space \mathcal{Y}=\{1,...,K\} , and data distribution \mathcal{D}_{\mathcal{X},\mathcal{Y}} over \mathcal{X}\times\mathcal{Y} . A selective classifier is a pair (f,g) , where f:\mathcal{X}\rightarrow\mathbb{R}^{K} is a base classifier, and g:\mathcal{X}\rightarrow\{0,1\} is a selector function that determines whether to make a prediction or abstain. Formally, (f,g)({\bm{x}})\triangleq\begin{cases}f({\bm{x}})&\text{if }g({\bm{x}})=1,\\ \text{abstain}&\text{if }g({\bm{x}})=0.\end{cases} (1) That is, the model abstains on input {\bm{x}} when g({\bm{x}})=0 . In practice, g is typically implemented by thresholding a real-valued confidence score: g_{s,\gamma}({\bm{x}})=\mathbbm{1}[s({\bm{x}})>\gamma], (2) where s:\mathcal{X}\rightarrow\mathbb{R} is a confidence scoring function (often adapted from OOD detection methods; see below), and \gamma is a tunable threshold. The performance of a selective classifier is typically evaluated using two metrics: Coverage: \displaystyle\phi_{s,\gamma}=\mathbb{E}_{{\bm{x}}\sim\mathcal{D}_{\mathcal{X},\mathcal{Y}}}[g_{s,\gamma}({\bm{x}})], (3) Selective Risk: \displaystyle R_{s,\gamma}=\frac{\mathbb{E}_{({\bm{x}},y)\sim\mathcal{D}_{\mathcal{X},\mathcal{Y}}}[\ell(f({\bm{x}}),y)\cdot g_{s,\gamma}({\bm{x}})]}{\phi_{s,\gamma}}, (4) where \ell(f({\bm{x}}),y) is the 0/1 loss [ 1 ] . Selective classification aims to optimize the tradeoff between selective risk and coverage by ideally reducing risk while maintaining high coverage. Improvements can stem from enhancing the base classifier f , or from refining the selector g to better identify error-prone inputs. In this work, we fix f to be a strong pretrained model and focus on designing more effective selector functions g . Covariate Shift. Covariate shift refers to a scenario where the marginal distribution over inputs, p({\bm{x}}) , changes between training and testing, while the label distribution p(y) remains unchanged. For example, a model trained on photographs of cats may face a covariate shift when evaluated on paintings of cats, as the input appearance changes but the semantic categories are preserved. This is in contrast to semantic shift , where both p({\bm{x}}) and p(y) change, typically due to the introduction of unseen classes. In this work, we focus on covariate shifts, which are increasingly relevant in modern applications such as VLMs, where the label set is large and can be adjusted to suit a given task. In such settings, distributional changes primarily manifest as covariate shifts, making them a critical yet underexplored challenge for robust selective classification. Out-of-Distribution (OOD) Detection. OOD detection is closely related to selective classification, as many selector functions s({\bm{x}}) are derived from or inspired by OOD scoring methods. Given an in-distribution (ID) data distribution p_{ID} , the goal of OOD detection is to construct a scoring function s:\mathcal{X}\rightarrow\mathbb{R} such that s({\bm{x}}) indicates the likelihood that {\bm{x}} originates from p_{ID} . A higher s({\bm{x}}) corresponds to higher confidence that {\bm{x}} is in-distribution. In selective classification, these scores are thresholded to determine whether to accept or abstain on a given input, as formalized in Eq. 2 . 3 Selective Classification via the Neyman–Pearson Lemma We begin by framing selective classification within the classical paradigm of hypothesis testing. Let \mathcal{H}_{0}:\mathcal{C} denote the hypothesis that the classifier makes a correct prediction, and \mathcal{H}_{1}:\neg\mathcal{C} the hypothesis that it makes