Conformal Prediction Meets Long-tail Classification

Shuqi Liu, Jianguo Huang, Luke Ong · 2025

Read Conformal Prediction Meets Long-tail Classification by Shuqi Liu, Jianguo Huang, Luke Ong (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-11345.txt.

Introduction

Shuqi Liu, Jianguo Huang, and Luke Ong fix a fairness flaw in conformal prediction when classes are heavily skewed, in a paper from 2025. Conformal prediction wraps a model's guess in a set of options, sized by confidence. Its promise holds on average, but rare classes often get sets that miss the truth too often.

Their method, Tail-Aware Conformal Prediction, uses the skew itself to close the gap between common and rare classes. Theory shows a smaller coverage gap than standard methods. A soft version adds reweighting to balance coverage across all classes at once. The frame works with many scoring rules, and tests on skewed benchmarks backed the claims.

Catalog record

CREATORShuqi Liu, Jianguo Huang, Luke Ong
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE54 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2508-11345

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Conformal Prediction Meets Long-tail Classification Shuqi Liu, Jianguo Huang, Luke Ong Abstract Conformal Prediction (CP) is a popular method for uncertainty quantification that converts a pretrained model’s point prediction into a prediction set, with the set size reflecting the model’s confidence. Although existing CP methods are guaranteed to achieve marginal coverage, they often exhibit imbalanced coverage across classes under long-tail label distributions, tending to over cover the head classes at the expense of under covering the remaining tail classes. This under coverage is particularly concerning, as it undermines the reliability of the prediction sets for minority classes, even with coverage ensured on average. In this paper, we propose the Tail-Aware Conformal Prediction (TACP) method to mitigate the under coverage of the tail classes by utilizing the long-tail structure and narrowing the head-tail coverage gap. Theoretical analysis shows that it consistently achieves a smaller head-tail coverage gap than standard methods. To further improve coverage balance across all classes, we introduce an extension of TACP: soft TACP (sTACP) via a reweighting mechanism. The proposed framework can be combined with various non-conformity scores, and experiments on multiple long-tail benchmark datasets demonstrate the effectiveness of our methods. 1 Introduction Uncertainty quantification (UQ) is crucial in building reliable machine learning systems, especially in high-stakes applications, such as autonomous driving [ 12 ] , where overconfident errors can be costly. A popular approach for UQ is Conformal Prediction (CP) [ 40 ] , which converts the point predictions of a pretrained model into prediction sets that come with rigorous statistical guarantees. Specifically, CP ensures that the prediction set contains the true label with a user-specified probability on average, and the size of the prediction set serves as a proxy for model confidence: smaller sets imply higher confidence. Despite their strong theoretical guarantees, standard CP methods often struggle under long-tail (LT) label distributions [ 5 ] leading to imbalanced coverage across classes. A long-tail distribution refers to a class frequency distribution in which a small subset of classes accounts for the majority of samples, whereas the remaining classes contain comparatively few samples. The high-frequency classes are commonly referred to as head classes, while the low-frequency classes are termed tail classes. CP methods typically achieve the desired marginal coverage on average, but display specific coverage imbalance: over covering the head classes at the cost of under covering the tail classes [ 23 , 24 ] . Conformal prediction sets often fail to include the ground-truth tail label with the target probability, resulting in unreliable uncertainty estimations. Such under coverage can mislead subsequent downstream decision-making, potentially causing costly or even harmful outcomes in fairness-sensitive or safety-critical scenarios. To address this coverage imbalance observed in standard CP, group-conditional and class-conditional approaches have been proposed [ 11 , 35 , 38 ] , which aim to achieve coverage guarantee for each group or individual class. However, these methods are often less practical when the calibration data is limited and exhibits a long-tail distribution, as the per-group or per-class acceptance thresholds estimates become highly unreliable and lead to excessively large prediction sets. These limitations highlight the need for conformal prediction methods that provide valid, efficient, and balanced uncertainty estimates between groups with head and tail classes or further across classes in LT settings. To tackle the problems above, we propose the T ail- A ware C onformal P rediction (TACP) method, which introduces a tailored regularization term that adapts to the underlying label imbalance, leading to more balanced coverage between head and tail classes. Then we provide a theoretical analysis demonstrating that TACP narrows the coverage gap between head and tail classes compared with standard CP. To further improve coverage balance across all individual classes, we introduced an extension of TACP: s oft T ail- A ware C onformal P rediction (sTACP), which employs a smooth reweighting strategy for adaptive penalty control. Our proposed frameworks are flexible and can be combined with a wide range of representative non-conformity scores. Extensive experiments on multiple long-tail benchmarks, including CIFAR100-LT and ImageNet-LT, demonstrate the effectiveness of the proposed TACP and sTACP frameworks. TACP significantly reduces the head-tail coverage gap compared to standard CP across four non-conformity scores, while maintaining informative prediction sets. For example, on ImageNet-LT ( \rho=0.6 ) using the APS score, TACP reduces CovGap-HT from 2.18 to 1.11 , accompanied by a decrease in AvgSize (36.43 \rightarrow 33.98). sTACP further improves class-conditional coverage balance, outperforming other baselines on ImageNet-LT. For instance, with APS as the base score at \alpha=10\% , sTACP reduces the class-conditional gap from 19.00\% (STANDARD) to 15.86\% while preserving a similar set size. These results highlight that TACP and sTACP provide robust, efficient, and balance uncertainty estimates under long-tail settings. We summarize our contributions as follows: • We study the problem of CP under LT label distributions and propose the TACP framework to mitigate the coverage gap between head and tail classes. • We provide a theoretical analysis to demonstrate the effectiveness of TACP and empirically compare its performance with several baselines across multiple long-tail benchmarks using four non-conformity scores. • To narrow class-conditional coverage gaps in LT scenarios, we propose sTACP, an enhanced variant of TACP. Extensive experiments demonstrate that sTACP consistently reduces the class-conditional coverage gap across a wide range of long-tail classification tasks. 2 Related Work Conformal prediction is an active area of research that provides a model-free and distribution-free framework for UQ by converting the point prediction of any model into a prediction set that contains the ground-truth label with a user-defined level. A widely used variant of CP is split CP [ 18 , 25 ] , which employs a held-out calibration set to improve the computational efficiency of full CP [ 40 ] . In addition, several variants of CP have been developed based on cross-validation [ 39 ] or leave-one-out [ 2 , 17 ] . There is an increasing amount of research focused on maintaining marginal coverage by relaxing the exchangeability assumption to account for covariate shift [ 37 ] and label shift [ 28 , 42 ] . CP has been applied to various tasks, including classification [ 1 , 19 , 30 , 43 ] , regression [ 29 , 32 , 33 ] , and outlier detection [ 3 , 13 ] . Its performance is typically assessed by coverage validity and length efficiency. Hence, a central research goal is to improve length efficiency [ 14 , 21 , 41 ] . Another important line of research seeks to move beyond marginal coverage guarantees, including methods that aim to improve class-conditional coverage [ 27 , 31 , 34 ] , feature-conditional coverage [ 30 ] . Such conditional guarantees are especially important in LT data settings, where marginal coverage can obscure significant disparities across different classes or subgroups [ 15 , 23 ] . Label distributions in real-world applications are often long-tail [ 5 ] , where a few head classes dominate the data while many tail classes have a limited number of samples. Under such LT settings, previous works focused on improving models’ prediction accuracy by data re-sampling [ 4 , 8 ] and surrogate loss function modification [ 6 , 36 ] . Previous work has observed

Read the complete text