Conformal Correction for Efficiency May be at Odds with Entropy
Senrong Xu, Tianyu Wang, Zenan Li, Yuan Yao, Taolue Chen, Feng Xu, Xiaoxing Ma · 2025
Read Conformal Correction for Efficiency May be at Odds with Entropy by Senrong Xu, Tianyu Wang, Zenan Li, Yuan Yao, Taolue Chen, Feng Xu, Xiaoxing Ma (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2512-02704.txt.
Introduction
Senrong Xu and six co-authors question a popular way to sharpen uncertainty sets from machine learning models. Conformal prediction wraps any black box model and returns sets with rigorous coverage guarantees. A correction step can fine tune the model to make those sets smaller and more efficient.
The authors argue that this correction may clash with entropy, a measure of information content. Chasing smaller sets could distort what the model expresses. The tension matters for high stakes uses, such as medicine, where an efficient but misleading uncertainty set is worse than a wider honest one. Wide sets that stay true beat tight sets that lie, a trade this work puts in plain sight for all to weigh.
Catalog record
| CREATOR | Senrong Xu, Tianyu Wang, Zenan Li, Yuan Yao, Taolue Chen, Feng Xu, Xiaoxing Ma |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 61 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2512-02704
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Conformal Correction for Efficiency May be at Odds with Entropy Senrong Xu, Tianyu Wang, Zenan Li, Yuan Yao, Taolue Chen, Feng Xu, Xiaoxing Ma Abstract Conformal prediction (CP) provides a comprehensive framework to produce statistically rigorous uncertainty sets for black-box machine learning models. To further improve the efficiency of CP, conformal correction is proposed to fine-tune or wrap the base model with an extra module using a conformal-aware inefficiency loss. In this work, we empirically and theoretically identify a trade-off between the CP efficiency and the entropy of model prediction. We then propose an entropy-constrained conformal correction method, exploring a better Pareto optimum between efficiency and entropy. Extensive experimental results on both computer vision and graph datasets demonstrate the efficacy of the proposed method. For instance, it can significantly improve the efficiency of state-of-the-art CP methods by up to 34.4%, given an entropy threshold. 1 Introduction For a decision-making process driven by machine learning (e.g., loan approval, fraud detection), it is essential for the predictions to be accompanied by a level of confidence to quantify uncertainty ( 41 ; 34 ) . Conformal prediction (CP) is a promising uncertainty quantification method, providing statistically rigorous uncertainty sets for black-box machine learning models ( 3 ; 36 ; 35 ; 8 ) . In standard classification, for any test input x , the posterior distribution \tilde{\pi}_{y}(x)=P(Y=y\mid X=x) on classes [K]:=\{1,\cdots,K\} is calculated. Conformal prediction leverages an additional calibration step to guarantee a user-specified (marginal) coverage: by producing a prediction set \mathcal{C}(x)\subseteq[K] , it guarantees the true class of x is included in \mathcal{C}(x) with a user-chosen probability, when the calibration samples are exchangeable with the test samples. The uncertainty typically manifests in two aspects in CP: (1) the efficiency of prediction sets; (2) the entropy of model predictions. For the former, \mathcal{C}(x) with a small size is considered to have high efficiency, providing more certainty for decision-making processes. For the latter, entropy directly quantifies the level of prediction uncertainty. Simply consider the two prediction sets for a patient, \{ Diabetes, Asthma \} with predictive probabilities 0.4 and 0.4, and \{ Diabetes, Asthma, Stroke \} with predictive probabilities 0.6, 0.1 and 0.1. It would be difficult to compare the goodness of these two sets in terms of guiding a doctor to make decisions. Recent progress in CP mainly focuses on the low-efficiency problem via introducing extra training on the base model, largely neglecting the important role of entropy. For example, 5 proposes the notion of conformal training and 37 simulates the conformal prediction process during training; this approach is further extended to graph-structure data ( 18 ) by introducing a conformal adapter, which performs an additional conformal-aware training step based on the fixed base model. In this paper, we adopt the latter setting as the conformal adapter only needs the output distribution of the base model (as input), which is more akin to traditional CP (in the sense that it is decoupled from the base model), and thus has broader applications in practice. We refer to this emerging class of approaches as conformal correction . To be more concrete, given a base classifier \widetilde{M} , we can obtain a conformal adapter \widehat{M} , which takes \tilde{\pi}(x) from \widetilde{M} as input and outputs \hat{\pi}(x) , together with \mathcal{C}(x) typically of a smaller size. (a) Pareto frontier (b) CIFAR100 (c) Cora-ML Figure 1: Fig.(a) plots the Pareto frontier between inefficiency and entropy. For both of them, the lower, the better; Fig.(b) and (c) are the results of training only with \mathcal{L}_{\mathrm{class}} on CIFAR100 and Cora-ML, respectively. (b) and (c) depict the efficiency and entropy on the test set during the conformal correction, and there is a trade-off between them when the accuracy reaches the top. Our motivation is to have an in-depth understanding of the potential catch when a smaller \mathcal{C}(x) is in place. We find that while the average size of \mathcal{C}(x) may be smaller, the entropy of the prediction \hat{\pi}(x) also increases, indicating that the prediction becomes more uncertain , which is not ideal. Nevertheless, high efficiency should not sacrifice prediction entropy too much! Indeed, our experiments show that, when conformal correction is applied on CIFAR100, the average size of CP sets is increased from 17.3 to 58.6 while the prediction entropy decreases from 6.3 to 1.1 (cf. Fig. 1(b) in Section 3 ). This indicates that a trade-off exists between the CP efficiency and the prediction entropy. We further confirm the finding by showing that, for APS ( 31 ) , the expected size of CP sets can be upper-bounded by the negative entropy (plus some positive constant; cf. Theorem 3 for a precise account). This gives theoretical evidence that the efficiency of CP sets produced by APS may be at odds with the prediction entropy. The trade-off between efficiency and entropy entails a Pareto perspective on CP, where different Pareto optima form a Pareto frontier as shown in Fig. 1(a) . Conformal correction can thus be viewed as a traversal of the Pareto frontier. Technically, one can reduce the inefficiency significantly, but at the cost of an increased entropy, rendering such a reduction less meaningful. Instead, we argue that seeking for a better Pareto frontier is more crucial for conformal correction than simply adapting the trade-off. To this end, we propose a new method, i.e., entropy-constrained conformal correction ( EC {}^{\text{3}} ) to ameliorate the trade-off by controlling the entropy of conformal adapters via focal loss ( 27 ) and temperature scaling ( 13 ) . We conduct extensive experiments on computer vision (CV) and graph datasets to evaluate the effectiveness of EC {}^{\text{3}} . The results show that our method can outperform the competitors by up to 34.4% in terms of efficiency given an entropy threshold; the qualitative analysis also indicates that our method can locate better Pareto optimality with strong control over the model entropy. Furthermore, when EC {}^{\text{3}} is adapted to provide (stronger) conditional coverage, it can significantly improve, for instance, the class coverage from 0.77 to 0.83 (for the CV dataset) and from 0.74 to 0.85 (for the graph dataset), respectively. To summarize, the main contributions of the paper are: (1) we identify a trade-off between efficiency and prediction entropy in CP, which has not been fully investigated before; (2) we propose a new conformal correction method based on entropy control to improve the efficiency of CP, the effectiveness of which is confirmed by extensive experiments. 2 Preliminary Notations. We focus on multiclass classification (with K classes). Assume D=\{(X_{i},Y_{i})\}_{i=1}^{n+1} of i.i.d. (or simply exchangeable) observations sampled from an (unknown) testing distribution P_{XY} . We denote the (oracle) conditional distribution P_{Y\mid X} by \pi_{y}(x)=P(Y=y\mid X=x) . Furthermore, a black-box base classifier undergoes adapting to prescribe prediction \hat{\pi}_{y}(x) . The prediction entropy is defined as H(\hat{\pi}(x)):=-\sum_{k=1}^{K}\hat{\pi}_{k}(x)\log\hat{\pi}_{k}(x) . CP Framework. Given D_{\mathrm{cal}}=\{(X_{i},Y_{i})\}_{i=1}^{n} as the calibration set and a user-defined miscoverage rate \alpha\in(0,1) , CP typically proceeds in the following three steps: (1) Non-conformity score definition. CP first heuristically defines a non-conformity score function V(x,y) , which indicates how the class y conforms to the predictive result \hat{\pi}(x)=[\hat{\pi}_{1}(x),\dots,\hat{\pi}_{K}(x)] . For example, the non-conformity score V(x,y) can be defined as the sum of the probabilities of all K classes in \hat{\pi}(x) except