Expectation Consistency Loss: Rethink Confidence Calibration under Covariate Shift

Jinzong Dong, Zhaohui Jiang, Bo Yang · 2026

Read Expectation Consistency Loss: Rethink Confidence Calibration under Covariate Shift by Jinzong Dong, Zhaohui Jiang, Bo Yang (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-21552.txt.

Introduction

Jinzong Dong, Zhaohui Jiang, and Bo Yang propose Expectation Consistency Loss, rethinking trust tuning under covariate shift. Trust tuning for sorting models is vital in safety-critical decisions. Yet general methods assume training and test data are of the same kind, limiting power under shift. They derive an exact rule for tuning under shift, named Expectation consistency rule. This gives a sound basis for a new tuning loss.

Catalog record

CREATORJinzong Dong, Zhaohui Jiang, Bo Yang
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE81358
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.21552

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Expectation Consistency Loss: Rethink Confidence Calibration under Covariate Shift Jinzong Dong, Zhaohui Jiang, Bo Yang Abstract Confidence calibration for classification models is vital in safety-critical decision-making scenarios and has received extensive attention. General confidence calibration methods assume training and test data are independent and identically distributed ( i . i . d . i.i.d. ), limiting their effectiveness under covariate shifts. Previous calibration methods under covariate shift struggle with class-wise or canonical calibrations and often rely on unstable importance weighting when density ratios are large or unbounded. Given the above limitations, this paper rethinks confidence calibration under covariate shifts. First, we derive a necessary and sufficient condition for confidence calibration under covariate shifts, named Expectation consistency condition , which reveals covariate shifts do not necessarily lead to uncalibrated confidence and provides a weaker condition for confidence calibration than global covariate distribution alignment. Then, utilizing Expectation consistency condition , this paper proposes an unsupervised domain adaptation loss to calibrate confidence of the target domain, named Expectation consistency loss (ECL), which is compatible with canonical calibration, class-wise calibration, and top-label calibration. Third, we prove that computing ECL loss has the same sample complexity as Expected Calibration Error (ECE) and provide a theoretically grounded mini-batch trainable scheme for ECL loss. Finally, we validate the effectiveness of our method on both simulated and real-world covariate shift datasets. 1 Introduction Modern machine learning classification models, such as deep neural networks, are becoming increasingly accurate and widely applied in safety-critical fields ( 25 ; 20 ) . Nevertheless, decision-making systems in such fields need not only high accuracy but also the ability to recognize when they might be wrong ( 30 ) . For example, in automatic disease diagnosis, if a model has low confidence in its prediction, it should defer to a medical professional ( 19 ) . Thus, classification models should provide accurate confidence estimates alongside their predictions to reflect the true likelihood of an event. Accurate confidence is more informative than mere class labels, e.g., stating “a patient has a 70% probability of having cancer” gives doctors more actionable information than just labeling the condition as “cancer” . Moreover, accurate confidence facilitates classification models to better integrate with other probabilistic models, e.g., helping active learning to select more representative samples ( 13 ) and improving the generalization performance of knowledge distillation ( 27 ) . Therefore, pursuing more accurate confidence in classification models is of great importance ( 9 ) . In recent years, confidence calibration has emerged as one of the most effective methods for producing more reliable confidence estimates and has attracted considerable attention ( 11 ; 41 ; 23 ; 6 ) . However, general confidence calibration methods typically assume that the target domain (or test set) and the source domain (or calibration set) are independent and identically distributed ( i.i.d. ). When this assumption is violated due to distribution shifts, calibration performance often deteriorates significantly ( 42 ) . Covariate shift, a common type of data distribution shift, often occurs in real-world tasks like medical diagnosis across different populations or image recognition under varying lighting conditions, where the input data distribution of models changes while the underlying relationship between inputs and outputs remains consistent ( 22 ) . Under covariate shift, models calibrated on the source domain frequently fail to generalize to the target domain, resulting in unreliable confidence estimates ( 1 ) . This highlights the importance of developing confidence calibration methods that remain robust under covariate shift ( 16 ) . Currently, the mainstream confidence calibration methods under covariate shift are based on importance weighting ( 32 ; 33 ; 38 ; 37 ) , which adjusts the objective function by assigning weights based on the importance of instances from the source domain, thereby guiding the model to generalize to the target domain unbiasedly ( 22 ) . However, it is well known that importance weighting has been criticized for its instability when the density ratio is large or unbounded ( 4 ) . 16 use mixup to synthesize pseudo-target data and generalize the calibration performance from the pseudo-target data to the target domain. However, the efficacy of this method hinges primarily on the degree of similarity between the pseudo-target data and the target domain data. Furthermore, existing methods primarily address the simplest prediction-based calibration (i.e., top-label calibration). To our knowledge, there remains a notable absence of class-wise and canonical calibration methods designed to handle covariate shift. Importance weighting in confidence calibration aims to globally align covariate distributions, inspired by accuracy improvement under covariate shift. However, confidence calibration differs fundamentally from accuracy improvement: it requires not learning new knowledge, but precisely conveying uncertainty. This raises a natural but often neglected question: Is global covariate distribution alignment necessary? To answer this, we first derive a necessary and sufficient condition for confidence calibration under covariate shifts, termed the Expectation consistency condition . This condition reveals that covariate shifts do not necessarily cause miscalibration and provides a weaker requirement than global distribution alignment. Based on this condition, we propose an unsupervised domain adaptation loss, Expectation consistency loss (ECL), with three variants for canonical, class-wise, and top-label calibration. We prove that ECL has sample complexity {\cal O}(B/{\varepsilon^{2}}) , comparable to histogram binning, where B denotes the number of confidence bins. To enable unbiased gradient backpropagation on mini-batch data, we also provide a theoretically sound mini-batch training scheme for ECL. Finally, we validate the method on simulated and real-world covariate shift datasets. Table 1: Comparison of ECL and related calibration methods. Method Covariate Shift Class-wise Calibration Canonical Calibration Density Ratio Unbounded Mini-batch Trainable SB - ECE ( 21 ) ✗ ✗ ✗ ✓ ✗ DECE ( 2 ) ✗ ✗ ✗ ✓ ✗ ECE^{KDE} ( 34 ) ✗ ✓ ✓ ✓ ✓ Weighted TS ( 32 ) ✓ ✗ ✗ ✗ ✗ FL+IW+Temp ( 33 ) ✓ ✗ ✗ ✗ ✗ TransCal ( 38 ) ✓ ✗ ✗ ✗ ✗ DRL ( 37 ) ✓ ✗ ✗ ✗ ✗ PseudoCal ( 16 ) ✓ ✗ ✗ ✓ ✗ ECL (Ours) ✓ ✓ ✓ ✓ ✓ 2 Background and Related Work Consider a K -class classification problem where X\in\mathcal{X} denotes the input feature and {Y}=(Y_{1},\cdots,Y_{K})\in\mathcal{Y} denotes the K -class one-hot encoded label variable, with \mathcal{X}\subset\mathbb{R}^{d} and \mathcal{Y}=\{e_{k}\}_{k=1}^{K} , where e_{k} is a unit vector whose k -th component is 1. Let f:\mathcal{X}\to\mathcal{S}\subset\Delta_{K-1} be a probabilistic classifier, where \Delta_{K-1} represents a ( K-1 )-dimensional simplex. The predicted confidence score vector is given by S=f(X)=(f_{1}(X),\cdots,f_{K}(X))=(S_{1},{\ldots},S_{K})\in\mathcal{S} . In general, the true class scalar is Y^{*}={{\mathop{\rm argmax}\nolimits}_{k}}{\{{Y_{k}}\}_{1\leq k\leq K}} , the predicted class is defined as \hat{Y}={{\mathop{\rm argmax}\nolimits}_{k}}{\{{S_{k}}\}_{1\leq k\leq K}} , and the confidence score of the predicted class is \hat{S}=\max{\{{S_{k}}\}_{1\leq k\leq K}} . In covariate shift, let P_{s}(\cdot) and P_{t}(\cdot) denote the probability density (for continuous variables, e.g., X , X|S , S|X , and X|Y ) or probability measure (for discrete variables, e.g., Y , Y|S and Y|X ) on the source domain and target domain, respectively. P denotes either P_{s} or P_{t} in cases where distinguishing between the source and target domains is not required. Let D_{s} and D_{t} represent the source domain and target domain data, respectively. 2.1 Confidence Calibration Confidence calibration aims

Read the complete text