ALSA: Anchors in Logit Space for Out-of-Distribution Accuracy Estimation

Chenzhi Liu, Mahsa Baktashmotlagh, Yanran Tang, Zi Huang, Ruihong Qiu · 2025

Read ALSA: Anchors in Logit Space for Out-of-Distribution Accuracy Estimation by Chenzhi Liu, Mahsa Baktashmotlagh, Yanran Tang, Zi Huang, Ruihong Qiu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-19613.txt.

Introduction

Chenzhi Liu and co-authors build ALSA, a way to guess how well a trained model will score on new, unlabeled data from a shifted world. Users rarely know that accuracy until labels arrive. Older guesses lean on the model's own confidence scores, which squash rich signals, or on data similarity, which is costly and narrow.

ALSA works in logit space, the raw scores before squashing. The team shows those scores bunch and spread in ways that track true accuracy. Learnable anchor points in that space, each with its own influence curve, read the pattern. Across vision, language, and graph tests, ALSA beat both confidence- and similarity-based rivals.

Catalog record

CREATORChenzhi Liu, Mahsa Baktashmotlagh, Yanran Tang, Zi Huang, Ruihong Qiu
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE66 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2508-19613

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

ALSA: Anchors in Logit Space for Out-of-Distribution Accuracy Estimation Chenzhi Liu, Mahsa Baktashmotlagh, Yanran Tang, Zi Huang, Ruihong Qiu Abstract Estimating model accuracy on unseen, unlabeled datasets is crucial for real-world machine learning applications, especially under distribution shifts that can degrade performance. Existing methods often rely on predicted class probabilities (softmax scores) or data similarity metrics. While softmax-based approaches benefit from representing predictions on the standard simplex, compressing logits into probabilities leads to information loss. Meanwhile, similarity-based methods can be computationally expensive and domain-specific, limiting their broader applicability. In this paper, we introduce ALSA (Anchors in Logit Space for Accuracy estimation), a novel framework that preserves richer information by operating directly in the logit space. Building on theoretical insights and empirical observations, we demonstrate that the aggregation and distribution of logits exhibit a strong correlation with the predictive performance of the model. To exploit this property, ALSA employs an anchor-based modeling strategy: multiple learnable anchors are initialized in logit space, each assigned an influence function that captures subtle variations in the logits. This allows ALSA to provide robust and accurate performance estimates across a wide range of distribution shifts. Extensive experiments on vision, language, and graph benchmarks demonstrate ALSA’s superiority over both softmax- and similarity-based baselines. Notably, ALSA’s robustness under significant distribution shifts highlights its potential as a practical tool for reliable model evaluation. Code has been released at https://github.com/chenzhi-liu/ALSA . ALSA † † email: [email protected] † † email: [email protected] † † email: [email protected] † † email: [email protected] † † email: [email protected] † † affiliation: School of Electrical Engineering and Computer Science The University of Queensland Brisbane, Australia 1 Introduction In real-world machine learning model deployment, the trustworthy and reliable performance of well-trained models largely depends on the in-distribution (ID) assumption. However, the user interactions in online environments frequently introduce diverse and dynamically changing data streams, where the target (test) data may differ significantly from the source (train) data, resulting in substantial model performance degradation [ Recht et al.(2019)Recht, Roelofs, Schmidt, and Shankar , Koh et al.(2021a)Koh, Sagawa, Marklund, Xie, Zhang, Balsubramani, Hu, Yasunaga, Phillips, Gao, Lee, David, Stavness, Guo, Earnshaw, Haque, Beery, Leskovec, Kundaje, Pierson, Levine, Finn, and Liang , Geirhos et al.(2019)Geirhos, Rubisch, Michaelis, Bethge, Wichmann, and Brendel ] . Consequently, accurately estimating model performance in these out-of-distribution (OOD) scenarios is of critical importance. Performance estimation methods mainly fall into two categories. (1) Using the softmax scores of models. For instance, the Average Confidence (AC) method [ Guillory et al.(2021)Guillory, Shankar, Ebrahimi, Darrell, and Schmidt ] uses the average of maximum softmax scores to estimate the error, while Average Thresholded Confidence (ATC) [ Garg et al.(2022)Garg, Balakrishnan, Lipton, Neyshabur, and Sedghi ] further refines the estimation by learning a threshold to better map softmax scores to error predictions. Similarly, Confidence Optimal Transport (COT) [ Lu et al.(2023)Lu, Qin, Zhai, Shen, Chen, Wang, Kolouri, Stepputtis, Campbell, and Sycara ] employs optimal transport between softmax scores under the assumption of a label distribution shift. (2) Relying on data similarity. These approaches suggest that a model’s accuracy is strongly associated with the similarity between the source and target datasets, and quantifying this similarity can lead to accurate performance estimation [ Deng and Zheng(2021) , Deng et al.(2021)Deng, Gould, and Zheng ] . Although data similarity-based methods hold promise for model evaluation, they are computationally intensive, as their effectiveness often necessitates the use of multiple models or extensive data augmentation. Furthermore, these approaches are specifically designed for certain data types, including images [ Deng et al.(2021)Deng, Gould, and Zheng , Deng and Zheng(2021) ] and graphs [ Zheng et al.(2023)Zheng, Zhang, Chen, Molaei, Zhou, and Pan ] , making them less suitable as a general framework for evaluating models under distribution shifts. Softmax-based methods are also limited as softmax compresses logits [ Goodfellow et al.(2016)Goodfellow, Bengio, and Courville ] , resulting in the loss of magnitude differences that are critical for accurate performance estimation. By operating in logit space instead of softmax outputs a greater amount of information is retained, thereby enhancing accuracy estimation. Additionally, existing methods overlook class-wise imbalances, which can adversely impact predictions when training distributions are skewed [ He and Garcia(2009) , Tang et al.(2020)Tang, Huang, and Zhang ] . For example, metrics like average confidence in AC [ Garg et al.(2022)Garg, Balakrishnan, Lipton, Neyshabur, and Sedghi ] and thresholds in ATC [ Garg et al.(2022)Garg, Balakrishnan, Lipton, Neyshabur, and Sedghi ] can be misleading when minority classes are scarce in training but prevalent in the target set. Thus, analyzing logits distribution and class-wise performance are both crucial for improving OOD performance estimation . This paper proposes Anchors in Logit Space for Accuracy estimation (ALSA), a novel framework for estimating the accuracy of machine learning models on unlabeled datasets under distribution shift. Unlike existing methods that rely on softmax probabilities, ALSA operates directly in the logit space to preserve richer information for more accurate performance estimation. The approach is motivated by the observation that logits tend to aggregate near a specific hyperplane, and certain regions in logit space are more predictive of correct classification outcomes. To formalize this insight, anchors are introduced as reference points in logit space that act as confidence indicators. A probabilistic model is then constructed, where each anchor contributes to the likelihood that a logit vector corresponds to a correct prediction. These contributions are aggregated and passed through a sigmoid function to yield the final probability estimate. ALSA is evaluated on a diverse set of benchmarks, including vision, language, and graph datasets, and demonstrates improved accuracy estimation compared to existing methods. In addition to providing useful insights into model behavior under distribution shifts, it shows strong robustness in scenarios where the marginal distribution of the target variable differs across datasets, which is common in real-world applications. • Identification and analysis of information loss caused by softmax compression, emphasizing the importance of logit distribution in accuracy estimation. • Development of ALSA, a novel framework that leverages logit-space distributions to estimate accuracy under distribution shifts without the need for labeled target data. • Extensive evaluation across vision, language, and graph datasets, as well as diverse model architectures, demonstrating ALSA’s superior performance, robustness, and generalization ability in real-world applications. Prior Work. Our research is related to but different from work in OOD detection [ Hendrycks and Gimpel(2016) , Liang et al.(2018)Liang, Li, and Srikant , Du and Mordatch(2019) ] and generalization estimation [ Recht et al.(2019)Recht, Roelofs, Schmidt, and Shankar , Miller et al.(2021)Miller, Taori, Raghunathan, Sagawa, Koh, Shankar, Liang, Carmon, and Schmidt , Garg et al.(2022)Garg, Balakrishnan, Lipton, Neyshabur, and Sedghi , Baek et al.(2022)Baek, Jiang, Raghunathan, and Kolter , Zheng et al.(2023)Zheng, Zhang, Chen, Molaei, Zhou, and Pan ] . Prior approaches use softmax scores [ Hendrycks and Gimpel(2016) , Garg et al.(2022)Garg, Balakrishnan, Lipton, Neyshabur, and Sedghi ] , calibration [ Liang et al.(2018)Liang, Li, and Srikant ] , or dataset similarity [ Deng and Zheng(2021) ] , often at the cost of reduced generality or informativeness. ALSA addresses these issues by leveraging richer structures in logit space. Detailed related work can be found in Appendix A . 2 Preliminaries and Motivations 2.1 Problem Definition Consider a multi-class classification problem with input space \mathcal{X}\in\mathbb{R}^{d} and

Read the complete text