Contrastive Residual Energy Test-time Adaptation

Yewon Han, Seoyun Yang, Taesup Kim · 2025

Read Contrastive Residual Energy Test-time Adaptation by Yewon Han, Seoyun Yang, Taesup Kim (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-19607.txt.

Introduction

Yewon Han, Seoyun Yang, and Taesup Kim put forth a method in 2025 to tune a trained model at test time. When new data drift from training data, a model can go off track. Most fixes tune by the model's own guesses, which can drift and are often badly scaled. An energy-based fix avoids guesses but needs slow sampling. Their method learns a spare energy term that sits on top of the frozen model.

The math makes the hard part of the energy sum cancel out, so no sampling is needed. A built-in reweighting stops the model from merely backing its own first guesses. Tests showed fast, well-scaled tuning under real limits on time and compute. The authors frame it as a workable route for field use.

Catalog record

CREATORYewon Han, Seoyun Yang, Taesup Kim
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE80 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-19607

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Contrastive Residual Energy Test-time Adaptation Yewon Han, Seoyun Yang, Taesup Kim Abstract Test-time adaptation (TTA) enhances model robustness by enabling adaptation to target distributions that differ from training distributions, improving real-world generalizability. However, most existing TTA approaches focus on adjusting the conditional distribution and therefore exhibit poor calibration, as they rely on uncertain predictions in the absence of labels. Energy-based TTA frameworks provide an alternative by modeling the marginal distribution of target data without depending on label predictions, but their reliance on costly sampling hinders scalability in real-world scenarios where decisions must be made without latency. In this work, we propose Contrastive Residual Energy Test-time Adaptation ( CreTTA ) , a practical solution for reliable adaptation. We theoretically reformulate the marginal distribution adaptation as learning a residual energy function. This formulation leads to a contrastive objective where the intractable partition function mathematically cancels out, removing sampling and approximation error. Crucially, our analysis reveals that this design prevents overfitting through an adaptive gradient reweighting mechanism that leverages relative energy differences, avoiding the self-confirming bias of entropy minimization. Extensive experiments demonstrate that CreTTA achieves scalable and well-calibrated adaptation under real-world computational constraints. † † affiliationnotice: Equal contribution 1 Introduction Deep learning models can achieve high accuracy on training and testing data from the same distribution. However, when the distribution of the test data diverges from the original training dataset, the performance of the deep learning models deteriorates. This distribution shift refers to changes in the underlying data statistics, such as feature distributions or environmental conditions, between training and deployment. It is a major challenge in real-world scenarios, where test samples are often drawn from a distribution that deviates from the training data. Figure 1 : Comparison of classification (mCE), calibration (ECE) performance gains over source with corresponding computational costs (GFLOPs) to achieve improvements on CIFAR10-C (Sev 1-5 Avg). CreTTA achieves a better balance between performance and computation than the existing energy-based method, TEA. To address distribution shifts during testing, test-time adaptation (TTA) strategy aims to adapt trained model instantly, thereby maintaining robust performance on unexpected out-of-distribution samples. Since ground-truth labels are unavailable at test-time, existing approaches ( Lee and others, 2013 ; Liang et al., 2020 ) often rely on the model’s own predictions. For instance, TENT ( Wang et al., 2020 ) uses the model’s predicted probability distribution as a surrogate ground-truth within an Entropy-Minimization(EM) objective. Formally, the entropy minimization objective for a test sample x_{t} is expressed as -\sum_{k=1}^{C}p_{\theta}(\hat{y}_{k}\mid x_{t})\log p_{\theta}(\hat{y}_{k}\mid x_{t}) , where C is the number of classes and p_{\theta}(\hat{y}_{k}\mid x_{t}) is the predicted probability of class \hat{y}_{k} . While this approach demonstrates promising accuracy, it fundamentally relies on uncertain predictions without ground-truth supervision. As a result, optimizing the entropy minimization objective creates a self-confirming bias , driving the predicted probabilities to collapse to extreme values of 0 or 1, leading to severe overconfidence ( Press et al., 2024 ) . This behavior increases calibration error as illustrated in Figure 10 , which can be detrimental in high-stakes real-world scenarios. Energy-based models (EBMs) ( LeCun et al., 2006 ) offer a promising alternative by directly modeling the marginal distribution, thereby avoiding the pitfalls of confidence-based objectives. TEA ( Yuan et al., 2024 ) applies Maximum Likelihood Estimation (MLE) in the TTA setting to align the model with unseen target distributions. While providing experimental evidence of stable and reliable adaptation, standard EBM approaches still face two critical limitations: • Unresolvable Approximation Error. The normalization constant requires approximation via short-run Markov chains, which yield biased gradient estimates . Consequently, parameter updates can be unstable or converge to suboptimal solutions ( Song and Kingma, 2021 ; Yair and Michaeli, 2021 ) , particularly during few-shot adaptation or with high-dimensional data as shown in Figure 2 . • High Computational Cost. Re-estimating the normalization constant for every incoming batch necessitates repeated sampling steps, incurring substantial computational overhead. This burden makes real-time deployment infeasible under strict latency constraints , highlighting the need for methods that reduce overhead while maintaining strong performance. Consequently, TTA faces a dilemma: existing methods are either (i) efficient but unreliable (entropy-based), or (ii) reliable but computationally prohibitive (sampling-based). In real-world settings, however, well-calibrated adaptation is critical to avoid overconfident errors while ensuring computationally efficient and scalable model behavior . To this end, we propose Contrastive Residual Energy Test-time Adaptation ( CreTTA ), a novel framework that optimizes the marginal distribution while eliminating the normalization constant approximation. We theoretically reformulate the TTA task as learning a residual energy function that captures the discrepancy between source and target distributions. By embedding this function into a pairwise contrastive objective , we mathematically offset the intractable partition function, removing sampling and approximation error. The primary contributions of our work are summarized as follows: • Sampling-free EBM Framework. We introduce CreTTA , a novel sampling-free residual energy-based TTA framework. By removing the sampling bottleneck, it offers significant computational efficiency and scalability , making it a practical solution for real-time applications. • Theoretical Reformulation and Stability. We redefine the marginal distribution adaptation as learning a residual energy function with a contrastive objective. Our analysis reveals that this formulation acts as an implicit regularizer via an adaptive gradient reweighting mechanism , which effectively mitigates overfitting and ensures stable adaptation. • Strong Performance with Reliability. Extensive experiments demonstrate that CreTTA is well-calibrated and achieves state-of-the-art classification performance across various distribution shifts. 2 Preliminaries Problem Setup. Let Q denote the marginal distribution of the source training data x_{s} . Consider a classifier f_{\phi}(x) with parameters \phi , which is pretrained on a labeled source dataset \{(x^{(i)}_{s},y^{(i)}_{s})\}_{i=1}^{M} . Although this pretrained model performs well on in-distribution test data (i.e., x_{s}\sim Q ), its performance can degrade substantially when tested on data from a different distribution P(\neq Q) , commonly referred to as out-of-distribution data. Test-time adaptation (TTA) aims to mitigate this issue by adapting the pretrained parameters \phi to better align with the target marginal distribution P . In this setting, we are given a set of N unlabeled target samples \{x_{t}^{(i)}\}_{i=1}^{N} drawn from P , which arrive in online batches. To cope with the absence of label information, existing methods often rely on unsupervised objectives, particularly entropy minimization. EATA ( Niu et al., 2022 ) and SAR ( Niu et al., 2023 ) build on this foundation by incorporating surrogate objectives and sample selection mechanisms to filter out unreliable predictions. However, these enhancements still fundamentally depend on uncertain model outputs, as dictated by the nature of entropy-based objectives. Moving beyond entropy-based methods, recent work by TEA ( Yuan et al., 2024 ) demonstrates the promise of energy-based modeling for TTA, where the central idea is to represent the test data marginal distribution through an energy function. Within this framework, TEA employs MLE on the marginal distribution of test samples \{x_{t}^{(i)}\}_{i=1}^{N} , so that the energy function is learned to assign lower energy (i.e., higher likelihood) to observed test inputs. By directly modeling and adapting to the marginal distribution, TEA mitigates distribution shifts without requiring labeled data or relying heavily on potentially unreliable model predictions. Building on this insight, our proposed method, CreTTA , approaches test-time adaptation not merely as an MLE procedure, but as effective learning of an

Read the complete text