Dual Strategies for Test-Time Adaptation
Nam Nguyen Phuong Duc Nguyen The Minh Phi Le Nguyen Ehsan Abbasnejad Minh Hoai · 2026
Read Dual Strategies for Test-Time Adaptation by Nam Nguyen Phuong Duc Nguyen The Minh Phi Le Nguyen Ehsan Abbasnejad Minh Hoai (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-17542.txt.
Introduction
Nguyen and four colleagues improve test-time adaptation, the practice of tuning a model on unlabeled test data when distributions shift. Standard methods adapt on only a small slice of confident predictions, wasting the rest. Their DualTTA framework splits samples into likely-correct and likely-wrong groups, reinforcing the first by minimizing prediction entropy and correcting the second by maximizing it. A new reliability test based on stability under meaning-preserving and meaning-changing transformations separates the groups more cleanly than entropy alone.
Catalog record
| CREATOR | Nam Nguyen Phuong Duc Nguyen The Minh Phi Le Nguyen Ehsan Abbasnejad Minh Hoai |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 41963 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.17542
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Dual Strategies for Test-Time Adaptation Nam Nguyen Phuong Duc Nguyen The Minh Phi Le Nguyen Ehsan Abbasnejad Minh Hoai Abstract Conventional test-time adaptation (TTA) approaches typically adapt the model using only a small fraction of test samples, often those with low-entropy predictions, thereby failing to fully leverage the available information in the test distribution. This paper introduces DualTTA, a novel framework that improves performance under distribution shifts by utilizing a larger and more diverse set of test samples. DualTTA identifies two distinct groups: one where the model’s predictions are likely consistent with the underlying semantics, and another where predictions are likely incorrect. For the first group, it minimizes prediction entropy to reinforce reliable decisions; for the second, it maximizes entropy to suppress overconfident errors and unlearn spurious behavior. These groups are adaptively selected using a new reliability criterion that measures prediction stability under both semantic-preserving and semantic-altering transformations, addressing the limitations of purely entropy-based selection. We further provide theoretical analysis and empirical justification showing that our approach enables a tighter separation between reliable and unreliable samples - in the context of their suitability for adaptation - leading to provably more effective model updates. The source code is available at https://github.com/namk65hust/DualTTA . 1 Introduction Deep learning models can struggle when training and test distributions differ, a challenge known as distribution shift. For example, in many computer vision applications, this occurs when statistical properties of the source and target data diverge due to factors like image corruption, lighting changes, or adverse weather 8 ; 11 . As deep networks are highly sensitive to such shifts, adapting models to test scenarios is crucial for maintaining reliability in real-world deployments. To address this issue, various approaches have been proposed, including domain generalization 37 ; 38 ; 26 , domain adaptation 4 ; 23 , test-time training 18 ; 25 , and Test-Time Adaptation (TTA) 1 ; 16 ; 12 ; 19 ; 20 ; 28 ; 35 ; 36 . This paper focuses on TTA, a practical setting in which a trained model must adapt to an unseen target domain during inference without access to test data labels or the original training data. Figure 1 : Comparison of sample utilization between DualTTA and the prior state-of-the-art method DeYO 12 across four datasets. DeYO adapts using only a small portion of test samples (green + red), with many of them (red) driving adaptation in the wrong direction. DualTTA, while not perfect, leverages a broader set of samples and misuses a much smaller fraction - achieving better data efficiency and more reliable adaptation. Given the absence of labeled data, TTA typically updates the model’s parameters by optimizing an unsupervised objective defined on selected test samples. The effectiveness of a TTA method depends critically on the choice of the objective function and the selection of samples for adaptation. One of the most widely adopted objectives in TTA is entropy minimization 28 ; 22 , as low entropy correlates with the decisiveness of the model. A notable approach, TENT 28 , adapts the model during testing by minimizing the entropy of its predictions on each test sample, encouraging the model to make more confident predictions. However, subsequent studies 12 ; 19 have shown that not all test samples are suitable for adaptation, and incorporating inappropriate samples can lead to model degradation. Motivated by this, several TTA methods have been proposed to focus on selecting the right samples for adaptation 12 ; 28 ; 20 ; 19 . For instance, EATA 19 and SAR 20 retain only test samples with low prediction entropy. More recently, DeYO 12 demonstrated that entropy alone is insufficient to filter out unsuitable samples and proposed an additional criterion based on probability differences. Existing TTA methods typically adapt using only a small subset of test samples deemed “reliable” based on low-entropy predictions, discarding most of the available data. This over-selective strategy suffers from two key failure modes: (i) insufficient utilization : the coverage is limited, as the pool of confident samples is small under significant distribution shifts; and (ii) misguided adaptation : confidence does not imply correctness, so high-confidence errors may be misclassified as reliable and, when used for adaptation, can reinforce spurious predictions. For example, DeYO 12 , a state-of-the-art TTA method, adapts on only \approx 14% of test samples (see Fig. 1 ), of which only 60-70% yield beneficial updates (about 9%). Relying solely on entropy minimization as the adaptation objective is thus both insufficient in scope and misguided in execution. To address these limitations, we propose DualTTA , a new TTA framework that expands the effective adaptation set through a more principled partitioning of test samples into likely-correct and likely-incorrect groups. This partitioning is guided by a novel reliability criterion that combines prediction entropy with stability under both semantic-preserving and semantic-altering transformations. By going beyond raw confidence, this criterion better distinguishes informative samples from overconfident outliers, enabling more effective and robust adaptation. Our paper introduces two core technical contributions. First, we propose a novel reliability criterion based on prediction stability under semantic-preserving and semantic-altering transformations. This criterion enables us to distinguish likely-correct samples - those with stable predictions under superficial changes but unstable under content changes - from likely-incorrect ones. Based on this classification, we maintain separate adaptation strategies for the two groups. Second, we introduce a dual-objective optimization for test-time adaptation: entropy minimization for likely-correct samples to reinforce accurate predictions, and entropy maximization for likely-incorrect ones to mitigate overconfidence and unlearn spurious patterns. We further provide theoretical analysis showing that our stability-based criterion, combined with the dual-objective formulation, yields a sharper separation between reliable and unreliable samples and leads to better adaptation. Details of this proof will be presented fully in the Supplementary material section. 2 Related work 2.1 Test-Time Adaptation TTA aims to enhance model performance in online settings by adapting to unlabeled test data without relying on original training data. Existing methods 34 ; 13 ; 17 ; 20 ; 21 ; 28 ; 30 ; 35 ; 36 mainly follow two approaches: (1) entropy minimization and (2) pseudo-label generation. Entropy minimization. These methods assume that confident model predictions (low entropy) indicate better performance, and adapt the model using an unsupervised entropy-based loss on selected test samples. TENT 28 initiated this line, with follow-ups like EATA 19 , which filtered high-entropy samples to avoid noisy gradients, and SAR 20 , which minimized both entropy and loss sharpness using the SAM optimizer 5 to address small batch sizes and label imbalance. DeYO 12 further identified that low-entropy samples could still lack discriminability and proposed filtering such samples. While effective, these methods struggle to fully leverage the available target data. Pseudo-label generation derives pseudo-labels for selected samples and uses them in adaptation loss. CoTTA 29 generated robust pseudo-labels via augmentation and multiple inferences. FATA 2 extended entropy-minimization methods by transforming feature-level representations of selected reliable samples. Though this approach improves sample utilization, it incurs high computational cost, limiting real-world applicability. 2.2 Prior use of transformation techniques Several TTA methods incorporate semantic-altering perturbations, such as spatial transformations, into self-training frameworks 29 , often assuming all augmented data is beneficial. This can lead to degraded performance when misleading samples are included. In contrast,