ZeroSiam: An Efficient Siamese for Test-Time Entropy Optimization without Collapse

Guohao Chen, Shuaicheng Niu, Deyu Chen, Jiahao Yang, Zitian Zhang, Mingkui Tan, Pengcheng Wu, Zhiqi Shen {guohao.chen, shuaicheng.niu, zqshen}@ntu.edu.sg, Nanyang Technological University South China University of Technology · 2025

Read ZeroSiam: An Efficient Siamese for Test-Time Entropy Optimization without Collapse by Guohao Chen, Shuaicheng Niu, Deyu Chen, Jiahao Yang, Zitian Zhang, Mingkui Tan, Pengcheng Wu, Zhiqi Shen {guohao.chen, shuaicheng.niu, zqshen}@ntu.edu.sg, Nanyang Technological University South China University of Technology (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-23183.txt.

Introduction

Adapting a model at test time by shrinking its own uncertainty can collapse into one lazy answer. Guohao Chen and 7 co-authors found that asymmetry blocks that collapse. Their ZeroSiam design is an asymmetric Siamese setup for test-time entropy work. A learnable predictor plus a stop-gradient step create the asymmetry cheaply. Constant one-hot outputs no longer trivially win.

Theory and tests both support the design. It also regularizes biased learning signals when no collapse threatens. Gains held on vision adaptation and language model reasoning tasks. Tiny, collapse-prone models benefited as well. Overhead stayed negligible next to prior methods.

Catalog record

CREATORGuohao Chen, Shuaicheng Niu, Deyu Chen, Jiahao Yang, Zitian Zhang, Mingkui Tan, Pengcheng Wu, Zhiqi Shen {guohao.chen, shuaicheng.niu, zqshen}@ntu.edu.sg, Nanyang Technological University South China University of Technology
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE105 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2509-23183

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

ZeroSiam: An Efficient Siamese for Test-Time Entropy Optimization without Collapse Guohao Chen, Shuaicheng Niu, Deyu Chen, Jiahao Yang, Zitian Zhang, Mingkui Tan, Pengcheng Wu, Zhiqi Shen {guohao.chen, shuaicheng.niu, zqshen}@ntu.edu.sg, Nanyang Technological University South China University of Technology Abstract Test-time entropy minimization helps adapt a model to novel environments and incentivize its reasoning capability, unleashing the model’s potential during inference by allowing it to evolve and improve in real-time using its own predictions, achieving promising performance. However, pure entropy minimization can favor non-generalizable shortcuts, such as inflating the logit norm and driving all predictions to a dominant class to reduce entropy, risking collapsed solutions ( e.g. , constant one-hot outputs) that trivially minimize the objective without meaningful learning. In this paper, we introduce ZeroSiam, an efficient asymmetric Siamese architecture tailored for test-time entropy minimization. ZeroSiam prevents collapse through asymmetric divergence alignment, which is efficiently achieved by a learnable predictor and a stop-gradient operator before the classifier. We provide empirical and theoretical evidence that ZeroSiam not only prevents collapse solutions, but also absorbs and regularizes biased learning signals, enhancing performance even when no collapse occurs. Despite its simplicity, extensive results show that ZeroSiam performs more stably over prior methods using negligible overhead, demonstrating efficacy on both vision adaptation and large language model reasoning tasks across challenging test scenarios and diverse models, including tiny models that are particularly collapse-prone. 1 Introduction Entropy measures the uncertainty of a model’s predictions. Since its emergence, entropy optimization has been widely adopted as an auxiliary objective alongside supervised, reinforcement signals, etc. To be specific, in semi-supervised learning it enforces low-entropy predictions on unlabeled data to refine decision boundaries ( Grandvalet & Bengio 2004 ; Lee et al. 2013 ; Sajjadi et al. 2016 ) ; in domain adaptation it mitigates distribution shifts by promoting confident outputs ( Jiang & Zhai 2007 ; Long et al. 2016 ; Morerio et al. 2017 ) ; and in reinforcement learning it is often maximized to encourage exploration or minimized to ensure deterministic policies ( Ziebart et al. 2008 ; Ahmed et al. 2019 ) , among others, showing its remarkable success in boosting the learning effectiveness. Of late, test-time entropy minimization has attracted growing interest, as it operates purely unsupervised during inference, without relying on any ground-truth supervision. For example, in test-time adaptation (TTA) ( Wang et al. 2021 ; Lee et al. 2024 ) it encourages confident predictions to mitigate domain shifts on unseen data streams; in large language models, it helps calibrate uncertainty and improve prediction consistency for alignment ( Hu et al. 2025a ; Jang et al. 2025 ) . These advances highlight the potential of entropy minimization in unleashing model power during inference, enabling adaptation to novel or out-of-distribution domains and moving toward greater intelligence. However, entropy minimization naturally drives the model to increase the maximum predicted logit, regardless of whether it aligns with the true label. In practice, testing often involves noisy real-world data, domain shifts, or novel environments, where models often tend to be highly sensitive and uncertain, and the predicted maximum class is frequently incorrect. In this sense, minimizing entropy alone does not ensure that the corresponding supervised loss ( e.g. , cross-entropy) on the target task is also reduced. Instead, the model can easily exploit a shortcut solution by producing near one-hot outputs for all inputs, which trivially minimizes entropy but fails to capture meaningful predictions, leading to degraded performance. For instance, after TTA, models can deteriorate to predicting all samples to a single class under challenging wild testing scenarios ( Niu et al. 2023 ) . Various TTA methods have been proposed to tackle the above issue, such as filtering unreliable gradients using predefined thresholds ( Niu et al. 2022 ; Lee et al. 2024 ; Hu et al. 2025a ) , or mitigating disturbances through optimization of a sharpness-aware loss surface ( Niu et al. 2023 ) . However, the use of heuristic thresholds for gradient selection remains problematic, as such thresholds are difficult to define and generalize across domains. In this sense, filtering or suppressing noisy gradients can only be partial, and the model remains exposed to trivial solutions by optimizing with the remaining gradients, for which entropy can still be minimized by collapsing into constant one-hot outputs regardless of the inputs. As a result, it is unavoidable that these methods still risk collapse during deployment, particularly under prolonged or more challenging testing scenarios, as in Tables 2 & 6 . In this paper, we resolve the collapse risk of test-time entropy minimization by drawing inspiration from negative-free self-supervised learning (SSL) ( Chen & He 2021 ; Zhang et al. 2022a ) , which seeks to learn meaningful representations by maximizing the similarity between two augmented views of the same image. Negative-free SSL introduces an asymmetric structure to prevent the two branches from collapsing toward the same constant, resolving collapse from the architecture design. However, asymmetric structure is originally developed for representation learning, and naively applying asymmetry to our context is infeasible or inferior, as: 1) test-time entropy minimization typically has only one prediction branch and optimizes entropy instead of similarity; 2) traditional Siamese design requires extra backbone passes, which impairs efficiency for our test-time learning. Figure 1: Comparisons on architectures. (a) Alignment-oriented SSL methods (BYOL ( Grill et al. 2020 ) , SimSiam ( Chen & He 2021 ) ). (b) Test-time entropy minimization (Tent) ( Wang et al. 2021 ) . (c) Our ZeroSiam, which designs a minimal asymmetry for entropy minimization with a lightweight predictor and a stop-gradient branch—without augmentations, extra encoder passes, or teacher models—to substantially enhance learning stability and boost performance while retaining efficiency. Therefore, we design ZeroSiam, an efficient asymmetric Siamese architecture tailored for test-time entropy minimization, using zero augmentations and extra backbone passes. To embed asymmetry within a single backbone pass, ZeroSiam decouples a prediction into two asymmetric outputs based on the same feature: an online branch with a learnable predictor and a target branch with stop-gradient. As shown in Figure 1 , we minimize entropy on the online branch to learn discriminative features, while performing asymmetric predictor–target alignment to prevent collapsed constant solutions. We investigate ZeroSiam’s behaviors empirically and theoretically during TTA in Section 3.2 , demonstrating that our architecture avoids collapse and is also meaningful in absorbing and regularizing biased learning signals at testing, which improves the performance of entropy optimization even when no collapse occurs. ZeroSiam achieves stable and effective entropy minimization using only a single predictor, introducing negligible overhead (see Table 1 ). Despite its simplicity, ZeroSiam yields superior robustness compared to prior methods across diverse architectures and a wide range of test scenarios, e.g. , adapting reliably even with incorrect pseudo labels (see Table 6 ). Main Novelty and Contributions : 1) We propose ZeroSiam, a simple yet effective asymmetric architecture for test-time entropy minimization without collapse, while requiring zero augmentations, no extra backbone passes, and no teacher models to achieve efficient online deployment. 2) We provide complementary empirical and theoretical insights into ZeroSiam’s behaviors (c.f. Section 3.2 ), demonstrating that ZeroSiam also helps absorb and regulate non-generalizable biased learning signals at testing, which enhances test-time learning performance even when no collapse occurs. 3) Extensive experiments on

Read the complete text