When Small Guides Large: Cross-Model Co-Learning for Test-Time Adaptation
Chang’an Yi * Foshan University Foshan, China [email protected] Xiaohui Deng * Foshan University Foshan, China [email protected] Guohao Chen * Nanyang Technological University Singapore [email protected] Yan Zhou Foshan University Foshan, China [email protected] Qinghua Lu · 2025
Read When Small Guides Large: Cross-Model Co-Learning for Test-Time Adaptation by Chang’an Yi * Foshan University Foshan, China [email protected] Xiaohui Deng * Foshan University Foshan, China [email protected] Guohao Chen * Nanyang Technological University Singapore [email protected] Yan Zhou Foshan University Foshan, China [email protected] Qinghua Lu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-23724.txt.
Introduction
Chang'an Yi and co-authors let small models guide large ones at test time in a 2025 paper. Test-time tuning bends a trained model to new data on the fly, with no labels to lean on. Past work tuned one model alone, and the team asked what peer models could add to the mix.
Even a small model, they found, can steer a far larger one with fair, sure facts the big one lacks. Size gaps did not block the help. Their frame, called COCA, has models teach each other as tests roll in. Joint tuning beat lone tuning across shifts.
Catalog record
| CREATOR | Chang’an Yi * Foshan University Foshan, China [email protected] Xiaohui Deng * Foshan University Foshan, China [email protected] Guohao Chen * Nanyang Technological University Singapore [email protected] Yan Zhou Foshan University Foshan, China [email protected] Qinghua Lu |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 59 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2506-23724
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
When Small Guides Large: Cross-Model Co-Learning for Test-Time Adaptation Chang’an Yi * Foshan University Foshan, China [email protected] Xiaohui Deng * Foshan University Foshan, China [email protected] Guohao Chen * Nanyang Technological University Singapore [email protected] Yan Zhou Foshan University Foshan, China [email protected] Qinghua Lu Foshan University Foshan, China [email protected] Shuaicheng Niu † Nanyang Technological University Singapore [email protected] Abstract Test-time Adaptation (TTA) adapts a given model to testing domain data with potential domain shifts through online unsupervised learning, yielding impressive performance. However, to date, existing TTA methods primarily focus on single-model adaptation. In this work, we investigate an intriguing question: how does cross-model knowledge influence the TTA process? Our findings reveal that, in TTA’s unsupervised online setting, each model can provide complementary, confident knowledge to the others, even when there are substantial differences in model size. For instance, a smaller model like MobileViT (10.6M parameters) can effectively guide a larger model like ViT-Base (86.6M parameters). In light of this, we propose COCA, a c ross-m o del c o-le a rning framework for TTA, which mainly consists of two main strategies. 1) Co-adaptation adaptively integrates complementary knowledge from other models throughout the TTA process, reducing individual model biases. 2) Self-adaptation enhances each model’s unique strengths via unsupervised learning, enabling diverse adaptation to the target domain. Extensive experiments show that COCA, which can also serve as a plug-and-play module, significantly boosts existing SOTAs, on models with various sizes—including ResNets, ViTs, and Mobile-ViTs—via cross-model co-learned TTA. For example, with Mobile-ViT’s guidance, COCA raises ViT-Base’s average adaptation accuracy on ImageNet-C from 51.7% to 64.5%. The code is publicly available at https://github.com/ycarobot/COCA. † † footnotetext: Equal contribution. † Corresponding author. 1 Introduction Deep neural networks have demonstrated remarkable performance when the training and test domains follow the same data distribution [ 24 ] . However, their accuracy drops significantly when testing data suffers from distribution shifts [ 24 , 22 ] . To address this, numerous attempts have been explored, such as unsupervised domain adaptation [ 9 ] , source-free domain adaptation [ 20 ] , and domain generalization [ 47 ] . However, in dynamic applications like autonomous driving, environmental conditions are constantly evolving, causing continuous data distribution shifts [ 21 , 32 ] . To maintain stable and excellent performance, models need to adapt continuously to new conditions—an especially challenging task given the unpredictability of new environments [ 31 ] . (a) Distinctness of model predictions (b) Benefits of complementary knowledge (c) TTA robustness against noisy labels Figure 1 : Motivation for COCA. (a) Pre-trained models exhibit distinct strengths due to differences in training strategies, architectures, or sizes. (b) TTA [ 36 ] , using complementary confident predictions from another model, significantly enhances adaptation performance. (c) Larger models are more vulnerable to erroneous guidance than smaller ones, highlighting differences in robustness during test time. Results are based on predicting ImageNet-C (Gaussian noise, level 5) with various ResNet-based [ 11 ] and Transformer-based [ 7 ] models. Confident samples are filtered using an entropy threshold of 0.4*ln1000 following EATA [ 28 ] . Recently, test-time adaptation (TTA) has emerged as a promising paradigm for tackling these challenges [ 33 , 23 , 29 , 18 , 22 ] . By directly leveraging test data through self-supervised or unsupervised objectives, TTA benefits from an online, source-free approach. This is especially true for fully TTA methods [ 36 , 28 ] , which can be applied to any pre-trained model, thereby significantly broadening its appeal in real-world applications compared to earlier techniques. However, existing TTA methods [ 36 , 28 , 18 ] typically focus on single-model adaptation and are excessively sensitive to pseudo label noise. Motivated by the success of co-learning [ 44 ] and knowledge distillation [ 15 , 26 ] in conventional supervised learning, we pose a natural question: Can co-learning also enhance TTA? To investigate this, we conduct two key experiments to examine the complementary capabilities of different models during TTA, revealing that: 1) Complementary knowledge between models : As from Fig. 1 (a), we show that different models can yield complementary predictions for the same data due to variations in training strategies, architectures, or sizes. By leveraging the complementary confident samples from the other model to construct a pseudo cross-entropy loss during TTA, we significantly improve the adaptation performance on both the large and small model compared to Tent [ 36 ] . 2) Differential robustness of models in TTA : We evaluate the TTA robustness of various models by adapting them to test samples with randomly erroneous labels, using the same learning rate as in Tent [ 36 ] . While larger models initially yield higher accuracy, they are more sensitive to incorrect pseudo labels during TTA, as shown in Fig. 1 (c). These findings indicate the potential of a small model to significantly improve the TTA performance and robustness of a larger model through cross-model co-learning. In light of the above motivation, we propose a novel approach called COCA — c ross-m o del c o-le a rning for enhanced TTA, which mainly consists of two strategies, i.e . , co-adaptation and self-adaptation. Co-adaptation integrates complementary knowledge between models during TTA to reduce individual biases. To achieve this, we introduce an adaptive temperature-scaled marginal entropy loss to promote mutual learning. The adaptive temperature is automatically determined via an anchor-guided alignment loss, using the larger model as an anchor. This enables robust knowledge aggregation, particularly when there is a significant size disparity between models, or the learning signals from the smaller models can otherwise be overstated or overlooked. Additionally, we propose a cross-model knowledge distillation loss, leveraging combined predictions as pseudo-labels to supervise both models and maximize the effective use of cross-model knowledge. On the other hand, self-adaptation allows each model to independently adapt using existing unsupervised learning objectives from single-model TTA, enhancing each model’s unique strengths and fostering diverse adaptation to the target domain. A key distinction of COCA from conventional methods is its bi-directional learning capability. Unlike typical teacher-student frameworks that distill knowledge unidirectionally from a powerful teacher to a weaker student [ 5 , 4 , 16 ] , or co-learning schemes where models with similar architectures are involved [ 38 , 6 , 48 ] , COCA enables every model—even those with weaker performance on out-of-distribution (OOD) data—to contribute valuable insights to others. This effective exchange of complementary knowledge allows COCA to perform well in various co-TTA scenarios, such as ResNet-18+ViT-Base and Mobile-ViT+ViT-Large, as shown in Table 12 in Appendix. Our key contributions are as follows: • We reveal that two distinct models in TTA can mutually enhance each other. Each model extracts useful information from the other’s predictions, improving TTA performance and stability, even when there is a large disparity in model size, such as between Mobile-ViT (10.6M parameters) and ViT-Large (304.2M parameters). • We propose COCA, a novel approach that seamlessly integrates co-adaptation and self-adaptation. Co-adaptation is achieved via adaptive temperature-scaled marginal entropy loss and cross-model distillation loss, enabling knowledge sharing to reduce biases. Self-adaptation enhances each model’s unique strengths via unsupervised learning, enabling diverse adaptation to target data. • Extensive experiments show that COCA, which is also a plug-and-play module, significantly boosts TTA performance and