ETTA: Efficient Test-Time Adaptation for Vision-Language Models through Dynamic Embedding Updates

Hamidreza Dastmalchi, Aijun An, Ali Cheraghian · 2025

Read ETTA: Efficient Test-Time Adaptation for Vision-Language Models through Dynamic Embedding Updates by Hamidreza Dastmalchi, Aijun An, Ali Cheraghian (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-05898.txt.

Introduction

Hamidreza Dastmalchi and co-authors present a 2025 paper on vision and language models used in new settings. Such models do well on tasks with no training examples. They falter when the test data shift away from what they saw before. One fix is to adapt at test time, using test items that have no labels.

Their method, ETTA, keeps a rolling store of all test items, not just the most sure ones. That store refines the line between classes step by step. A second part picks the best text prompts for each class and blends scores by confidence. On two test sets, ETTA beat the best past methods. It did so with low memory and compute cost. Code is free online.

Catalog record

CREATORHamidreza Dastmalchi, Aijun An, Ali Cheraghian
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE42 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2508-05898

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

ETTA: Efficient Test-Time Adaptation for Vision-Language Models through Dynamic Embedding Updates Hamidreza Dastmalchi, Aijun An, Ali Cheraghian Abstract Pretrained vision-language models (VLMs) like CLIP show strong zero-shot performance but struggle with generalization under distribution shifts. Test-Time Adaptation (TTA) addresses this by adapting VLMs to unlabeled test data in new domains. While some TTA methods rely on prompt-tuning, training-free cache-based approaches are preferred for efficiency. However, current cache-based TTA models store only a limited set of high-confidence samples, restricting the decision boundary to these samples and ignoring the influence of other incoming test data. To address this, we propose Efficient Test-Time Adaptation (ETTA), introducing a Recursive Updating module that integrates all incoming test samples, progressively refining the decision boundary. This strategy mimics an unbounded cache, dynamically updating contextual embeddings for improved accuracy with minimal memory and computational overhead. ETTA also includes an Adaptive Ensemble module to reduce prompt dependency in image-to-text scores by dynamically selecting optimal prompts for each class. Furthermore, ETTA adaptively combines scores from both modules based on confidence levels, leveraging their complementary strengths. Extensive experiments on two benchmarks confirm that ETTA surpasses the state-of-the-art TTA models in computational complexity and accuracy, setting a new standard for effective, efficient test-time adaptation. The code has been released at https://github.com/hamidreza-dastmalchi/ETTA . ETTA † † email: [email protected] † † email: [email protected] † † email: [email protected] † † affiliation: York University Toronto, Canada † † affiliation: The Australian National University Canberra, Australia † † affiliation: Data61-CSIRO, Australia 1 Introduction Vision-Language Models (VLMs) like CLIP [ Radford et al.(2021)Radford, Kim, Hallacy, Ramesh, Goh, Agarwal, Sastry, Askell, Mishkin, Clark, et al. ] and ALIGN [ Jia et al.(2021)Jia, Yang, Xia, Chen, Parekh, Pham, Le, Sung, Li, and Duerig ] excel at merging visual and textual information for tasks such as classification [ Gondal et al.(2024)Gondal, Gast, Ruiz, Droste, Macri, Kumar, and Staudigl , Huang et al.(2023)Huang, Dong, Yang, Huang, Lau, Ouyang, and Zuo , Jeong et al.(2023)Jeong, Zou, Kim, Zhang, Ravichandran, and Dabeer ] , generation [ Patashnik et al.(2021)Patashnik, Wu, Shechtman, Cohen-Or, and Lischinski , Frans et al.(2022)Frans, Soros, and Witkowski , Nichol et al.(2021)Nichol, Dhariwal, Ramesh, Shyam, Mishkin, McGrew, Sutskever, and Chen , Ramesh et al.(2022)Ramesh, Dhariwal, Nichol, Chu, and Chen ] , and recognition [ Ke et al.(2018)Ke, Bennamoun, An, Sohel, and Boussaid , Wang et al.(2021)Wang, Xing, and Liu , Zhang et al.(2022b)Zhang, Guo, Zhang, Li, Miao, Cui, Qiao, Gao, and Li , Chen et al.(2024)Chen, Liu, Kong, Chen, Zhu, Ma, Liu, and Wang ] . CLIP’s core strength lies in its ability to generalize to new tasks without fine-tuning, enabled by training on large-scale image-text pairs that align visual and textual features in a shared embedding space via contrastive learning. Using crafted prompts like “a photo of a <category>,” CLIP generates text embeddings that match visual embeddings, enabling classification across tasks it was not explicitly trained for. Figure 1 : (Left): Conventional cache-based adaptation, where a few high-confidence prototypes (shown as larger data points) define the decision boundary, often leading to misclassification of samples near the boundary. (Right): Our Recursive Update cache incorporates all data points in forming the decision boundary, resulting in enhanced classification performance. Inspired by prompt learning in NLP [ Li and Liang(2021) , Lester et al.(2021)Lester, Al-Rfou, and Constant , Gao et al.(2020)Gao, Fisch, and Chen ] , researchers have introduced prompt tuning as an efficient way to align CLIP with various downstream tasks. However, prompt tuning requires labeled data, which is often unavailable in real-world scenarios. To mitigate this limitation, Test-time Prompt Tuning (TPT) [ Shu et al.(2022)Shu, Nie, Huang, Yu, Goldstein, Anandkumar, and Xiao ] was introduced to enhance zero-shot classification by minimizing prediction entropy across image augmentations. C-TPT [ Yoon et al.(2024)Yoon, Yoon, Tee, Hasegawa-Johnson, Li, and Yoo ] builds upon TPT, addressing its calibration issue. However, the computational demand of optimizing learnable prompts through backpropagation remains misaligned with the efficiency requirements of test-time tasks. To address this issue, training-free adaptation strategies [ Karmanov et al.(2024)Karmanov, Guan, Lu, El Saddik, and Xing , Zhang et al.(2024)Zhang, Zhu, Tang, Ma, Zhou, and Zhang , Udandarao et al.(2023)Udandarao, Gupta, and Albanie , Pratt et al.(2023)Pratt, Covert, Liu, and Farhadi ] have been proposed, enabling VLMs to adapt to new samples at test time without backpropagation. Nevertheless, the training-free TTA approaches in [ Udandarao et al.(2023)Udandarao, Gupta, and Albanie ] and [ Pratt et al.(2023)Pratt, Covert, Liu, and Farhadi ] rely on external models, specifically diffusion models [ Ho et al.(2020)Ho, Jain, and Abbeel ] and large language models (LLMs), respectively. In contrast, cache-based methods [ Karmanov et al.(2024)Karmanov, Guan, Lu, El Saddik, and Xing , Zhang et al.(2024)Zhang, Zhu, Tang, Ma, Zhou, and Zhang ] are more efficient and independent of external models. These methods store a limited number of highly confident test samples, allowing for the computation of affinities within the image embedding space. Cache-based TTA approaches are efficient but rely exclusively on a small set of highly confident test samples for predictions, limiting the model’s adaptability. In Fig. 1 (Left), this limitation is demonstrated using a 2D toy example, where only three prototypes per class (shown as larger data points) define the decision boundary. As a result, this boundary depends solely on these prototypes, leading to misclassification within the hashed region (indicated by a shaded area in the figure). In contrast, we propose a Recursive Updating strategy in which all points contribute to forming the decision boundary, as illustrated in Fig. 1 (Right), resulting in more reliable classifications. Our approach achieves accuracy equal to an unbounded cache while requiring significantly less memory and computational overhead. Additionally, cache-based TTA models have another limitation: they still rely on prompts. While combining cache scores with VLM scores improves predictions, the latter are highly sensitive to prompt wording and vary across domains and classes. Although prompt ensembling [ Radford et al.(2021)Radford, Kim, Hallacy, Ramesh, Goh, Agarwal, Sastry, Askell, Mishkin, Clark, et al. ] aims to improve robustness, our experiments show it does not consistently enhance performance. To address this, we propose Adaptive Ensembling, which filters out irrelevant prompts based on their similarity to the input image. To this end, we introduce Efficient Test-Time Adaptation (ETTA) to address the prompt dependency and limited adaptability of the conventional cache-based model. ETTA comprises three key components: the Adaptive Ensemble module, the Recursive Update module, and an Adaptive Fusion module. The Adaptive Ensemble module enhances the reliability of image-to-text scores by reducing sensitivity to prompt quality. Similar to [ Radford et al.(2021)Radford, Kim, Hallacy, Ramesh, Goh, Agarwal, Sastry, Askell, Mishkin, Clark, et al. ] , we prepend a set of templates with class names to create class-specific prompts. However, rather than simply ensembling the scores from these prompts, our module filters out prompts less representative of each class based on their similarity to the input image. It then averages the embeddings of only the most relevant prompts, leading to more accurate predictions for each test sample. Additionally, the Recursive Update module matches the performance of an unbounded cache by progressively refining the decision boundary using all incoming test samples. Unlike traditional caches that store multiple prototypes, it maintains just one contextual embedding per class, updated recursively. This design minimizes memory and

Read the complete text