TAPS : Frustratingly Simple Test Time Active Learning for VLMs

Dhruv Sarkar Aprameyo Chakrabartty Bibhudatta Bhanja · 2025

Read TAPS : Frustratingly Simple Test Time Active Learning for VLMs by Dhruv Sarkar Aprameyo Chakrabartty Bibhudatta Bhanja (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-20028.txt.

Introduction

Dhruv Sarkar, Aprameyo Chakrabartty, and Bibhudatta Bhanja pose a live-stream puzzle in a 2025 paper. Smart vision tools that read both image and text can adapt as they run. But in a true stream, each item comes once, and a call on whether to ask a human guide must be made at once.

Their frame asks the guide only on the items it finds most unsure, within tight time and memory caps. It then updates its text hints on the fly from what it learns. Past methods assumed grouped data or many update passes. The simple frame aims to make live help from a guide cheap enough to use in real feeds.

Catalog record

CREATORDhruv Sarkar Aprameyo Chakrabartty Bibhudatta Bhanja
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE66 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2507-20028

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

TAPS : Frustratingly Simple Test Time Active Learning for VLMs Dhruv Sarkar Aprameyo Chakrabartty Bibhudatta Bhanja Abstract Test-Time Optimization enables models to adapt to new data during inference by updating parameters on-the-fly. Recent advances in Vision-Language Models (VLMs) have explored learning prompts at test time to improve performance in downstream tasks. In this work, we extend this idea by addressing a more general and practical challenge: Can we effectively utilize an oracle in a continuous data stream where only one sample is available at a time, requiring an immediate query decision while respecting latency and memory constraints? To tackle this, we propose a novel Test-Time Active Learning (TTAL) framework that adaptively queries uncertain samples and updates prompts dynamically. Unlike prior methods that assume batched data or multiple gradient updates, our approach operates in a real-time streaming scenario with a single test sample per step. We introduce a dynamically adjusted entropy threshold for active querying, a class-balanced replacement strategy for memory efficiency, and a class-aware distribution alignment technique to enhance adaptation. The design choices are justified using careful theoretical analysis. Extensive experiments across 10 cross-dataset transfer benchmarks and 4 domain generalization datasets demonstrate consistent improvements over state-of-the-art methods while maintaining reasonable latency and memory overhead. Our framework provides a practical and effective solution for real-world deployment in safety-critical applications such as autonomous systems and medical diagnostics. 1 Introduction The emergence of large Vision-Language Models (VLMs) [ 29 , 20 , 9 ] has revolutionized visual understanding by enabling zero-shot generalization through vision-language alignment. While prompts like “a photo of a class” guide VLM’s text-visual comparisons, crafting optimal prompts remains heuristic-driven, spurring parameter-efficient prompt learning [ 52 , 45 ] that tunes soft prompts without modifying pretrained models. Additional background on Prompt Learning and VLMs is provided in section 7 of the appendix for interested readers. While supervised prompt learning [ 51 , 16 ] achieves strong discriminative performance, its practical limitations become apparent when facing real-world deployment challenges: annotation scarcity in specialized domains (e.g., rare medical pathologies), dynamic environments requiring continuous adaptation (e.g., autonomous driving), and latency-critical applications prohibiting offline retraining. Contemporary Test-Time Adaptation (TTA) approaches [ 42 , 47 ] circumvent these barriers through self-supervision rather than pure unsupervised learning—employing surrogate training signals like entropy minimization over test-time augmentations [ 36 ] or feature distribution alignment via moment matching [ 22 ] . Nevertheless, such unsupervised adaptation mechanisms frequently exhibit performance disparities relative to fully supervised baselines. To mitigate this limitation, active learning methodologies [ 31 , 49 ] offer a hybrid solution, strategically curating instances for annotation to enhance predictive performance with minimal labeling expenditure. TAPS ( T est-Time A ctive P rompt Learning for S ingle-Sample Streams) is a novel generalization framework that dynamically identifies uncertainty in model predictions, triggering oracle annotation requests for critical samples. Upon label acquisition, the system jointly optimizes prompts using both annotated instances and the broader unlabeled test distribution. The principal innovation lies in addressing the non-trivial challenge of active sampling under streaming data constraints—specifically, operating on individual data points rather than batched inputs. This work represents the inaugural effort to address active sampling within a continuous data stream paradigm, processing individual instances sequentially while maintaining real-time adaptation capabilities. The method is frustratingly simple but keeps both latency and memory constraints in mind. Further, the simplicity of the design choices makes them amenable to conceptual and mathematical justification. We believe that as a pioneering work in this area, we should start with something that is simple yet has strong conceptual foundations. Motivation, Challenges and Novelty. Models are often deployed in many safety-critical scenarios. In such scenarios, it is often better to consult an oracle and learn from its advice rather than taking the risk of wrong prediction. However, as this is essentially a test-time setting, we have to keep certain things in mind- • The decision to consult an oracle cannot be indefinitely postponed. This is because in many real life situations, the expert feedback is needed instantaneously, like in the case of a self-driving car or other autopilot systems. • There cannot be a presupposition of the data arriving in a batched manner - that is, we have to work with the assumption of only a single sample arriving. The assumption of multiple test points being present at a time is restrictive and could prevent widespread use of the model [ 50 ] . It is also not wise to wait for the arrival of multiple samples to form a batch. • The latency of the system cannot be too high as this is a test-time setting. • The memory requirements of the system should be limited. Incorporating these characteristics in a methodology would make it highly applicable but significantly more challenging than regular active learning applications. This is because - • Making the query decision at the time of arrival of the sample in real-time would mean that we are doing so without knowing what the future data stream would look like. Under query budget constraints, this becomes an important factor where, ideally one should prioritise the most uncertain samples for labelling. Thus, any such decision-making mechanism would be fundamentally heuristic. • In a batch, one can choose the the most uncertain samples of the batch. This is a reasonable method and has been tried in various works [ 10 ] before. However, this method won’t work in our case as we only have a single sample at a time. • The active learning mechanism should be simple yet effective - more complicated mechanisms that could yield slightly better results while significantly compromising latency are not acceptable - while the former is important, the latter is equally important in a test-time setting. • There is only limited time to learn from labelled samples as they cannot be held on to for a long time due to memory constraints. We come up with a method that is simple, but non-trivially takes into account all the above considerations. Thus, while we have demonstrated our work on VLMs due to their ubiquity, it must be appreciated in its full generality. It is our strong belief that this general framework would spur on further developments in this nascent field. Comparison with prior works. A contemporary work ATTA [ 10 ] shows the impact of active learning in the standard TTA setting. Their analysis established more generalization ability in the model than standard TTA setting. However, their methodology and framework cannot be directly applied to large VLMs like CLIP [ 29 ] because they do not focus on the adaptation of a specific parameter group in a large model, while ours does so by incorporating the parameter-efficient Prompt Tuning. Further, they assumed a batch setting whereas we have a single test sample at a time and they also did multiple gradient updates per minibatch while we have only one gradient update per sample. As our approach updates and improves the model after encountering a single test example without waiting for additional examples to arrive, it is more flexible in continuous data streaming scenarios [ 50 ] . Further, relying on the batch setting, they use a complicated clustering mechanism which while effective, has great latency cost

Read the complete text