Segmentation Assisted Incremental Test Time Adaptation in an Open World

Manogna Sreenivas, Soma Biswas · 2025

Read Segmentation Assisted Incremental Test Time Adaptation in an Open World by Manogna Sreenivas, Soma Biswas (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-20029.txt.

Introduction

Manogna Sreenivas and Soma Biswas present a 2025 arXiv paper on vision-language models that keep learning at test time in an open world. Deployed models meet new object classes and new settings mid-stream. The authors set a benchmark where both arrive continuously, mixing classic test-time adaptation with an oracle that can be asked to label suspicious samples.

Their module, SegAssist, borrows the model's own segmentation skill to pick which samples most likely hide unseen classes, and it needs no training. Experiments across datasets showed the selection sharpening adaptation. The authors pitch segmentation-guided questioning as cheap vigilance for models in changing worlds.

Catalog record

CREATORManogna Sreenivas, Soma Biswas
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE48 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2508-20029

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Segmentation Assisted Incremental Test Time Adaptation in an Open World Manogna Sreenivas, Soma Biswas Abstract In dynamic environments, unfamiliar objects and distribution shifts are often encountered which challenge the generalization abilities of the deployed trained models. This work addresses Incremental Test Time Adaptation (ITTA) of Vision-Language Models (VLMs), tackling scenarios where unseen classes and unseen domains continuously appear during testing. Unlike traditional Test Time Adaptation approaches, where the test stream comes only from a predefined set of classes, our framework allows models to adapt simultaneously to both covariate and label shifts, actively incorporating new classes as they emerge. Towards this goal, we establish a new benchmark for ITTA, integrating single-image TTA methods for VLMs with active labeling techniques that query an oracle for samples potentially representing unseen classes during test time. We propose a segmentation assisted active labeling module, termed SegAssist, which is training-free and repurposes the VLM’s segmentation capabilities to refine active sample selection, prioritizing samples likely to belong to unseen classes. Extensive experiments on several benchmark datasets demonstrate the potential of SegAssist to enhance the performance of VLMs in real-world scenarios, where continuous adaptation to emerging data is essential. 1 Introduction Consider a model deployed in real world, navigating a dynamic environment where it encounters familiar objects in unfamiliar settings and new objects it has never seen before. For instance, a model trained to recognize urban entities like cars, trucks, and pedestrians might suddenly come across unfamiliar elements, such as bicycles or animals, when it transitions to a rural landscape. In these situations, the model must not only adapt to the changed appearance of known objects, affected by varying factors like lighting and weather, but also correctly identify unfamiliar objects as new classes instead of misclassifying them. Moreover, it must seamlessly integrate this new information to remain accurate and relevant in real-time. This is the underlying motivation of Incremental Test Time Adaptation (ITTA) , where a deployed model must continuously adapt to changes in data distribution and incorporate emerging classes in an online manner. Traditional TTA methods [ 27 , 2 , 24 , 1 ] aim to recognize known classes in unseen domains with batches of test data. Recently, VLMs such as CLIP [ 19 ] , have shown excellent zero-shot generalization abilities due to their pre-training on web-scale image-text data, making them promising candidates for the challenging single-image TTA scenario [ 22 , 11 ] . In this work, we propose a more realistic Incremental TTA setting, since data in the open-world can come not only from unseen distributions, but from unseen classes as well. Our protocol reimagines TTA as an evolving learning process, suited for real-world applications where environments are constantly changing. This is done by actively querying an oracle for labels of uncertain samples to discover and incorporate new classes, thereby continuously expanding its knowledge in an online setting. Towards this goal, we propose a novel active sample selection module termed SegAssist, which leverages pixel-wise segmentation to determine whether an uncertain sample should be chosen for active labeling. This ensures that we utilize the limited active labeling budget judiciously, primarily for labeling samples of new classes. SegAssist is a training-free plug-in module which can be seamlessly integrated with existing single-sample TTA frameworks to adapt them to the ITTA scenario. Our key contributions are summarized as follows: • We address the real-world challenging task of Incremental Test Time Adaptation (ITTA) in an open-world with both covariate and label shift using VLMs. • We establish strong baselines using different active labeling strategies and state-of-the-art CLIP-based TTA methods, paving the way for future research in this direction. • We propose a novel training-free active labeling module, named SegAssist, which dynamically queries and incorporates new classes, supporting class recognition in a continuously expanding set of classes in an online setting. SegAssist can be seamlessly plugged in with existing TTA methods to equip them for ITTA. • We propose a novel metric termed Incremental Class Detection Delay (ICDD) to quantify an ITTA method in terms of the timeliness of detecting new classes. • We demonstrate the effectiveness of our method through extensive experiments on several benchmark datasets, showcasing its robustness in handling both covariate and label shifts during test time. 2 ITTA: A step towards TTA in real world Training TTA Covariate Label Single Classification Methods shift shift image task ✓ ✓ ✓ ✗ ✗ C_{s} TENT [ 27 ] , CoTTA [ 29 ] , ROID [ 16 ] ✗ ✓ ✓ ✗ ✓ C_{s} TPT [ 22 ] , TDA [ 11 ] , DPE [ 32 ] ✗ ✓ ✓ ✓ ✗ C_{s}+1 Open world TTA [ 13 , 14 ] ✓ ✗ ✗ ✓ ✗ C_{s} + C_{u} Class Incremental Learning [ 15 , 20 , 9 ] ✗ ✓ ✓ ✓ ✓ C_{s} + C_{u} Proposed ITTA framework Table 1 : Comparison of the proposed ITTA framework with existing research directions. The motivation for addressing Incremental Test Time Adaptation (ITTA) stems from the growing demand for models that can handle dynamically changing, real-world environments, where assumptions of a fixed distribution between training and test data rarely hold. Existing approaches have attempted to address isolated aspects of these challenges, yet they often fall short in real-world scenarios such as autonomous systems, surveillance, etc. which demand that models adapt to changes in both the nature of input data (covariate shift) and the introduction and assimilation of entirely new object categories (label shift). We first describe the existing closely related research directions (Table 1 ), motivating the need for the proposed ITTA framework. • TTA with only Covariate Shift: Traditional TTA methods like TENT [ 27 ] , CoTTA [ 29 ] , ROID [ 16 ] focus on adapting models trained on clean data to handle changes in data distribution during test-time without changing the underlying classes. These methods cannot accommodate dynamically emerging new classes. • Single-Image TTA with Covariate Shift: With the advent of pretrained VLMs, methods like TPT [ 22 ] , TDA [ 11 ] , DPE [ 32 ] address single-image TTA, which is closer to a real-time scenario, but with fixed label space. Thus, they also cannot handle test samples from unseen classes outside the C_{s} predefined classes of interest. • Open-World TTA with Covariate and Label Shift: To handle unknown classes, only recently, open-world TTA methods [ 13 , 14 , 23 ] propose to do (C_{s}+1) -way classification, where C_{s} represents the seen classes and an additional class is reserved for all unknown classes. However, these methods do not support incremental learning of these new classes, which is crucial in real-world. • Class Incremental Learning (CIL): Traditional CIL techniques [ 15 , 20 ] aim to incorporate new classes with time. However, they often assume access to training data for these new classes to continually learn the model. Hence, they are ill-suited to dynamic TTA settings where only one unlabeled test sample is available at a time and immediate adaptation is desired. Our work combines these considerations by establishing a novel Incremental Test Time Adaptation (ITTA) protocol reflecting the online nature of real-world deployment. ITTA requires the model to not only adapt to covariate shifts, but also incrementally learn

Read the complete text