Bootstrapping Video Semantic Segmentation Model via Distillation-assisted Test-Time Adaptation

Jihun Kim*, Hoyong Kwon*, Hyeokjun Kweon*, Kuk-Jin Yoon · 2026

Read Bootstrapping Video Semantic Segmentation Model via Distillation-assisted Test-Time Adaptation by Jihun Kim*, Hoyong Kwon*, Hyeokjun Kweon*, Kuk-Jin Yoon (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-10950.txt.

Introduction

Kim and three colleagues turn an image segmentation model into a video model without annotated videos, in a method called DiTTA. Frame-by-frame image models ignore time. The model SAM2 spreads masks well but lacks semantic sense and costs too much to run on video directly. DiTTA distills SAM2 time knowledge into the image model in one brief pass, plus a light module merging context across frames. With only the first tenth of a video for adaptation, it rivals fully supervised video methods on VSPW and Cityscapes.

Catalog record

CREATORJihun Kim*, Hoyong Kwon*, Hyeokjun Kweon*, Kuk-Jin Yoon
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE36113
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.10950

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Bootstrapping Video Semantic Segmentation Model via Distillation-assisted Test-Time Adaptation Jihun Kim*, Hoyong Kwon*, Hyeokjun Kweon*, Kuk-Jin Yoon Abstract Fully supervised Video Semantic Segmentation (VSS) relies heavily on densely annotated video data, limiting practical applicability. Alternatively, applying pre-trained Image Semantic Segmentation (ISS) models frame-by-frame avoids annotation costs but ignores crucial temporal coherence. Recent foundation models such as SAM2 enable high-quality mask propagation yet remain impractical for direct VSS due to limited semantic understanding and computational overhead. In this paper, we propose DiTTA (Distillation-assisted Test-Time Adaptation) , a novel framework that converts an ISS model into a temporally-aware VSS model through efficient test-time adaptation (TTA), without annotated videos. DiTTA distills SAM2’s temporal segmentation knowledge into the ISS model during a brief, single-pass initialization phase, complemented by a lightweight temporal fusion module to aggregate cross-frame context. Crucially, DiTTA achieves robust generalization even when adapting with highly limited partial video snippets ( e.g. , initial 10%), significantly outperforming zero-shot refinement approaches that repeatedly invoke SAM2 during inference. Extensive experiments on VSPW and Cityscapes demonstrate DiTTA’s effectiveness, achieving competitive or superior performance relative to fully-supervised VSS methods, thus providing a practical and annotation-free solution for real-world VSS tasks. The code is available at https://github.com/jihun1998/DiTTA . * * footnotetext: These authors contributed equally to this work. 1 Introduction Video Semantic Segmentation (VSS) is a core task in understanding dynamic scenes. However, conventional works typically rely on large-scale annotated video datasets such as VSPW 37 , whose construction demands extensive pixel-level labeling and substantial human effort. In contrast, Image Semantic Segmentation (ISS) models are trained solely on static images 4 ; 59 ; 66 ; 29 ; 27 ; 28 ; 62 ; 64 ; 63 , which are significantly easier to collect and organize into large-scale annotated datasets. Consequently, a common workaround for VSS is to apply pre-trained ISS models in a frame-by-frame manner (Fig. 1 A). By directly employing ISS models, this strategy circumvents the need for expensive annotated video datasets. However, it processes each frame in isolation, disregarding the temporal continuity in video. Moving objects typically follow coherent trajectories, which provide strong temporal cues, especially under occlusion. Neglecting these cues often leads to inconsistent predictions across frames. Figure 1 : VSS using ISS model and SAM2. \mathbf{I} and \mathbf{S} are frames and their semantic segmentation results, respectively. In (C), DiTTA adapts the ISS model to the VSS model at test time by distilling temporal knowledge from SAM2 over a few initial frames. The pink arrow denotes an temporal fusion add-on. To address this lack of temporal reasoning in frame-wise inference, we explore the use of SAM2 43 , a recently introduced open-source foundation model for promptable video segmentation. Our motivation stems from its capability for high-precision temporal mask propagation and strong generalization. A straightforward way to leverage this potential is to refine frame-wise ISS predictions using SAM2 in a zero-shot post-processing manner (Fig. 1 B). Specifically, object-level visual prompts extracted from the initial ISS output are propagated temporally using SAM2, refining the segmentation predictions on subsequent frames. While this refinement approach leverages SAM2’s temporal capabilities, it still suffers from several limitations that hinder practical deployment. First, it incurs substantial computational and memory overhead, as SAM2 must be invoked on every frame to track multiple objects throughout the video. New objects must be assigned for tracking whenever they appear, and it remains unclear when to discard obsolete ones. Second, this zero-shot approach relies heavily on the initial ISS results, which may be inaccurate. Any errors in the initial predictions directly propagate across frames, offering limited capacity for correction. We propose a novel and adaptive strategy, named Distillation-assisted Test-Time Adaptation (DiTTA) . DiTTA adapts a pre-trained ISS model to a given test video, effectively transforming it into a video-specific VSS model (Fig. 1 C). Since standard ISS models operate on a single frame at a time, we devise a lightweight add-on for temporal fusion that aggregates information across frames. The key novelty of DiTTA lies in test-time knowledge distillation from SAM2. Specifically, DiTTA enables the ISS model to acquire SAM2’s temporal segmentation capability by mimicking its spatio-temporal mask predictions. Importantly, this distillation is performed using only the initial few frames of the test video—without access to the full video. During this, the ISS model learns SAM2’s temporal segmentation behavior by mimicking its outputs, and is bootstrapped into a temporally-aware VSS model tailored to the input video. Once adaptation is completed, the resulting model is used for inference on subsequent frames without any further adaptation or reliance on SAM2. In contrast to computationally intensive zero-shot refinement that invoke SAM2 repeatedly at inference time, DiTTA enables fast and consistent video segmentation. We validate our method through comprehensive experiments, comparing against three representative baselines: (1) standard ISS/VSS models, (2) existing TTA methods, and (3) zero-shot refinement with SAM2. We further demonstrate strong generalization to domain-shifted scenarios and different training data regimes, as well as competitive performance under full-video adaptation. These results highlight DiTTA as a practical, data-efficient, and scalable solution for real-world video segmentation tasks. 2 Related Works 2.1 Video Semantic Segmentation (VSS) VSS 11 ; 14 ; 19 ; 21 ; 30 ; 31 ; 32 ; 35 ; 37 ; 38 ; 44 ; 45 ; 47 ; 54 ; 3 ; 60 ; 5 aims to predict pixel-wise classes for each frame in a video. Previous VSS studies have primarily focused on leveraging temporal information between frames to exploit the sequential nature of video data. One widely used approach is to incorporate optical flow 9 between sequential frames. Methods following this approach 8 ; 11 ; 18 ; 19 ; 38 have devised additional modules that estimate optical flow and integrate it at the feature level to capture temporal dynamics. Recently, MPVSS 58 introduces a query-based flow method to propagate masks from keyframes, improving efficiency. Another approach leverages attention mechanisms to capture temporal dependencies across frames 16 ; 31 ; 42 ; 44 ; 45 ; 47 ; 54 . For example, TDNet 16 employs an attention mechanism within a temporally distributed network, enabling faster inference. CFFM 45 proposes coarse-to-fine feature mining modules, while MRCFA 47 introduces cross-frame affinity to make effective use of temporal information. VPseg 14 , specifically designed for driving scenes, incorporates the concept of a vanishing point as a valuable prior for this setting. Despite significant progress in VSS, we find that conventional VSS studies have been heavily dependent on specialized VSS datasets, which limits their practicality in real-world applications. 2.2 Using External Source for Video Segmentation Extensive research on video segmentation has leveraged external sources, which can generally be categorized into two approaches: (1) boundary estimation models ( e.g. , superpixel) designed to effectively capture segment structures 12 ; 13 ; 22 ; 51 , and (2) temporal knowledge sources, such as optical flow, that provide insights into motion context between frames 11 ; 18 ; 19 ; 38 ; 17 ; 36 . Additionally, recent studies have explored segmentation using segmentation knowledge from SAM/SAM2 24 ; 43 . This includes applications not only in general video segmentation 34 ; 40 ; 49 ; 61 but also in specialized domains such as camouflaged object detection 67 ; 71 and medical imaging

Read the complete text