SemiDAViL: Semi -supervised D omain A daptation with Vi sion- L anguage Guidance for Semantic Segmentation

Hritam Basak*, Zhaozheng Yin · 2025

Read SemiDAViL: Semi -supervised D omain A daptation with Vi sion- L anguage Guidance for Semantic Segmentation by Hritam Basak*, Zhaozheng Yin (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-06389.txt.

Introduction

Hritam Basak and Zhaozheng Yin add language guidance to image labeling across domains, in this 2025 paper. Their task, semi-supervised domain adaptation, moves a model from a labeled source set to a target set with few labels. Plain merges of the two parent methods confuse look-alike classes and lean too hard on the source data.

Their framework leans on vision-language models, whose text side carries rich class meaning that can separate similar classes. They also add a class-balanced loss to protect rare classes. Across several domain shift tests, the method beat current rivals by clear margins, the authors report.

Catalog record

CREATORHritam Basak*, Zhaozheng Yin
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE60 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2504-06389

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

SemiDAViL: Semi -supervised D omain A daptation with Vi sion- L anguage Guidance for Semantic Segmentation Hritam Basak*, Zhaozheng Yin Abstract Domain Adaptation (DA) and Semi-supervised Learning (SSL) converge in Semi-supervised Domain Adaptation (SSDA), where the objective is to transfer knowledge from a source domain to a target domain using a combination of limited labeled target samples and abundant unlabeled target data. Although intuitive, a simple amalgamation of DA and SSL is suboptimal in semantic segmentation due to two major reasons: (1) previous methods, while able to learn good segmentation boundaries, are prone to confuse classes with similar visual appearance due to limited supervision; and (2) skewed and imbalanced training data distribution preferring source representation learning whereas impeding from exploring limited information about tailed classes. Language guidance can serve as a pivotal semantic bridge, facilitating robust class discrimination and mitigating visual ambiguities by leveraging the rich semantic relationships encoded in pre-trained language models to enhance feature representations across domains. Therefore, we propose the first language-guided SSDA setting for semantic segmentation in this work. Specifically, we harness the semantic generalization capabilities inherent in vision-language models (VLMs) to establish a synergistic framework within the SSDA paradigm. To address the inherent class-imbalance challenges in long-tailed distributions, we introduce class-balanced segmentation loss formulations that effectively regularize the learning process. Through extensive experimentation across diverse domain adaptation scenarios, our approach demonstrates substantial performance improvements over contemporary state-of-the-art (SoTA) methodologies. Code is available: GitHub . 1 Introduction Figure 1 : Major contributions of SemiDAVil: (1) We propose the first language-guided SSDA framework for semantic segmentation, (2) Utilizing spatial context via dense language guidance (DLG) improves segmentation performance, (3) Our proposed DyCE loss dynamically reweighs imbalanced class distributions, resulting in precise segmentation of minority classes . The remarkable progress in deep learning has significantly enhanced the performance of visual understanding tasks, including image classification [ 9 ] , object detection [ 109 ] , and, more recently, semantic segmentation [ 66 , 76 ] . These advancements have been particularly notable when a wealth of labeled training data is available. However, as noted in [ 73 ] , their performance degrades precipitously when confronted with annotation-scarce environments, especially in the context of semantic segmentation, where dense pixel-wise annotations are essential. Furthermore, these sophisticated models exhibit substantial vulnerability when tasked with generalizing across domains characterized by significant distributional shifts [ 39 , 33 ] - a challenge particularly evident in real-world applications where models trained on synthetic data must maintain robust performance in naturalistic settings, such as autonomous navigation systems [ 71 , 21 ] . This inherent limitation in cross-domain generalization has catalyzed the emergence of two pivotal research paradigms: Domain Adaptation (DA) and Semi-supervised Learning (SSL). The confluence of DA and SSL has given rise to Semi-supervised Domain Adaptation (SSDA), a hybrid approach that strategically leverages three distinct data streams: comprehensively labeled source domain data, sparsely labeled target domain samples, and a wealth of unlabeled target domain instances [ 70 , 46 ] . While SSDA holds intuitive appeal for real-world applications, existing methods encounter critical limitations when applied to semantic segmentation tasks. Specifically, (1) despite achieving accurate segmentation boundaries, current approaches [ 97 , 57 ] often suffer from misclassification among visually similar classes , due to restricted supervision within the target domain; (2) the SSDA framework tends to over-prioritize source domain features, driven by abundant source labels, while generating error-prone pseudo-labels for target data , which hampers adaptation performance; (3) class-imbalance, a common issue in real-world datasets, exacerbates these challenges, limiting effective exploration and representation of minority (tail) classes in the target domain. To address the identified SSDA challenges, we augment the SSDA paradigm with vision-language (VL) guidance using VLMs (e.g., CLIP [ 65 ] ) to enrich semantic representation, leveraging their large-scale image-caption pretraining. By incorporating VLM features into a global-local context exploration module, we mitigate misclassification among visually similar classes . To tackle the over-reliance on source features, we introduce a joint embedding space guided by language priors, enhancing instance separability and reducing domain bias , unlike traditional divergence-based alignment methods [ 99 , 45 ] . Finally, to combat class imbalance , we design a tailored cross-entropy loss that dynamically reweighs minority classes, thereby facilitating more equitable exploration and representation of tail classes in the target domain. Specifically, our contributions can be summarized as: 1. Language-Guided SSDA Framework : We pioneer the first language-guided SSDA framework for semantic segmentation by harnessing the rich semantic knowledge encoded in pre-trained Vision-Language Models (VLMs). Our novel attention-based fusion mechanism seamlessly integrates visual features with dense language embeddings, establishing a robust semantic bridge between source-target domains while providing enhanced contextual understanding. 2. Enhanced Feature Localization : Recognizing that VL pre-training primarily operates at the image level, we address the critical challenge of feature localization in semantic segmentation through targeted fine-tuning. To mitigate the risks of overfitting and semantic knowledge degradation inherent in limited-annotation scenarios, we develop a sophisticated consistency regularization framework that preserves the rich semantic representations acquired during pre-training. 3. Adaptive Class-Balanced Loss : To tackle class imbalance in a limited annotation scenario, we introduce a Dynamic Cross-Entropy (DyCE) loss formulation that dynamically calibrates the learning emphasis on tail classes. This innovative, plug-and-play loss mechanism demonstrates broad applicability across various class-imbalanced learning scenarios. 4. State-of-the-Art Performance : Through detailed evaluation across diverse domain-adaptive and class-imbalanced segmentation benchmarks, our methodology demonstrates superior performance and robustness, consistently surpassing contemporary state-of-the-art approaches by significant margins. 2 Related Works 2.1 Semi-supervised Domain Adaptation Recent advances in Semi-Supervised Domain Adaptation (SSDA) for semantic segmentation have focused on utilizing limited labeled target data and abundant unlabeled data to bridge the domain gap at the pixel level [ 2 , 3 ] . Early approaches like MME [ 70 ] and ASDA [ 63 ] used entropy minimization for feature alignment, but their classification-centric strategies struggled with fine-grained segmentation tasks, leading to suboptimal boundary delineation. To address this, SSL-based methods such as DECOTA [ 96 ] and SS-ADA [ 94 ] employed teacher-student frameworks with consistency constraints, generating pseudo-labels for unlabeled target data. However, these methods faced issues with noisy pseudo-labels, particularly for minority and boundary classes. More recent methods have explored novel directions: S-Depth [ 35 ] leverages self-supervised depth estimation as an auxiliary task to enhance feature learning, while DSTC [ 26 ] introduces a domain-specific teacher-student framework that dynamically adapts to target domain characteristics. IIDM [ 25 ] proposes an innovative inter-intra-domain mixing strategy to address domain shift and limited supervision simultaneously. However, these methods often struggle with two critical limitations: class confusion due to limited supervision and skewed data distribution favoring source domain representations. 2.2 Vision Language Model Vision-Language Models (VLMs), like CLIP and its extensions [ 65 , 37 , 55 , 105 ] , leverage large-scale image-text pre-training for semantic segmentation via a shared embedding space that aligns visual and textual features. Initial zero-shot methods, such as MaskCLIP [ 104 ] and GroupViT [ 92 ] , struggled with boundary precision due to reliance on high-level features. Later, fine-tuned models like OpenSeg [ 27 ] and LSeg [ 44 ] improved segmentation accuracy using labeled data and text embeddings. Techniques such as ZegFormer [ 19

Read the complete text