DynAlign: Unsupervised Dynamic Taxonomy Alignment for Cross-Domain Segmentation

Han Sun, Rui Gong, Ismail Nejjar, Olga Fink · 2025

Read DynAlign: Unsupervised Dynamic Taxonomy Alignment for Cross-Domain Segmentation by Han Sun, Rui Gong, Ismail Nejjar, Olga Fink (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-16410.txt.

Introduction

Sun, Gong, Nejjar, and Fink teach image-labeling models to work in new places whose label lists do not match the training set. Standard transfer methods assume both sets share labels, yet real target sets often split one class into finer ones or name things differently. Their 2025 framework, DynAlign, leans on large foundation models whose broad prior knowledge can bridge that label gap.

Prior knowledge first maps source labels onto the target list, which may hold new or finer classes. The foundation models then label pixels and reassign classes, while a fusion step adapts choices to each scene. On street-scene tests from a game world to two real photo sets, the method beat prior approaches by a wide margin. No hand labels in the target set were needed.

Catalog record

CREATORHan Sun, Rui Gong, Ismail Nejjar, Olga Fink
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE68 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2501-16410

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

DynAlign: Unsupervised Dynamic Taxonomy Alignment for Cross-Domain Segmentation Han Sun, Rui Gong, Ismail Nejjar, Olga Fink Abstract Current unsupervised domain adaptation (UDA) methods for semantic segmentation typically assume identical class labels between the source and target domains. This assumption ignores the label-level domain gap, which is common in real-world scenarios, thus limiting their ability to identify finer-grained or novel categories without requiring extensive manual annotation. A promising direction to address this limitation lies in recent advancements in foundation models, which exhibit strong generalization abilities due to their rich prior knowledge. However, these models often struggle with domain-specific nuances and underrepresented fine-grained categories. To address these challenges, we introduce DynAlign , a framework that integrates UDA with foundation models to bridge both the image-level and label-level domain gaps. Our approach leverages prior semantic knowledge to align source categories with target categories that can be novel, more fine-grained, or named differently (e.g., ‘vehicle’ to { ‘car’ , ‘truck’ , ‘bus’ }). Foundation models are then employed for precise segmentation and category reassignment. To further enhance accuracy, we propose a knowledge fusion approach that dynamically adapts to varying scene contexts. DynAlign generates accurate predictions in a new target label space without requiring any manual annotations, allowing seamless adaptation to new taxonomies through either model retraining or direct inference. Experiments on the street scene semantic segmentation benchmarks GTA → \rightarrow Mapillary Vistas and GTA → \rightarrow IDD validate the effectiveness of our approach, achieving a significant improvement over existing methods. Our code will be publically available. 1 Introduction Semantic segmentation is a crucial computer vision task that assigns category labels to each pixel in an image, enabling detailed scene understanding. Driven by advancements in deep learning, the field has recently seen significant progress, with applications ranging from autonomous driving ( Cheng et al., 2022 ; Jain et al., 2023 ) to medical image diagnosis ( Cao et al., 2022 ) . Despite this progress, models trained on labeled source datasets often struggle to generalize to data with different distributions due to variations in weather, illumination, or object appearance, resulting in degraded performance. While re-training or fine-tuning models can address this issue, these approaches require annotated in-domain data, which is particularly costly for semantic segmentation. Unsupervised Domain Adaptation (UDA) addresses the challenge of adapting a model trained on a labeled source domain to an unlabeled target domain by mitigating the adverse effects of domain shift without requiring costly data annotations. However, most UDA are constrained by domain-specific knowledge from the available datasets and typically operate under closed-set assumptions, where the label spaces of the two domains are identical. This assumption limits their applicability in real-world scenarios, where source and target datasets often exhibit the label-level taxonomy gap - including variations in class categories, semantic contexts, and category granularity. Figure 1: DynAlign and taxonomy adaptation. Current UDA methods focus solely on domain-specific knowledge transfer and assume consistent class labels across domains, limiting their flexibility in adapting to different taxonomies. Open-vocabulary segmentation models excel with broader taxonomies through large-scale pretraining but lack the precision of domain-specific models for specialized tasks. In contrast, DynAlign integrates with any UDA model and flexibly adapts to diverse taxonomies and scene contexts, leveraging the prior knowledge of foundational models. To overcome this limitation, open-set Domain Adaptation (DA) methods have been developed to recognize novel classes in the target domain ( Saito & Saenko, 2021 ; Li et al., 2023 ) . However, these methods are only capable of distinguishing unknown classes from known ones and do not provide detailed classifications for new target classes. To solve this, taxonomy-adaptive domain adaptation has been proposed, enabling UDA in settings where the target domain adopts a label space different from that of the source domain ( Gong et al., 2022 ; Fan et al., 2023a ) . These methods aim to align differing taxonomies by leveraging relationships between source and target classes but still require prior knowledge about the target labels through annotated samples from the target domain. These challenges highlight the need for a more flexible approach that can simultaneously manage domain shifts while accommodating new taxonomies without requiring additional annotations from the target domain . Despite the significance of this problem, fully unsupervised methods that can address both domain shifts and taxonomy discrepancies remain largely underexplored. Recent advancements in foundation models offer a promising direction to overcome these limitations. By leveraging large-scale pretraining on diverse datasets, these models can generalize across varied domains, tackling the challenges posed by limited domain knowledge and unseen categories. For instance, recent open-vocabulary semantic segmentation works ( Ghiasi et al., 2022 ; opensam ) leverage the knowledge in foundation models like CLIP ( Radford et al., 2021 ) and Segment Anything (SAM) ( Kirillov et al., 2023 ) , enabling segmentation across a broad range of unbounded class labels. However, despite their strong generalization ability, foundation models often struggle with inferior performance compared to domain-specific models trained on in-domain datasets. Their broad focus can limit their effectiveness in specialized tasks, such as street scene understanding, where fine-grained segmentation requires detailed, domain-specific knowledge. To conclude, existing works have been tackling image-level and label-level domain gaps separately, while the intersection of these two problems remains underexplored. In this work, we propose a new benchmark of unsupervised taxonomy adaptation, addressing both image-level and label-level domain gaps without supervision, as shown in Figure 1 (c). To achieve this, we propose DynAlign , a novel approach that integrates both domain-specific knowledge and rich open-world prior knowledge from foundation models. The framework first leverages domain-specific knowledge by aligning the data distributions of the source and target domains within the UDA paradigm. Subsequently, to incorporate prior knowledge from foundation models, DynAlign dynamically adapts to new scene contexts by retrieving in-domain predictions and extending the knowledge with foundation models to generate predictions in the new target label space, accommodating the different taxonomy in the target domain. In DynAlign , foundational model knowledge is fused in three modules. First, Large Language Model (LLM) is used for semantic taxonomy mapping and context-aware descriptions. For instance, the source domain label ‘road’ is mapped to { ‘road’ , ‘sidewalk’ , ‘lane marking’ , etc. } in the target domain, and then each label such as ‘lane marking’ can be further enriched with more precise descriptions like { ‘traffic lane marking’ , ‘double lines’ } to capture the current context and improve semantic granularity. Then, SAM is employed to refine the coarse semantic masks generated by the UDA model, providing more fine-grained masks within precise boundaries (e.g., segmenting ‘lane marking’ , ‘catch basin’ within the ‘road’ region). Lastly, a knowledge fusion mechanism is introduced, where CLIP is leveraged to extract textual features based on the context-aware taxonomy provided by LLM and reassign semantic labels to the proposed mask regions, effectively fusing UDA knowledge with prior knowledge. Predictions from DynAlign can be directly used or leveraged as pseudo-labels to train an offline segmentation model on the target domain, enabling instant inference without the need for target domain annotations. Our approach operates in a fully unsupervised manner, integrating seamlessly with any UDA-based semantic segmentation model. By leveraging the prior knowledge of foundation models, it strengthens in-domain predictions and flexibly adapts to new classes

Read the complete text