Unified Modality Separation: A Vision-Language Framework for Unsupervised Domain Adaptation
Xinyao Li, Jingjing Li*, Zhekai Du, Lei Zhu, Heng Tao Shen · 2025
Read Unified Modality Separation: A Vision-Language Framework for Unsupervised Domain Adaptation by Xinyao Li, Jingjing Li*, Zhekai Du, Lei Zhu, Heng Tao Shen (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-04987.txt.
Introduction
Xinyao Li and co-authors split vision and language features into shared and private parts so image models can adapt to new domains unlabeled, in work from 2025. Vision-language models carry a built-in gap between how images and text sit in their space. Adapting with the gap intact passes only the shared knowledge across domains.
Their frame disentangles the parts during training and treats each in one unified scheme. At test time, blend weights adjust on their own to get the best mix per sample. A new gap metric sorts samples into shared, private, and unsure camps. The camps then get handled by fit. The goal is target skill that uses all the model knows.
Catalog record
| CREATOR | Xinyao Li, Jingjing Li*, Zhekai Du, Lei Zhu, Heng Tao Shen |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 73 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2508-04987
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Unified Modality Separation: A Vision-Language Framework for Unsupervised Domain Adaptation Xinyao Li, Jingjing Li*, Zhekai Du, Lei Zhu, Heng Tao Shen Abstract Unsupervised domain adaptation (UDA) enables models trained on a labeled source domain to handle new unlabeled domains. Recently, pre-trained vision-language models (VLMs) have demonstrated promising zero-shot performance by leveraging semantic information to facilitate target tasks. By aligning vision and text embeddings, VLMs have shown notable success in bridging domain gaps. However, inherent differences naturally exist between modalities, which is known as modality gap . Our findings reveal that direct UDA with the presence of modality gap only transfers modality-invariant knowledge, leading to suboptimal target performance. To address this limitation, we propose a unified modality separation framework that accommodates both modality-specific and modality-invariant components. During training, different modality components are disentangled from VLM features then handled separately in a unified manner. At test time, modality-adaptive ensemble weights are automatically determined to maximize the synergy of different components. To evaluate instance-level modality characteristics, we design a modality discrepancy metric to categorize samples into modality-invariant, modality-specific, and uncertain ones. The modality-invariant samples are exploited to facilitate cross-modal alignment, while uncertain ones are annotated to enhance model capabilities. Building upon prompt tuning techniques, our methods achieve up to 9% performance gain with 9 times of computational efficiencies. Extensive experiments and analysis across various backbones, baselines, datasets and adaptation settings demonstrate the efficacy of our design. I Introduction Unsupervised domain adaptation (UDA) [ 1 , 2 ] transfers knowledge from a labeled source domain to a related but unlabeled target domain. The primary challenge in UDA is to mitigate the distribution shift between the source and target domains [ 3 ] . Prior efforts either optimize a criterion to minimize the distribution discrepancy [ 4 , 5 ] , or employ adversarial learning [ 6 ] to learn domain-invariant features [ 1 , 7 ] . Another line of study resort to more capable networks for better transferability [ 8 , 9 ] . While these methods have shown promising success, they have concentrated on single vision modality, while neglecting the rich semantic structures in data. Recent advances in vision-language models (VLMs) [ 10 , 11 ] offer a promising solution. The contrastive learning in VLMs pulls closer paired image and text features, endowing them with rich knowledge on general semantic concepts. Such large-scale pretraining (e.g., CLIP [ 10 ] is trained on 400 million image-text pairs) also contributes to the capability in closing domain gap, making them suitable for domain adaptation tasks. (a) Illustration of (i) conventional vision UDA, (ii) vision-language UDA, (iii) modality gap hindering adaptation on modality-specific data, and (iv) the proposed modality-aware adaptation concept. (b) Right: Examples on modality-specific information from OfficeHome-RealWorld. Left: Visualizations of the disentangled VAC and LAC. Fig. 1: Motivations and effects of our modality separation design. Vision-associated components (VAC), language-associated components (LAC) are inherent visual details and semantic cues in the image, each focused by VLM’s modality-specific branches, respectively. Pioneering works on VLM-based UDA either adopt prompt tuning [ 12 ] to adapt text embeddings [ 13 , 14 ] , or tune the pretrained parameters to match target distribution [ 15 ] . These methods all rely on similarities between vision and text embeddings to perform classification, as shown in Fig. 1a (ii). However, recent studies on the modality gap [ 16 ] phenomenon reveal that, despite pretraining efforts, the vision and text features are distinctly distributed, each containing private modality-specific information. Fig. 1b (right) offers clear examples on modality-specific knowledge. The vision and text classifiers are independently adapted to the target data 1 1 1 Vision classifier is adapted by linear-probing on pseudo labels from vision KMeans [ 17 ] . Text classifier includes prompts tuned on text pseudo labels following method in [ 14 ] . , whose classification results vary across different modalities. The vision classifier excels at identifying visually straightforward items like a chair, while the text classifier better interprets semantically rich pictures like a file cabinet. Such modality-specific information cannot be well aligned by current methods with the existence of modality gap, as illustrated in Fig. 1a (iii). Rather than attempting to reduce [ 18 ] or bridge [ 19 ] the modality gap, we propose to bypass it with modality separation on CLIP features, so that modality information vital for classification can be preserved. Specifically, a multimodal framework is designed to transfer and exploit modality-specific information for finer adaptation, as depicted in Fig. 1a (iv). As fine-tuning CLIP parameters is expensive and may disturb the pre-aligned multi-modal space [ 20 ] , we opt to the feature disentanglement and adaptation with a Unified Modality Separation (UniMoS) framework. During training, the modality separation network disentangles the pre-extracted vision features into vision- and language-associated components (VAC, LAC) to capture respective modality-specific characteristics. Fig. 1b (left) visualizes disentangled VAC and LAC using GradCAM [ 21 ] . VAC and LAC focus on distinct image areas that represent modality-specific information. Such distinction is ensured by an orthogonal regularization loss. LAC and VAC are then handled separately in a unified manner, with a learnable weight to balance their contributions. To transfer source modality-specific knowledge to target domain, VAC and LAC are aligned separately with a modality discriminator. During inference, we combine the strengths of each modality with a Modality-adaptive Ensemble (MaE) mechanism. MaE first evaluates ensemble thresholds for target samples, then determines optimal weights by approximating the vision and text distributions. These weights can adapt dynamically to various training phases and tasks automatically. Fig. 2: Illustration of the MDI metric and model optimization goal. Both modality classifiers are optimized towards the active samples and each other, achieving maximum harmony on target samples. To handle samples with various modality gap, we further design a Modality DIscrepancy (MDI) metric to categorize target data according to their cross-modal prediction confidences and consistencies. The resultant MDI categories are conceptualized in Fig. 2 . The modality-invariant samples help learn more robust decision boundaries. On the other hand, modality-specific information is extracted and utilized by the modality ensemble processes. Uncertain samples might interfere the modality separation and adaptation processes. The absence of target labels makes them hard to learn. Therefore, we draw inspirations from active domain adaptation (ADA) [ 22 , 23 ] to annotate these hard samples, which enhances the adaptation process with limited annotation costs. Fig. 2 indicates that exploitation of annotated uncertain samples extend VLM’s representation boarders. The main contributions of this work can be summarized as: 1. We identify the negative effects of the modality gap on domain adaptation tasks, emphasizing the modality-specific information that is often overlooked by existing VLM-based UDA methods. 2. We propose a unified modality separation framework to handle both modality-specific and modality-invariant components for comprehensive UDA. 3. We present a novel MDI score that quantifies instance-level modality characteristics, which not only enhances training but also provides insights and solutions for multimodal active learning. 4. Our method is compatible with various prompt tuning techniques and adaptation settings including UDA, ADA, source-free ADA and multi-source DA. Extensive evaluations on various benchmarks, backbones and prompt tuning methods demonstrate the efficacy of our approach. Comprehensive analysis further demonstrate the validness and efficacy of our