Rethinking the Need for Source Models: Source-Free Domain Adaptation from Scratch Guided by a Vision-Language Model Journal: Knowledge-Based Systems

Zhou Bingtao, Xiang Mian, Ning Qian* · 2026

Read Rethinking the Need for Source Models: Source-Free Domain Adaptation from Scratch Guided by a Vision-Language Model Journal: Knowledge-Based Systems by Zhou Bingtao, Xiang Mian, Ning Qian* (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-02604.txt.

Introduction

Zhou Bingtao, Xiang Mian, and Ning Qian question a quiet assumption in domain adaptation, the task of moving a vision model to a new setting without access to its original training data. Standard source-free methods still start from a model pretrained on that source data, so they are not truly source-free. The authors also note that swapping the source model changes final results very little.

They propose a stricter setting that starts from a randomly initialized model, guided only by a vision-language model and unlabeled target images. Their two-stage framework first warms the model up under that guidance, then finds a cleaner shared region between the guide and the learner for distillation. On three standard benchmarks, the method matched or beat approaches that still rely on source models.

Catalog record

CREATORZhou Bingtao, Xiang Mian, Ning Qian*
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE87 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.02604

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Rethinking the Need for Source Models: Source-Free Domain Adaptation from Scratch Guided by a Vision-Language Model Journal: Knowledge-Based Systems Zhou Bingtao, Xiang Mian, Ning Qian* Abstract Source-Free Domain Adaptation (SFDA) adapts source models to target domains without accessing source data, addressing privacy and transmission issues. However, existing methods still initialize from a source pre-trained model and thus are not truly source-free. Recent works have introduced Vision-Language (ViL) models to guide the adaptation process, in these methods, we observe that for the same target domain, different source models yield minimal variation in final results, indicating the source model itself has limited impact. Motivated by this, we propose ViL-Only Domain Adaptation (VODA) , a stricter setting that eliminates all dependencies on source domain , relying solely on a randomly initialized model, a ViL model, and unlabeled target data. We analyze the adaptation dynamics of VODA and introduce Two-Stage Denoised-Region Distillation (TS-DRD) , a two-stage framework that first warms up the model with ViL guidance, then seek a Denoised-Region inherent in both the ViL and adapting model, yielding cleaner supervision for distillation. Experiments on Office-Home, VisDA, and DomainNet-126 show that under VODA, TS-DRD achieves competitive or superior performance to existing SFDA methods that still use source models, demonstrating its effectiveness and the potential of the VODA setting. † † highlights: Research highlight 1 We introduce ViL-Only Domain Adaptation (VODA), a strictly source-information-free paradigm that eliminates reliance on both source data and source models. VODA operates solely with a randomly initialized model, a vision-language model, and unlabeled target data. Through geometric analysis of the adaptation dynamics, we uncover a convergent behavior: under strong vision-language guidance, models initialized from vastly different states converge to nearly identical final representations. This finding questions the foundational necessity of source model initialization in current SFDA and opens a new, more accessible direction for domain adaptation. Research highlight 2 We propose Two-Stage Denoised-Region Distillation (TS-DRD), a novel framework that first warms up the model using pure vision-language guidance, then constructs a theoretically-grounded Denoised-Region to suppress noise and produce cleaner supervision for distillation. TS-DRD achieves state-of-the-art or highly competitive performance across Office-Home, VisDA, and DomainNet-126, outperforming even source-dependent SFDA methods, while introducing negligible computational overhead. 1 Introduction Source-Free Domain Adaptation (SFDA) tackles the challenge of adapting a model to an unlabeled target domain without access to source data, which is often required in privacy-sensitive scenarios [ 22 , 21 , 20 ] . Despite being termed "source-free", this paradigm is not entirely free of source information, as the source data and labels are required to train the source model in the first place, and the source domain’s characteristics are implicitly embedded within the model’s parameters. Recent works leverage Vision-Language (ViL) models as an external guide for the adaptation process [ 35 , 34 , 43 ] . By providing high-quality supervision signals from ViL models, these methods significantly improve the adaptation performance. However, we observe a thought-provoking phenomenon in this line of work: when a powerful ViL model is employed for guidance, the final adaptation performance on the same target domain shows a surprisingly weak dependence on different source models. For instance, as shown in 1(a) of the current state-of-the-art approach method ProDe [ 34 ] , the results of adapting from different source domains to the same target domain exhibit minimal variance, with statistically insubstantial differences. Other work like DIFO [ 35 ] and DTKI [ 43 ] also shows very small differences across different source domains on the same target domain adaptation task, and across multiple datasets, differences exceeding 2% are rare. This indicates that in ViL model’ guided SFDA methods, the domain-specific information carried by the source model plays a negligible role in determining the final outcome. (a) (b) Figure 1 : (a) Performance of ViL-guided SFDA methods (e.g., ProDe [ 34 ] in Office-Home) exhibits minimal variance across different source domains for the same target domain, indicating weak dependence on the source model. (b) Our proposed VODA setting: a truly source-free adaptation paradigm that eliminates all source dependencies. This observation encourages us to rethink the necessity of this prevailing SFDA paradigm: If the impact of the source models is so marginal, is it possible to discard them altogether to achieve a truly source-free adaptation? To this end, we propose a new learning setting: training the target model from scratch using only an untrained initial model, a ViL model and unlabeled target data, termed ViL-Only Domain Adaptation (VODA), as illustrated in 1(b) . In this setting, dependencies of information and resources on source domain is thoroughly eliminated, thus achieving genuine source-information-free adaptation. In this paper, we first propose a dynamic process for VODA, which captures a key insight: under proper ViL model’s guidance, the initial domain’s position becomes less critical to the final adaptation objective. This claim is empirically validated in our experiments. Guided by this dynamic perspective, the central challenge of VODA becomes how to leverage ViL model’s guidance at different stages of the adaptation trajectory. So we propose TS-DRD, a two-stage framework. It first uses pure ViL guidance to warm up the adapting model, then in the second stage, we construct a Denoised-Region by exploiting the complementary of both models, providing cleaner supervision for adaptation. Our contributions are summarized as follows: 1. We propose ViL-Only Domain Adaptation (VODA), a novel and stricter adaptation setting that eliminates any dependency on source domains, and characterize its adaptation dynamics. 2. Inspired by the dynamic process analysis of VODA, we propose TS-DRD, a two-stage framework that progressively transfers knowledge from the ViL model to the target model for stable and efficient adaptation. 3. Experiments on Office-Home, VisDA, and DomainNet-126 show that TS-DRD performs on par with or outperforms state-of-the-art SFDA approaches, all without leveraging any source information. This demonstrates the feasibility of VODA, establishing it as a viable paradigm for truly source-free adaptation. The code can be found in https://github.com/Zhoubingtao/VODA-TS-DRD . 2 Related Work 2.1 Traditional Source-Free Domain Adaptation A dominant strategy of SFDA is self-training, which generates pseudo-labels on target data for supervision. Pioneering works like SHOT [ 22 ] established a baseline by aligning target features with the source hypothesis via information maximization. Following its idea, a major research thrust has focused on denoising these pseudo-labels using techniques such as neighborhood consistency [ 40 ] , adaptive thresholding [ 42 ] , or noise transition modeling [ 4 ] . Other approaches generate surrogate supervision through data synthesis [ 17 , 14 ] or selective sampling [ 5 , 7 ] . Crucially, the pseudo-labels generated in all these methods are fundamentally grounded in the knowledge encoded within the source-pretrained model. Consequently, they are "source-free" only in terms of data access but still rely on source models, making it inherently difficult to achieve genuine "source-information-free" adaptation. 2.2 ViL-Guided Source-Free Domain Adaptation Large-scale ViL models like CLIP [ 28 ] , with their strong zero-shot generalization, offer a powerful external prior for SFDA. A common strategy is to adapt the ViL model to the target domain via prompt tuning [ 9 , 44 , 30 ] and then leverage it to guide the adaptation of the source model. Recently, several methods have integrated ViL models into the SFDA pipeline.

Read the complete text