Preserving Domain Generalization in Fine-Tuning via Joint Parameter Selection
Bin Pan, Shiyu Shen, Zongbin Wang, Zhenwei Shi, Xia Xu · 2025
Read Preserving Domain Generalization in Fine-Tuning via Joint Parameter Selection by Bin Pan, Shiyu Shen, Zongbin Wang, Zhenwei Shi, Xia Xu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-16976.txt.
Introduction
Bin Pan and four colleagues present Joint Parameter Selection, a fine-tuning method that protects domain generalization, in a paper from twenty twenty-five. Domain generalization means doing well on new target domains after training on few source domains. Large pretrained vision models are the usual starting point. Full fine-tuning, though, can damage the generalization built into them.
Some methods tune only a subset of parameters. That balances fitting the task against keeping generalization. Joint Parameter Selection limits updates to a small, sparse subset. The authors prove a generalization error bound that reflects update sparsity in a direct way. A selection mechanism with dual operators picks parameters. It looks for gradients that are steady and strong across source domains. Tests on benchmarks showed scores above leading domain generalization methods. The results back selective fine-tuning as both efficient and effective.
Catalog record
| CREATOR | Bin Pan, Shiyu Shen, Zongbin Wang, Zhenwei Shi, Xia Xu |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 63 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2508-16976
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Preserving Domain Generalization in Fine-Tuning via Joint Parameter Selection Bin Pan, Shiyu Shen, Zongbin Wang, Zhenwei Shi, Xia Xu Abstract Domain generalization seeks to develop models trained on a limited set of source domains that are capable of generalizing effectively to unseen target domains. While the predominant approach leverages large-scale pre-trained vision models as initialization, recent studies have highlighted that full fine-tuning can compromise the intrinsic generalization capabilities of these models. To address this limitation, parameter-efficient adaptation strategies have emerged, wherein only a subset of model parameters is selectively fine-tuned, thereby balancing task adaptation with the preservation of generalization. Motivated by this paradigm, we introduce Joint Parameter Selection (JPS), a novel method that restricts updates to a small, sparse subset of parameters, thereby retaining and harnessing the generalization strength of pre-trained models. Theoretically, we establish a generalization error bound that explicitly accounts for the sparsity of parameter updates, thereby providing a principled justification for selective fine-tuning. Practically, we design a selection mechanism employing dual operators to identify and update parameters exhibiting consistent and significant gradients across all source domains. Extensive benchmark experiments demonstrate that JPS achieves superior performance compared to state-of-the-art domain generalization methods, substantiating both the efficiency and efficacy of the proposed approach. I Introduction Fig. 1 : Illustration of risks in full fine-tuning. When using a pre-trained model, some parameters might already be close to the unseen target domain. Optimizing all parameters might improve performance on source domains but harm generalization. Domain generalization addresses the fundamental challenge posed by distributional discrepancies between training and testing data [ 1 ] . While the assumption of independent and identically distributed (i.i.d.) data underpins much of machine learning, it is frequently violated in practical applications, resulting in notable performance degradation when models encounter previously unseen domain shifts [ 2 ] . In computer vision, such domain shifts may arise from variations in style, texture, background, or camera viewpoint [ 3 , 4 , 5 ] . Domain generalization approaches commonly employ fine-tuning strategies on pre-trained backbone models such as CLIP pre-trained ViT [ 6 ] , leveraging the robust initial representations provided by large-scale pre-training [ 7 , 8 ] . These methods typically adapt pre-trained vision models by optimizing them with respect to task-specific feature constraints or specialized losses [ 9 , 10 ] , with the objective of preserving the generalization capability of the pre-trained models to the downstream task. Nevertheless, full fine-tuning can undermine the general-purpose knowledge embedded in pre-trained vision models [ 11 , 12 , 13 ] , potentially altering well-optimized parameters to enhance source domain performance at the expense of generalization. As illustrated in Figure 1 , certain pre-trained parameters may already be optimally aligned with unseen domains, and indiscriminate updates based on source domain data can inadvertently degrade their utility. Furthermore, the need to retrain fully fine-tuned models for related, yet distinct, tasks imposes considerable computational and communication overhead, which is particularly prohibitive in resource-constrained environments such as satellite platforms or autonomous vehicles [ 14 , 15 ] . Consequently, selective fine-tuning —wherein only a sparse subset of parameters is updated— has emerged as a promising alternative, balancing adaptation and generalization while minimizing parameter modifications [ 16 ] . Recent advances indicate that sparse parameter optimization not only enhances stability but also yields superior generalization compared to full model updates [ 11 , 17 , 18 ] . However, the core challenge lies in carefully selecting which parameters to update. Existing methods often rely on gradient magnitude as a criterion, presuming that parameters with the largest gradients are most relevant to the current task. This assumption is questionable in the context of domain generalization, where parameters significant in one domain may be uninformative or even detrimental in others, thereby introducing the risk of overfitting [ 19 , 20 ] . This underscores the necessity for parameter selection strategies that explicitly account for domain-specific gradient behavior. To address these limitations, we propose Joint Parameter Selection (JPS), a novel framework for domain generalization that leverages gradient information across multiple domains to identify parameters with robust generalization potential. Our approach is grounded in rigorous theory, introducing a generalization error upper bound that incorporates update sparsity, thereby elucidating the interaction between sparsity and model generalization. Motivated by this analysis, we propose a two-stage selection mechanism: we choose sparse and small sets of ViT parameters with consistent, significant gradients across source domains in specific layers for updating. This adaptive framework affords flexible control over the computational budget by modulating both the degree of sparsity and the choice of layers for fine-tuning, yielding two principal operational regimes: 1. Enhanced Generalization with Comparable Cost. By selecting enough layers, we improve generalization while keeping the computational cost similar to full fine-tuning. 2. Reduced Resource Consumption with Competitive Generalization. By selecting limited layers, we reduce training resources while achieving competitive performance. We empirically validate our method on the DomainBed benchmark [ 21 ] , demonstrating that JPS consistently outperforms state-of-the-art domain generalization and parameter-efficient adaptation methods. In summary, the key contributions of this work are as follows: • We introduce a selective fine-tuning method that updates only a small set of parameters to preserve and leverage the generalization ability of the pre-trained model for domain generalization. • We propose a new generalization upper bound and show that a suitable parameter selection strategy reduces prediction error on unseen domains. • We present a generic implementation with two operators to select and update the parameters with strong generalization potential. II Related Work and Background II-A Large-Scale Pre-trained Vision Models Large-scale pre-trained vision models have become foundational in computer vision, offering powerful and transferable feature representations. Early approaches relied on supervised pre-training using large-scale datasets such as ImageNet, with models like ResNet [ 22 ] and Vision Transformer (ViT) [ 23 ] achieving impressive performance across various tasks. In recent years, self-supervised pre-training methods [ 24 , 25 ] have emerged, enabling models to learn rich and generalizable features without requiring labeled data. Among these, CLIP [ 6 ] introduced a paradigm shift by jointly training on image-text pairs, resulting in models that understand both visual and semantic concepts. Notably, CLIP-pretrained ViT models have become some of the most widely used and effective backbones for downstream vision tasks due to their strong generalization capabilities and versatility. II-B Domain Generalization Domain generalization aims to use multiple source domains S=\{S_{1},S_{2},\dots,S_{N}\} to train a model that performs robustly on unseen target domains T,T\cap S=\emptyset . Each S_{i} is a distribution from which data (x,y) are drawn: (x,y)\sim S_{i} , where x and y are the samples and labels. The objective is to learn a model within the hypothesis space h\in\mathcal{H} and parameter \theta\in\Theta , guided by a loss function l(h_{\theta}(x),y) . The ideal loss function in domain generalization is [ 26 ] : R(h_{\theta},D)=E_{(x,y)\sim D}(l(h_{\theta}(x),y))\ ,D=S\cup T. (1) In practice, T and the distribution of S are inaccessible during training [ 2 ] . Only the training data from source domains \hat{S}=\{x_{i},y_{i}\}_{i=1}^{n},\hat{S}\subset S is available. The empirical loss function in domain generalization is: R(h_{\theta},\hat{S})=\frac{1}{n}\sum_{i=1}^{n}l(h_{\theta}(x_{i}),y_{i})\ ,(x_{i},y_{i})\in\hat{S}. (2) This loss in domain generalization, known as Empirical Risk Minimization (ERM) [ 27 ] , minimizes task loss on source domains. While ERM