CoDoL: Conditional Domain Prompt Learning for Out-of-Distribution Generalization DOI: XXXXXXX.XXXXXXX Conference: Make sure to enter the correct conference title from your rights confirmation email; June 03–05, 2025; Woodstock, NY ISBN: 978-1-4503-XXXX-X/2025/06 CCS: Do Not Use This Code Generate th
Min Zhang, Bo Jiang, Jie Zhou, Yimeng Liu, Xin Lin · 2025
Read CoDoL: Conditional Domain Prompt Learning for Out-of-Distribution Generalization DOI: XXXXXXX.XXXXXXX Conference: Make sure to enter the correct conference title from your rights confirmation email; June 03–05, 2025; Woodstock, NY ISBN: 978-1-4503-XXXX-X/2025/06 CCS: Do Not Use This Code Generate th by Min Zhang, Bo Jiang, Jie Zhou, Yimeng Liu, Xin Lin (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-15330.txt.
Introduction
Min Zhang and six co-authors condition prompt learning on domain information so vision-language models generalize to unseen distributions in a 2025 machine learning paper. Contrastive models like CLIP promise representations that survive distribution shift, yet prompt-based tuning still suffers from sloppy text descriptions and weak alignment between image and text embeddings.
Their method, CoDoL, builds prompts from readily available domain information rather than generic templates. Better prompts pull the vision and language embeddings into closer alignment, which the authors identify as a driver of out-of-distribution gains. The design captures both instance-specific detail and domain-level context in the learned prompts. Stronger alignment at training time is offered as the mechanism behind the improved performance on distributions the model never saw.
Catalog record
| CREATOR | Min Zhang, Bo Jiang, Jie Zhou, Yimeng Liu, Xin Lin |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 45 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2509-15330
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
CoDoL: Conditional Domain Prompt Learning for Out-of-Distribution Generalization DOI: XXXXXXX.XXXXXXX Conference: Make sure to enter the correct conference title from your rights confirmation email; June 03–05, 2025; Woodstock, NY ISBN: 978-1-4503-XXXX-X/2025/06 CCS: Do Not Use This Code Generate the Correct Terms for Your Paper CCS: Do Not Use This Code Generate the Correct Terms for Your Paper CCS: Do Not Use This Code Generate the Correct Terms for Your Paper CCS: Do Not Use This Code Generate the Correct Terms for Your Paper Min Zhang, Bo Jiang, Jie Zhou, Yimeng Liu, Xin Lin Abstract . Recent advances in pre-training vision-language models (VLMs), e.g. , contrastive language-image pre-training (CLIP) methods, have shown great potential in learning out-of-distribution (OOD) representations. Despite showing competitive performance, the prompt-based CLIP methods still suffer from: i) inaccurate text descriptions , which leads to degraded accuracy and robustness, and poses a challenge for zero-shot CLIP methods. ii) limited vision-language embedding alignment , which significantly affects the generalization performance. To tackle the above issues, this paper proposes a novel Co nditional D omain pr o mpt L earning (CoDoL) method, which utilizes readily-available domain information to form prompts and improves the vision-language embedding alignment for improving OOD generalization. To capture both instance-specific and domain-specific information, we further propose a lightweight D omain M eta N etwork (DMN) to generate input-conditional tokens for images in each domain. Extensive experiments on four OOD benchmarks (PACS, VLCS, OfficeHome and DigitDG) validate the effectiveness of our proposed CoDoL in terms of improving the vision-language embedding alignment as well as the out-of-distribution generalization performance. 1. Introduction Figure 1. Comparison of the framework based on CLIP models. (a) Zero-shot CLIP uses the fixed context. (b) Prompt-tuning CLIP designs the learnable context. (c) CoDoL not only models the learnable context for the class prompt but also drafts the domain prompt to align the vision and language representations. Moreover, a lightweight domain meta-net is also proposed to generate the input-conditional domain-specific information. The snowflake and fire mean frozen and learnable, respectively. Deep learning methods generally rely on the independent and identically distributed (IID) assumption that the distributions of training data and testing data are the same ( Gänssler and Stute 1979 ; He et al. 2015 ; He et al. 2016a ; He et al. 2016b ) . However, in real-world applications, out-of-distribution (OOD) generalization is a ubiquitous problem, where the distribution of testing data differs from the training data, leading to the significant performance degradation ( Zhang et al. 2022 ; Yao et al. 2023 ) . Many methods have been proposed to solve the out-of-distribution generalization problem, including: (1) causal learning ( Cha et al. 2022 ; Lin et al. 2022a ; Arjovsky et al. 2019 ) , adversarial learning ( Ajakan et al. 2014 ; Li et al. 2018b ) and meta learning (or named learning to learn) ( Balaji et al. 2018 ; Khattak et al. 2022 ) , and so on. Previous studies used supervised pre-trained models and carefully-designed transfer learning algorithms for achieving OOD generalization ( Gulrajani and Lopez-Paz 2020 ; Kim et al. 2022 ) . Recently, vision-language pre-training models (VLMs) seek to learn from naturally formed supervision of web-scale image-language pairs ( Radford et al. 2021a ; Jia et al. 2021 ) , which exhibits impressive zero-shot learning performance and outperforms models trained from only labeled images. The VLMs such as the contrastive language-image pre-training model (CLIP) ( Radford et al. 2021b ) also empirically outperform traditional supervised learning methods on OOD downstream tasks in terms of zero-shot performance (see Tables 1 and 2 for more details), which reveals a promising research direction toward OOD generalization in real-world downstream tasks. Specifically, as shown in Figure 1 , CLIP methods first compute the similarity between images and embedded words for each category, then train the text and image encoders by maximizing the similarity. Therefore, the vision-language embedding alignment heavily affects the generalization performance of the standard CLIP methods. Similar conclusions also could be found in previous works ( Shu et al. 2023 ; Menon and Vondrick 2022 ; Goyal et al. 2022 ) . Figure 1 (a) illustrates the zero-shot CLIP method, which use a fixed text to test the performance of the pre-training encoder on downstream tasks. Figure 1 (b) illustrates the prompt-tuning CLIP methods, which use the learnable context and update only a small number of additional parameters during the fine-tuning phase, thereby reducing the training and storage burdens. Despite the competitive performance in OOD downstream tasks, these methods still suffer from the following issues: i) inaccurate text descriptions , which leads to degraded accuracy and robustness, and poses a challenge for zero-shot CLIP methods ( Zhou et al. 2022a ; Zhou et al. 2022b ; Jia et al. 2022 ; Khattak et al. 2022 ) . ii) limited vision-language embedding alignment , which serves as a key factor affecting the generalization performance of the prompt-tuning CLIP methods ( Zhang et al. 2021 ; Li et al. 2022 ; Bose et al. 2023 ) . The two core challenge significantly affect the generalization performance for pre-training vision-language models under the out-of-distribution setting. To tackle the above issues, in this paper, we propose to utilize domain information to improve the alignment of the embedding spaces of CLIP image and text encoders, thus improving the OOD generalization performance. This domain information could be different measuring circumstances, locations, times, experimental conditions, external interventions, contexts, and so forth, which are readily available in practice ( Arjovsky et al. 2019 ; Krueger et al. 2021 ; Lin et al. 2022b ) . Figure 2 illustrates a toy example of the benefits of utilizing domain information for improving OOD generalization using the PACS benchmark dataset ( Li et al. 2017 ) , which contains four domains, i.e. , Photo, Art Painting, Cartoon and Sketch. Compared with the zero-shot CLIP method, we additionally introduce the domain token and obtain a novel prompt form, i.e. , “a photo of a [CLASS] [DOMAIN] ”, with the added domain vectors. The proposed zero-shot CLIP with domain information achieves 93.17\% accuracy, whereas the previous zero-shot CLIP method only achieves 90.32\% accuracy, validating the effectiveness of introducing the domain information for OOD generalization. Given that the domain information can improve out-of-distribution (OOD) generalization, we propose Co nditional D omain pr o mpt L earning (CoDoL), a simple and effective parameter-efficient method to improve the OOD generalization of CLIP on downstream tasks. Specifically, CoDoL first models a class prompt’s context words with learnable vectors following the previous works ( Zhou et al. 2022a ; Zhou et al. 2022b ) . Then, we design the learnable domain prompt’s context vectors to capture the domain information in the prompt-tuning phase. Because OOD generalization aims to learn knowledge from multiple training domains and transfer it to unseen testing domains, in this paper, the proposed domain prompt is domain-agnostic parameters that are used to learn domain-invariant representation and generalize to unseen testing domains. Furthermore, to capture instance-specific information, we design a lightweight neural network named D omain M eta N etwork (DMN) to generate input-conditional tokens for images in each domain, which is then concatenated with the learnable domain-specific context vectors. Extensive experiments are