Meta-Contrastive Learning for Vision-Language Models via Task-Adaptive CLIP Training
Merham Fouladvand, Peuroly Batra · 2026
Read Meta-Contrastive Learning for Vision-Language Models via Task-Adaptive CLIP Training by Merham Fouladvand, Peuroly Batra (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-27091.txt.
Introduction
Fouladvand and Batra propose domain-conditioned meta-contrastive learning for vision-language models like CLIP. CLIP trains with one global objective that ignores domain shift between datasets. Their framework turns training into a bilevel meta-learning problem over domain tasks. Learned domain embeddings modulate both image and text representations. Inner-loop gradient steps adapt the model quickly to each domain's distribution. A cross-domain alignment term pushes representations toward domain invariance. The method fits standard contrastive pipelines and spans natural and medical data.
Catalog record
| CREATOR | Merham Fouladvand, Peuroly Batra |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 19775 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.27091
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Meta-Contrastive Learning for Vision-Language Models via Task-Adaptive CLIP Training Merham Fouladvand, Peuroly Batra Abstract We propose Domain-Conditioned Meta-Contrastive Learning , a framework for improving the cross-domain generalization of vision-language models. While contrastive models such as CLIP achieve strong performance through large-scale training, they rely on a global objective that does not explicitly account for domain shift. To address this limitation, we formulate multimodal learning as a bilevel meta-learning problem over domain-conditioned tasks. Specifically, we introduce domain embeddings that modulate image and text representations, and optimize the model for rapid adaptation to domain-specific distributions via gradient-based inner-loop updates. In addition, we incorporate a cross-domain alignment regularization to encourage domain-invariant representations. Our approach is compatible with standard contrastive training pipelines and can be applied to heterogeneous datasets spanning natural and medical domains. We expect improved robustness under domain shift and enhanced few-shot adaptation performance, highlighting a promising direction for scalable multimodal learning. 1 Introduction Vision-language models (VLMs) trained with contrastive objectives, such as CLIP [ 12 ] , have emerged as a powerful paradigm for learning aligned multimodal representations at scale. By leveraging large collections of image-text pairs, these models achieve strong performance across a wide range of downstream tasks, including cross-modal retrieval, zero-shot classification, and multimodal reasoning. Despite these successes, a fundamental limitation remains: standard contrastive training optimizes a single global objective , implicitly assuming that all data are drawn from a unified distribution. In practice, multimodal data are highly heterogeneous and exhibit significant domain shift . For example, natural images, medical imaging data, and satellite imagery differ substantially in visual statistics, semantic structure, and annotation styles. As a result, models trained with global contrastive objectives often fail to generalize effectively across domains, leading to degraded performance under distribution shift and limited adaptability in low-resource settings. This limitation is closely related to the broader challenge of domain generalization [ 7 ] . To address this limitation, we propose Domain-Conditioned Meta-Contrastive Learning , a framework that explicitly models domain heterogeneity within a bilevel meta-learning formulation [ 5 ] . Rather than optimizing a single set of parameters for all data, we treat each domain as a task distribution and train the model to rapidly adapt to domain-specific data through gradient-based inner-loop updates. This enables the model to capture both shared structure across domains and domain-specific variations. A key component of our approach is the introduction of domain-conditioned representation learning . We incorporate learnable domain embeddings that modulate both visual and textual representations, inspired by conditional feature modulation methods such as FiLM [ 11 ] . This allows the model to dynamically adjust its embedding space according to domain context. In addition, we introduce a cross-domain alignment regularization that encourages semantically consistent representations across domains, improving robustness while preserving discriminative power. Our framework generalizes standard contrastive learning by shifting the objective from finding a single global optimum to learning parameters that are adaptable across domains . This perspective is particularly important in real-world multimodal applications, where domain shifts are unavoidable and data distributions evolve over time. Contributions. Our main contributions are: • We introduce a domain-conditioned meta-learning framework for contrastive vision-language training that explicitly models domain shift. • We propose a domain-aware representation learning mechanism using learnable domain embeddings to modulate multimodal features. • We formulate a bilevel optimization objective that jointly optimizes for representation alignment and rapid domain adaptation. • We incorporate cross-domain alignment regularization to improve generalization across heterogeneous datasets. — 2 Related Work 2.1 Contrastive Vision-Language Learning Contrastive learning has become the dominant approach for training large-scale vision-language models. Methods such as CLIP [ 12 ] align image and text representations by maximizing agreement between paired samples while contrasting against negative pairs. This paradigm enables strong zero-shot transfer and scalable training across diverse datasets. However, existing approaches typically optimize a global objective over all data, without explicitly accounting for domain-specific variations or distribution shifts. Recent extensions have explored improving contrastive learning through better negative sampling, temperature scaling, and architectural modifications. While these methods improve representation quality, they do not address the fundamental challenge of adapting to heterogeneous domains. — 2.2 Meta-Learning and Bilevel Optimization Meta-learning aims to train models that can rapidly adapt to new tasks using limited data. Gradient-based methods such as Model-Agnostic Meta-Learning (MAML) [ 5 ] formulate learning as a bilevel optimization problem, where an inner loop performs task-specific adaptation and an outer loop optimizes for generalization across tasks. Recent work has begun to explore meta-learning in multimodal settings, including methods that bridge vision and language representations via task-adaptive modules [ 10 ] . However, the integration of meta-learning with contrastive objectives in large-scale vision-language models remains underexplored. In particular, prior work typically defines tasks based on class labels or few-shot splits, rather than explicitly modeling domain heterogeneity. — 2.3 Domain Generalization and Adaptation Domain generalization methods aim to learn representations that are robust to distribution shifts across domains. Existing approaches include domain-invariant feature learning [ 7 ] , adversarial training methods such as domain-adversarial neural networks (DANN) [ 6 ] , as well as invariant risk minimization and related techniques [ 1 ] . Furthermore, meta-learning-based approaches to parallel MRI reconstruction reduce reliance on coil sensitivity estimation and enhance reconstruction fidelity [ 2 ] . More recently, vision-language models have also explored domain robustness through prompt adaptation and conditional representations [ 14 , 13 ] . Meta-learning has shown strong potential in addressing data heterogeneity, particularly in medical imaging. Prior work has demonstrated that optimization-based meta-learning enables robust MRI reconstruction across diverse acquisition settings [ 4 ] . These results highlight the effectiveness of meta-learning for handling domain variability in high-dimensional inverse problems. Beyond medical imaging, meta-learning has also been explored for improving adaptability in multimodal systems. For example, recent work has proposed meta-learning frameworks to bridge vision and language representations for few-shot learning [ 10 ] , while test-time adaptation and prompt-based methods further improve robustness under distribution shift [ 9 , 14 ] . Multi-task meta-learning frameworks further improve adaptability across imaging conditions and protocols [ 3 ] . Despite these advances, existing approaches typically focus either on learning invariant representations or on parameter-efficient adaptation, without explicitly integrating domain-aware modeling into contrastive learning. In contrast, our approach combines domain-aware representation learning with meta-learning, enabling both shared representation learning and flexible domain adaptation within a unified contrastive framework. While these methods focus on learning invariant representations, they often overlook the benefits of task-specific adaptation. In contrast, our approach combines domain-aware modeling with meta-learning, enabling both shared representation learning and flexible domain adaptation. — 2.4 Parameter-Efficient Adaptation in Vision-Language Models Recent work has explored parameter-efficient adaptation techniques, such as prompt tuning [ 14 , 13 ] , adapters [ 8 ] , and low-rank adaptation (LoRA) [ 9 ] , to adapt large pretrained models to downstream tasks. These methods reduce computational cost and enable flexible fine-tuning without modifying the full model. Our work is complementary to these approaches. Instead of focusing solely on parameter efficiency, we introduce a principled meta-learning framework that explicitly models domain shift and optimizes for adaptability. Our method can be naturally combined with parameter-efficient techniques to further improve scalability. — 2.5 Multimodal Learning under Domain Shift Handling domain shift in multimodal