Domain Generalization in-the-Wild: Disentangling Classification from Domain-Aware Representations
Ha Min Son, Zhe Zhao, Shahbaz Rezaei, Xin Liu · 2025
Read Domain Generalization in-the-Wild: Disentangling Classification from Domain-Aware Representations by Ha Min Son, Zhe Zhao, Shahbaz Rezaei, Xin Liu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-21769.txt.
Introduction
Ha Min Son and colleagues study domain generalization for foundation models in the wild. Their 2025 arXiv paper focuses on CLIP, trained on web-scale data. Such pretraining may already cover many standard test sets. That makes honest tests of unseen-domain skill hard. The team built tougher ways to judge CLIP on new domains.
They scored many diverse datasets by how far they sit from training data. They also made CLIP forget some domains through unlearning. CLIP performance fell sharply on the most distant datasets. Their fix keeps domain-aware features instead of forcing them away. Standard invariance losses can harm foundation models, they argue. The new method separates classification from domain-aware representation. Tests support better handling of truly unseen data.
Catalog record
| CREATOR | Ha Min Son, Zhe Zhao, Shahbaz Rezaei, Xin Liu |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 57 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2508-21769
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Domain Generalization in-the-Wild: Disentangling Classification from Domain-Aware Representations Ha Min Son, Zhe Zhao, Shahbaz Rezaei, Xin Liu Abstract Evaluating domain generalization (DG) for foundational models like CLIP is challenging, as web-scale pretraining data potentially covers many existing benchmarks. Consequently, current DG evaluation may neither be sufficiently challenging nor adequately test genuinely unseen data scenarios. To better assess the performance of CLIP on DG in-the-wild, a scenario where CLIP encounters challenging unseen data, we consider two approaches: (1) evaluating on 33 diverse datasets with quantified out-of-distribution (OOD) scores after fine-tuning CLIP on ImageNet, and (2) using unlearning to make CLIP ‘forget’ some domains as an approximation. We observe that CLIP’s performance deteriorates significantly on more OOD datasets. To address this, we present CLIP-DCA ( D isentangling C lassification from enhanced domain A ware representations). Our approach is motivated by the observation that while standard domain invariance losses aim to make representations domain-invariant, this can be harmful to foundation models by forcing the discarding of domain-aware representations beneficial for generalization. We instead hypothesize that enhancing domain awareness is a prerequisite for effective domain-invariant classification in foundation models. CLIP-DCA identifies and enhances domain awareness within CLIP’s encoders using a separate domain head and synthetically generated diverse domain data. Simultaneously, it encourages domain-invariant classification through disentanglement from the domain features. CLIP-DCA shows significant improvements within this challenging evaluation compared to existing methods, particularly on datasets that are more OOD. 1 Introduction Domain generalization (DG) aims to train models that maintain robust performance when encountering out-of-distribution (OOD) data zhou2022domain . A key assumption of DG is that the target domains represent novel data distributions for evaluation. However, this assumption is challenged when evaluating pretrained foundation models like CLIP radford2021learning_CLIP and ALIGN jia2021scaling . These models have been trained on comprehensive web-scale datasets, thus have likely been exposed to most existing domains, contributing to its impressive zero-shot capabilities. Consequently, much research has focused on adapting CLIP through parameter-efficient finetuning zhou2022learning_coop ; zhou2022conditional_cocoop ; gao2024clipadapter ; zhang2022tip , regularization using the original weights wortsman2022robust_wise ; nam2024lipsum ; oh2024towards_carot ; shu2023clipood , and even transductive methods wallingford2023neuralpriming ; martin2024transductive , largely preserving its pretrained knowledge. However, a critical question arises: how would CLIP perform on genuinely unseen domains? A recent study mayilvahanan2024search found that retraining CLIP from scratch using only natural images significantly degrades performance on OOD benchmarks, resulting in performance similar to models trained only on ImageNet. Our results (Figure 1 ) align with these findings, suggesting current DG evaluations for CLIP may overestimate its true OOD robustness because standard evaluation settings like leave-one-domain-out and existing cross-dataset evaluations may not be sufficiently challenging (Sec. 4.1 , 4.2 ). Figure 1 : Improvement over zeroshot after finetuning on ImageNet (in %). OOD scores are quantified relative to ImageNet (source dataset), illustrating the challenge of DG in-the-wild. We therefore propose that DG evaluation for foundation models, such as CLIP, should be more challenging, to approximate “domain generalization in-the-wild," where CLIP might encounter diverse and challenging new data in the real-world. We evaluate CLIP on 33 target datasets spanning a diverse range of OODness. To systematically approach evaluation, we quantify a multi-modal OOD score (Sec. 4.2 ), using ImageNet as both an anchor and a source dataset owing to its inclusion of many classes and concepts. We find that after finetuning on ImageNet, CLIP’s DG performance degrades on datasets with higher OOD scores with respect to ImageNet (Figure 1 ), consistent with the domain contamination findings mayilvahanan2024search . In addition, to further simulate truly unseen domains, we use an unlearning technique sepahvandselective to make CLIP forget some domains (Sec. 1 ), and find significant performance degradation for existing robust finetuning methods. Our results (Figure 9 ), alongside findings on domain contamination mayilvahanan2024search , suggest that for DG in-the-wild, different robust finetuning algorithms are needed for genuinely unseen data. In light of this, we present CLIP-DCA ( D isentangling C lassification from enhanced domain A ware representations), an end-to-end finetuning method to improve the robustness of CLIP on truly OOD data. A key idea in DG is that learning domain-invariant features is beneficial for robust generalization zhou2022domain ; ganin2016domain_dann . However, naively enforcing domain invariance for a pretrained foundation model could cause catastrophic forgetting of useful features learned from diverse domains during pretraining as the model is forced to make its representations entirely domain-invariant. We hypothesize that to learn effective domain invariance, domain awareness is a prerequisite. This awareness is critical to maintain CLIP’s vast knowledge, which includes generalizable features that support capabilities like zero-shot classification. By enhancing domain awareness, CLIP can also selectively disentangle classification from domain-specific aspects, thereby achieving robust generalization without forgetting valuable information. We combine the idea of domain awareness and domain invariance by encouraging them simultaneously within CLIP-DCA (Figure 2 ). Specifically, we encourage domain awareness within CLIP’s image and text encoders, while promoting domain invariance specifically at the final classification layer through disentanglement. Our premise is that while domain awareness is a requirement to maintain pre-existing knowledge, this awareness can be disentangled for domain-invariant classification and robust generalization. To achieve this, we add a new head to the CLIP image encoder, called the domain head, which is trained to understand domains. The original classification head is then disentangled from the domain head, effectively learning domain awareness within its encoders and achieving domain invariance at the classification stage. Additionally, since many datasets lack distinct domains or textual descriptions, and the definition of ‘domain’ is often vague in DG in-the-wild, we address this by using diffusion models to create images of artificial domains and MLLMs to generate descriptions for these artificial domains (Sec. 3.2 ). Our contributions are summarized as follows: • We demonstrate potential limitations in current DG evaluations of foundation models, supported by our results and a recent study. Existing benchmarks may overestimate true OOD robustness, potentially leading finetuning strategies towards in-distribution improvement rather than OOD. • We propose more challenging and holistic evaluations for DG in-the-wild. We use an expanded cross-dataset evaluation setting spanning 33 datasets from diverse domains, indexed by multi-modal OOD scores. We also use an unlearned model to further approximate unseen domains. • We introduce CLIP-DCA, a novel finetuning method that improves OOD robustness by disentangling classification from enhanced domain-aware representations. We find that on more OOD target datasets, CLIP-DCA performs significantly better compared to existing robust finetuning methods, while performance is similar across all methods on less OOD target datasets. 2 Related Work Domain Generalization. Learning domain-invariant representations has historically been a central idea in domain generalization blanchard2011generalizing ; zhou2022domain . The intuition is that when classifying images from entirely new distributions, learning abstract features common across source domains should provide better robustness for classification in new domains blanchard2011generalizing ; muandet2013domain . Among these, domain-adversarial learning methods have become a relatively standard approach within the DG field due to its conceptual simplicity and effectiveness zhou2022domain . For instance, Domain Adversarial Neural Networks (DANN) ganin2016domain_dann uses an auxiliary domain classifier trained adversarially against the encoder, encouraging the encoder to produce features indistinguishable across source domains. Given the focus of DANN on the central idea of domain invariance, we focus on DANN and its adaptation to CLIP in our analysis. Notably, despite