Bayesian Cross-Modal Alignment Learning for Few-Shot Out-of-Distribution Generalization
Lin Zhu, Xinbing Wang, Chenghu Zhou, Nanyang Ye · 2025
Read Bayesian Cross-Modal Alignment Learning for Few-Shot Out-of-Distribution Generalization by Lin Zhu, Xinbing Wang, Chenghu Zhou, Nanyang Ye (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-09448.txt.
Introduction
Lin Zhu, Xinbing Wang, and their co-authors present a Bayesian method that teaches vision models to generalize from a few examples to new kinds of data. Their 2025 paper notes that large pretrained models do well on few shot tasks within their training world, but shifts in correlation or diversity break them. Even heavy training often fails to beat a plain baseline under such shifts.
Their method aligns image and text representations, tuning only the text side. Bayesian modeling guards against overfitting the few examples, while two extra losses push the model to ignore spurious links. The gradient step is also adjusted to keep directions from clashing. Tests on few shot out of distribution tasks showed gains over standard training. The approach targets domains where labeled examples are scarce.
Catalog record
| CREATOR | Lin Zhu, Xinbing Wang, Chenghu Zhou, Nanyang Ye |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 51 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2504-09448
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Bayesian Cross-Modal Alignment Learning for Few-Shot Out-of-Distribution Generalization Lin Zhu, Xinbing Wang, Chenghu Zhou, Nanyang Ye Abstract Recent advances in large pre-trained models showed promising results in few-shot learning. However, their generalization ability on two-dimensional Out-of-Distribution (OoD) data, i.e., correlation shift and diversity shift, has not been thoroughly investigated. Researches have shown that even with a significant amount of training data, few methods can achieve better performance than the standard empirical risk minimization method (ERM) in OoD generalization. This few-shot OoD generalization dilemma emerges as a challenging direction in deep neural network generalization research, where the performance suffers from overfitting on few-shot examples and OoD generalization errors. In this paper, leveraging a broader supervision source, we explore a novel Bayesian cross-modal image-text alignment learning method (Bayes-CAL) to address this issue. Specifically, the model is designed as only text representations are fine-tuned via a Bayesian modelling approach with gradient orthogonalization loss and invariant risk minimization (IRM) loss. The Bayesian approach is essentially introduced to avoid overfitting the base classes observed during training and improve generalization to broader unseen classes. The dedicated loss is introduced to achieve better image-text alignment by disentangling the causal and non-casual parts of image features. Numerical experiments demonstrate that Bayes-CAL achieved state-of-the-art OoD generalization performances on two-dimensional distribution shifts. Moreover, compared with CLIP-like models, Bayes-CAL yields more stable generalization performances on unseen classes. Our code is available at https://github.com/LinLLLL/BayesCAL . Shanghai Jiao Tong University, Shanghai, China [email protected], [email protected], [email protected], [email protected] . Introduction Few-shot learning is an emerging research topic that aims to generalize from only a few training samples ( Wang et al. 2020 ) . Despite the success of recent few-shot learning methods on independent and identically distributed (I.I.D) settings ( Finn, Abbeel, and Levine 2017 ; Rusu et al. 2018 ; Sung et al. 2018 ; Vuorio et al. 2019 ; Fan et al. 2021 ) , these few-shot learning methods would suffer significant performance drop in the presence of domain differences between source and target datasets ( Chen et al. 2019 ; Guo et al. 2020 ) . This domain shift issue commonly exists in real scenarios, especially in the few-shot setting. For example, it is difficult to construct large training datasets for rare species, there can be huge differences between the training and test environments due to the large randomness of the few-shot setting. Figure 1: Illustration of the proposed Bayesian cross-modal alignment learning paradigms. To avoid overfitting on the few-shot training samples, \varphi^{(t)} is modelled by a variational distribution q(\varphi^{(t)}) to approximate its posterior distribution. To address this domain shift problem, i.e., training and test data access different conditional distributions, there have been plenty of methods proposed to achieve domain generalization in few-shot learning scenarios ( Tseng et al. 2020 ; Liu et al. 2021 ; Liang et al. 2021 ; Zhou and Tan 2021 ) . However, most of these studies focused on this issue without considering the different characteristics of distribution shifts between training and test domains. In this paper, we consider a more practical setting of few-shot Out-of-Distribution (OoD) generalization , focusing on the few-shot image recognition task of generalizing under two major types of distribution shifts (as described in OoD-Bench ( Ye et al. 2021 ) ). Specifically, models usually have access to multi-domain ”K-shot N-way” training samples of OoD data dominated by diversity shift or correlation shift. According to ( Ye et al. 2021 ) , the diversity shift is defined by the support set’s differences on latent environment’s distributions (e.g., changes in the image style), and the correlation shift is defined by the probability density functions’ differences caused by spurious correlations. Under this condition, models are expected to learn feature distributions only from the seen domains and generalize to unseen domains within the same downstream task. In recent few-shot learning methods, studies ( Rahman, Khan, and Barnes 2020 ; Zhou et al. 2022b ; Zhang et al. 2021a ; Zhu et al. 2021 ) based on large-scale vision-language pre-trained models achieved striking performances in various downstream tasks. The conventional visual deep learning models that only focus on closed-set visual concepts, are susceptible to overfitting on the pre-defined list of classes due to their limited supervision source ( Zhou et al. 2022a ) . In contrast, vision-language models leverage a broader source of supervision coming from natural language, which has been proven effective in learning transferable representations ( Jia et al. 2021 ; Radford et al. 2021 ) . A widely utilized method in vision-language pre-trained models is the cross-modal representation learning ( Fang et al. 2022 ; Li et al. 2020 ; Wehrmann, Kolling, and Barros 2020 ; Zheng et al. 2020 ) . Inspired by the advantages of the cross-modal alignment and Bayesian methods ( Lin et al. 2022 ) for alleviating overfitting, instead of focusing on improving learning algorithms for learning OoD generalizable image features, we propose the Bayesian cross-modal alignment learning method (Bayes-CAL) to achieve few-shot OoD generalization. Since fine-tuning the entire model is impractical and might damage the well-learned representation space ( Zhou et al. 2022a ) and adjusting the text representations are more flexible, as shown in Figure 1 (a), we design the model architecture as only a task-specific network in text feature extractor is tuned in each specific downstream task. In this paper, we instantiate the text representation learning of Bayes-CAL in three methods–prompt leaning ( Ding et al. 2021 ) , directly utilizing learnable vectors, and Word2Vector ( Mikolov et al. 2013 ) to detail how Bayes-CAL works on few-shot OoD generalization. Our key contributions are as follows: 1. We propose a Bayesian treatment for cross-modal alignment learning for few-shot OoD generalization. The superiority of the Bayesian method is demonstrated by stable generalization performances on unseen classes. We have also carefully designed experiments to gain insight into the superiority of image-text alignment learning. 2. Under the proposed architecture (see Figure 2 ), gradient orthogonalization loss is introduced to achieve better alignment learning by disentangling image features. Invariant risk minimization (IRM) loss is utilized to improve OoD generalization ability further. 3. Bayes-CAL has achieved state-of-the-art performances on OoD-Bench datasets with both diversity shift and correlation shift, especially 10%-20% performance improvements compared with algorithms in OoD-Bench ( Ye et al. 2021 ) . Moreover, it outperforms CLIP-like solid models by more stable generalization performance on both I.I.D and OoD unseen classes. Related Work Foundation Models In this paper, we focus on foundation models of large-scale vision-language pre-training, which have recently emerged ( Gu et al. 2021 ; Dai et al. 2021 ; Radford et al. 2021 ) for various image-text retrieval tasks. Especially for image recognition, a contrastive language image pre-training model (CLIP, ( Radford et al. 2021 ) ) is proposed. In CLIP, images and texts are encoded to the feature space. Then, the model is optimized to maximize the similarity of image features and texts features. There are also many efficient CLIP-based models to enhance generalization performance via prompt tuning or image feature adapters. Prompt tuning is a type of method to get better vision-language alignment via only fine-tuning the input prompts, such as CoOp ( Zhou et al. 2022b ) , CoCoOp ( Zhou et al. 2022a )