CIBR: Cross-modal Information Bottleneck Regularization for Robust CLIP Generalization
Chenrui Ma 6 Lijing Zhu 5 Aokun Liang 4 Jiansheng Chen 1† · 2025
Read CIBR: Cross-modal Information Bottleneck Regularization for Robust CLIP Generalization by Chenrui Ma 6 Lijing Zhu 5 Aokun Liang 4 Jiansheng Chen 1† (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-24182.txt.
Introduction
Yingrui Ji and co-authors explain CLIP's strong generalization through an information bottleneck lens, then train it better. CLIP aligns images and text so well that it sorts unseen classes with no extra training. Why that works so broadly lacked a firm theory. Their frame reads the training goal as squeezing out sense-specific clutter.
Shared meaning is kept while image-only and text-only extras are shed. A new penalty term, CIBR, enforces that squeeze during training. Zero-shot sorting across seven image sets improved under the rule. Retrieval on two standard caption sets improved as well. The gains back the theory with practical payoff.
Catalog record
| CREATOR | Chenrui Ma 6 Lijing Zhu 5 Aokun Liang 4 Jiansheng Chen 1† |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 24 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-24182
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
CIBR: Cross-modal Information Bottleneck Regularization for Robust CLIP Generalization Chenrui Ma 6 Lijing Zhu 5 Aokun Liang 4 Jiansheng Chen 1† Abstract Contrastive Language-Image Pretraining (CLIP) has achieved remarkable success in cross-modal tasks such as zero-shot image classification and text–image retrieval by effectively aligning visual and textual representations. However, the theoretical foundations underlying CLIP’s strong generalization remain unclear. In this work, we address this gap by proposing the Cross-modal Information Bottleneck (CIB) framework. CIB offers a principled interpretation of CLIP’s contrastive learning objective as an implicit Information Bottleneck optimization. Under this view, the model maximizes shared cross-modal information while discarding modality-specific redundancies, thereby preserving essential semantic alignment across modalities. Building on this insight, we introduce a Cross-modal Information Bottleneck Regularization (CIBR) method that explicitly enforces these IB principles during training. CIBR introduces a penalty term to discourage modality-specific redundancy, thereby enhancing semantic alignment between image and text features. We validate CIBR on extensive vision–language benchmarks, including zero-shot classification across seven diverse image datasets and text–image retrieval on MSCOCO and Flickr30K. The results show consistent performance gains over standard CLIP. These findings provide the first theoretical understanding of CLIP’s generalization through the IB lens. They also demonstrate practical improvements, offering guidance for future cross-modal representation learning. 1 Introduction Contrastive Language-Image Pretraining (CLIP) [ 20 ] has demonstrated significant progress in multimodal representation learning, enabling effective generalization across diverse vision-language tasks. Unlike traditional supervised visual models, CLIP employs large-scale, noisy image-text datasets and leverages contrastive learning to align visual and textual embeddings. This training paradigm results in remarkable zero-shot capabilities, facilitating transfer to a wide range of downstream tasks without task-specific fine-tuning [ 20 , 39 ] . Despite these empirical successes, the underlying theoretical reasons for CLIP’s strong cross-modal generalization remain insufficiently understood. Current research primarily aims at enhancing CLIP through empical methods, including sophisticated prompt engineering and advanced fine-tuning strategies [ 39 , 16 , 3 ] , yet these approaches provide limited theoretical insights. The critical question remains unresolved: what theoretical mechanism enables CLIP’s strong generalization capabilities across modalities? To address this gap, we revisit CLIP through the lens of Information Bottleneck (IB) theory [ 25 , 26 ] , aiming to fundamentally explain CLIP’s extraordinary generalization capability. IB theory offers a rigorous framework for understanding generalization by positing that effective representations compress input data while retaining task-relevant information [ 23 , 22 , 13 ] . We hypothesize that CLIP implicitly achieves an IB-like optimization, simultaneously compressing irrelevant, modality-specific redundancies and preserving essential cross-modal semantic correlations. In this paper, we formalize this intuition by introducing a novel theoretical framework, named Cross-modal Information Bottleneck (CIB), explicitly linking CLIP’s contrastive learning objectives to IB principles. Figure 1: Illustration of our Cross-modal Information Bottleneck Regularization (CIBR) strategy. Given image-text pairs, the visual and textual encoders produce embeddings Z_{v} and Z_{t} , which are aligned through contrastive learning. • We provide the first comprehensive theoretical interpretation of CLIP from the Information Bottleneck perspective, elucidating the intrinsic reasons behind its strong cross-modal generalization. • Based on IB theory, we propose a novel mutual-information regularization strategy, explicitly guiding the CLIP model towards optimal cross-modal representation learning. • Our theoretical insights are validated through preliminary experiments on representative multimodal datasets, highlighting the effectiveness and practical value of theory-informed approaches in contrastive multimodal learning. 2 Related Work 2.1 Contrastive Language-Image Pretraining (CLIP) Contrastive Language-Image Pretraining (CLIP) [ 20 , 39 , 24 , 36 ] has emerged as a significant breakthrough in multimodal representation learning by effectively bridging vision and language modalities. Trained on millions of noisy image-text pairs, CLIP optimizes a contrastive objective that aligns representations from the two modalities into a unified embedding space. This enables zero-shot transfer learning across numerous downstream tasks, significantly outperforming traditional supervised methods on various vision benchmarks [ 20 , 39 ] . Despite its empirical success, current studies mainly concentrate on further empirical improvements, such as CoOp [ 38 ] and MaPLe [ 16 ] , and ALIGN [ 14 ] , which explore effective prompting mechanisms and large-scale multimodal alignment. Moreover, LiT [ 37 ] further enhances the transferability by leveraging pre-trained vision and text models. Despite these empirical successes, theoretical understanding of CLIP’s generalization capabilities across modalities remains under-explored. While some recent theoretical works examine the representational geometry [ 32 ] or learning dynamics [ 21 ] in unimodal contrastive settings, rigorous analyses specifically addressing CLIP’s cross-modal generalization remain scarce. Understanding the underlying theoretical mechanisms driving CLIP’s remarkable zero-shot capabilities thus represents an important yet open research direction. [ 20 , 39 ] . Despite its empirical success, current studies mainly concentrate on further empirical improvements. Moreover, LiT [ 37 ] enhances transferability by leveraging pre-trained vision and text models. Nonetheless, theoretical understanding of CLIP’s cross-modal generalization remains under-explored. While recent theoretical works examine representational geometry [ 32 ] or learning dynamics [ 21 ] in unimodal contrastive settings, rigorous analyses explicitly addressing CLIP’s cross-modal generalization remain scarce. Therefore, uncovering the theoretical foundations behind CLIP’s remarkable zero-shot capabilities is an important yet open research direction. 2.2 Prompt Tuning in Vision Models Prompt tuning originated in the Natural Language Processing (NLP) domain as a parameter-efficient fine-tuning strategy, allowing large pretrained models to adapt quickly to downstream tasks by learning additional soft prompts [ 17 ] . Prompt tuning was soon adapted to vision foundation models, notably Vision Transformers (ViT) [ 7 , 15 , 34 ] . For instance, Visual Prompt Tuning (VPT) [ 15 ] introduced learnable prompts to the input and intermediate feature spaces of ViT, achieving promising performance with minimal parameter updates. Recent advances further refine prompt tuning by incorporating semantic priors, such as textures, shapes, and color histograms, significantly enhancing generalization and interpretability [ 3 , 9 , 35 ] . More recent works, such as Multi-modal Prompt Learning [ 16 ] and Cross-modal Prompt Tuning [ 31 ] , have demonstrated the effectiveness of prompt-based methods in cross-modal learning scenarios. However, these methods predominantly remain empirically driven, with limited theoretical insights guiding the design and selection of prompts in the visual domain. 2.3 Information Bottleneck in Deep Learning The IB theory, first introduced by Tishby [ 25 ] and [ 23 , 26 ] , provides a principled framework for understanding representation learning in neural networks. The IB principle aims to find the optimal representation by compressing the input data, preserving only information relevant to target labels. [ 23 , 26 ] . Specifically, IB theory explains the generalization capabilities of neural networks through the lens of mutual information between input, representations, and task labels [ 4 , 22 ] . Goldfeld [ 8 ] and Alemi [ 2 ] further extend these theoretical analyses by providing formal connections between neural network optimization and IB principles. Nonetheless, existing IB-driven analyses have mainly focused on single-modality data, lacking rigorous extension to cross-modal scenarios, especially for contrastive learning frameworks such as CLIP. 2.4 Theoretical Analysis of Contrastive Learning Contrastive learning has attracted substantial theoretical interest due to its impressive empirical performance in unsupervised and self-supervised learning scenarios [ 5 , 11 ] . Recent theoretical works primarily focus on deriving generalization bounds using the Rademacher complexity and PAC-Bayes frameworks, highlighting how contrastive objectives implicitly