OVA-IB: One vs All Information Bottleneck for Multi-Modal Alignment

Tianchao Li 1 Shujian Yu 2,3 Xinrui Zu 2 Zhaolong Wei 6 Jeremy Gummeson 6 Jack C.P. Cheng 1 Robert Jenssen 3,4,5 1 Hong Kong University of Science and Technology, Hong Kong 2 Vrije Universiteit Amsterdam, Netherlands 3 UiT – The Arctic University of Norway, Norway 4 University of Copenhagen, Denmark 5 Norwegian Computing Center, Norway 6 University of Massachusetts Amherst, USA · 2026

Read OVA-IB: One vs All Information Bottleneck for Multi-Modal Alignment by Tianchao Li 1 Shujian Yu 2,3 Xinrui Zu 2 Zhaolong Wei 6 Jeremy Gummeson 6 Jack C.P. Cheng 1 Robert Jenssen 3,4,5 1 Hong Kong University of Science and Technology, Hong Kong 2 Vrije Universiteit Amsterdam, Netherlands 3 UiT – The Arctic University of Norway, Norway 4 University of Copenhagen, Denmark 5 Norwegian Computing Center, Norway 6 University of Massachusetts Amherst, USA (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-29900.txt.

Introduction

Tianchao Li and six colleagues rethink lining up three-plus data modes, where pairwise losses miss all-mode links. Through the Information Bottleneck, each mode keeps what others foretell and sheds detail no other mode backs. OVA-IB pits each mode against the rest, with a plain geometry-aware score and a bound capping each encoding's tie to its input. Tests across labeling, regression, mode-free scoring, and retrieval show strong, steady gains. The result gives multi-mode alignment a rule for what each mode should keep and drop.

Catalog record

CREATORTianchao Li 1 Shujian Yu 2,3 Xinrui Zu 2 Zhaolong Wei 6 Jeremy Gummeson 6 Jack C.P. Cheng 1 Robert Jenssen 3,4,5 1 Hong Kong University of Science and Technology, Hong Kong 2 Vrije Universiteit Amsterdam, Netherlands 3 UiT – The Arctic University of Norway, Norway 4 University of Copenhagen, Denmark 5 Norwegian Computing Center, Norway 6 University of Massachusetts Amherst, USA
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE70327
LICENSEhttp://creativecommons.org/licenses/by-sa/4.0/
RIGHTSCC BY SA
SOURCEarXiv

Source: https://arxiv.org/abs/2605.29900

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

OVA-IB: One vs All Information Bottleneck for Multi-Modal Alignment Tianchao Li 1 Shujian Yu 2,3 Xinrui Zu 2 Zhaolong Wei 6 Jeremy Gummeson 6 Jack C.P. Cheng 1 Robert Jenssen 3,4,5 1 Hong Kong University of Science and Technology, Hong Kong 2 Vrije Universiteit Amsterdam, Netherlands 3 UiT – The Arctic University of Norway, Norway 4 University of Copenhagen, Denmark 5 Norwegian Computing Center, Norway 6 University of Massachusetts Amherst, USA Abstract Contrastive learning is effective for aligning paired views or modalities, but alignment beyond two modalities remains non-trivial and comparatively underexplored. Pairwise CLIP-style losses decompose multi-modal alignment into independent two-way comparisons and therefore do not explicitly model higher-order dependencies among multiple modalities. Recent beyond-pairwise objectives approach this problem from statistical or geometric perspectives, but arbitrary-modality alignment still lacks a principled criterion for defining what each modality should preserve and compress relative to the others. We revisit arbitrary-modality alignment through the Information Bottleneck principle. In multi-modal learning, sufficiency should preserve information predictable from the remaining modalities, while minimality should compress modality-specific information not supported by them. This naturally leads to a One-vs-All view, where each modality is characterized with respect to the remaining modalities. We propose OVA-IB, an Information Bottleneck framework for arbitrary-modality alignment. OVA-IB optimizes a tractable One-vs-All contrastive lower bound for sufficiency connected to a Dual Total Correlation-style objective, uses a parameter-free geometry-aware projection score, and derives a tractable upper-bound regularizer for minimality by bounding each representation’s dependence on its own input with representation distributions induced by the remaining modalities. Experiments on classification, regression, modality-agnostic evaluation, and cross-modal retrieval benchmarks demonstrate strong and robust performance. 1 Introduction Contrastive Learning (CL) has become a central paradigm for self-supervised representation learning, as it enables models to learn transferable representations without relying on extensive human annotations. In the single-modality setting, different views of the same samples are typically generated through data augmentation, and the model is trained to bring positive views closer while separating representations from different samples. As a widely used objective for this purpose, InfoNCE 31 ; 7 maximizes agreement between paired views and has been interpreted from both information-theoretic and geometric perspectives. From the information-theoretic perspective, InfoNCE maximizes a lower bound on mutual information between positive views 37 ; from the geometric perspective, contrastive learning simultaneously promotes alignment between positive pairs and uniformity of representations on the hypersphere 32 . These two perspectives provide the foundation for understanding contrastive learning as both an information-preserving and geometry-shaping representation learning framework. For two-modality representation learning, paired modalities provide natural positive views for contrastive alignment. A prominent example is CLIP 25 , which applies a symmetric InfoNCE objective to image–text pairs and learns a shared embedding space with strong transfer performance. However, this formulation is intrinsically pairwise. For three or more modalities, a common extension is to sum CLIP-style losses over all modality pairs. While simple and scalable, this strategy decomposes multi-modal alignment into independent two-modality comparisons and therefore fails to explicitly capture higher-order dependencies jointly supported by multiple modalities. This motivates alignment objectives that move beyond pairwise agreement. Recent beyond-pairwise methods address this limitation from complementary directions. Symile 27 models higher-order statistical dependence among an arbitrary number of modalities, while Gram 10 and TRIANGLE 9 introduce geometry-aware objectives in the shared representation space. These works provide important statistical and geometric formulations of beyond-pairwise alignment. However, alignment beyond two modalities remains comparatively underexplored, and it is still unclear how to define what each modality representation should preserve and compress when multiple remaining modalities are available. To instantiate this principle, we propose OVA-IB, a One-vs-All Information Bottleneck framework for multi-modal alignment with an arbitrary number of modalities. The key idea is to define both sufficiency and minimality in a modality-wise manner: for each modality, the remaining modalities serve as the reference for determining what information should be preserved and what modality-specific variation should be compressed. Specifically, the sufficiency term encourages each modality representation to retain information predictable from the remaining modalities. We derive a tractable One-vs-All contrastive lower bound for this objective and show that it is connected to a Dual Total Correlation 34 (DTC)-style dependence measure. To make the alignment geometry-aware, we score each modality by its projection onto the subspace spanned by the remaining modality embeddings. For minimality, we derive a tractable upper-bound regularizer that controls each representation’s dependence on its own input using representation distributions induced by the remaining modalities. Our contributions are summarized as follows: • We propose OVA-IB, a One-vs-All Information Bottleneck framework for multi-modal alignment with an arbitrary number of modalities. • We derive a DTC-style sufficiency objective that aligns each modality with the remaining modalities as complementary evidence, and introduce a geometry-aware projection score that aligns each embedding with the subspace spanned by the remaining modalities. • We derive a tractable minimality regularizer that suppresses modality-specific nuisance information by bounding each representation’s dependence on its own input using distributions induced by the remaining modalities. • Experiments on classification, regression, modality-agnostic evaluation, and cross-modal retrieval benchmarks demonstrate that OVA-IB achieves strong and robust performance. 2 Related Work 2.1 Contrastive Learning for Multi-Modal Representation Learning Contrastive learning (CL) has been widely adopted for self-supervised representation learning by aligning positive views and separating negative samples 31 ; 7 ; 37 ; 26 ; 32 ; 35 ; 38 ; 23 . In multi-modal learning, different modalities of the same sample naturally define positive pairs, enabling contrastive objectives to learn shared information across modalities. CLIP 25 applies a symmetric InfoNCE to image-text pairs and has become a standard paradigm for two-modality alignment: \displaystyle\mathcal{L}^{(1\rightarrow 2)}_{\text{InfoNCE}}(\theta) \displaystyle=-\frac{1}{N}\sum_{i=1}^{N}\log\frac{\exp\left(\text{sim}(\mathbf{z}^{(1)}_{i},\mathbf{z}^{(2)}_{i})/\tau\right)}{\sum_{k=1,k\neq i}^{N}\exp\left(\text{sim}(\mathbf{z}^{(1)}_{i},\mathbf{z}^{(2)}_{k})/\tau\right)}, (1) \displaystyle\mathcal{L}^{(1,2)}_{\text{CLIP}}(\theta) \displaystyle=\frac{1}{2}(\mathcal{L}^{(1\rightarrow 2)}_{\text{InfoNCE}}(\theta)+\mathcal{L}^{(2\rightarrow 1)}_{\text{InfoNCE}}(\theta)), (2) where \mathbf{z}_{i}^{(m)} represents the representation of the m -th modality of the i -th sample, \tau is a temperature parameter, and \theta represents all the learnable parameters. For more than two modalities, a common extension is to apply CLIP-style objectives over all modality pairs 29 ; 1 ; 13 ; 6 ; 2 ; 5 ; 22 ; 33 ; 8 . In the simplest case, for three modalities, the pairwise CLIP loss corresponds to \displaystyle\mathcal{L}^{(1,2,3)}_{\text{CLIP}}(\theta)=\mathcal{L}^{(1,2)}_{\text{CLIP}}(\theta)+\mathcal{L}^{(1,3)}_{\text{CLIP}}(\theta)+\mathcal{L}^{(2,3)}_{\text{CLIP}}(\theta). (3) It reduces multi-modal alignment to a collection of independent pairwise comparisons and therefore does not explicitly capture higher-order dependencies among multiple modalities. Compared with the extensive literature on pairwise multi-modal contrastive learning, alignment objectives for more than two modalities remain comparatively underexplored, with only a small number of recent works explicitly addressing this setting. Symile 27 extends contrastive learning by modeling higher-order statistical dependence among arbitrary modalities, while Gram 10 and TRIANGLE 9 introduce geometry-aware objectives based on the volume or area formed by the same-sample modality embeddings. These works provide important initial steps toward beyond-pairwise alignment from statistical and geometric perspectives. Complementary to them, we revisit arbitrary-modality alignment from the Information Bottleneck 30 perspective and ask how sufficiency and minimality should be defined for each modality when multiple remaining modalities are available. 2.2 Information Bottleneck for Multi-Modal Representation Learning Information Bottleneck (IB) 30 provides an information-theoretic framework for learning representations that are sufficient for a target variable while remaining minimal with respect to the input. Given an input \mathbf{x} , a target \mathbf{y} , and a representation \mathbf{t} , the standard IB objective is formulated as \max_{\mathbf{t}}\ I(\mathbf{y};\mathbf{t})-\beta I(\mathbf{x};\mathbf{t}),

Read the complete text