Aligning Multimodal Representations through an Information Bottleneck
Antonio Almudévar, José Miguel Hernández-Lobato, Sameer Khurana, Ricard Marxer, Alfonso Ortega · 2025
Read Aligning Multimodal Representations through an Information Bottleneck by Antonio Almudévar, José Miguel Hernández-Lobato, Sameer Khurana, Ricard Marxer, Alfonso Ortega (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-04870.txt.
Introduction
Antonio Almudévar and co-authors tackle a flaw in how AI systems learn to match different kinds of data, such as images and text. Popular contrastive training pulls matched pairs together, yet the resulting shared space still fails to line up well. The authors argue the cause is leftover information that belongs to one kind of data alone.
Seen through the information bottleneck lens, training that only boosts shared information never strips out the private parts. The team supports the claim with controlled tests showing how settings affect the gap. They then add a penalty term, derived from a standard approximation, that pushes the two views into closer alignment. Tests in both controlled setups and real tasks show the penalty helps, offering a fix for systems that search or reason across data types.
Catalog record
| CREATOR | Antonio Almudévar, José Miguel Hernández-Lobato, Sameer Khurana, Ricard Marxer, Alfonso Ortega |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 52 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2506-04870
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Aligning Multimodal Representations through an Information Bottleneck Antonio Almudévar, José Miguel Hernández-Lobato, Sameer Khurana, Ricard Marxer, Alfonso Ortega Abstract Contrastive losses have been extensively used as a tool for multimodal representation learning. However, it has been empirically observed that their use is not effective to learn an aligned representation space. In this paper, we argue that this phenomenon is caused by the presence of modality-specific information in the representation space. Although some of the most widely used contrastive losses maximize the mutual information between representations of both modalities, they are not designed to remove the modality-specific information. We give a theoretical description of this problem through the lens of the Information Bottleneck Principle. We also empirically analyze how different hyperparameters affect the emergence of this phenomenon in a controlled experimental setup. Finally, we propose a regularization term in the loss function that is derived by means of a variational approximation and aims to increase the representational alignment. We analyze in a set of controlled experiments and real-world applications the advantages of including this regularization term. 1 Introduction Multimodal Learning is an area of AI that is focused on processing and integrating information from multiple modalities (e.g., text, image or audio). It is becoming a pivotal topic in the community because of multiple reasons, including, but not limited to, (i) it permits to mimic human cognition processes ( Fei et al. 2022 ; Lee et al. 2023 ) ; (ii) it allows to use a greater amount of training data from different modalities, which tends to improve the performance of the models ( Kaplan et al. 2020 ; Cuervo & Marxer 2024 ) ; and (iii) it is essential in real-world applications such as autonomous vehicles ( Xiao et al. 2020 ) , healthcare ( Kline et al. 2022 ) or human-computer interaction ( Sinha et al. 2010 ) . Similarly to humans, most AI systems work through obtaining intermediate representations, which are compressed versions of the raw data that preserve useful information to solve different downstream tasks ( Bengio et al. 2013 ; Cadieu et al. 2014 ) . One of the most widely used ways of training multimodal systems is Contrastive Representation Learning ( Karpathy & Fei-Fei 2015 ; Oord et al. 2018 ; Tian et al. 2020a ) . In this paradigm, representations corresponding to similar input data are brought closer than dissimilar ones. For example, the caption “a photo of a dog” should become closer to an image of a dog than to that of a cat . The most widely used of the contrastive losses is the \mathrm{InfoNCE} ( Oord et al. 2018 ) . Minimizing this loss is equivalent to maximizing a lower bound of the mutual information of the representations from both modalities. In other words, when minimizing this loss, representations from each modality should maximize the information that they contain about what is common between them. However, the above does not imply that representations from both modalities contain the same information. Representations could contain all the information about what is common to both modalities, but still preserve much of the information that is specific to their own modality (a.k.a. nuisances from now on). We argue that this can translate into a substantial representational misalignment ( Klabunde et al. 2023 ) , especially when the inputs contain a high level of nuisances. In other words, representations from two modalities of a positive pair could be not so similar to each other due to the fact that they are encoding different information. Figure 1 illustrates a trivial example in which two similar, yet different, images have exactly the same caption. Thus, the text representations are exactly the same, while the image representations are different from each other, since they can be encoding information about aspects like the color of the dog, the number of clouds in the sky or the number of blades of grass. This misalignment phenomenon has been already observed and denominated modality gap ( Liang et al. 2022 ) . However, to the best of our knowledge, the present is the first work in which this phenomenon is explained from an information theory perspective. Figure 1 : Different modalities usually contain different information. A trivial example of this is the case in which two different images have exactly the same caption. As a consequence of this, representations from different modalities tend to contain different information (thus leading to misalignment) if the opposite is not explicitly imposed. It is precisely this explanation which allows us to propose a solution to this phenomenon. Concretely, we propose to apply an Information Bottleneck (IB) ( Tishby et al. 2000 ) in the representation space. With this, apart from maximizing the mutual information between the representations of both modalities through the contrastive loss, we reduce the nuisances that can be found in the representations. This IB is applied by means of a regularization term in the loss function that is derived using a variational approximation. Thus, it is efficient, straightforward to implement and modality-agnostic, which is advantageous over alternative approaches ( Li et al. 2021 ) . 2 Preliminaries Contrastive Representation Learning (CRL) This encompasses a set of techniques that learns a representation space in which representations of similar inputs are closer than those of dissimilar ones. It has emerged as one of the most competitive methods for learning representations without labels in a self-supervised way ( Oord et al. 2018 ; Hjelm et al. 2018 ; Wu et al. 2018 ; Logeswaran & Lee 2018 ; Bachman et al. 2019 ; Tian et al. 2020a ; Chen et al. 2020a ; Henaff 2020 ) . The most widely used among the contrastive losses is the \mathrm{InfoNCE} ( Oord et al. 2018 ) and it has been shown that minimizing this is equivalent to maximizing a lower bound of the mutual information (MI) between a pair of representations ( Bachman et al. 2019 ; Tian et al. 2020a ) . Multimodal Contrastive Representation Learning One of the tasks in which contrastive losses have gained popularity is Multimodal Representation Learning, which consists in designing systems that map inputs from different modalities (e.g. image and text) into a joint representation space. Some foundation works used rank-based losses ( Yager 1988 ; Usunier et al. 2009 ; Schroff et al. 2015 ) to learn multimodal representation spaces ( Weston et al. 2010 ; Frome et al. 2013 ; Karpathy & Fei-Fei 2015 ) while more modern approaches have used the \mathrm{InfoNCE} loss ( Tian et al. 2020a ; Radford et al. 2021 ; Jia et al. 2021 ; Xu et al. 2021 ; Girdhar et al. 2023 ) for this purpose. However, it has been observed that, when trained in a contrastive way, representations from different modalities tend to be located in different regions of the space, a phenomenon called modality gap ( Liang et al. 2022 ; Udandarao 2022 ; Ramasinghe et al. 2024 ; Fahim et al. 2024 ; Schrodi et al. 2024 ) . This phenomenon can be an issue in some applications such as Image Captioning or Visual Question Answering, so sophisticated training methods have been proposed to palliate it ( Chen et al. 2020b ;