The “Law” of the Unconscious Contrastive Learner: Probabilistic Alignment of Unpaired Modalities

Yongwei Che, Benjamin Eysenbach · 2025

Read The “Law” of the Unconscious Contrastive Learner: Probabilistic Alignment of Unpaired Modalities by Yongwei Che, Benjamin Eysenbach (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-11326.txt.

Introduction

Yongwei Che and Benjamin Eysenbach wrote a 2025 theory paper on how AI systems link different kinds of data. The internet offers paired data, such as photos with captions or sound with video. But we often need to connect kinds that never appear together, like sound directly to text. A common hope is that chaining pair-trained spaces lines the unpaired kinds up for free.

The authors prove that hope is justified, under stated assumptions. Starting from the principled approach of averaging over the middle kind of data, they show direct comparison of the two unpaired kinds recovers the same answer. Their work builds on known results about the shape and meaning of contrastive representations. The theory suggests two fresh uses: working with ready-made contrastive models, and handling vague language in reinforcement learning. Computer experiments check how much the assumptions matter and demo the new uses.

Catalog record

CREATORYongwei Che, Benjamin Eysenbach
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE59 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2501-11326

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

The “Law” of the Unconscious Contrastive Learner: Probabilistic Alignment of Unpaired Modalities Yongwei Che, Benjamin Eysenbach Abstract While internet-scale data often comes in pairs (e.g., audio/image, image/text), we often want to perform inferences over modalities unseen together in the training data (e.g., audio/text). Empirically, this can often be addressed by learning multiple contrastive embedding spaces between existing modality pairs, implicitly hoping that unseen modality pairs will end up being aligned. This theoretical paper proves that this hope is well founded, under certain assumptions. Starting with the proper Bayesian approach of integrating out intermediate modalities, we show that directly comparing the representations of data from unpaired modalities can recover the same likelihood ratio. Our analysis builds on prior work on the geometry and probabilistic interpretation of contrastive representations, showing how these representations can answer many of the same inferences as probabilistic graphical models. Our analysis suggests two new ways of using contrastive representations: in settings with pre-trained contrastive models, and for handling language ambiguity in reinforcement learning. Our numerical experiments study the importance of our assumptions and demonstrate these new applications. Code: https://github.com/YongweiChe/UnconsciousContrastiveLearner 1 Introduction Much of the appeal of contrastive learning is that it gives a “plug-n-play” approach for swapping one modality for another. Because representations from different modalities are trained to be aligned when representing the same object, the hope is that (say) a language representation and image representation of the same scene can be used as substitutes. This property is practically appealing for at least two reasons. First , it allows us to make use of pre-trained models. If you have a model that wants to make use of (say) language input and you have access to a pre-trained image-language contrastive model, you might simply train your model on the pre-trained image representations and hope that it will continue to work when you swap in the language representations. Second , it allows you to merge datasets that have different modalities, in the same way that a data scientist might merge two SQL tables. This is especially useful in settings where we have datasets with pairs of modalities, where collecting datasets with triplets of modalities (e.g., image, audio, and text) is challenging. While it seems intuitive that this “plug-n-play” approach should work, to the best of our knowledge prior work has not provided a rigorous argument for why it works or when it should fail. The main contribution of our paper is to answer these questions, and to provide an alternative algorithm for when this “naive” approach fails. Our analysis is based on two ideas. The first idea , building on prior work ( Poole et al., 2019 ; Ma & Collins, 2018 ) , is to think about contrastive learning probabilistically. Rather than thinking about representations as encoding “this image maps to this text,” we think about them as encoding a likelihood – that this text is more likely to describe this image, but could plausibly also describe this other image. The second idea , building on top of other prior work ( Wang & Isola, 2020 ) , is to consider the marginal distribution over the representations. We follow prior works in making the assumption that the marginal distribution is either (1) Gaussian and (2) Uniform over the hypersphere. The main contribution of this paper is a probabilistic analysis of multimodal contrastive learning. The first part of our analysis is to show that, when certain assumptions are satisfied, the “plug-n-play” approach is principled, correctly inferring the posterior mean. We also introduce an approach that works agnostic to the marginal distribution of the representations. The second part looks at applications of our results. We start by verifying our theoretical results with experiments over synthetic datasets. One such experiment includes a setting where users want to bridge modalities A and C using pre-trained or black-box models that connect A and C through an intermediate modality B . Then, we extend our results to large pre-trained contrastive models over image, language, and audio. Finally, we explore an application of our method in language-conditioned reinforcement learning, testing our method’s ability to handle natural ambiguity in language. 2 Related Work Figure 1: Aligning vision and audio by integrating over the intermediate ‘language’ modality. While prior work has shown that aligning modalities A\leftrightarrow B and B\leftrightarrow C results in representations that can compare modalities A and C, it remains unclear when and why this is guaranteed to work. This paper provides the assumptions under which this approach is principled, and our analysis unlocks new ways of comparing unpaired modalities and new applications for contrastive learning. Our work builds on a wide range of prior work, ranging from representation learning, multimodal machine learning, and probabilistic graphical models. Contrastive learning methods aim to learn compact representations of examples from one or more modalities ( Chen et al., 2020 ; Hadsell et al., 2006 ; Mikolov et al., 2013 ) , and have found applications ranging from ecology to NLP to to reinforcement learning. These methods typically are trained on pairs of examples, with the aim of acquiring representations that are similar for paired examples and dissimilar for unpaired examples. 1 1 1 Our work will not consider the successful line of prior work that studies negative-free contrastive learning methods ( Grill et al., 2020 ; Chen et al., 2021 ) . While these pairs are often generated via data augmentation ( Oord et al., 2018 ) , our work builds on prior work that looks at paired examples coming from different modalities (e.g., an image accompanied by its caption) ( Elizalde et al., 2023 ; Girdhar et al., 2023 ; Zhu et al., 2023 ) . Prior work has shown that these methods can work effectively with more than two modalities Xue et al. (2023) ; Guzhov et al. (2021) , even in settings where there are no paired examples from certain modalities ( Girdhar et al., 2023 ; Zhu et al., 2023 ; Shah et al., 2023 ; Yuan et al., 2023 ; Ferraro et al., 2023 ) . However, from a theoretical perspective it remains unclear why these sorts of “plug-n-play” methods can combine modalities for which there are no training examples, a problem that ordinarily requires marginalization (which is very challenging on the high-dimensional inputs to which these methods have been applied). Are these methods implicitly performing this marginalization? Are they correctly accounting for uncertainty? Our paper will aim to answer some of these questions by building on top of ideas from two lines of prior work. First, we will exploit the geometry of contrastive representations ( Wang & Isola, 2020 ; Eysenbach et al., 2024 ) , borrowing (theoretically-motivated) assumptions about the asymptotic behavior of contrastive representation distributions. Second, we build on prior work ( Poole et al., 2019 ) that provides a probabilistic interpretation of contrastive learning: when trained with a cross entropy loss, the learned representations encode a probability ratio. When combined, these two ideas allow us to prove that reasoning across unpaired modalities takes the form of a certain integral over representations, which admits a closed form solution. Our theoretical analysis will explain why the “plug-n-play” approach often works, elucidate its assumptions, and provide new ways of combining

Read the complete text