Contrasting with Symile: Simple Model-Agnostic Representation Learning for Unlimited Modalities

Adriel Saporta, Aahlad Puli, Mark Goldstein, Rajesh Ranganath · 2024

Read Contrasting with Symile: Simple Model-Agnostic Representation Learning for Unlimited Modalities by Adriel Saporta, Aahlad Puli, Mark Goldstein, Rajesh Ranganath (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2411-01053.txt.

Description

Abstract Contrastive learning methods, such as CLIP , leverage naturally paired data—for example, images and their corresponding text captions—to learn general representations that transfer efficiently to downstream tasks. While such approaches are generally applied to two modalities, domains such as robotics, healthcare, and video need to support many types of data at once.

Catalog record

CREATORAdriel Saporta, Aahlad Puli, Mark Goldstein, Rajesh Ranganath
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE110 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2411-01053

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Contrasting with Symile: Simple Model-Agnostic Representation Learning for Unlimited Modalities Adriel Saporta, Aahlad Puli, Mark Goldstein, Rajesh Ranganath Abstract Contrastive learning methods, such as CLIP , leverage naturally paired data—for example, images and their corresponding text captions—to learn general representations that transfer efficiently to downstream tasks. While such approaches are generally applied to two modalities, domains such as robotics, healthcare, and video need to support many types of data at once. We show that the pairwise application of CLIP fails to capture joint information between modalities, thereby limiting the quality of the learned representations. To address this issue, we present Symile , a simple contrastive learning approach that captures higher-order information between any number of modalities. Symile provides a flexible, architecture-agnostic objective for learning modality-specific representations. To develop Symile ’s objective, we derive a lower bound on total correlation, and show that Symile representations for any set of modalities form a sufficient statistic for predicting the remaining modalities. Symile outperforms pairwise CLIP , even with modalities missing in the data, on cross-modal classification and retrieval across several experiments including on an original multilingual dataset of 33M image, text and audio samples and a clinical dataset of chest X-rays, electrocardiograms, and laboratory measurements. All datasets and code used in this work are publicly available at https://github.com/rajesh-lab/symile . 1 Introduction Contrastive learning leverages naturally paired data to learn general representations that transfer efficiently to downstream tasks [ 53 , 35 , 3 ] . A common contrastive approach is to maximize the mutual information between the paired modalities, ensuring that the learned representations retain sensitivity to all correlations between them. While SimCLR [ 12 ] popularized the use of the mutual information estimator InfoNCE [ 38 ] for data augmentations, CLIP [ 40 ] applied the approach to distinct modalities—for example, images and their corresponding text captions—where representations are learned using any encoder for each modality. While contrastive approaches are generally applied to two modalities, there is a rapidly expanding range of domains that require the integration of many types of data at once. For example, in robotics, agents combine information from visual, proprioceptive, and tactile sensors [ 18 , 28 ] ; healthcare providers analyze various types of patient data including imaging, biosignals, and genomics [ 10 , 29 ] ; and video encompasses RGB frames, audio waveforms, and text transcripts [ 55 ] . One strategy for handling multimodal data has been to design specialized architectures capable of processing all data types at once, which limits their general applicability and increases operational complexity [ 2 , 47 ] . Another common approach is to apply two-modality contrastive objectives, such as CLIP , to pairs of available modalities [ 15 , 44 ] . In this paper, we show that, despite its popularity, the pairwise application of CLIP fails to capture higher-order conditional information between modalities, thereby limiting the quality of the representations it learns. For instance, given three modalities \mathbf{a} , \mathbf{b} , and \mathbf{c} , pairwise CLIP captures dependencies between \mathbf{a} and \mathbf{b} , \mathbf{b} and \mathbf{c} , and \mathbf{a} and \mathbf{c} , yet cannot capture any conditional dependencies, such as between \mathbf{a} and \mathbf{b} given \mathbf{c} . We show in Section 2.2 that even in a simple one-dimensional controlled setting where the target \mathbf{b} is perfectly predictable from \mathbf{a} and \mathbf{c} , CLIP performs no better than random chance. Effective contrastive learning for more than two modalities requires a model-agnostic approach capable of learning modality-specific representations—like CLIP—yet also captures higher-order information between any number of modalities—unlike CLIP. Methodological contributions. This paper presents Symile , a simple contrastive learning approach that captures higher-order information between any number of modalities. Symile provides a flexible, architecture-agnostic objective for learning modality-specific representations. To develop Symile ’s objective, we derive a total correlation estimator, employing a generalization of inner products to more than two vectors that allows for the simultaneous contrasting of all modalities and enables zero-shot applications such as classification and retrieval. We then show that the representations produced by Symile for any set of modalities form a sufficient statistic for predicting the remaining modalities not considered in the set. Because it targets total correlation, Symile captures strictly more information than CLIP , guaranteeing performance that matches or surpasses CLIP , except in cases where it known that only pairwise statistics are relevant. Given that such prior knowledge is rarely available, Symile should be favored over CLIP . Empirical contributions. We demonstrate that Symile outperforms pairwise CLIP on cross-modal classification and retrieval across several experiments including on a multilingual dataset of images, text and audio of over 33M examples and a clinical dataset of chest X-rays, electrocardiograms, and laboratory measurements. We show that Symile retains its advantage over pairwise CLIP even with modalities missing in the data. We publicly release both the multilingual and the clinical datasets, which are specifically designed to test a model’s ability to capture higher-order information between three distinct high-dimensional data types. 2 Background and motivation In this section, we first provide background on the original CLIP objective for two modalities, and describe how it has been extended to additional modalities. We then present a simple problem set up for three modalities that illustrates where pairwise contrastive objectives fall short. 2.1 Pairwise contrastive learning Given a batch of (\mathbf{x},\mathbf{y}) pairs, separately encoded by f_{\mathbf{x}}^{{\boldsymbol{\mathbf{\theta}}}} and f_{\mathbf{y}}^{{\boldsymbol{\mathbf{\theta}}}} , respectively, contrastive objectives such as CLIP maximize the similarity between representations of correctly paired ( positive ) samples and minimize the similarity between representations of incorrectly paired ( negative ) samples. As is now standard in contrastive learning, in order to construct a batch of data, each modality is treated as the anchor in turn and used to construct a set of positive and negative samples. Letting \tau\in\mathbb{R}^{+} be a temperature parameter, the CLIP objective when \mathbf{x} is the anchor modality is the categorical cross-entropy of correctly classifying the positive pair out of N possible pairs: \displaystyle\ell^{(\mathbf{x}\rightarrow\mathbf{y})}({\boldsymbol{\mathbf{\theta}}},\tau) \displaystyle=-\frac{1}{N}\sum_{i=1}^{N}\log\frac{\exp\big[\big(f_{\mathbf{x}}^{{\boldsymbol{\mathbf{\theta}}}}(\mathbf{x}_{i})^{\top}f_{\mathbf{y}}^{{\boldsymbol{\mathbf{\theta}}}}(\mathbf{y}_{i})\big)/\tau\big]}{\sum_{j=1}^{N}\exp\big[\big(f_{\mathbf{x}}^{{\boldsymbol{\mathbf{\theta}}}}(\mathbf{x}_{i})^{\top}f_{\mathbf{y}}^{{\boldsymbol{\mathbf{\theta}}}}(\mathbf{y}_{j})\big)/\tau\big]}. (1) The final CLIP objective is an average of the losses in each direction: \mathcal{L}_{\textrm{\lx@glossaries@gls@link{main}{clip}{{{}}CLIP} }}^{(\mathbf{x},\mathbf{y})}({\boldsymbol{\mathbf{\theta}}},\tau)=\frac{1}{2}\big[\ell^{(\mathbf{x}\rightarrow\mathbf{y})}({\boldsymbol{\mathbf{\theta}}},\tau)+\ell^{(\mathbf{y}\rightarrow\mathbf{x})}({\boldsymbol{\mathbf{\theta}}},\tau)\big] . The dot product in Equation 1 serves as a scoring function that is trained to assign high values to positive pairs, which are sampled from the joint distribution p_{\mathbf{x},\mathbf{y}} , and low values to negative pairs, which are sampled from the product of marginals p_{\mathbf{x}}p_{\mathbf{y}} . Contrastive methods are typically designed to maximize the mutual information between \mathbf{x} and \mathbf{y} , which is defined as the Kullback–Leibler divergence from the joint distribution to the product of the marginal distributions: \mathbf{I}(\mathbf{x};\mathbf{y})=D_{\mathrm{KL}}\big(p(\mathbf{x},\mathbf{y})\;\|\;p(\mathbf{x})p(\mathbf{y})\big) . It has been shown that Equation 1 maximizes a lower bound on the mutual information between \mathbf{x} and \mathbf{y} [ 38 , 39 ] . This information maximization ensures that the learned representations preserve all correlations between the modalities, which is essential for downstream tasks. Incorporating additional modalities. In order to learn a joint embedding space for more than two modalities, existing work has applied the CLIP objective in a pairwise fashion [ 2 , 43 , 1 , 9 , 14 , 47 , 52 , 34 , 33 , 11 , 21 , 44 ] . For example, Guzhov et al. [19] extend CLIP to incorporate audio alongside image and text, and ImageBind

Read the complete text