Multimodal Datasets with Controllable Mutual Information

Raheem Karim Hashmani, Garrett W. Merz, Helen Qu, Mariel Pettee, Kyle Cranmer · 2025

Read Multimodal Datasets with Controllable Mutual Information by Raheem Karim Hashmani, Garrett W. Merz, Helen Qu, Mariel Pettee, Kyle Cranmer (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-21686.txt.

Introduction

Raheem Karim Hashmani and four co-authors manufacture multimodal data sets whose shared information content is known exactly. The field has lacked such a measuring stick. Mutual information between data channels drives how estimators are judged. It also drives how self-supervised learners are tuned. Real data never reveal the true value to check against.

Their generator pairs a flow-based model with a structured causal scheme for the hidden variables. Those variables correlate the channels. The result is realistic data at any chosen level of true mutual information. A suite of estimators is benchmarked across those levels. A sanity law is verified too: prediction improves as true shared information rises. The authors sketch uses from multi-detector astrophysics to self-supervised learning in the strongly multimodal regime.

Catalog record

CREATORRaheem Karim Hashmani, Garrett W. Merz, Helen Qu, Mariel Pettee, Kyle Cranmer
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE35 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2510-21686

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Multimodal Datasets with Controllable Mutual Information Raheem Karim Hashmani, Garrett W. Merz, Helen Qu, Mariel Pettee, Kyle Cranmer Abstract We introduce a framework for generating highly multimodal datasets with explicitly calculable mutual information between modalities. This enables the construction of benchmark datasets that provide a novel testbed for systematic studies of mutual information estimators and multimodal self-supervised learning techniques. Our framework constructs realistic datasets with known mutual information using a flow-based generative model and a structured causal framework for generating correlated latent variables. 1 1 1 Our code is publicly available at https://github.com/RKHashmani/MmMi-Datasets . 1 Introduction Self-supervised learning (SSL) has become a core component of many state-of-the-art large-scale machine learning models [ 1 ] . Such models are also increasingly multimodal , i.e. designed to learn from varied input sources such as text, images, and audio [ 2 , 3 ] . A prevailing intuition is that multimodal SSL is effective because different modalities provide complementary “views” of the same underlying concepts, enabling the learning process to exploit their shared information. The precise relationship between the mutual information (MI) between modalities and SSL performance, however, is not fully understood. Contrastive SSL methods built using the InfoNCE loss function [ 4 , 5 , 6 , 7 ] have a clear information-theoretic interpretation: for example, the learned similarity scores estimate the pointwise mutual information (PMI) between paired samples. By contrast, no analogous theoretical connection has been established for either highly multimodal settings (i.e. N>2 modalities) or for non-contrastive SSL methods such as multimodal masked modeling [ 8 , 9 , 10 , 11 ] , despite the fact that both of these directions are quickly gaining prominence in the field [ 12 , 13 ] . A theoretically-grounded understanding of the fundamental relationship between inter-modality mutual information and SSL representations (and their corresponding performance on downstream tasks) will be increasingly critical as models continue to scale to larger numbers of input modalities. In particular, principled frameworks will be needed to evaluate how the distribution of shared information across modalities influences the quality of the learned embeddings. Complicating matters further, MI is notoriously difficult to estimate from samples, particularly in high-dimensional, real-world datasets [ 14 , 15 ] . A wide range of MI estimators have been proposed using techniques such as kernel estimation, k -nearest neighbor, and neural estimators [ 16 , 17 , 18 , 19 , 20 , 21 , 22 , 23 ] . However, these estimators are typically only validated on synthetic datasets of simple distributions for which the MI is analytically tractable [ 24 , 25 , 15 , 26 , 27 ] . Datasets with controllable MI that emulate the challenges of real-world data are needed to better understand the advantages, disadvantages, and tradeoffs of different SSL learning objectives. Such datasets can enable systematic, reproducible studies of how multimodal SSL representations depend on information overlap and shared features, offering both theoretical insights and practical guidance for model design. Finally, they provide a reliable testbed for evaluating the performance of various mutual information estimation strategies designed for use on real-world datasets. Figure 1: Schematic of our dataset generation framework. a) An example DAG showing the linear mixing of proto-latents \mathbf{u} via coefficients \mathbf{\eta},\mathbf{\rho} into interpretable correlated Gaussian latent variables \mathbf{z} . b) Overview of sampling from a multidimensional Gaussian to draw latent inputs z_{1} , z_{2} , and z_{\theta} that are fed into invertible maps f_{1} and f_{2} to a realistic feature space. In this work, we introduce a framework to generate realistic multimodal data with controllable mutual information. Figure 1 shows an overview of our data generation framework: (a) First, we use directed acyclic graphs (DAGs) to generate easily interpretable correlated Gaussian latent variables \mathbf{z} with known mutual information. (b) We then feed the outputs \mathbf{z} of these DAGs into invertible bijective transformations to construct multimodal datasets where the amount and distribution of shared information can be explicitly controlled across multiple modalities. 2 Background Mutual Information (MI). Mutual information I(X;Y) is a fundamental quantity from information theory that measures the statistical dependence between two random variables X and Y . It is formally defined as the Kullback-Leibler (KL) divergence of the joint distribution p(X,Y) and the product of the marginal distributions p(X)p(Y) : I(X;Y)\equiv D_{KL}\left(p(X,Y)\text{ }||\text{ }p(X)p(Y)\right) Alternatively, it can also be expressed in terms of the Pointwise Mutual Information (PMI): I(X;Y)\equiv\mathbb{E}_{x,y\sim p(X,Y)}\left[\text{PMI}(x;y)\right] The MI quantifies the extent to which one variable reduces uncertainty in the other. For instance, when X and Y are fully independent, their joint distribution p(X,Y) reduces to the product of the marginal distributions p(X)p(Y) , therefore the MI is zero. Pointwise Mutual Information (PMI). When evaluated on specific values x\sim p(X) and y\sim p(Y) , the pointwise MI (PMI) captures the probability of these two values occurring together compared with that same probability if they were fully independent. The PMI is formally expressed as: \text{PMI}(x;y)\equiv\text{log}\left(\frac{p(x,y)}{p(x)p(y)}\right). Multimodal Self-Supervised Learning (SSL). Often compared to the primary human senses such as sight, hearing, or touch, modalities in machine learning refer to distinct forms of sensing the world and the corresponding representations of the observed data. Modalities can have radically different formats (e.g. RGB images and timeseries data), or they can exhibit similar formats but describe distinct information sources (e.g. RGB images and segmentation maps). In this paper, our operational definition for a modality is a random variable X_{m} and a corresponding sample space \mathcal{X}_{m} . In particular, if X_{1} and X_{2} correspond to two different distributions, then we consider them to be two separate modalities regardless of their data format. Multimodal SSL often involves learning a joint representation of many data modalities, which we view as distinct from multi- view SSL, which generally consists of learning a joint representation of multiple views derived from the same data modality, e.g. different crops of a single image. Multimodal SSL uses the relationships between modalities to learn joint representations without explicit labels. Flow-based generative modeling. Flow-based generative models are designed to facilitate the direct transformation between probability distributions using invertible mappings applied to a simple base distribution such as a Gaussian. Each transformation is designed to be bijective with a tractable Jacobian determinant, enabling exact computation of both likelihoods and samples. Because they provide both efficient sampling and exact density evaluation, flow-based models are increasingly used not only in generative modeling but also in scientific applications requiring tractable likelihoods and explicit control over distributions. Flow-matching [ 28 , 29 ] is a recent approach within the family of flow-based generative models that enables efficient training of Continuous Normalizing Flows (CNFs) [ 30 ] by directly regressing the velocity field that transports a base distribution to the data distribution instead of optimizing the exact maximum-likelihood objective. 3 Creating Datasets with Controlled Mutual Information Our goal is to enable rigorous, scalable experiments using multimodal datasets where the MI between modalities is precisely specified and easy to interpret. To accomplish this, we design an expressive three-step framework \mathbf{u}\to\mathbf{z}\to\mathbf{x} . This begins with uncorrelated, normally distributed ‘proto-latent’ variables \mathbf{u} , which are related by linear structural equations to form an easy-to-interpret causal model for latent variables \mathbf{z} , for which mutual information is easy to compute. Finally, we use blocks

Read the complete text