Accurate Estimation of Mutual Information in High Dimensional Data
Eslam Abdelaleem [email protected], K. Michael Martini [email protected], Ilya Nemenman [email protected] · 2025
Read Accurate Estimation of Mutual Information in High Dimensional Data by Eslam Abdelaleem [email protected], K. Michael Martini [email protected], Ilya Nemenman [email protected] (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-00330.txt.
Introduction
Mutual information measures how much knowing one variable tells you about another, and science leans on it everywhere. Estimating it from limited data in many dimensions is notoriously treacherous, and Abdelaleem and co-authors aim to make neural estimators trustworthy.
Their key insight is that when the link between variables runs through a hidden low-dimensional layer, the data needed depends on that small dimension, not the huge ambient one. On that basis they build a protocol with consistency checks, bias correction, and confidence intervals, plus a new family of critics that stays steady where older estimators break. Tests ran from synthetic data to image sets with thousands of pixels. Their method matched or beat rivals and, uniquely, flagged its own unreliable answers.
Catalog record
| CREATOR | Eslam Abdelaleem [email protected], K. Michael Martini [email protected], Ilya Nemenman [email protected] |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 75 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2506-00330
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Accurate Estimation of Mutual Information in High Dimensional Data Eslam Abdelaleem [email protected], K. Michael Martini [email protected], Ilya Nemenman [email protected] Abstract Mutual information (MI) is a measure of statistical dependencies between two variables, widely used in data analysis. Thus, accurate methods for estimating MI from empirical data are crucial. Such estimation is a hard problem, and there are provably no estimators that are universally good for finite datasets. Common estimators struggle with high-dimensional data, which is a staple of modern experiments. Recently, promising machine learning-based MI estimation methods have emerged. Yet it remains unclear if and when they produce accurate results, depending on dataset sizes, statistical structure of the data, and hyperparameters of the estimators, such as the embedding dimensionality or the duration of training. There are also no accepted tests to signal when the estimators are inaccurate. Here, we systematically explore these gaps. We propose and validate a protocol for MI estimation that includes explicit checks ensuring reliability and statistical consistency. Contrary to accepted wisdom, we demonstrate that reliable MI estimation is achievable even with severely undersampled, high-dimensional datasets, provided these data admit accurate low-dimensional representations. These findings broaden the potential use of machine learning-based MI estimation methods in real-world data analysis and provide new insights into when and why modern high-dimensional, self-supervised algorithms perform effectively. a Department of Physics b Department of Biology c Initiative in Theory and Modeling of Living Systems Emory University, Atlanta, GA 30322, USA 1 Introduction Mutual information (MI) is a fundamental measure of statistical dependence between two variables [ 1 ] . It captures both linear and nonlinear associations, is invariant under reparameterizations, and is zero if and only if the variables are statistically independent. These properties make MI a key tool across disciplines, from neuroscience to computer vision [ 2 ] . In systems neuroscience, MI estimation plays an important role in understanding neural coding and decoding processes, analyzing spike trains in single neurons and neural populations, and studying patterns of information transfer across brain areas and behaviors [ 3 , 4 , 5 , 6 , 7 , 8 , 9 ] . Similarly, in brain imaging studies, MI quantifies functional connectivity between brain regions, illuminating cognitive processes and effects of neurological disorders [ 10 , 11 , 12 ] . MI has also proven useful in domains like protein sequence alignment and contact prediction [ 13 , 14 , 15 ] , and in the inference of gene regulatory networks [ 16 ] . In computer vision, state-of-the-art models, such as CLIP [ 17 ] , utilize loss functions related to MI to align visual and textual representations. Similarly, self-supervised learning frameworks such as Barlow Twins [ 18 ] can be interpreted as mutual information-based objectives designed to enforce cross-modal correspondences [ 19 ] . For continuous variables X and Y , MI is I(X;Y)=\int dx\ dy\,p(x,y)\log_{2}\frac{p(x,y)}{p(x)p(y)} (measured in bits 1 1 1 Most estimators are naturally expressed in nats , where \log is the natural logarithm. Thus in all derivations in this paper \log=\ln . However, when reporting the MI values, we convert to bits for clarity, \log=\log_{2} . ), where x and y are specific values of the variables, p(\cdot) are the corresponding probability density functions, and the integration is over the domain of the variables. (For discrete X and Y , sums over distributions are used instead.) Since MI is a nonlinear function of p(x,y) , substituting an unbiased estimate of p into the definition of MI results in a biased estimate of MI. Typically, the bias of estimators is a more serious problem than their variance, particularly for continuous variables, because MI is invariant under reparameterization, while it is impossible to construct an estimator that is covariant under all reparameterizations [ 20 ] . Traditional methods, such as histogram-based, k-nearest neighbors (kNN), box-counting, or kernel-based [ 21 , 22 , 23 , 24 , 25 , 26 , 27 , 28 , 29 ] , struggle to reduce the bias for high-dimensional data since they require the number of samples that grows exponentially with the dimensionality [ 30 , 25 , 22 , 31 ] . Recent advances in machine learning have introduced neural network (NN)-based estimators for MI, which aim to circumvent the limitations of traditional methods. These estimators frame MI estimation as optimization over a family of functions [ 32 , 33 , 34 , 35 , 36 , 37 ] . In principle, they can work even for very high-dimensional data, producing estimates in scenarios that are out of reach for traditional methods. For instance, they supposedly can compute MI between images, with the dimensionality of thousands. However, the practical accuracy of these methods remains unclear. First, most of them have been tested primarily on synthetic data with simple dependence structures and essentially infinite sample sizes, raising questions about their real-world applicability. Second, since universally good MI estimation is impossible, internal consistency checks are essential to signal whether the output can be trusted. Such checks are not widely adopted. Third, NN estimators depend on hyperparameters, such as criteria for stopping training, and how to choose these parameters remains unclear. Here, we systematically address these gaps. We argue that successful neural MI estimation requires: (i) even high-dimensional data having a low-dimensional latent structure ; (ii) the critic sufficiently expressive (e.g., embedding dimension and nonlinearity) to capture it; and (iii) sufficient data to resolve statistical dependencies in the latent (not full data) space. Our specific contributions include: (1) development of max-test heuristic for early stopping to mitigate overfitting in MI estimation; (2) introduction of probabilistic critics , which improve estimation for large MI values; (3) unification of correlation-based and neural MI estimators under the Donsker-Varadhan [ 38 ] framework; (4) benchmarking estimators on synthetic and real-world data, demonstrating when and why MI estimation succeeds; and (5) development of a practical guide for MI network estimation. 2 Background Traditional MI Estimation Methods. Estimating mutual information (MI) from finite data is notoriously challenging, especially for high-dimensional continuous variables [ 39 ] . To illustrate this, consider a simple argument: suppose each component of X and Y lies within a bounded range of size A , and the joint density p(x,y) is smooth, with its smallest feature on the scale of a . Then accurate MI estimation requires N\gg(A/a)^{K_{\text{tot}}} samples, where K_{\text{tot}}=K_{X}+K_{Y} is the total dimensionality of X and Y . Even when A/a is only slightly greater than 1, the required sample size is exponential in K_{\text{tot}} , illustrating the classic curse of dimensionality. The situation is even worse when the coordinate system in which p is smooth is unknown, or when the variables are unbounded. Thus, while many methods have been developed to estimate MI from continuous data, they often break down beyond K_{\rm tot}\sim 10 dimensions [ 40 ] (cf. SI Fig. 10 ). In contrast, most modern datasets are high-dimensional, e.g., images with thousands of pixels, or neural recordings from thousands of units. Neural Network-Based Estimators. The inability of traditional methods to overcome the curse of dimensionality has motivated neural network (NN)-based approaches. Deep NNs can capture complex nonlinear dependencies in high-dimensional data and, in some cases, generalize well even from modest sample sizes [