InfoAtlas: A Foundation Model for Zero-Shot Statistical Dependence Estimate
Zhengyang Hu, Yanzhi Chen, Hanxiang Ren, Qunsong Zeng, Youyi Zheng, Adrian Weller, Kaibin Huang, Yanchao Yang · 2026
Read InfoAtlas: A Foundation Model for Zero-Shot Statistical Dependence Estimate by Zhengyang Hu, Yanzhi Chen, Hanxiang Ren, Qunsong Zeng, Youyi Zheng, Adrian Weller, Kaibin Huang, Yanchao Yang (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-00241.txt.
Introduction
Hu and co-authors build a base model that gauges links between data sets in one pass. Gauging how much two rich data sets share is a core data task. Net-based gauges work well but need fresh slow training per set. That cost blocks live use.
InfoAtlas is trained once on vast fake data with rich link forms. It learns to spot a broad range of link shapes. At test time, it reads a new set and calls the link strength at once. Tests show it ties top gauges on scores while running far faster. One model takes in varied widths and set sizes. It also holds up on hard, real-world sets. The shift turns link gauging into a quick read task.
Catalog record
| CREATOR | Zhengyang Hu, Yanzhi Chen, Hanxiang Ren, Qunsong Zeng, Youyi Zheng, Adrian Weller, Kaibin Huang, Yanchao Yang |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 66 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2606.00241
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
InfoAtlas: A Foundation Model for Zero-Shot Statistical Dependence Estimate Zhengyang Hu, Yanzhi Chen, Hanxiang Ren, Qunsong Zeng, Youyi Zheng, Adrian Weller, Kaibin Huang, Yanchao Yang Abstract Measuring statistical dependency between high-dimensional random variables is a fundamental task in data science and machine learning. Neural mutual information (MI) estimators offer a promising avenue, but they typically require costly iterative optimization for each new dataset, making them impractical for real-time applications. We present InfoAtlas , a foundation model-like architecture that eliminates this bottleneck by directly inferring MI in a single forward pass. Pretrained on large-scale synthetic data with rich dependence patterns, InfoAtlas learns to identify diverse dependence structures and predict MI directly from the dataset. Comprehensive experiments demonstrate that InfoAtlas matches state-of-the-art neural estimators in accuracy while achieving 100× speedup, can flexibly handle varying dimensions and sample sizes through a single unified model, and generalizes effectively to complex, real-world scenarios. By reformulating MI estimation as an inference task, InfoAtlas establishes a foundation for real-time dependency analysis. Project page: InfoAtlas-project † † affiliationnotice: Equal contribution 1 Introduction Understanding statistical dependencies between variables is fundamental to data science and machine learning. Quantifying how variables influence each other uncovers hidden structures and causal mechanisms that drive complex systems. Applications span a wide range of domains: in healthcare, identifying dependencies between lifestyle factors and disease risks enables personalized prevention strategies ( 18 ) ; in autonomous driving, modeling dependencies between sensor signals and road conditions improves safety and decision making ( 33 ) ; in biology, assessing the dependence between protein sequences reveals insights for understanding their functional relationship ( 23 ) ; and in robotics, maximizing statistical dependence between observational states is shown useful for policy discovery ( 58 ) . Mutual information (MI) ( 44 ) has long served as a principled measure for dependency, uniquely capturing complex nonlinear relationships for multivariate variables in interpretable units of bits. Its generality has made it a core tool in data analysis, generative modeling and representation learning ( 6 ; 36 ; 9 ) . However, computing MI from empirical samples is notoriously difficult: closed-form solutions exist only for certain distributions, and neural estimators ( 2 ; 12 ; 20 ; 51 ; 11 ; 50 ) require costly gradient-based optimization for every dataset, making them impractical for real-time or large-scale applications. Figure 1 : Conceptual comparison: prior methods vs our method . Existing neural MI estimators (left) requires iterative gradient-based optimization to train a neural network for each new dataset. In contrast, we uses a pre-trained architecture to directly generate MI estimates in a single forward pass (right), eliminating per-dataset training and achieving speedup while maintaining comparable accuracy. In this work, we introduce InfoAtlas , a foundation model-style architecture for fast and accurate estimation of statistical dependence between multivariate random variables. InfoAtlas predicts the strength of dependence in a single forward pass—an ability reminiscent of foundation models ( 26 ; 13 ) . This ability is acquired through large-scale pretraining on massive synthetic datasets that capture a wide range of dependence structures and data patterns, enabling InfoAtlas to directly infer statistical relationships without per-dataset optimization. Crucially, InfoAtlas preserves full differentiability, facilitating seamless integration into larger AI pipelines. Extensive experiments demonstrate that InfoAtlas generalizes effectively from synthetic benchmarks to complex real-world data, accurately capturing a broad spectrum of dependencies, being a versatile tool for rapid understanding of variable relationships. Our main contributions are: • We introduce InfoAtlas , the first pretrained architecture for zero-shot estimation of mutual information between multivariate variables. InfoAtlas achieves accuracy on par with state-of-the-art neural methods without any gradient-based optimization, and flexibly handles variables of varying dimensionalities and sample sizes with a single model. • We propose an attentive dual-path hypernetwork-based architecture, which is pretrained on large-scale synthetic datasets covering diverse dependency structures. This design enables InfoAtlas to predict dependency strength in a single inference step, and generalizes effectively to unseen real-world scenarios without task-specific finetuning. • We comprehensively evaluate InfoAtlas on both synthetic benchmarks and real-world tasks, including independence testing, CLIP embedding analysis ( 41 ) , motion trajectory modeling and robotics manipulation. Results demonstrate its robust performance and accurate perception of a wide spectrum of dependencies. 2 Problem Statement In this work, we consider the problem of quantifying statistical dependence between two multivariate random variables \mathbf{x}\in\mathbb{R}^{d_{x}} and \mathbf{y}\in\mathbb{R}^{d_{y}} , with d_{x}\geq 1 and d_{y}\geq 1 . Measuring dependence via mutual information. Mutual information (MI) offers a principle measure for quantifying statistical dependence between multivariate variables. Unlike linear correlation coefficients that capture only linear relationships, MI effectively captures both linear and nonlinear correlations. Formally, MI is defined as the Kullback-Leibler (KL) divergence between the joint distribution p_{\mathbf{x},\mathbf{y}} and the product of marginals p_{\mathbf{x}}\otimes p_{\mathbf{y}} ( 31 ) : \displaystyle\mathbb{I}(\mathbf{x},\mathbf{y}) \displaystyle=\text{KL}(p_{\mathbf{x},\mathbf{y}}\|p_{\mathbf{x}}\otimes p_{\mathbf{y}}) (1) \displaystyle=\int_{\mathcal{Y}}\int_{\mathcal{X}}p_{\mathbf{x},\mathbf{y}}(\mathbf{x},\mathbf{y})\log\left(\frac{p_{\mathbf{x},\mathbf{y}}(\mathbf{x},\mathbf{y})}{p_{\mathbf{x}}(\mathbf{x})p_{\mathbf{y}}(\mathbf{y})}\right)d\mathbf{x}d\mathbf{y}. Strong correlation manifests as significant divergence between p(\mathbf{x},\mathbf{y}) and p(\mathbf{x})p(\mathbf{y}) , yielding large MI, while uncorrelated variables satisfy p(\mathbf{x},\mathbf{y})\approx p(\mathbf{x})p(\mathbf{y}) , resulting in MI near zero. While MI offers a principled dependence measure, it rarely admits closed-form solutions except for certain known distributions ( 14 ; 15 ) . Thus, practical applications require estimation from finite samples \mathcal{D}=\{\mathbf{x}^{i},\mathbf{y}^{i}\}_{i=1}^{n} drawn from p_{\mathbf{x},\mathbf{y}} . Recent advances have produced powerful neural estimators ( 2 ; 19 ; 20 ; 50 ; 39 ; 47 ; 32 ; 52 ) , with the most prominent one leveraging the Donsker-Varadhan (DV) representation ( 17 ) : \displaystyle\mathbb{I}(\mathbf{x},\mathbf{y})\coloneqq\sup_{\theta}\mathbb{E}_{p_{\mathbf{x},\mathbf{y}}}[\theta]-\log(\mathbb{E}_{p_{\mathbf{x}}\otimes p_{\mathbf{y}}}[e^{\theta}]), (2) where \theta:\mathcal{X}\times\mathcal{Y}\rightarrow\mathbb{R} is a critic function. Mutual Information Neural Estimation (MINE) ( 2 ) parametrizes \theta as a neural network and approximates the supremum through gradient-based optimization. Besides MINE, there also exist a wide range of neural estimators based on different bounds and learning objectives; see § 6 . Challenge of real-time MI estimation. Despite differences in theory and algorithm, all existing neural estimators share a critical computational bottleneck in practice: they require training a network \theta from scratch for each incoming dataset \mathcal{D}=\{\mathbf{x}^{i},\mathbf{y}^{i}\}_{i=1}^{n} via gradient descent: \vskip-1.00006pt\theta^{t+1}\leftarrow\theta^{t}-\eta\nabla_{\theta^{t}}\mathcal{L}(\theta^{t}),\quad t=1,...,T (3) where \mathcal{L}(\theta) is an estimator-specific objective (e.g., the negative DV bound for MINE). Achieving accurate MI estimates typically requires thousands of gradient steps, resulting in a \mathcal{O}(T) computational complexity. This prohibitive cost limits real-time applications such as high-frequency financial correlation monitoring or large-scale genomic screening. The recent InfoNet ( 27 ) aimed to address this inefficiency by pretraining a network to directly output optimal critic value via lookup tables, eliminating inference-time optimization. However, InfoNet is fundamentally limited to univariate inputs, and extending it to d -dimensional variables would require storing \mathcal{O}(e^{d}) values in its lookup table, which quickly becomes intractable even for d=8 , and it can not process data with varying data dimensionality. These limitations motivate our fundamentally different approach for real-time measurement of statistical dependence, where a unified model is developed to directly process multivariate data with varying dimensionalities and sample sizes. 3 Method We present InfoAtlas , a pretrained architecture to address the above challenge of real-time correlation estimation between multivariate random variables \mathbf{x}\in\mathbb{R}^{d_{x}} and \mathbf{y}\in\mathbb{R}^{d_{y}} . Unlike existing neural estimators that require iterative optimization, InfoAtlas directly outputs mutual information (MI) in a single forward pass. This capability is enabled by two key innovations: (i) a dual-path attentive hypernetwork-based architecture, which directly generates distribution-specific critic parameters from observed samples with varying sizes