A Random Matrix Theory Perspective on the Consistency of Diffusion Models
Binxu Wang, Jacob Zavatone-Veth, Cengiz Pehlevan · 2026
Read A Random Matrix Theory Perspective on the Consistency of Diffusion Models by Binxu Wang, Jacob Zavatone-Veth, Cengiz Pehlevan (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2602-02908.txt.
Introduction
Binxu Wang and 2 co-authors explain a puzzle about diffusion models in a 2026 paper. Models trained on separate, non-overlapping data subsets often generate strikingly similar images from the same starting noise. The authors trace much of that similarity to shared Gaussian statistics across the subsets.
They build a random matrix theory framework for the linear case. It quantifies how finite datasets shape the learned denoiser's average behavior and its variation. Limited data, they show, shrinks low-variance directions and pulls samples toward the dataset mean. Three factors drive disagreement between splits. Tests on larger network architectures in a non-memorization regime matched the theory's predictions, giving a baseline for judging reproducibility in diffusion training.
Catalog record
| CREATOR | Binxu Wang, Jacob Zavatone-Veth, Cengiz Pehlevan |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 124 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2602-02908
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
A Random Matrix Theory Perspective on the Consistency of Diffusion Models Binxu Wang, Jacob Zavatone-Veth, Cengiz Pehlevan Abstract Diffusion models trained on different, non-overlapping subsets of a dataset often produce strikingly similar outputs when given the same noise seed. We trace this consistency to a simple linear effect: the shared Gaussian statistics across splits already predict much of the generated images. To formalize this, we develop a random matrix theory (RMT) framework that quantifies how finite datasets shape the expectation and variance of the learned denoiser and sampling map in the linear setting. For expectations, sampling variability acts as a renormalization of the noise level through a self-consistent relation σ 2 ↦ κ ( σ 2 ) \sigma^{2}\mapsto\kappa(\sigma^{2}) , explaining why limited data overshrink low-variance directions and pull samples toward the dataset mean. For fluctuations, our variance formulas reveal three key factors behind cross-split disagreement: anisotropy across eigenmodes, inhomogeneity across inputs, and overall scaling with dataset size. Extending deterministic-equivalence tools to fractional matrix powers further allows us to analyze entire sampling trajectories. The theory sharply predicts the behavior of linear diffusion models, and we validate its predictions on UNet and DiT architectures in their non-memorization regime, identifying where and how samples deviates across training data split. This provides a principled baseline for reproducibility in diffusion training, linking spectral properties of data to the stability of generative outputs. Figure 1 : Motivating observation and the linear theory . A. Diffusion models trained on non-overlapping data splits generate similar images from the same initial noise, even with different neural network architectures, consistent with results in ( Kadkhodaie et al. 2024 ; Zhang et al. 2024 ) . Notably, generated samples from both splits are visually similar to the prediction from the Gaussian linear theory ( Wang & Vastola 2024b ) . B. Quantification of A by paired image distances (MSE) averaging from 512 initial noises. The low-MSE block structure of the four DNNs and linear solution emphasize that this consistency effect is related to the linear structure. CNN1 denotes the CNN trained on split1, similar for CNN2, DiT1, DiT2; CNN1 nearest denotes the set of closest training set sample for the 512 generated image. We hide results for linear predictor of two splits since their samples are nearly identical with the linear predictor for the full dataset. Similar analysis for FFHQ64 is showed in Fig. 6 . 1 Introduction Diffusion models and their relatives such as flow matching have become the dominant generative modeling paradigm across diverse domains, including images, video, and proteins. By learning a time-dependent vector field, these models transform Gaussian noise into structured samples through an ordinary differential equation (ODE) or its stochastic variants ( Song et al. 2021 ; Albergo et al. 2023 ) . A distinctive feature of diffusion models is their striking consistency across training runs (Figure 1 ). When trained on the same distribution, even with disjoint datasets, different architectures, or repeated initializations, diffusion models often map the same noise seed to highly similar outputs under the deterministic probability flow ( Kadkhodaie et al. 2024 ; Zhang et al. 2024 ) . This phenomenon contrasts with other generative modeling frameworks including GANs and VAEs, where the isotropic Gaussian latent space admits arbitrary rotations, leading to run-to-run variability in the mapping from latent codes to data ( Martinez & Pearson 2022 ) . Why does consistency matter? Consistency across non-overlapping data splits suggests that diffusion models recover aspects of the underlying data manifold that are insensitive to the specific training set. This raises fundamental questions about how such models generalize beyond their training samples, to what extent they memorize idiosyncratic data, and whether their outputs reflect universal statistical regularities of the distribution. These issues connect to emerging theoretical and empirical debates on generalization, memorization, and creativity in diffusion models ( Kamb & Ganguli 2024 ; Niedoba et al. 2024 ; Kadkhodaie et al. 2024 ; Chen 2025 ; Vastola 2025 ; Bonnaire et al. 2025 ) ; see also further discussion in App. A . Our approach. We analyze this phenomenon through the lens of random matrix theory (RMT), beginning with the observation that the consistency effect can already be predicted by a linear Gaussian model (Fig. 1 ). Building on the linear denoiser framework, we develop a precise RMT analysis of how finite-sample variability in the empirical covariance affects both the expectation and fluctuation of denoisers and sampling maps (Fig. 2 A ). We then validate these theoretical predictions against deep diffusion models (CNNs and DiTs), showing that the same RMT principles still govern their inhomogeneity of consistency across data splits. Our main contributions are as follows: • Linear origin of consistency: We show that shared Gaussian statistics i.e. linear denoiser already predict cross-split agreement. • Finite-sample RMT: We prove that randomness enters through a renormalized noise scale \sigma^{2}\!\mapsto\!\kappa(\sigma^{2}) , explaining overshrinkage of low-variance modes. • Variance law: We derive a factorized form for fluctuation—explaining structures in cross-split deviation: anisotropy across eigenmodes, inhomogeneity across inputs, and global scaling with training set size. • Fractional-power DE: We extend deterministic equivalence (DE) to fractional matrix powers, enabling analysis of full sampling trajectories. • Deep-net validation: We qualitatively confirm overshrinkage, anisotropy, and inhomogeneity phenomenon in UNet and DiT models beyond the linear regime. 2 Notation and Set up Score-based Diffusion Models. Let p_{0}(\mathbf{x}) be the target data distribution. For each noise scale \sigma>0 , define the noised distribution as p(\mathbf{x};\sigma)=\bigl(p_{0}*\mathcal{N}(0,\sigma^{2}\mathbf{I})\bigr)(\mathbf{x})=\int p_{0}(\mathbf{y})\,\mathcal{N}(\mathbf{x}\mid\mathbf{y},\sigma^{2}\mathbf{I})\,d\mathbf{y}. The corresponding score function is \nabla_{\mathbf{x}}\log p(\mathbf{x};\sigma), i.e. the gradient of the log–density. In the EDM formulation ( Karras et al. 2022 ) , the probability flow ODE ( PF -ODE) reads, \frac{d\mathbf{x}}{d\sigma}=-\,\sigma\,\nabla_{\mathbf{x}}\log p(\mathbf{x};\sigma). (PF) This ODE transports samples from p(\,\cdot\,;\sigma_{2}) to p(\,\cdot\,;\sigma_{1}) when integrating \sigma from \sigma_{2} to \sigma_{1} . In particular, by starting from Gaussian noise \mathcal{N}(0,\sigma_{T}^{2}I) and integrating the PF-ODE from a sufficiently large \sigma_{T} down to \sigma=0 , one recovers clean samples from p_{0} . We adopt the EDM parametrization for its notational simplicity; other common diffusion formalisms are equivalent up to simple rescalings of time and space ( Karras et al. 2022 ) . To estimate the score function of p_{0}(\mathbf{x}) , we minimize the denoising score matching (DSM) objective ( Vincent 2011 ) . Parametrizing the denoiser \mathbf{D}_{\theta} with a function approximator, then at noise level \sigma the DSM objective reads \displaystyle\mathcal{L}_{\sigma}\;=\;\mathbb{E}_{\mathbf{x}_{0}\sim p_{0},\;\mathbf{z}\sim\mathcal{N}(0,\mathbf{I})}\Bigl\|\mathbf{D}_{\theta}(\mathbf{x}_{0}+\sigma\mathbf{z};\sigma)\;-\;\mathbf{x}_{0}\Bigr\|_{2}^{2}. (DSM) Using Tweedie’s formula ( Robbins 1992 ) , the score function can be obtained via the optimized ‘denoiser’ \mathbf{s}_{\theta}(\mathbf{x},\sigma)=(\mathbf{D}_{\theta}(\mathbf{x},\sigma)-\mathbf{x})/\sigma^{2} . In practice ( Karras et al. 2022 ) , diffusion models balance these scale-specific objectives with a weighting function w(\sigma) , yielding the overall training loss \mathcal{L}=\int_{\sigma}d\sigma\,\,w(\sigma)\,\mathcal{L}_{\sigma} . Data distribution. Consider a ground truth data distribution p_{0}(\mathbf{x}),\ \mathbf{x}\in\mathbb{R}^{d} , with population mean \bm{\mu} and covariance \bm{\Sigma} . From this underlying distribution, we construct an empirical distribution \{\mathbf{x}_{i}\} with n samples, stacked as X\in\mathbb{R}^{n\times d} , then we denote the empirical mean \hat{\bm{\mu}} and covariance \hat{\bm{\Sigma}} . Here we are interested in the effect of the number of samples n , and different realizations of X on the expectation (mean) and fluctuation (variance) of learned diffusion model. More specifically, we will study how randomness in the empirical covariance \hat{\bm{\Sigma}} drives variability in the denoiser, relative to the