Diffusion Model’s Generalization Can Be Characterized by Inductive Biases toward a Data-Dependent Ridge Manifold
Ye He, Yitong Qiu, Molei Tao · 2026
Read Diffusion Model’s Generalization Can Be Characterized by Inductive Biases toward a Data-Dependent Ridge Manifold by Ye He, Yitong Qiu, Molei Tao (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2602-06021.txt.
Introduction
When a diffusion model does not just memorise its training set, where do its new samples land? Ye He and two co-authors answer with geometry, building a time-varying family of ridge manifolds from the smoothed data distribution, tracing log-density ridges. Their main result follows generated samples through a reach-align-slide motion, first entering a ridge neighbourhood, then closing in, then sliding along the ridge.
How far a sample sits from the ridge tracks the normal part of training error, while motion along the ridge tracks the tangential part. The authors tie that picture to training dynamics and spell it out for random feature models, where architecture bias and optimisation error split cleanly. Synthetic multimodal tests and MNIST latent-diffusion trials back the geometry in low and high dimensions. The work gives diffusion generalisation a map, not just a mystery.
Catalog record
| CREATOR | Ye He, Yitong Qiu, Molei Tao |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 147 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2602-06021
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Diffusion Model’s Generalization Can Be Characterized by Inductive Biases toward a Data-Dependent Ridge Manifold Ye He, Yitong Qiu, Molei Tao Abstract When a diffusion model is not memorizing the training data set, how does it generalize exactly? A quantitative understanding of the distribution it generates would be beneficial to, for example, an assessment of the model’s performance for downstream applications. We thus explicitly characterize what diffusion model generates, by proposing a log-density ridge manifold and quantifying how the generated data relate to this manifold as inference dynamics progresses. More precisely, inference undergoes a reach–align–slide process centered around the ridge manifold: trajectories first reach a neighborhood of the manifold, then align as being pushed toward or away from the manifold in normal directions, and finally slide along the manifold in tangent directions. Within the scope of this general behavior, different training errors will lead to different normal and tangent motions, which can be quantified, and these detailed motions characterize when inter-mode generations emerge. More detailed understanding of training dynamics will lead to more accurate quantification of the generation inductive bias, and an example of random feature model will be considered, for which we can explicitly illustrate how diffusion model’s inductive biases originate as a composition of architectural bias and training accuracy, and how they evolve with the inference dynamics. Experiments on synthetic multimodal distributions and MNIST latent diffusion support the predicted directional effects, in both low- and high-dimensions. 1 Introduction Diffusion models ( Sohl-Dickstein et al. 2015 , Ho et al. 2020 , Song et al. 2021 ) now deliver state-of-the-art sample quality across a wide range of generative tasks, which has made them a central tool for image, audio and video generation ( Dhariwal and Nichol 2021 , Kong et al. 2021 , Brooks et al. 2024 ) . At the same time, understanding how innovative these models can be (or in other words, what exactly do they generate) is becoming increasingly important. In fact, certain diffusion models can memorize the training data and act like a stochastic parrot that just randomly repeats training data, which means the generation quality may appear to be high but in fact no new knowledge is created; in addition, it can lead to safety and privacy concerns ( Carlini et al. 2023 , Somepalli et al. 2023 , Duan et al. 2023 , Liu et al. 2024 ) . Naturally, what triggers memorization of diffusion model has been extensively investigated ( Kadkhodaie et al. 2023 , Zhang et al. 2023 , e.g.,) . By now, it is fairly well accepted that it is the various sources of error inside diffusion model that give its non-memorization; in fact, criteria for (non-)memorization can already be made rather quantitative ( Ye et al. 2025 , e.g.,) , and ‘generalization’ became a synonym of non-memorization in this literature. One important question still being actively investigated though is, when diffusion model is not memorizing, how/where does the model produce new samples different from the training set, i.e. how does it innovate knowledge. A quantitative, explicit answer would help distinguish generalization from hallucination and assess the applicability of the generative model to specific downstream tasks. The powerful classical theory of statistical learning can help bound the gap between the generated distribution and the latent population distribution (e.g., Wang et al. 2024 ), but in practice such a worst-case bound can be loose as it needs to account for all possible population distributions that are consistent with the finite dataset which the model trains on. This motivates an alternative, fully data-dependent perspective: compare the generated distribution directly to the empirical distribution of the training data, and regard this discrepancy as the model’s generalization. Many recent progress along this line provide rich insights. For example, Aithal et al. 2024 ’s seminal empirical investigation suggests some geometric bias, in which generated data concentrate in structured regions that connect distinct modes of the data. More systematically speaking, generalization in diffusion models may arise from several mechanisms, and the literature studies it from complementary perspectives. One perspective concerns the target , asking whether stochasticity in the finite training target is itself a driver of generalization ( Vastola 2025 , Bertrand et al. 2025 ) . A second perspective examines training inductive bias ( Kamb and Ganguli 2024 , Shah et al. 2025 , Wu et al. 2025 , Bonnaire et al. 2025 ) , characterizing how the learned reverse-time dynamics depend on the model class and learning procedure. A third perspective focuses on inference ( Baptista et al. 2025 , Chen 2025 , Farghly et al. 2025 ) , understanding what reverse-time inference dynamics generate under manually assumed structured training error. While extremely insightful, the literature still lacks a rigorous and quantitative understanding of diffusion model’s generalization. Motivated by this gap, we develop a purely data-driven (no population distribution involved) framework to explicitly quantify the inductive biases of diffusion model’s generation. Our work integrates the perspectives in the literature: On the target side, we treat the finite training dataset as the target and analyze how the generated distribution relates to the empirical distribution of training data, rather than attributing generalization to the stochasticity of sampling an unknown population distribution. On the inference side, we make the data-structure explicit by constructing a time-indexed family of log-density ridge sets from the smoothed empirical density, and we analyze how inference trajectories evolve relative to this geometry. This differs from the work ( Farghly et al. 2025 ) that motivates log-density smoothing as a useful analytical lens but does not study the simulated inference dynamics or provides a ridge-manifold description based on the data. On the training side, building on random-feature analyses of diffusion training ( George et al. 2025 , Bonnaire et al. 2025 ) , we go beyond test/training error comparisons and show how limited-expressivity parametrization (due to finite width) and inaccurate optimization translate into directional geometric effects during inference. Our results confirm and explain when and why samples will populate structured intermediate regions beyond the training dataset, how this behavior depends on both the time of the sampling dynamics and what is learned during training. Compared to recent analytical explanations of interpolation bias that rely on stylized settings or imposed error ansatz (e.g., two-point data and smoothed empirical score) ( Chen 2025 ) , our results are derived under verifiable regularity conditions and explicitly connect the resulting geometric bias to the dataset and to training parameters. We also complement the interesting work of ( Li et al. 2025 ) , which investigates the transition between sampling uniform from some low-dimensional manifold and sampling a target distribution supported on the manifold, as our setting differs in two key ways: first, we study the full denoising diffusion, where noise level is time-dependent and vanishing as sampling approaches the data, as opposed to constant-noise level considered; second, rather than assuming there is some underlying manifold without spelling out what it is, we explicitly characterize the geometric object that governs generation in our framework and quantify how the learned score/mean estimator drives the inference dynamics around this object. (a) early phase (b) reach (c) align (d) slide Figure 1 : Illustration of the generation inductive bias. 13 training points (