Generalization error bound for denoising score matching under relaxed manifold assumption
Konstantin Yakovlev, Nikita Puchkin · 2025
Read Generalization error bound for denoising score matching under relaxed manifold assumption by Konstantin Yakovlev, Nikita Puchkin (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-13662.txt.
Introduction
Konstantin Yakovlev and Nikita Puchkin bound the errors of a learning method behind modern generative models. Denoising score matching trains a model to undo noise, and theory for it usually assumes data lie exactly on a thin surface. Their 2025 paper softens that assumption, letting samples stray from the surface while keeping a mixture model structure.
Even with the looser setting, the authors derive firm bounds on both approximation and generalization error. The convergence speed turns on the data's intrinsic dimension, not the full space size. The bounds hold even when the ambient dimension grows polynomially with sample size. The result widens the conditions under which score based generators carry guarantees.
Catalog record
| CREATOR | Konstantin Yakovlev, Nikita Puchkin |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 194 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2502-13662
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Generalization error bound for denoising score matching under relaxed manifold assumption Konstantin Yakovlev, Nikita Puchkin Abstract We examine theoretical properties of the denoising score matching estimate. We model the density of observations with a nonparametric Gaussian mixture. We significantly relax the standard manifold assumption allowing the samples step away from the manifold. At the same time, we are still able to leverage a nice distribution structure. We derive non-asymptotic bounds on the approximation and generalization errors of the denoising score matching estimate. The rates of convergence are determined by the intrinsic dimension. Furthermore, our bounds remain valid even if we allow the ambient dimension grow polynomially with the sample size. \coltauthor HSE University, Russian Federation and HSE University, Russian Federation keywords diffusion models, denoising score matching, score estimation, manifold hypothesis. 1 Introduction Denoising diffusion probabilistic models ( Song and Ermon, 2019 ; Song et al., 2020 ) provide a state-of-the art tool for generating high-quality data including images, audio and video synthesis ( Dhariwal and Nichol, 2021 ; Ho et al., 2022 ; Kong et al., 2021 ) . It is based on an insight from random processes theory that the Ornstein-Uhlenbeck process {\mathrm{d}}X_{t}=-X_{t}\,{\mathrm{d}}t+\sqrt{2}\,{\mathrm{d}}W_{t},\quad 0\leqslant t\leqslant T, (1) with an initial condition X_{0}\sim\mathsf{p}_{0}^{*} admits an inverse one {\mathrm{d}}Z_{t}=\left(Z_{t}+2\nabla\log\mathsf{p}_{T-t}^{*}(Z_{t})\right){\mathrm{d}}t+\sqrt{2}\,{\mathrm{d}}B_{t},\quad Z_{0}\sim\mathsf{p}_{T}^{*}. (2) Here W_{t} and B_{t} are independent Wiener processes in \mathbb{R}^{D} and \mathsf{p}_{t}^{*} is the density of X_{t} , t\in[0,T] . Since the diffusion ( 1 ) converges to the standard Gaussian distribution {\mathcal{N}}(0,I_{D}) quite fast (see, e.g., ( Bakry et al., 2014 , Section 4.1) ), one can model the initial density \mathsf{p}_{0}^{*} running the inverse process ( 2 ) with Z_{0}\sim{\mathcal{N}}(0,I_{D}) . The problem is that the score function s^{*}(y,t)=\nabla\log\mathsf{p}_{t}^{*}(y),\quad(y,t)\in\mathbb{R}^{D}\times[0,T], is unknown and should be estimated from i.i.d. samples Y_{1},\dots,Y_{n}\sim\mathsf{p}_{0}^{*} . In practice, one usually replaces ( 2 ) with {\mathrm{d}}\widehat{Z}_{t}=\left(\widehat{Z}_{t}+2\,\widehat{s}(\widehat{Z}_{t},T-t)\right){\mathrm{d}}t+\sqrt{2}\,{\mathrm{d}}B_{t},\quad\widehat{Z}_{0}\sim{\mathcal{N}}(0,I_{D}), (3) where \widehat{s}(y,t) is an estimate of s^{*}(y,t) . Recently, a lot of researchers (for instance, Bortoli et al. (2021) ; De Bortoli (2022) ; Lee et al. (2023) ; Chen et al. (2023a) ; Chen et al. (2023c) ; Chen et al. (2023d) ; Benton et al. (2024) ; Li and Yan (2024) to name a few) considered generative diffusion models through the lens of Markov processes theory. As one should have expected, the rates of convergence of the density of \widehat{Z}_{T-t_{0}} , where t_{0}>0 is a fixed number referred to as stopping time , to the target distribution obtained in those papers heavily depend on the accuracy of estimation of s^{*}(y,t) . For this reason, the problem of score estimation attracted attention of many statisticians. In ( Sriperumbudur et al., 2017 ; Wibisono et al., 2024 ; Zhang et al., 2024 ) , the authors tackled this problem via widespread tools from nonparametric statistics. Namely, Sriperumbudur et al. (2017) considered an infinite-dimensional exponential family of probability densities, which are parametrized by functions in a reproducing kernel Hilbert space, while Wibisono et al. (2024) and Zhang et al. (2024) used kernel smoothing. On the other hand, Oko et al. (2023) studied theoretical properties of the denoising score matching estimate (see Section 2 for the definition), which is often used in practice. The authors took a class of feed-forward neural networks with ReLU activations as a class of admissible scores. Under the condition that \mathsf{p}_{0}^{*} is supported on the cube [-1,1]^{D} and bounded away from zero on this set, Oko et al. (2023) derived approximation and generalization error bounds. Unfortunately, the rates of convergence in Sriperumbudur et al. (2017) ; Wibisono et al. (2024) ; Zhang et al. (2024) ; Oko et al. (2023) deteriorate extremely fast as the dimension D grows and are not applicable in real-world scenarios. At the same time, in various applications the data distribution has a nice low-dimensional structure ( Bengio et al., 2013 ; Pope et al., 2021 ) despite its high-dimensional representation. For this reason, one can hope for more optimistic rates of convergence depending on the effective dimension rather than on the ambient one. For instance, Chen et al. (2023b) attempted to escape the curse of dimensionality imposing additional assumptions on the underlying density \mathsf{p}_{0}^{*} . They showed that the risk of the denoising score matching estimate with probability at least 1-1/n does not exceed {\mathcal{O}}\left(\frac{1}{t_{0}}\left(n^{-2/(d+5)}+Dn^{-(d+3)/(d+5)}\right)\mathrm{polylog}(\log D,\log(1/t_{0}),\log n)\right). Here t_{0} is a stopping time and d is an intrinsic dimension which may be much smaller than D . However, this upper bound estimate (as well as the one in the subsequent work ( Boffi et al., 2024 ) ) was obtained under very restrictive assumption that \mathsf{p}_{0}^{*} is supported on a linear d -dimensional subspace. In the recent papers ( Tang and Yang, 2024 ) and ( Azangulov et al., 2024 ) the authors considered a much more general and challenging setup when the distribution of Y_{1},\dots,Y_{n} is supported on a smooth low-dimensional manifold {\mathcal{M}} . They reported that the excess risk of their estimates converges to zero at the rates depending on the manifold smoothness, smoothness of \mathsf{p}^{*}_{0} , and the dimension of {\mathcal{M}} . However, the works ( Tang and Yang, 2024 ; Azangulov et al., 2024 ) have several significant issues. • In ( Tang and Yang, 2024 ) the hidden constants in the rates of convergence depend exponentially on the ambient dimension D . This makes the derived rates of convergence useless when D is of order \log n . In practice, D is usually much larger than \log n . • The estimate of Azangulov et al. (2024) exploits the fact that Y_{1},\dots,Y_{n} lie exactly on the manifold {\mathcal{M}} . However, this assumption seems to be unrealistic. In contrary, in several papers ( Daras et al., 2024a ; Daras et al., 2024b ; Kawar et al., 2024 ; Bai et al., 2024 ) the authors note that adding moderate noise to initial samples improves quality of image generation and prevents mode collapse. • Both Tang and Yang (2024) and Azangulov et al. (2024) assume that {\mathcal{M}} has no boundary and a positive reach. Furthermore, they suppose that the density \mathsf{p}_{0}^{*} (with respect to the volume measure on {\mathcal{M}} ) is bounded away from zero and infinity. In ( Zhang et al., 2024 , p. 2) , the authors write: “The density lower bound greatly simplifies the proof of the score estimation error bound; however, it excludes natural distribution classes, such as multi-modal distributions or mixtures with well-separated components.” • Both Tang and Yang (2024) and Azangulov et al. (2024) refer to ( Oko et al., 2023 , Theorem C.4) when derive the estimation error bound. However, Theorem C.4 in ( Oko et al., 2023 ) has a critical flaw. To be more precise, the last implication in (69) on page 41 does not hold. Hence, the rates of convergence in all the papers ( Oko et al., 2023 ; Tang and Yang, 2024 ; Azangulov et al., 2024 ) are incorrect. Our Contribution. In this paper, we attempt to overcome the aforementioned drawbacks in the existing results on score estimation. • We suggest a statistical model with relaxed manifold assumption. In contrast to Tang and Yang (2024) ; Azangulov et al. (2024) , we do not require the samples Y_{1},\dots,Y_{n}\sim\mathsf{p}_{0}^{*} to lie exactly on a