Dimension-free Score Matching and Time Bootstrapping for Diffusion Models
Syamantak Kumar, Dheeraj Nagaraj, Purnamrita Sarkar · 2025
Read Dimension-free Score Matching and Time Bootstrapping for Diffusion Models by Syamantak Kumar, Dheeraj Nagaraj, Purnamrita Sarkar (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-10354.txt.
Introduction
Syamantak Kumar, Dheeraj Nagaraj, and Purnamrita Sarkar give new learning bounds for diffusion models in this 2025 paper. Diffusion models create data by learning a score function that reverses a noising process. Past bounds on how many samples that learning needs grew steeply with data dimension.
The authors prove the first nearly dimension-free bounds, a double-exponential improvement in that factor. One approximator learns the scores at all noise levels at once, which helps it generalize across time steps. A martingale-based error split handles dependent data. They also propose a bootstrapping trick that reuses learned scores to sharpen estimates at high noise levels.
Catalog record
| CREATOR | Syamantak Kumar, Dheeraj Nagaraj, Purnamrita Sarkar |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 174 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2502-10354
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Dimension-free Score Matching and Time Bootstrapping for Diffusion Models Syamantak Kumar, Dheeraj Nagaraj, Purnamrita Sarkar Abstract Diffusion models generate samples by estimating the score function of the target distribution at various noise levels. The model is trained using samples drawn from the target distribution, progressively adding noise. Previous sample complexity bounds have a polynomial dependence on the dimension d d , apart from log ( | ℋ | ) \log\left(|\mathcal{H}|\right) , where ℋ \mathcal{H} is the hypothesis class. In this work, we establish the first (nearly) dimension-free sample complexity bounds, modulo any dependence due to log ( | ℋ | ) \log(|\mathcal{H}|) , for learning these score functions, achieving a double exponential improvement in dimension over prior results. A key aspect of our analysis is to use a single function approximator to jointly estimate scores across noise levels, a critical feature in practice which enables generalization across timesteps. We introduce a novel martingale-based error decomposition and sharp variance bounds, enabling efficient learning from dependent data generated by Markov processes, which may be of independent interest. Building on these insights, we propose Bootstrapped Score Matching (BSM), a variance reduction technique that utilizes previously learned scores to improve accuracy at higher noise levels. These results provide crucial insights into the efficiency and effectiveness of diffusion models for generative modeling. 1 Introduction Score-based diffusion models [ SDWMG15 , HJA20 ] are generative models that have transformed image and video generation [ RBL+22 , SCS+22 , RDN+22 , PEL+23 ] , enabling foundation models to produce photorealistic and stylized images from text prompts. Their adaptability extends diverse domains such as audio [ KPH+21 , ECT+24 ] , text [ GH24 , HKT22 , LME23 , VNS24 ] , molecule [ HSVW22 , HLX+24 ] , and layout generation [ IKSS+23 , LBMP23 ] . They generate additional samples given m i.i.d. samples from a target distribution ( \pi ) using a trained neural network that learns the score function \pi at different noise levels. In contrast, the classical Markov Chain Monte Carlo (MCMC) methods which seek to sample from a distribution given access to underlying density function. While MCMC methods can be slow for multi-modal distributions, diffusion models can sample efficiently with minimal assumptions, provided the score functions are learned accurately [ CCL+22 , BDBDD24 ] . Given m i.i.d. samples from the target distribution, the first step (called the forward process) obtains noised samples from a noising Markov process converging to the Gaussian distribution at various noise levels. The second step estimates score functions of the distribution at each noise level using Denoising Score Matching (DSM) [ Vin11 ] . This approach relies on learning from dependent data from multiple trajectories of a Markov process in contrast to learning with i.i.d. data in traditional settings. Prior works [ BMR20 , GPPX24 ] provide theoretical guarantees for score function approximation separately at each noise level using the same samples. However, in practice, a single function approximator is commonly used at all noise levels, which is considered by [ HRX24 ] . [ BJTZ24 ] show that despite the problem of distribution estimation suffering from the curse of dimensionality [ CHZW23 , OAS23 ] , the existence of low-dimensional structures allows neural networks to learn the score functions. All of these existing bounds exhibit polynomial dependence on the dimension, d . We establish that under suitable smoothness conditions for a given function class, score matching with a single function approximator jointly across all timesteps achieves a nearly dimension-free sample complexity that depends on the smoothness parameter and grows only as \log\log(d) . 1.1 Our Contributions (1) We analyze the sample complexity of denoising score matching across noise levels using a single function approximator, achieving a double-exponential reduction in dimension dependence . (2) We present a novel martingale decomposition of the error, which allows us to bound the error despite being composed of samples from multiple trajectories of dependent data . (3) We use second-order Tweedie-type formulae to obtain a sharp bound on the error variance , crucial for establishing almost dimension-free convergence rates. (4) Inspired by the above, we present Bootstrapped Score Matching . Here the score at a given noise level is learned by bootstrapping to the learned score function at a lower noise level, achieving variance reduction. This shows improved performance compared to DSM in simple empirical studies. 1.2 Related work Score matching and diffusion models: Score Matching was introduced in the context of statistical estimation in [ HD05 ] with an algorithm now called Implicit Score Matching (ISM). Diffusion models are trained using Denoising Score Matching (DSM) introduced in [ Vin11 ] , and is based on Tweedie’s formula. Several algorithms have been introduced since, such as Sliced Score Matching [ SGSE20 ] and Target Score Matching [ DBHWD24 ] . Prior works have analyzed the complexity of Denoising Score Matching in various settings [ CHZW23 , OAS23 , GPPX24 , BMR20 , FYWC24 ] . We consider the setting in [ GPPX24 , BMR20 ] , where the score functions of the given distribution can be accurately approximated by a function approximator class (ex: neural networks), instead of the worst case non-parametric analysis in [ FYWC24 ] . These bounds can then be used with the discretization analyses in [ BDBDD24 , CCL+22 , LLT23 ] to guarantee the quality of the generated samples. Learning from dependent data: Learning with data from a markov trajectory has been explored in literature in the context of system identification, time series forecasting and reinforcement learning [ DAJJ12 , SMT+18 , NWB+20 , KNJN21 , TFS24 , ZT22 , BRS18 , KS24 , Sri24 ] Many of these works analyze the rates of convergence with data derived from a mixing Markov chain, when the number of data points available is much higher than the mixing time, \tau_{\mathsf{mix}} . In our context, the Markov chain contains \tilde{O}(\tau_{\mathsf{mix}}) data points created by progressively noising samples from the target distributions, where \tilde{O} hides logarithmic factors. This is similar to the setting in [ TFS24 ] , which considered linear regression and linear system identification. We outline our paper as follows: Section 2 introduces the problem setup and preliminaries, followed by the main results and a comparison with prior work in Section 3 . Section 4 presents key technical results from our proof technique. Finally, Section 5 introduces Bootstrapped Score Matching, a novel training method that shares information explicitly across time by modifying the learning objective. 2 Problem setup and preliminaries Notation: We use [n] to denote \left\{i\in\mathbb{N}\;|\;i\leq n\right\} . \mathbf{I}\in\mathbb{R}^{d\times d} represents the d -dimensional identity matrix. We use \mathcal{N}(\mu,\boldsymbol{\Sigma}) to denote the multivariate normal distribution with specified mean, \mu and covariance matrix \boldsymbol{\Sigma} . \left\lVert.\right\rVert_{2} denotes the \ell_{2} euclidean norm for vectors and \left\lVert.\right\rVert_{\textup{op}} denotes the operator norm for matrices. \mathbb{E}\left[X\right] denotes the expectation of the random variable X and \mathsf{Cov}(X) denotes its covariance matrix. For a,b\in\mathbb{R} , we write a\lesssim b\text{ if and only if there exists an absolute constant }C>0\text{ such that }a\leq Cb. \tilde{O},\tilde{\Omega} represent order notations with logarithmic factors. We also define a coarser notion of subGaussianity used subsequently in our proof sketch, Definition 1 ( \left(\beta^{2},K\right) -subGaussianity) . A mean-zero