Implicit score matching meets denoising score matching: improved rates of convergence and log-density Hessian estimation

Konstantin Yakovlev, Anna Markovich, Nikita Puchkin · 2025

Read Implicit score matching meets denoising score matching: improved rates of convergence and log-density Hessian estimation by Konstantin Yakovlev, Anna Markovich, Nikita Puchkin (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2512-24378.txt.

Introduction

Konstantin Yakovlev, Anna Markovich, and Nikita Puchkin compare two ways to learn a score function. The score, the gradient of the log density, drives modern generative samplers. Implicit score matching and denoising score matching estimate it differently. Data with low-dimensional structure is the setting. Such structure should make estimation easier in principle.

The authors prove implicit score matching adapts to the intrinsic dimension. Its convergence rates match denoising score matching in sample size. Both methods also estimate log-density Hessians by simple differentiation. No curse of dimensionality appears in that step. The result justifies convergence of ODE-based samplers for diffusion models. The proofs rest on Gagliardo-Nirenberg inequalities. Those relate weighted norms of functions to norms of their derivatives. Two estimation traditions end up equally strong on structured data.

Catalog record

CREATORKonstantin Yakovlev, Anna Markovich, Nikita Puchkin
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE145 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2512-24378

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Implicit score matching meets denoising score matching: improved rates of convergence and log-density Hessian estimation Konstantin Yakovlev, Anna Markovich, Nikita Puchkin Abstract We study the problem of estimating the score function using both implicit score matching and denoising score matching. Assuming that the data distribution exhibiting a low-dimensional structure, we prove that implicit score matching is able not only to adapt to the intrinsic dimension, but also to achieve the same rates of convergence as denoising score matching in terms of the sample size. Furthermore, we demonstrate that both methods allow us to estimate log-density Hessians without the curse of dimensionality by simple differentiation. This justifies convergence of ODE-based samplers for generative diffusion models. Our approach is based on Gagliardo-Nirenberg-type inequalities relating weighted L 2 L^{2} -norms of smooth functions and their derivatives. 1 Introduction Given i.i.d. samples Y_{1},\dots,Y_{n}\in\mathbb{R}^{D} drawn from an absolutely continuous distribution with a density \mathsf{p}^{*} , we are interested in estimation of the log-density gradient s^{*}(y)=\nabla\mathsf{p}(y) also referred to as score function. This task arises naturally in generative modelling, where a learner has to deal with intractable densities and must use either classical algorithms (such as Langevin ( Cheng et al. 2018 ) or Hamiltonian Monte Carlo ( Neal et al. 2011 ) ) or modern score-based diffusion models ( Song and Ermon 2019 ; Song et al. 2020 ; Song et al. 2021 ) to produce a new sample. Besides, the problem of interest can be considered as a particular case of density estimation where the performance is measured by the Fisher divergence. The advantages of the Fisher divergence were discussed, for instance, by Sriperumbudur et al. 2017 . For these reasons, statisticians have paid a lot of attention to theoretical analysis of various score estimation methods including kernel-based approaches ( Li et al. 2005 ; Wibisono et al. 2024 ; Zhang et al. 2024 ) , Stein’s method ( Li and Turner 2018 ; Shi et al. 2018 ) , implicit score matching ( Hyvärinen 2005 ; Hyvärinen 2007 ; Sriperumbudur et al. 2017 ; Sutherland et al. 2018 ; Sasaki et al. 2018 ; Zhou et al. 2020 ; Koehler et al. 2023 ) , and denoising score matching ( Oko et al. 2023 ; Tang and Yang 2024 ; Azangulov et al. 2024 ; Yakovlev and Puchkin 2025a ) . In the present paper, we examine denoising ( Vincent 2011 ) and implicit ( Hyvärinen 2005 ) score matching. The former algorithm is the cornerstone for learning score-based diffusion models ( Song et al. 2021 ) . Many advances on score estimation with deep feedforward neural networks appeared in few recent years. In Oko et al. 2023 , the authors considered a nonparametric setup, where the target density belongs to the Besov space B^{\beta}_{p,q}([-1,1]^{D}) . Assuming that \mathsf{p}^{*} is bounded away from zero, they showed 1 1 1 According to Yakovlev and Puchkin 2025a , the analysis of estimation error in Oko et al. 2023 has a flaw (in particular, the issue occurs in the proof of Theorem C.4, see ( Yakovlev and Puchkin 2025a , page 9) for the discussion). However, following the approach of Yakovlev and Puchkin 2025a , one can obtain the rates of convergence (with respect to the sample size n ) announced by Oko et al. 2023 with slightly worse dependence on the stopping time \underline{T} . that denoising score matching can achieve the expected squared error of order {\mathcal{O}}(n^{-2\beta/(2\beta+D)}) . While this rate of convergence is common for nonparametric statistics, it obviously suffers from the curse of dimensionality when D=\Omega(\log n) (as well as kernel-based approaches Wibisono et al. 2024 ; Zhang et al. 2024 ). Fortunately, real-world data sets, such as high-resolution images, often have intrinsic low-dimensional structures ( Bengio et al. 2013 ; Pope et al. 2021 ) . To mitigate the curse of dimensionality, Chen et al. 2023b suggested a model with a target distribution supported on a low-dimensional linear subspace and derived the score estimation rates that do not deteriorate fast when the ambient dimension is large. However, this setup looks oversimplified and poorly reflects non-linear structures in the read-world data. This issue was addressed in subsequent works ( Tang and Yang 2024 ; Azangulov et al. 2024 ) , where the authors assumed that \mathsf{p}^{*} is supported on a smooth submanifold of dimension d<D . Under some technical assumptions, they proved 2 2 2 The papers of Tang and Yang 2024 ; Azangulov et al. 2024 inherit the mistake of Oko et al. 2023 , as their proofs rely on Theorem C.4, which has a flaw. Similarly to Oko et al. 2023 , one can obtain the rates of convergence (with respect to the sample size n ) announced in these papers (with slightly worse dependence on the stopping time) following the approach of Yakovlev and Puchkin 2025a . that the rate of convergence for for denoising score matching in this setup is determined by the intrinsic dimension d , rather than the ambient one. Despite a significant progress in understanding score-based diffusion models, both Tang and Yang 2024 and Azangulov et al. 2024 required \mathsf{p}^{*} to be bounded away from zero. According to Zhang et al. 2024 , this may be too demanding. In Yakovlev and Puchkin 2025a , the authors bypassed the restrictive assumptions of density lower bounds and bounded data distribution support, thereby capturing a broader and more realistic class of data distributions. On the other hand, the ability of implicit score matching to adapt to low intrinsic dimension (as in the setups of Tang and Yang 2024 ; Azangulov et al. 2024 ; Yakovlev and Puchkin 2025a ) is much less explored. In Sriperumbudur et al. 2017 ; Sutherland et al. 2018 ; Zhou et al. 2020 , the authors studied nonparametric score estimation using reproducing kernel Hilbert space (RKHS). In addition, Sasaki et al. 2018 use similar machinery to derive the estimation error bounds for the score function and its derivatives under analogous conditions. Of course, it is known that diffusion kernels can keep information about underlying manifold structure, and some popular dimension reduction techniques, such as diffusion maps Coifman and Lafon 2006 or Laplacian eigenmaps Belkin and Niyogi 2003 , are based on this fact. However, this property mostly holds in the situation when the data points lie exactly on the manifold or in its very small vicinity. For this reason, even if the kernel is chosen properly, the rates of convergence in Sriperumbudur et al. 2017 ; Sutherland et al. 2018 ; Sasaki et al. 2018 ; Zhou et al. 2020 may significantly deteriorate in noisy setups, which are common in generative diffusion models (see, for example, ( Tang and Yang 2024 ; Azangulov et al. 2024 ; Yakovlev and Puchkin 2025a ) ). A similar issue appears in Shen et al. 2024 , where the authors assumed that the target density is supported on a thin tube around a low-dimensional manifold. They leave the case of unbounded support beyond the scope and, moreover, require the tube to shrink as the sample size grows. This leads us to the following question. Q1 : Can a score estimator trained with an implicit score matching objective

Read the complete text