The geometry of Stein’s method of moments: A canonical decomposition via score matching

Mitsuki Nagai, Keisuke Yano · 2026

Read The geometry of Stein’s method of moments: A canonical decomposition via score matching by Mitsuki Nagai, Keisuke Yano (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-12843.txt.

Introduction

Mitsuki Nagai and Keisuke Yano clarify the geometry of Stein's method of moments for parameter estimation. The method builds estimators that need no normalizing constant. They give a canonical decomposition of such an estimator after centering the score matching one. That split shows score matching sits at the core of the frame. From it, they build an estimator with lower asymptotic variance. Links to Wasserstein geometry explain the gap to maximum likelihood.

Catalog record

CREATORMitsuki Nagai, Keisuke Yano
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE91544
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.12843

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

The geometry of Stein’s method of moments: A canonical decomposition via score matching Mitsuki Nagai, Keisuke Yano Abstract In this paper, we elucidate the geometry of Stein’s method of moments (SMoM). SMoM is a parameter estimation method based on the Stein operator, and yields a wide class of estimators that do not depend on the normalizing constant. We present a canonical decomposition of an SMoM estimator after centering the score matching estimator, which sheds light on the central role of the score matching within the SMoM framework. Using this decomposition, we construct an SMoM estimator that improves upon the score matching estimator in the asymptotic variance. We also discuss the connection between SMoM and the Wasserstein geometry. Specifically, using the Wasserstein score function, we provide a geometrical interpretation of the gap in the asymptotic variance between the score matching estimator and the maximum likelihood estimator. Furthermore, it is shown that the score matching estimator is asymptotically efficient if and only if the Fisher score functions span the same space as the Wasserstein score functions. 1 Introduction We consider estimation of statistical models whose probability density q_{\theta} has the form q_{\theta}(x)=\frac{1}{Z(\theta)}\tilde{q}_{\theta}(x), where \tilde{q}_{\theta} is a non-normalized density, and Z(\theta)\coloneqq\int\tilde{q}_{\theta}(x)\,\mathrm{d}x . Such models are known as non-normalized models or energy-based models, and appear in various fields ( 21 ; 33 , cf.) . Although flexible, non-normalized models often involve a computationally intractable normalizing constant Z(\theta) , and thus the maximum likelihood estimator (MLE) is computationally infeasible for non-normalized models. 15 introduced score matching as an estimation method for non-normalized models. Score matching estimates the parameter by minimizing the distance between the score function \nabla_{x}\log q_{\theta} and the true one \nabla_{x}\log q_{\theta^{\star}} . This procedure does not depend on the normalizing constant or the true score function \nabla_{x}\log q_{\theta^{\star}} since the minimization problem can be rewritten in a form that avoids \nabla_{x}\log q_{\theta^{\star}} using integration by parts. Recently, 8 proposed yet another approach for non-normalized models, Stein’s method of moments (SMoM). This approach utilizes a Stein operator \mathcal{A}_{\theta} , an operator satisfying the identity \E_{\theta}[\mathcal{A}_{\theta}f]=0 for a class of test functions f on \mathbb{R}^{p} . In this paper, we allow test functions f_{\theta} that may depend on \theta . Since the Stein operator is not unique, we need to choose an appropriate one, and employ the divergence-based Stein operator ( 26 ) : \mathcal{A}_{\theta}f_{\theta}\coloneqq\frac{\div(q_{\theta}f_{\theta})}{q_{\theta}},\quad f_{\theta}\colon\mathbb{R}^{p}\to\mathbb{R}^{p}. Under the boundary condition \lim_{\lVert x\rVert\to\infty}q_{\theta}(x)f_{\theta}(x)=0 , we have \E_{\theta}[\mathcal{A}_{\theta}f_{\theta}]=0 . This Stein operator does not depend on the normalizing constant, implying that SMoM yields a wide class of normalizing-constant-free estimators. In this paper, we provide a geometric framework for analyzing SMoM on the basis of score matching. We derive the asymptotic linear representation of SMoM estimators after centering the score matching estimator (Theorem 1 ). The representation reveals that, within the framework of SMoM, the relevant orthogonal structure arises from the space of test functions, rather than from the space of estimating functions ( 5 , e.g.,) . Using the difference of orthogonality between these spaces, we construct an SMoM estimator that improves upon the score matching estimator in the asymptotic variance (Theorem 2 ), and show the condition for the asymptotic efficiency of the score matching estimator (Theorem 3 ). One intriguing aspect of our construction is its connection to the Wasserstein geometry ( 28 ; 29 ; 22 ; 1 ; 3 ; 27 ; 36 , e.g.,) . In fact, our construction of the SMoM estimator improving the score matching estimator admits a geometric interpretation as the approach to the space spanned by the Wasserstein score functions ( 22 ) . Also, using the Wasserstein score function, we show that the gap in the asymptotic variance between the score matching estimator and the MLE also has a geometrical interpretation. Furthermore, we show that the score matching estimator is asymptotically efficient if and only if the Fisher score functions span the same space as the Wasserstein score functions. 1.1 Literature review and contributions There is an extensive literature on the estimation for non-normalized models, beginning with Hyvärinen’s pioneering work ( 15 ) . Regarding the denoising autoencoder as a non-normalized model, 40 proposed denoising score matching. To improve computational efficiency on score matching, 32 introduced sliced score matching. The framework of scoring rules is extended to cover score matching ( 30 ; 17 ; 35 ) . 34 proposed density estimation using infinite dimensional exponential families with score matching. 14 introduced yet another estimation method called noise contrastive estimation. 13 provided a unified framework based on the Bregman divergence that encompasses the score matching and noise contrastive estimation. 25 derived model selection criteria for non-normalized models. In settings with missing data, 38 combined imputation techniques with estimators for non-normalized models, and 12 utilized marginal score functions. 37 developed asymptotically efficient estimators that do not depend on the normalizing constant by employing density-ratio matching with a nonparametric density estimator. Despite the broad literature, the statistical efficiency of the original score matching has not been well understood. Using the isoperimetric constant, 19 analyzed the condition under which the score matching estimator is comparable to the maximum likelihood estimator. Our work provides a further characterization within the SMoM framework, and constructs an estimator that improves upon the score matching when the latter is not asymptotically efficient. Non-normalized models also naturally arise on more general domains, such as truncated domains in \mathbb{R}^{p} (e.g., the non-negative orthant \mathbb{R}_{+}^{p} ) or Riemannian manifolds (e.g., the unit sphere \mathbb{S}^{p-1} ). Score matching has been generalized to general domains by changing the metric \langle\,\cdot\,,\,\cdot\,\rangle to a weighted metric w\langle\,\cdot\,,\,\cdot\,\rangle ( 16 ; 23 , e.g.,) , or replacing differential operators with their manifold counterparts ( 7 ; 24 ; 41 , e.g.,) . SMoM also has been generalized in the same way ( 10 ; 11 , c.f.,) . In line with these generalizations, our results are extended to these settings; see Appendix C . For simplicity, we primarily focus on non-normalized models on \mathbb{R}^{p} in the main text. Our analysis is motivated by the fact that the score matching estimator corresponds to the SMoM estimator based on the test functions f_{\theta,j}\coloneqq\nabla_{x}\partial_{\theta_{j}}\log q_{\theta} ; see Lemma 1 . This lemma has been previously mentioned by 8 , 9 , and 20 ; it stands as a clue to investigate the relationship between score matching and the Stein operator, which has delivered various estimators. 9 extended Lemma 1 using the \gamma -divergence and developed the \gamma -score matching estimator, a robust estimator based on the \gamma -Stein operator. 4 proposed an estimation method based on Stein discrepancy and derived its relationship with score matching. Closely related to the present paper is 20 , where Lemma 1 together with the generalized method of moments (GMM) framework is employed to construct an estimator that improves upon score matching in exponential families. We emphasize that our geometric construction delivers a different perspective on score matching in general statistical models, and reveals an unexpected connection between score matching and the Wasserstein geometry. 1.2 Organization The rest of this paper is organized as follows. In Section 2 , we prepare notations and review the score matching and the SMoM. In Section 3 , we give our main results concerning

Read the complete text