Efficiently Access Diffusion Fisher: Within the Outer Product Span Space

Fangyikang Wang, Hubery Yin, Shaobin Zhuang, Huminhao Zhu, Yinan Li, Lei Qian, Chao Zhang, Hanbin Zhao, Hui Qian, Chen Li · 2025

Read Efficiently Access Diffusion Fisher: Within the Outer Product Span Space by Fangyikang Wang, Hubery Yin, Shaobin Zhuang, Huminhao Zhu, Yinan Li, Lei Qian, Chao Zhang, Hanbin Zhao, Hui Qian, Chen Li (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-23264.txt.

Introduction

Fangyikang Wang and co-authors find a fast path to a hard score in spread models that paint from noise. The score, called the diffusion Fisher, is the bend of a log mass curve, and it feeds odds tests and tune tasks. Stock craft bends the learned score net by auto steps, with no error bound and a slow toll. Their 2025 proof shows the score lives in a span built from the start data and the score's own outer goods.

That shape spawns two fast ways: one for the trace sum, one for the push of a vector. Both swap auto steps for cheap vector goods, with error bounds set in proof. Runs on odds tests and tune tasks beat the stock way on truth and cost. The shape also seeds the first check test for such scores.

Catalog record

CREATORFangyikang Wang, Hubery Yin, Shaobin Zhuang, Huminhao Zhu, Yinan Li, Lei Qian, Chao Zhang, Hanbin Zhao, Hui Qian, Chen Li
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE86 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-23264

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Efficiently Access Diffusion Fisher: Within the Outer Product Span Space Fangyikang Wang, Hubery Yin, Shaobin Zhuang, Huminhao Zhu, Yinan Li, Lei Qian, Chao Zhang, Hanbin Zhao, Hui Qian, Chen Li Abstract Recent Diffusion models (DMs) advancements have explored incorporating the second-order diffusion Fisher information (DF), defined as the negative Hessian of log density, into various downstream tasks and theoretical analysis. However, current practices typically approximate the diffusion Fisher by applying auto-differentiation to the learned score network. This black-box method, though straightforward, lacks any accuracy guarantee and is time-consuming. In this paper, we show that the diffusion Fisher actually resides within a space spanned by the outer products of score and initial data. Based on the outer-product structure, we develop two efficient approximation algorithms to access the trace and matrix-vector multiplication of DF, respectively. These algorithms bypass the auto-differentiation operations with time-efficient vector-product calculations. Furthermore, we establish the approximation error bounds for the proposed algorithms. Experiments in likelihood evaluation and adjoint optimization demonstrate the superior accuracy and reduced computational cost of our proposed algorithms. Additionally, based on the novel outer-product formulation of DF, we design the first numerical verification experiment for the optimal transport property of the general PF-ODE deduced map. 1 Introduction The emerging diffusion models (DMs) ( Sohl-Dickstein et al., 2015 ; Ho et al., 2020 ; Song & Ermon, 2019 ; Song et al., 2020 ) , generating samples of data distribution from initial noise by learning a reverse diffusion process, have been proven to be an effective technique for modeling data distribution, especially in generating high-quality images ( Nichol et al., 2022 ; Dhariwal & Nichol, 2021a ; Saharia et al., 2022 ; Ramesh et al., 2022 ; Rombach et al., 2022 ; Ho et al., 2022 ) . The training process of DMs can be seen as employing a neural network to match the first-order diffusion score \nabla_{\bm{x}}\log q_{t}({\bm{x}}) at varying noise levels. There has been a growing trend to recognize the importance of the diffusion Fisher (DF) in DMs, defined as the negative Hessian of the diffused distributions’ log density, -\nabla^{2}_{\bm{x}}\log q_{t}({\bm{x}}) . The DF provides valuable second-order information on DMs and plays a crucial role in downstream tasks, e.g., likelihood evaluation ( Lu et al., 2022a ; Zheng et al., 2023 ) , adjoint optimization ( Pan et al., 2023a ; Pan et al., 2023b ; Blasingame & Liu, 2024 ) , and the optimal transport analysis of DMs ( Zhang et al., 2024a ) . It is straightforward to access DFs by applying auto-differentiation to the learned first-order diffusion score. Nevertheless, the repeated auto-differentiation operations would be time-consuming for large-scale tasks, even when incorporating the Vector-Jacobian-product (VJP) and Hutchinson’s estimation technique ( Song & Lai, 2024 ) . Recently, certain research efforts investigated the time-evolving property of DM and demonstrated that the DF can be explicitly expressed in terms of the score and its covariance matrix ( Benton et al., 2024 ; Lu et al., 2022a ) . However, these results predominantly concentrate on the theoretical aspects and fail to enable efficient access to the DF. In this paper, we derive a novel form for the diffusion Fisher, which is composed of weighted outer-product sums. This innovative formulation uncovers that the diffusion Fisher invariably lies within the span of a set of outer-product bases. Notably, these bases solely depend on the initial data and the noise schedule. Leveraging this outer-product structure, we devise two efficient approximation algorithms to access the trace and matrix-vector multiplication of DF, respectively. Moreover, we design the first numerical verification experiment to investigate the optimal transport property of the diffusion-ODE deduced map under diverse types of initial data and noise schedules. The main contributions of our paper are listed as follows: • We derive an analytical formulation of the diffusion Fisher, which is the weighted summation of the outer product, under the setting when the initial distribution is a sum of Dirac. The derivation process involves applying the consecutive partial differential chain rule to the data-dependent form of marginal distributions. Subsequently, we extend this formulation to a more general setting. In this general case, we only assume that the initial distribution has a finite second moment. These novel formulations reveal that the diffusion Fisher invariably lies within the span of a set of outer-product bases. • Based on the outer-product structure of DF, we propose two efficient approximation algorithms, each tailored to access the trace and matrix-vector multiplication of the DF, respectively. When it comes to evaluating the trace of the diffusion Fisher, we introduce a parameterized network to learn the trace, significantly reducing the time complexity of trace evaluation from quadratic to linear w.r.t. the dimension. In situations where we need to access the matrix-vector multiplication of the diffusion Fisher, we present a training-free method that simplifies the complex linear transformation into several simple vector-product calculations. Furthermore, we rigorously establish the approximation error bounds for these two algorithms. • Utilizing the novel outer-product formulation of DF, we derive a corollary for the optimal transport (OT) property of the diffusion-ODE deduced map. Subsequently, we design the first numerical experiment to examine the OT property of any general diffusion ODE deduced map based on the corollary. Our numerical experiments indicate that for all commonly employed noise schedules (VE ( Ho et al., 2020 ) , VP ( Song & Ermon, 2019 ) , sub-VP ( Song et al., 2020 ) , and EDM ( Karras et al., 2022 ) ), the diffusion ODE map exhibits the OT property when the initial data follows a single Gaussian distribution or has an affine form. Conversely, it does not exhibit this property when the initial data is non-affine. These numerical results not only validate the conclusions presented in ( Khrulkov et al., 2023 ; Zhang et al., 2024a ; Lavenant & Santambrogio, 2022 ) , but also open up new avenues for the exploration of more general cases. We evaluate our DF access algorithms on likelihood evaluation and adjoint optimization tasks. The empirical results demonstrate our DF methods’ enhanced accuracy and reduced time cost. 2 Preliminaries Notation . The Euclidean norm over {\mathbb{R}}^{d} is denoted by \norm{\cdot} , and the Euclidean inner product is denoted by \innerproduct{\cdot}{\cdot} . Throughout, we simply write \int g to denote the integral with respect to the Lebesgue measure: \int g(x){\textnormal{d}}x . When the integral is with respect to a different measure \mu , we explicitly write \int g{\textnormal{d}}\mu . When clear from context, we sometimes abuse notation by identifying a measure \mu with its Lebesgue density. We also use \delta(\cdot) to denote the Dirac Delta function. For a vector {\bm{v}}\in{\mathbb{R}}^{d} , we denote the d\times d matrix {\bm{v}}{\bm{v}}^{\top} as the outer-product of {\bm{v}} . 2.1 Diffusion Models and Diffusion SDEs Suppose that we have a d-dimensional random variable {\bm{x}}_{0}\in{\mathbb{R}}^{d} following an unknown target distribution q_{0}({\bm{x}}_{0}) . Diffusion Models (DMs) define a forward process \{{\bm{x}}_{t}\}_{t\in[0,T]} with T>0 starting with {\bm{x}}_{0} , such that the distribution of {\bm{x}}_{t} conditioned on {\bm{x}}_{0} satisfies \displaystyle q_{t|0}({\bm{x}}_{t}|{\bm{x}}_{0})=\mathcal{N}({\bm{x}}_{t};\alpha(t){\bm{x}}_{0},\sigma^{2}(t)\mathbf{I}), (1) where \alpha(\cdot),\sigma(\cdot)\in\mathcal{C}(\left[0,T\right],\mathbb{R}^{+}) have bounded derivatives, and we denote them as \alpha_{t} and \sigma_{t} for simplicity. The choice for \alpha_{t} and \sigma_{t} is referred to as the noise schedule of a DM. According to (

Read the complete text