An explicit formulation of the learned noise predictor ϵ θ ​ ( 𝐱 t , t ) \epsilon_{\theta}({\bf x}_{t},t) via the forward-process noise ϵ t \epsilon_{t} in denoising diffusion probabilistic models (DDPMs)

KiHyun Yun · 2025

Read An explicit formulation of the learned noise predictor ϵ θ ​ ( 𝐱 t , t ) \epsilon_{\theta}({\bf x}_{t},t) via the forward-process noise ϵ t \epsilon_{t} in denoising diffusion probabilistic models (DDPMs) by KiHyun Yun (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-04203.txt.

Introduction

KiHyun Yun writes out a learned noise gauge in terms of the noise fed in at the front of a spread model. The work dates to 2025. Such models learn to guess the front noise from a blurred input. A known slope tie links that guess to the chance curve of the input.

His form shows just how the front noise feeds the learned guess, step by step. From that form he builds a fresh strict proof of the slope tie. The proof tells where the tie comes from, not just that it holds. The work adds firm ground to the math of spread-based make models.

Catalog record

CREATORKiHyun Yun
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE14 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2507-04203

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

An explicit formulation of the learned noise predictor ϵ θ ​ ( 𝐱 t , t ) \epsilon_{\theta}({\bf x}_{t},t) via the forward-process noise ϵ t \epsilon_{t} in denoising diffusion probabilistic models (DDPMs) KiHyun Yun Abstract In denoising diffusion probabilistic models (DDPMs), the learned noise predictor ϵ θ ​ ( 𝐱 t , t ) \epsilon_{\theta}({\bf x}_{t},t) is trained to approximate the forward-process noise ϵ t \epsilon_{t} . The equality ∇ 𝐱 t ​ log ​ q ​ ( 𝐱 t ) = − 1 1 − α ¯ t ​ ϵ θ ​ ( 𝐱 t , t ) \nabla_{{\bf x}_{t}}\log q({\bf x}_{t})=-\frac{1}{\sqrt{1-{\bar{\alpha}}_{t}}}\epsilon_{\theta}({\bf x}_{t},t) plays a fundamental role in both theoretical analyses and algorithmic design, and thus is frequently employed across diffusion-based generative models. In this paper, an explicit formulation of ϵ θ ​ ( 𝐱 t , t ) \epsilon_{\theta}({\bf x}_{t},t) in terms of the forward-process noise ϵ t \epsilon_{t} is derived. This result show how the forward-process noise ϵ t \epsilon_{t} contributes to the learned predictor ϵ θ ​ ( 𝐱 t , t ) \epsilon_{\theta}({\bf x}_{t},t) . Furthermore, based on this formulation, we present a novel and mathematically rigorous proof of the fundamental equality above, clarifying its origin and providing new theoretical insight into the structure of diffusion models. 1 Introduction Denoising diffusion probabilistic models (DDPMs) is a basic framework for generative modeling, wherein a neural network \epsilon_{\theta}({\bf x}_{t},t) is trained to predict the noise \epsilon_{t} added in a forward diffusion process. In this paper, we use the definitions and notations as presented in DDPM paper [ 3 ] . A fundamental identity that appears repeatedly in diffusion-based generative models is \nabla_{{\bf x}_{t}}\log q({\bf x}_{t})=-\frac{1}{\sqrt{1-{\bar{\alpha}}_{t}}}\epsilon_{\theta}({\bf x}_{t},t) (1.1) which connects the score function of the marginal distribution q({\bf x}_{t}) to the learned noise predictor \epsilon_{\theta}({\bf x}_{t},t) . Refer to [ 2 , 4 , 5 ] . This identity can be derived via Tweedie’s formula, or alternatively as a byproduct of the reverse-time SDE framework established by Anderson [ 1 ] . Also refer to [ 6 ] . However, both derivations do not clearly show how the forward-process noise \epsilon_{t} contributes to the learned predictor \epsilon_{\theta}({\bf x}_{t},t) , and why it ends up matching the score function of the marginal distribution q({\bf x}_{t}) precisely. In this work, we investigate this connection and provide an explicit formulation that expresses \epsilon_{\theta}({\bf x}_{t},t) as the conditional expectation of the forward-process noise \epsilon_{t} under the posterior q({\bf x}_{0}|{\bf x}_{t}) . This perspective not only yields a novel and mathematically rigorous proof of the identity above, but also offers a clearer understanding of how the denoising objective implicitly captures the score of the marginal distribution. Our result describes the mathematical structure underlying DDPMs and reinforces the interpretation of noise prediction as a form of conditional averaging. We begin by clarifying the definition of \epsilon_{\theta}({\bf x}_{t},t) . We use the definitions and notations as presented in DDPM paper [ 3 ] , and thus refer to the paper [ 3 ] for details. Let {\bf x}_{0} be a sample from the data distribution q({\bf x}_{0}) . The forward diffusion process generates a sequence of latent variables {\bf x}_{1},~\cdots,{\bf x}_{T} by gradually adding Gaussian noise. For each timestep t=1,\cdots,T , the variable {\bf x}_{t} in defined by {\bf x}_{t}=\sqrt{{\bar{\alpha}}_{t}}{\bf x}_{0}+\sqrt{1-{\bar{\alpha}}_{t}}\epsilon_{t},~\mbox{where}~\epsilon_{t}\sim{\mathcal{N}}({\bf 0},\mathbb{I}). Here, {\bar{\alpha}}_{t}\in(0,1) is the cumulative product of the noise schedule. It is worthy to note that \epsilon_{t} is given by {\bf x}_{0} and {\bf x}_{t} . Thus, we define a formal notation \epsilon_{t}({\bf x}_{t}|{\bf x}_{0}) for \epsilon_{t} above as \epsilon_{t}({\bf x}_{t}|{\bf x}_{0})=\epsilon_{t}. From the definition of \epsilon_{\theta}(\cdot,t) in DDPM, we find the parameter \tilde{\theta} minimizing the expection \displaystyle\mathbb{E}_{{\bf x}_{0}\sim q({\bf x}_{0})}\left[\mathbb{E}_{\epsilon_{t}\sim{\mathcal{N}}({\bf 0},\mathbb{I})}\left[\left\|\epsilon_{t}-\epsilon_{\tilde{\theta}}\left(\sqrt{{\bar{\alpha}}_{t}}{\bf x}_{0}+\sqrt{1-{\bar{\alpha}}_{t}}\epsilon_{t},t\right)\right\|^{2}\right]\right]. We define the function \epsilon_{\theta}(\cdot,t) of {\bf x}_{t} as \displaystyle\epsilon_{\theta}(\cdot,t)=f_{*}(\cdot) \displaystyle=\mbox{argmin}_{f(\cdot)}\mathbb{E}_{{\bf x}_{0}\sim q({\bf x}_{0})}\left[\mathbb{E}_{\epsilon_{t}\sim{\mathcal{N}}({\bf 0},\mathbb{I})}\left[\left\|\epsilon_{t}-f\left(\sqrt{{\bar{\alpha}}_{t}}{\bf x}_{0}+\sqrt{1-{\bar{\alpha}}_{t}}\epsilon_{t}\right)\right\|^{2}\right]\right]. Please note that the above f_{*} is not used commonly regardless of the choice of t . There is a different function f_{*} for each t . Theorem 1.1 Let \epsilon_{\theta}(\cdot,t) and \epsilon_{t}({\bf x}_{t}|{\bf x}_{0}) be as given above. Then, we have \epsilon_{\theta}({\bf x}_{t},t)=\int_{\Omega({\bf X}_{0})}\epsilon_{t}({\bf x}_{t}|{\bf x}_{0})q({\bf x}_{0}|{\bf x}_{t})d{\bf x}_{0}. where \Omega({\bf X}_{0}) is the domain of {\bf X}_{0} . The proof is presented in Section 2. The representation of \epsilon_{\theta}({\bf x}_{t},t) is reasonable by noting that, since \epsilon_{\theta}({\bf x}_{t},t) is a function of {\bf x}_{t} , it should be considered under the condition with {\bf x}_{t} is given, and consequently, the distribution of q({\bf x}_{0}|{\bf x}_{t}) is used to describe \epsilon_{\theta}({\bf x}_{t},t) . The preceding theorem provides a natural derivation of the fundamental identity ( 1.1 ), which is also stated in the following corollary. Corollary 1.2 \nabla_{{\bf x}_{t}}\log q({\bf x}_{t})=-\frac{1}{\sqrt{1-{\bar{\alpha}}_{t}}}\epsilon_{\theta}({\bf x}_{t},t). The equality is found in (11) in [ 2 ] . The following provides a new derivation the equality above derived from Theorem 1.1 . Proof of Corollary 1.2 . We begin by considering \epsilon_{t}({\bf x}_{t}|{\bf x}_{0})=\epsilon_{t},~\mbox{ and }{\bf x}_{t}=\sqrt{{\bar{\alpha}}_{t}}{\bf x}_{0}+\sqrt{1-{\bar{\alpha}}_{t}}\epsilon_{t}, where \epsilon_{t}\sim{\mathcal{N}}({\bf 0},\mathbb{I}). Then we have \displaystyle\frac{{\bf x}_{0}-\sqrt{{\bar{\alpha}}_{t}}{{\bf x}_{0}}}{\sqrt{1-{{\bar{\alpha}}_{t}}}} \displaystyle=\epsilon_{t}({\bf x}_{t}|{\bf x}_{0})\sim{\mathcal{N}}({\bf 0},\mathbb{I}), (1.2) \displaystyle q({\bf x}_{t}|{\bf x}_{0}) \displaystyle=p(\epsilon_{t}({\bf x}_{t}|{\bf x}_{0})|{\bf x}_{0})={\mathcal{N}}(\epsilon_{t}({\bf x}_{t}|{\bf x}_{0})|{\bf 0},\mathbb{I}). (1.3) In details, \displaystyle q({\bf x}_{t}|{\bf x}_{0}) \displaystyle={\mathcal{N}}({\bf x}_{t}|\sqrt{{\bar{\alpha}}_{t}}{{\bf x}_{0}},(1-{\bar{\alpha}}_{t})\mathbb{I})=\frac{1}{Z}\exp\left(-{\frac{1}{2(1-{\bar{\alpha}}_{t})}}\left\|{\bf x}_{t}-\sqrt{{\bar{\alpha}}_{t}}{{\bf x}_{0}}\right\|^{2}\right). Then, by the relation ( 1.2 ), \displaystyle\nabla_{{\bf x}_{t}}\log q({\bf x}_{t}|{\bf x}_{0}) \displaystyle=-{\frac{1}{1-{\bar{\alpha}}_{t}}}\left({\bf x}_{t}-\sqrt{{\bar{\alpha}}_{t}}{{\bf x}_{0}}\right) \displaystyle=-{\frac{1}{\sqrt{1-{\bar{\alpha}}_{t}}}}\epsilon_{t}({\bf x}_{t}|{\bf x}_{0}). (1.4) Applying ( 1.4 ) to the result of Theorem 1.1 , \displaystyle\epsilon_{\theta}({\bf x}_{t},t)q({\bf x}_{t}) \displaystyle=\left(\int_{\Omega{({\bf x}_{0})}}\epsilon_{t}({\bf x}_{t}|{\bf x}_{0})q({\bf x}_{0}|{\bf x}_{t})d{\bf x}_{0}\right)q({\bf x}_{t}) \displaystyle=\int_{\Omega{({\bf x}_{0})}}\epsilon_{t}({\bf x}_{t}|{\bf x}_{0})q({\bf x}_{t},{\bf x}_{0})d{\bf x}_{0} \displaystyle=\int_{\Omega{({\bf x}_{0})}}-{\sqrt{1-{\bar{\alpha}}_{t}}}\Big(\nabla_{{\bf x}_{t}}\log q({\bf x}_{t}|{\bf x}_{0})\Big)q({\bf x}_{t}|{\bf x}_{0})q({\bf x}_{0})d{\bf x}_{0} \displaystyle=-{\sqrt{1-{\bar{\alpha}}_{t}}}\int_{\Omega{({\bf x}_{0})}}\left(\frac{\nabla_{{\bf x}_{t}}q({\bf x}_{t}|{\bf x}_{0})}{q({\bf x}_{t}|{\bf x}_{0})}\right)q({\bf x}_{t}|{\bf x}_{0})q({\bf x}_{0})d{\bf x}_{0} \displaystyle=-{\sqrt{1-{\bar{\alpha}}_{t}}}\nabla_{{\bf x}_{t}}\int_{\Omega{({\bf x}_{0})}}\left(\frac{q({\bf x}_{t},{\bf x}_{0})}{q({\bf x}_{0})}\right)q({\bf x}_{0})d{\bf x}_{0} \displaystyle=-{\sqrt{1-{\bar{\alpha}}_{t}}}\nabla_{{\bf x}_{t}}\int_{\Omega{({\bf x}_{0})}}{q({\bf x}_{t},{\bf x}_{0})}d{\bf x}_{0} \displaystyle=-{\sqrt{1-{\bar{\alpha}}_{t}}}\nabla_{{\bf x}_{t}}q({\bf x}_{t}). Then, \epsilon_{\theta}({\bf x}_{t},t)=-{\sqrt{1-{\bar{\alpha}}_{t}}}\frac{\nabla_{{\bf x}_{t}}q({\bf x}_{t})}{q({\bf x}_{t})}=-{\sqrt{1-{\bar{\alpha}}_{t}}}\nabla_{{\bf x}_{t}}\log q({\bf x}_{t}). Thus, we have this corollary. \square 2 Proof of the theorem 1.1 Although a fully rigorous proof would require explicitly specifying the appropriate function space and proceeding accordingly, we have chosen instead to present a concise proof that reveals the core idea behind the argument. This approach is intended to make the content more accessible to a broader readers, including readers without a background in mathematics. From the definition of \epsilon_{t}({\bf x}_{t}|{\bf x}_{0}) , the expectation to be minimized for defining the function \epsilon_{\theta}({\bf x}_{t},t) can be reformulated and understood in the following form: \displaystyle\mathbb{E}_{{\bf x}_{0}\sim q({\bf x}_{0})}\left[\mathbb{E}_{\epsilon_{t}=\epsilon_{t}({\bf x}_{t}|{\bf x}_{0})\sim{\mathcal{N}}({\bf 0},\mathbb{I})}\left[\left\|\epsilon_{t}-\epsilon_{\theta}\left(\sqrt{{\bar{\alpha}}_{t}}{\bf x}_{0}+\sqrt{1-{\bar{\alpha}}_{t}}\epsilon_{t},t\right)\right\|^{2}\right]\right] \displaystyle=\mathbb{E}_{{\bf x}_{0}\sim q({\bf x}_{0})}\left[\mathbb{E}_{\epsilon_{t}\sim{\mathcal{N}}({\bf 0},\mathbb{I})}\left[\left\|\epsilon_{t}-\epsilon_{\theta}\left({\bf x}_{t},t\right)\right\|^{2}~\big|~{\bf x}_{t}=\sqrt{{\bar{\alpha}}_{t}}{\bf x}_{0}+\sqrt{1-{\bar{\alpha}}_{t}}\epsilon_{t}\right]\right], and the function \epsilon_{\theta}\left(\cdot,t\right) of {\bf x}_{t} is a function f_{*}(\cdot) minimizing the expectation as follows: \displaystyle\epsilon_{\theta}\left(\cdot,t\right)=f_{*}(\cdot) (2.1) \displaystyle=\mbox{argmin}_{f(\cdot)} \displaystyle\mathbb{E}_{{\bf x}_{0}\sim q({\bf x}_{0})}\left[\mathbb{E}_{\epsilon_{t}({\bf x}_{t}|{\bf x}_{0})\sim{\mathcal{N}}({\bf 0},\mathbb{I})}\left[\left\|\epsilon_{t}({\bf x}_{t}|{\bf x}_{0})-f\left({\bf x}_{t}\right)\right\|^{2}~\big|~{\bf x}_{t}=\sqrt{{\bar{\alpha}}_{t}}{\bf x}_{0}+\sqrt{1-{\bar{\alpha}}_{t}}\epsilon_{t}({\bf x}_{t}|{\bf x}_{0})\right]\right] \displaystyle=\mbox{argmin}_{f(\cdot)}\mathbb{E}_{{\bf x}_{0}\sim q({\bf x}_{0})}\left[\mathbb{E}_{{\bf x}_{t}\sim q({\bf x}_{t}|{\bf x}_{0})}\left[\left\|\epsilon_{t}({\bf x}_{t}|{\bf x}_{0})-f\left({\bf x}_{t}\right)\right\|^{2}\right]\right]. Let h(\cdot) be arbitrary function, and s be a real number. Then, (f_{*}+sh)({\bf x}_{t})=f_{*}({\bf x}_{t})+sh({\bf x}_{t}). At s=0 , the function (f_{*}+sh)({\bf x}_{t}) minimizes the following expection F_{h}(s)=\mathbb{E}_{{\bf x}_{0}\sim q({\bf x}_{0})}\left[\mathbb{E}_{{\bf x}_{t}\sim q({\bf x}_{t}|{\bf x}_{0})}\left[\left\|\epsilon_{t}({\bf x}_{t}|{\bf x}_{0})-(f_{*}+sh)\left({\bf x}_{t}\right)\right\|^{2}\right]\right]. Let \Omega({\bf x}_{t}) be the set of all {\bf x}_{t} , and it is the common domain of f_{*} and h . Then, \displaystyle 0 \displaystyle=\frac{dF_{h}}{ds}(0) \displaystyle=\mathbb{E}_{{\bf x}_{0}\sim q({\bf x}_{0})}\left[\mathbb{E}_{{\bf x}_{t}\sim q({\bf x}_{t}|{\bf x}_{0})}\Big[2\left(\epsilon_{t}({\bf x}_{t}|{\bf x}_{0})-f_{*}\left({\bf x}_{t}\right)\right)h({\bf x}_{t})\Big]\right] \displaystyle=2\int_{\Omega{({\bf x}_{0})}}\left(\int_{\Omega_{({\bf x}_{t})}}\left(\epsilon_{t}({\bf x}_{t}|{\bf x}_{0})-f_{*}\left({\bf x}_{t}\right)\right)h({\bf x}_{t})q({\bf x}_{t}|{\bf x}_{0})d{\bf x}_{t}\right)q({\bf x}_{0})d{\bf x}_{0} \displaystyle=2\int_{\Omega{({\bf x}_{t})}}\left(\int_{\Omega_{({\bf x}_{0})}}\left(\epsilon_{t}({\bf x}_{t}|{\bf x}_{0})-f_{*}\left({\bf x}_{t}\right)\right)q({\bf x}_{t}|{\bf x}_{0})q({\bf x}_{0})d{\bf x}_{0}\right)h({\bf x}_{t})d{\bf x}_{t} \displaystyle=2\int_{\Omega{({\bf x}_{t})}}g({\bf x}_{t})h({\bf x}_{t})d{\bf

Read the complete text