Exploring Diffusion and Flow Matching Under Generator Matching
Zeeshan Patel, James DeLoye, Lance Mathias · 2024
Read Exploring Diffusion and Flow Matching Under Generator Matching by Zeeshan Patel, James DeLoye, Lance Mathias (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-11024.txt.
Introduction
Zeeshan Patel, James DeLoye, and Lance Mathias compare two leading ways to train image-generating models, diffusion and flow matching, inside one shared framework. The two methods look different: one learns to remove noise step by step, the other learns a flow that carries noise into data. A recent framework, called Generator Matching, claims both are cases of one scheme.
Recasting both methods in that single frame, the authors derive why flow matching often proves more robust in practice. They also show how mixing steady and random parts creates new model types that sit between the two. The comparison gives builders a clearer map of the design space for generative models.
Catalog record
| CREATOR | Zeeshan Patel, James DeLoye, Lance Mathias |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 23 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2412-11024
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Exploring Diffusion and Flow Matching Under Generator Matching Zeeshan Patel, James DeLoye, Lance Mathias Abstract In this paper, we present a comprehensive theoretical comparison of diffusion and flow matching under the Generator Matching framework. Despite their apparent differences, both diffusion and flow matching can be viewed under the unified framework of Generator Matching. By recasting both diffusion and flow matching under the same generative Markov framework, we provide theoretical insights into why flow matching models can be more robust empirically and how novel model classes can be constructed by mixing deterministic and stochastic components. Our analysis offers a fresh perspective on the relationships between state-of-the-art generative modeling paradigms. 1 Introduction Recent techniques in deep generative modeling have leveraged Markov generative processes to learn complex, high-dimensional probability distributions in a more structured and flexible manner [ 17 ] . By integrating Markov chain methods with deep neural architectures, these approaches aim to exploit the representational power of deep networks while maintaining a tractable and theoretically grounded training procedure. In contrast to early generative models that relied heavily on direct maximum likelihood estimation or adversarial objectives, this class of methods employs iterative stochastic transformations—often expressed as Markovian updates—to gradually refine initial noise samples into samples drawn from the desired target distribution. Diffusion and flow matching models represent two prominent classes of generative approaches that construct data samples through a sequence of continuous transformations. Diffusion models [ 6 , 13 ] introduce a forward-noising and reverse-denoising process, progressively refining a simple noise distribution into a complex target distribution by learning to undo incremental noise corruption at each step. Flow matching models [ 10 , 11 , 12 ] , on the other hand, directly learn continuous-time transformations that morph a base distribution into the target distribution under a prescribed flow field. Both families benefit from well-defined likelihoods and stable training objectives, allowing for clearer theoretical insights, improved sample quality, and often more reliable convergence than prior approaches such as GANs [ 3 , 5 ] . Generator Matching [ 7 ] is a framework that unifies generative modeling with Markov processes on arbitrary state spaces. This framework allows combining different Markov processes in two ways: Markov superposition and creating multimodal generative models by combining unimodal generators. In this work, we aim to leverage the Generator Matching framework to provide a detailed theoretical comparison of diffusion and flow matching models. We show we aim to provide an overview of Generator Matching, how it connects to diffusion and flow matching models, and how specific properties of some Markov generative process make them more robust than others. 2 Connecting Diffusion and Flow Matching Diffusion and Flow Matching are two different generative Markov processes that have many similarities. In this section, we briefly define diffusion and flow matching models and provide some intuition on how diffusion and Gaussian flow matching models are theoretically equivalent. 2.1 Diffusion Models Diffusion models learn to reverse a time-dependent destructive process that corrupts data x by adding noise [ 6 ] . In practice, the learned component of a diffusion model is a neural network f_{\theta} that tries to estimate the denoised data given the noisy sample \mathbf{z_{t}}=\alpha_{t}\mathbf{x}+\sigma_{t}\boldsymbol{\epsilon} at time t , where typically \boldsymbol{\epsilon}\sim\mathcal{N}(0,\mathbf{I}) is Gaussian noise and (\alpha_{t},\sigma_{t}) define the noise schedule. We can generate samples from diffusion models by reversing the forward corruption process. We initialize our sample as pure Gaussian noise, \mathbf{z_{1}} , and we estimate the sample at time t by using a neural network to compute \mathbf{\hat{x_{t}}}\sim f_{\theta}(\mathbf{z_{t}};t) . Then, we perform the same forward process again at a lower noise level r where r<t such that \mathbf{z_{r}}=\alpha_{r}\mathbf{\hat{x}}+\sigma_{r}\mathbf{\hat{\epsilon}} , where \hat{\boldsymbol{\epsilon}}=(\mathbf{z_{t}}-\alpha_{t}\hat{\mathbf{x}})/\sigma_{t} . After repeatedly executing this process of predicting the clean sample and performing the forward process at a lower noise level, we finally arrive at the clean sample. In this algorithm, the randomness is only induced from the initial sample \mathbf{z_{1}} , and the rest of the reverse sampling process is deterministic. In practice, this is the DDIM sampling technique [ 14 ] , which is used for efficient sampling from diffusion models without performing the reverse process across the entire noise schedule. We can also formulate the forward corruption process of diffusion models with a stochastic differential equation (SDE): d\mathbf{z_{t}}=f_{t}\mathbf{z_{t}}dt+g_{t}d\mathbf{z}, where d\mathbf{z} is a Brownian motion that characterizes an infinitesimal Gaussian, and (f_{t},g_{t}) are the parameters of the noise schedule [ 4 , 8 , 15 ] . To sample from the model, we perform the reverse process using the score \nabla\log p_{t} of the forward process as the following: d\mathbf{z_{t}}=\left(f_{t}\mathbf{z_{t}}-\frac{1+\eta_{t}^{2}}{2}g_{t}^{2}\nabla\log p_{t}(\mathbf{z_{t}})\right)dt+\eta_{t}g_{t}d\mathbf{z}. This formulation contains an additional parameter \eta_{t} which controls the level of stochasticity at inference time. In practice, this term is also related to the “churn,” which is the fraction of the reverse step to undo by renoising [ 8 ] . 2.2 Flow Matching Flow matching models learn a transformation function that maps data from a simple base distribution to the complex target data distribution by matching the flow of probability densities [ 7 , 10 ] . Flow matching views the forward noising process as a linear interpolation between the data \mathbf{x} and noise \boldsymbol{\epsilon} : \mathbf{z_{t}}=t\boldsymbol{\epsilon}+(1-t)\mathbf{x}. We can see that if we set \alpha_{t}=1-t and \sigma_{t}=t , then this formulation is equivalent to the Gaussian diffusion forward process [ 1 , 4 ] . Assume that our initial sample \mathbf{z_{1}} is pure Gaussian noise. If we let r<t , then we can derive \mathbf{z_{t}}=\mathbf{z_{r}}+(t-r)\mathbf{v} , where \mathbf{v}=\boldsymbol{\epsilon}-\mathbf{x} , which is the “flow” or “vector field” learned by a neural network f_{\theta} . We can sample \mathbf{z_{r}} from \mathbf{z_{t}} by simply going reverse in time and estimating the flow at time t as \hat{\mathbf{v}}=f_{\theta}(\mathbf{z_{t}};t)=\hat{\boldsymbol{\epsilon}}-\hat{\mathbf{x}} . We then compute \mathbf{z_{r}} as \mathbf{z_{r}}=\mathbf{z_{t}}+(r-t)\hat{\mathbf{v}}, and we repeatedly perform this reverse process starting with \mathbf{z_{1}} till we reach the clean sample. We can also formulate flow matching with an ordinary differential equation (ODE) [ 10 ] : d\mathbf{z_{t}}=\mathbf{u_{t}}dt, where \mathbf{z_{t}}=\alpha_{t}\mathbf{x}+\sigma_{t}\boldsymbol{\epsilon} and \mathbf{u_{t}}=\dot{\alpha_{t}}\mathbf{x}+\dot{\sigma_{t}}\boldsymbol{\epsilon} . To generate from the model, we simply reverse the ODE in time. However, in practice, we typically also include a score-correction term since we are approximating \mathbf{u_{t}} . Formally, the reverse-time probability flow ODE to generate samples is: d\mathbf{z_{t}}=\left(\mathbf{u_{t}}-\frac{1}{2}\epsilon_{t}^{2}\nabla\log p_{t}(\mathbf{z_{t}})\right)dt, where \epsilon_{t}^{2} is now a deterministic scaling term. We can also write it as an SDE under a special case of stochastic interpolants [ 1 , 2 ] : d\mathbf{z_{t}}=\left(\mathbf{u_{t}}-\frac{1}{2}\epsilon_{t}^{2}\nabla\log p_{t}(\mathbf{z_{t}})\right)dt+\epsilon_{t}d\mathbf{z}, where in this formulation, \epsilon_{t} can be tuned to control the level of stochasticity during generation. 2.3 Equivalence of Diffusion and Flow Matching One way to show equivalence between diffusion and flow matching is by deriving the parameters of one generative Markov process from the other. This is simpler to derive using the SDE / ODE perspective of both diffusion / flow matching [ 1 , 2 , 4 ] . In diffusion, we found an SDE parameterized by f_{t},g_{t},\eta_{t} . Consider the deterministic evolution that would occur if we only followed the drift f_{t}\mathbf{z_{t}} without noise. If we start with \mathbf{z_{0}}=\mathbf{x} and have it evolve under the ODE \mathbf{\dot{z}_{t}}=f_{t}\mathbf{z_{t}} then \mathbf{z_{t}}=\alpha_{t}\mathbf{x} . Thus, \alpha_{t} encodes how a deterministic initial condition would scale over time, so we can formulate it as: \alpha_{t}=\exp\left(\int_{0}^{t}f_{r}dr\right). Next, we consider how noise accumulates relative to