On The Hidden Biases of Flow Matching Samplers

Soon Hoe Lim 1,2 1 Department of Mathematics, KTH Royal Institute of Technology 2 Nordita, KTH Royal Institute of Technology and Stockholm University · 2025

Read On The Hidden Biases of Flow Matching Samplers by Soon Hoe Lim 1,2 1 Department of Mathematics, KTH Royal Institute of Technology 2 Nordita, KTH Royal Institute of Technology and Stockholm University (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2512-16768.txt.

Introduction

Soon Hoe Lim studies hidden biases in flow matching samplers, a popular way to build generative models that carry a simple distribution into a complex target along smooth paths. The focus is on what changes when real, finite data replace the true target.

Lim lays out a ladder of such plug-in models, from the raw data measure to smoothed stand-ins. For affine flows he finds the exact best fit and a smoothed regime where the end law is a kernel mixture. Three biases stand out: the target itself shifts, the best fit may not be a gradient field, and one path can hide many particle motions. Tail bounds for kinetic energy hinge on the base distribution shape.

Catalog record

CREATORSoon Hoe Lim 1,2 1 Department of Mathematics, KTH Royal Institute of Technology 2 Nordita, KTH Royal Institute of Technology and Stockholm University
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE50 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2512-16768

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

On The Hidden Biases of Flow Matching Samplers Soon Hoe Lim 1,2 1 Department of Mathematics, KTH Royal Institute of Technology 2 Nordita, KTH Royal Institute of Technology and Stockholm University Abstract We study the implicit bias of flow matching (FM) samplers via the lens of empirical flow matching. Although population FM may produce gradient-field velocities resembling optimal transport (OT), we show that the empirical FM minimizer is generally not a gradient field, even when each conditional flow is. Consequently, empirical FM is intrinsically not OT-optimal in the Benamou-Brenier sense. In view of this, we analyze the kinetic energy of generated samples. With Gaussian sources, both instantaneous and integrated kinetic energies exhibit exponential concentration, while heavy-tailed sources lead to polynomial tails. These behaviors are governed primarily by the choice of source distribution rather than the data. Overall, these notes provide a concise mathematical account of the structural and energetic biases arising in empirical FM. 1 Introduction The main goal of generative modeling is to use finitely many samples from a distribution to construct a sampling scheme capable of generating new samples from the same distribution. Among the families of existing generative models, flow matching (FM) [ 27 , 28 ] is notable for its flexibility and simplicity. Given a target probability distribution, FM utilizes a parametric model (e.g., neural network) to learn the velocity vector field that defines a deterministic, continuous transformation (a normalizing flow) and transports a source probability distribution (e.g., standard Gaussian) to the target distribution. While the population formulation of FM often exhibits appealing structure—sometimes even admitting gradient-field velocities—practical models are trained on finite datasets and therefore optimize empirical objectives. This empirical setting substantially alters the geometry of the learned velocity field and the energetic properties of the resulting sampler. These notes aim to clarify how empirical FM behaves, how it differs from its population counterpart, and what implicit biases arise in the learned sampling dynamics. From now on, to keep the mathematical machinery light, we assume that all probability distributions/measures of the random variables considered (except the empirical distribution itself) are absolutely continuous, i.e., they admit densities with respect to the Lebesgue measure. We shall abuse notation and use the same symbol to denote both a distribution and its density. To maintain the flow of the main text, we defer the discussion of related work and all proofs of the theoretical results to the appendix. 2 Flow Matching (FM) and Conditional Flow Matching (CFM) Let p_{0} be the source distribution and p_{1} the target distribution (e.g., the data distribution p^{*} or a smoothed version) on \mathbb{R}^{d} . We say that T is a transport map if Z\sim p_{0} implies that T(Z)\sim p_{1} , in which case we write T_{\#}p_{0}=p_{1} , and there exist many such maps. A common generative modeling paradigm aims to learn a transport map T from p_{0} to p_{1} on \mathbb{R}^{d} using N i.i.d. samples x^{(i)}\sim p_{1} , where p_{1} is typically unknown [ 34 ] . One popular approach under this paradigm is flow matching (FM). FM. The goal of FM is to find a velocity field v:[0,1]\times\mathbb{R}^{d}\to\mathbb{R}^{d} , such that, if we solve the ODE: \frac{dz(t)}{dt}=v(t,z(t)),\ z(0)=z_{0}\in\mathbb{R}^{d}, then the law of z(1) when z_{0}\sim p_{0} is p_{1} (in which case we say that v drives p_{0} to p_{1} ). The law of z(t) for t\in[0,1] is described by a probability path p:[0,1]\times\mathbb{R}^{d}\to\mathbb{R} , denoted p_{t}(z) , that evolves from p_{0} at t=0 to p_{1} at t=1 . If we know v , then we can first sample z_{0}\sim p_{0} and then evolve the ODE from t=0 to t=1 to generate new samples. The velocity field v generates the flow \psi:[0,1]\times\mathbb{R}^{d} given as \psi_{t}(z)=z(t) , and the probability path via the push-forward distributions: p_{t}=[\psi_{t}]_{\#}p_{0} , i.e., \psi_{t}(Z)\sim p_{t} for Z\sim p_{0} . In particular, Z\sim p_{0} implies that \psi_{1}(Z)\sim p_{1} , i.e., \psi_{t} can be viewed as a dynamical transport map. The ODE corresponds to the Lagrangian description (the v -generated trajectories viewpoint), and a change of variables link it to the Eulerian description (the evolving probability path p_{t} viewpoint). Indeed, a necessary and sufficient condition for v to generate p_{t} is given by the continuity equation [ 2 ] : \frac{\partial p_{t}}{\partial t}+\nabla\cdot(p_{t}v)=0, (1) where \nabla\cdot denotes the divergence operator. This equation ensures that the flow defined by v conserves the mass (or probability) described by p_{t} . In general, even for p_{t} that linear interpolates between p_{0} and p_{1} , the velocity field does not admit a closed-form expression when p_{0} and p_{1} are known, except in special cases such as Gaussians, mixture of Gaussians and uniform distributions [ 33 ] . The above description gives us a population FM model, which we aim to learn using a finite number of samples in practice. Given such a v , it is standard to learn it with a parametric model v_{\theta} (e.g., neural network) by minimizing the FM objective: L_{\text{FM}}[v_{\theta}]=\mathbb{E}_{t\sim\mathcal{U}[0,1],\ Z_{t}\sim p_{t}}[\|v_{\theta}(t,Z_{t})-v(t,Z_{t})\|^{2}]. (2) CFM. In CFM [ 27 , 41 ] , we consider a probability path in the mixture form: p_{t}(z)=\int p_{t}(z|x)p_{1}(x)dx, (3) where p_{t}(\cdot|x):\mathbb{R}^{d}\to\mathbb{R}^{+} is a conditional probability path generated by some vector field v(t,\cdot|x):\mathbb{R}^{d}\to\mathbb{R}^{d} for x\in\mathbb{R}^{d} . Moreover, consider the vector field: v(t,z)=\int v(t,z|x)\frac{p_{t}(z|x)p_{1}(x)}{p_{t}(z)}dx. (4) In this setting, it can be shown in [ 27 ] that minimizing the FM objective L_{\text{FM}} is equivalent to minimizing the CFM objective: L_{\text{CFM}}[v_{\theta}]=\mathbb{E}_{t\sim\mathcal{U}[0,1],\ X\sim p_{1},\ Z_{t}\sim p_{t}(\cdot|X)}[\|v_{\theta}(t,Z_{t})-v(t,Z_{t}|X)\|^{2}]. (5) In order to apply CFM, we need to specify the boundary distributions p_{0} and p_{1} , and the conditional probability path p_{t}(z|x) . Below are some examples. Example 1 (Rectified Flow) . A canonical choice [ 30 ] is p_{0}=\mathcal{N}(0,I_{d}) , p_{1}=p^{*} , and p_{t}(z|X=x_{1})=\mathcal{N}(z;tx_{1},(1-t)^{2}I_{d}), (6) which corresponds to the conditional velocity field v(t,z|X=x_{1})=\frac{x_{1}-z}{1-t} . This conditional probability path realizes linear interpolating paths of the form Z_{t}=(1-t)x_{0}+tx_{1} between a (reference) Gaussian sample x_{0} and a data sample x_{1} . In practice, regularized versions of rectified flow are preferred for numerical stability (since v blows up as t\to 1 ). A simple version is to modify the conditional probability path to p_{t}(\cdot|X=x_{1})=\mathcal{N}(tx_{1},(1-(1-\sigma_{min})t)^{2}I_{d}), for some small \sigma_{min}>0 , which corresponds to the regularized conditional velocity field v(t,z|X=x_{1})=\frac{x_{1}-(1-\sigma_{min})z}{1-(1-\sigma_{min})t} . Another version is to consider a smoothed version of the data distribution p^{*} ; e.g., p_{1}=p^{*}\star\mathcal{N}(0,\sigma_{min}^{2}I_{d}) , where \star denotes convolution. Example 2 (Affine Flows) . More generally, consider a latent variable Z\sim\mathbb{Q} with probability density function (PDF) K (not necessarily Gaussian) and, for t\in[0,1] , the affine conditional flow defined by \psi_{t}(Z|X)=m_{t}(X)+\sigma_{t}(X)Z for some time-differentiable functions m:[0,1]\times\mathbb{R}^{d}\to\mathbb{R}^{d} and \sigma:[0,1]\times\mathbb{R}^{d}\to\mathbb{R}^{+} . Since \psi_{t} is linear in Z , we can obtain its density via the change of variables: p_{t}(z|X)=\frac{1}{\sigma_{t}^{d}(X)}K\left(\frac{z-m_{t}(X)}{\sigma_{t}(X)}\right). (7) Then, as in Theorem 3 in [ 27 ] , we can show that the unique vector field that defines \psi_{t}(\cdot|X) via the ODE \frac{d}{dt}\psi_{t}(z|X)=v(t,\psi_{t}(z|X)|X) has the form: v(t,z|X)=a_{t}(X)z+b_{t}(X), (8) where \displaystyle a_{t}(X) \displaystyle=\frac{\frac{\partial\sigma_{t}}{\partial t}(X)}{\sigma_{t}(X)},\quad b_{t}(X)=\frac{\partial m_{t}}{\partial t}(X)-m_{t}(X)a_{t}(X). (9) The rectified flow in previous example is a special case of this family of conditional flows (with K=\mathcal{N}(0,I_{d}) , m_{t}(X)=tX and \sigma_{t}(X)=1-t ). The Gaussian flows considered in [ 27 , 41 , 1 ] are also special cases. All the formulations thus far are in the idealized continuous-time setting. In practice, we work with Monte Carlo estimates

Read the complete text