Training Latent Diffusion Models with Interacting Particle Algorithms
Tim Y. J. Wang, Juan Kuntz, O. Deniz Akyildiz · 2025
Read Training Latent Diffusion Models with Interacting Particle Algorithms by Tim Y. J. Wang, Juan Kuntz, O. Deniz Akyildiz (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-12412.txt.
Introduction
Tim Wang, Juan Kuntz, and Deniz Akyildiz train latent diffusion models, the hidden-space engines behind image generators, with swarms of interacting particles. Their paper appeared in two thousand twenty-five. Training such models usually leans on variational shortcuts that can blur the fit. A fresh angle treats training as lowering a free energy.
That view yields a flow that slides downhill on the energy surface. The team mimics the flow with a set of particles that push and pull on each other. The result is a training algorithm with proven error bounds. In tests it held its own against earlier particle methods and variational rivals. The approach suits latent models where the hidden space does the heavy lifting. It also ties generative modeling to classic physics-style methods.
Catalog record
| CREATOR | Tim Y. J. Wang, Juan Kuntz, O. Deniz Akyildiz |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 65 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-12412
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Training Latent Diffusion Models with Interacting Particle Algorithms Tim Y. J. Wang, Juan Kuntz, O. Deniz Akyildiz Abstract We introduce a novel particle-based algorithm for end-to-end training of latent diffusion models. We reformulate the training task as minimizing a free energy functional and obtain a gradient flow that does so. By approximating the latter with a system of interacting particles, we obtain the algorithm, which we underpin theoretically by providing error guarantees. The novel algorithm compares favorably in experiments with previous particle-based methods and variational inference analogues. 1 Introduction Diffusion Models ( DM s) introduced in Sohl-Dickstein et al., 2015 , and further developed in Ho et al., 2020 ; Song et al., 2021c , excel in numerous generative modelling tasks. Examples include image synthesis ( Dhariwal and Nichol, 2021 ) , protein design ( Watson et al., 2023 ) , and language modelling ( Nie et al., 2025 ) . They work by progressively adding noise to data to transform the data distribution into an easy-to-sample reference distribution, and then learn to revert this noising process. However, these steps take place in the data’s ambient space which is typically high dimensional. For this reason, DM training and inference proves computationally expensive. To alleviate this issue, Rombach et al., 2022 ; Vahdat et al., 2021 ; Wehenkel and Louppe, 2021 and others proposed using a Variational Autoencoder ( VAE ) ( Kingma and Welling, 2013 ) to map back-and-forth between the high-dimensional ambient space and a low-dimensional latent space, and carrying out the noising/de-noising steps in the latter. To date, the world’s most popular DM s (e.g., Podell et al., 2024 ) fall into this Latent Diffusion Model ( LDM ) category. Recently, Kuntz et al., 2023 ; Lim et al., 2024 ; Lim and Johansen, 2024 have reported performance gains when training simpler latent-variable models and related tasks by replacing variational approximations with particle-based ones. Here, we investigate whether this is also the case for LDM s and introduce the, to the best of our knowledge, first particle-based method for LDM training. Contributions. (C1) We recast the problem of LDM training as the minimization of a free energy functional, characterize the functional’s minima (Proposition 2.1 ), identify a gradient flow that minimizes it, and establish its exponential convergence under standard assumptions (Theorem 3.1 ). (C2) Approximating the flow in (C1), we obtain Interacting Particle Latent Diffusions ( IPLD s) —a simple, particle-based, and encoder-free, algorithm for LDM training well-adapted to modern compute environments—and we derive non-asymptotic bounds on its error (Theorem 3.2 ). (C3) Through several practical improvements, we obtain an efficient and scalable version of the algorithm (Section 3.3 ) and demonstrate its effectiveness in numerical experiments (Section 5 ). (C4) Lastly, by approaching LDM s from the above unexplored angle, we open the door to other novel LDM training methods. In particular, our approach connects latent diffusion models to the rich body of work on gradient flows and interacting particle systems stemming from the optimal transport literature ( Villani et al., 2008 ; Chaintron and Diez, 2022 ) –a connection that can spur the design of new algorithms as we demonstrate here. Paper structure. The paper is organized as follows. First, we review the necessary background on LDM s and identify relevant loss functions (Section 2 ). Next, we obtain IPLD : an algorithm for minimizing this loss (Section 3 ). We do so by, identifying a gradient flow that minimizes the loss (Section 3.1 ), approximating it (Section 3.2 ), and incorporating a series of practical improvements (Section 3.3 ). We then survey the related literature (Section 4 ) and experimentally compare IPLD with relevant baselines (Section 5 ). We conclude with a discussion of our results, IPLD ’s limitations, and future research directions (Section 6 ). Notation. We use \mathsf{X}=\mathbb{R}^{d_{x}} and \mathsf{Z}=\mathbb{R}^{d_{z}} to denote the ambient and latent spaces, \Theta=\mathbb{R}^{d_{\theta}} and \Phi=\mathbb{R}^{d_{\phi}} the decoder’s and DM ’s parameter spaces (c.f. Section 2 ), \{x_{1},\dots,x_{M}\} the training set, and \mathcal{P}(\mathbb{R}^{d}) the space of probability distributions on \mathbb{R}^{d} . 2 Background on latent diffusion models We consider latent-space versions of Denoising Diffusion Probabilistic Models ( DDPM s) Ho et al., 2020 similar to those in Rombach et al., 2022 ; Vahdat et al., 2021 ; Wehenkel and Louppe, 2021 : (1) p_{\theta,\phi}(x,z_{0:K})=p_{\phi}(x|z_{0})p_{\theta}(z_{0:K}) where {p_{\phi}(x|z_{0})=\mathcal{N}(x;g_{\phi}(z_{0}),\sigma^{2}I)} is an isotropic Gaussian decoder, with g_{\phi}:{\mathsf{Z}}\to{\mathsf{X}} denoting a neural network, parametrized by \phi , that maps from the latent space to the ambient space, and the prior, p_{\theta}(z_{0:K}):=p(z_{K})\prod_{k=1}^{K}p_{\theta,k}(z_{k-1}|z_{k}), is a DDPM parametrized by \theta . The DDPM ’s distribution at time K is a standard Gaussian, p_{K}(z_{K}):=\mathcal{N}(z_{K};0,I) , and the backward kernels are also Gaussian: p_{\theta,k}(z_{k-1}|z_{k}):=\mathcal{N}(z_{k-1};\mu_{\theta,k}(z_{k}),\beta_{k}^{2}I) where (k,z)\mapsto\mu_{\theta,k}(z) denotes a neural network, parametrized by \theta , that maps from \{1,\dots,K\}\times{\mathsf{Z}} to {\mathsf{Z}} and \{\beta_{k}\}_{k=1}^{K} a fixed noise schedule. We fit the model by searching for parameters (\theta_{\star},\phi_{\star}) that maximize the log evidence averaged over the training set \{x_{1},\dots,x_{M}\} : (2) \ell(\theta,\phi):=\frac{1}{M}\sum_{m=1}^{M}\log p_{\theta,\phi}(x_{m}),\quad\text{where}\quad p_{\theta,\phi}(x):=\int p_{\theta,\phi}(x,z_{0:K}){\mathrm{d}}z_{0:K} denotes the probability density the model assigns to a given datapoint x . Assuming the training points are independent, this is equivalent to fitting the model through maximum likelihood. Because the log-evidence is intractable, we instead consider the following lower bound: (3) -F(\theta,\phi,q^{1:M}):=\frac{1}{M}\sum_{m=1}^{M}\mathbb{E}_{q^{m}}\left[\log\frac{p_{\theta,\phi}(x,z_{0})}{q^{m}(z_{0})}\right]\leq\ell(\theta,\phi)\quad\forall(\theta,\phi,q^{1:M})\in\Theta\times\Phi\times\mathcal{P}({\mathsf{Z}})^{M}, where p_{\theta,\phi}(x,z_{0}):=\int p_{\theta,\phi}(x,z_{0:K}){\mathrm{d}}z_{1:K} and F denotes the free energy . Maximizing ( 3 ) above over all parameters \theta,\phi and M -tuples q^{1:M}=(q^{1},\dots,q^{M}) of distributions q^{1},\dots,q^{M} over the latent space is equivalent to maximizing \ell(\theta,\phi) over all \theta,\phi ; see, e.g., Neal and Hinton, 1998 . However, p_{\theta,\phi}(x,z_{0}) is intractable, and we must resort to one more lower bound: (4) \displaystyle-\tilde{F}(\theta,\phi,q^{1:M}):=\frac{1}{M}\sum_{m=1}^{M}\mathbb{E}_{q^{m}}\left[\log\frac{p_{\theta,\phi}(x,z_{0:K})}{q(z_{1:K}|z_{0})q^{m}(z_{0})}\right]\leq F(\theta,\phi,q^{1:M})\leq\ell(\theta,\phi), for all \theta , \phi , and q^{1:M} , where q(z_{1:K}|z_{0}) denotes the forward process : a fixed user-chosen kernel as in DDPM . \tilde{F} amounts to a regularized version of F that penalizes deviations from the forward process: (5) \tilde{F}=F+\frac{1}{M}\sum_{m=1}^{M}\mathbb{E}_{q^{m}}[\mathcal{R}],\quad\text{where}\quad\mathcal{R}(\theta,z_{0}):=D_{\mathrm{KL}}(q(z_{1:K}|z_{0})||p_{\theta}(z_{1:K}|z_{0})), where D_{\mathrm{KL}} denotes the Kullback-Leibler divergence. Alternatively, we can view \tilde{F} as the free energy obtained replacing p_{\theta,\phi} in ( 3 ) with the following tilted version: (6) \tilde{p}_{\theta,\phi}(x,z_{0}):=p_{\theta,\phi}(x,z_{0})\exp(-\mathcal{R}(\theta,z_{0})). For this reason, similar arguments to those behind ( Neal and Hinton, 1998 , Theorem 2) yield: Proposition 2.1 . (\theta_{\star},\phi_{\star}) maximize \tilde{\ell}(\theta,\phi):=M^{-1}\sum_{m=1}^{M}\log\tilde{p}_{\theta,\phi}(x_{m}) iff (\theta_{\star},\phi_{\star},q_{\star}) minimize \tilde{F} for some q_{\star} , where \tilde{p}_{\theta,\phi}(x_{m}):=\int\tilde{p}_{\theta,\phi}(x,z_{0:K}){\mathrm{d}}z_{0:K} . Note that \tilde{\ell}(\theta,\phi) in the above equals \ell(\theta,\phi) in ( 2 ) whenever \theta is such that the forward and reverse processes coincide (that is, when \theta belongs to \Theta_{0}:=\{\theta\in\Theta:\mathcal{R}(\theta,\cdot)\equiv 0\} ). Hence, if the \theta -component of one of \ell ’s maximizers belongs to \Theta_{0} , it follows that \ell and \tilde{\ell} share the same maxima and maximizers, and Proposition 2.1 implies that the (\theta,\phi) -components of \tilde{F} ’s minimizers maximize \ell , our objective of interest. Because \ell only depends on \theta via p_{\theta}(z_{0}):=\int p_{\theta}(z_{0:K})dz_{1:K} , a sufficient condition for the \theta -component of one of \ell ’s maximizers to belong to \Theta_{0} is (7) \{p_{\theta}(z_{0}):\theta\in\Theta\}=\{p_{\theta}(z_{0}):\theta\in\Theta_{0}\}. In practice, the above will not hold exactly (in fact, \Theta_{0} will be empty). However, we believe a close approximation is possible when the forward and reverse processes are chosen with care (generally, the more expressive the latter is, the better). Previous works such as Wehenkel and Louppe, 2021 restrict q^{m} to mean-field Gaussian distributions of the sort \mathcal{N}(z_{0};\mu_{\psi}(x^{m}),\Sigma_{\psi}(x^{m})) , where