Variational Inference via Entropic Transport Descent
Vincent Pacelli, Akash Ratheesh, Evangelos Theodorou · 2026
Read Variational Inference via Entropic Transport Descent by Vincent Pacelli, Akash Ratheesh, Evangelos Theodorou (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-25265.txt.
Introduction
Approximating a distribution that cannot be sampled directly can be done by herding a swarm of particles toward high density regions. Vincent Pacelli, Akash Ratheesh, and Evangelos Theodorou introduce entropic transport descent in 2026, where each update solves a small optimal transport problem softened by entropy. The transport plan coordinates the whole swarm at once, steering every particle toward promising proposals while keeping distinct modes populated. Earlier particle methods push particles apart with kernel forces and often collapse in high dimensions.
Each iteration reduces to a standard Sinkhorn computation, and the method can run using only pointwise density values, with no gradient information at all. Across collapse diagnostics, logistic regression, neural networks, and molecular energy distributions, it matched or beat the leading alternatives. Gains were largest exactly where older methods struggle, in high dimensional and multimodal problems.
Catalog record
| CREATOR | Vincent Pacelli, Akash Ratheesh, Evangelos Theodorou |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 83 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2606.25265
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Variational Inference via Entropic Transport Descent Vincent Pacelli, Akash Ratheesh, Evangelos Theodorou Abstract Particle-based variational inference (ParVI) methods approximate an intractable target distribution by evolving an ensemble of interacting samples. Existing approaches rely predominantly on kernel-based repulsion (e.g., SVGD), which suffers from variance collapse in high dimensions and mode collapse on multimodal targets—pathologies caused by the absence of global transport structure. We introduce entropic transport descent (ETD), a ParVI family that frames each particle update as an entropy-regularized optimal transport problem. Derived from the JKO proximal scheme by lifting to the space of couplings and relaxing via the KL chain rule, each ETD iteration reduces to a Sinkhorn computation. The resulting transport plan provides global coordination, guiding each particle to nearby high-density proposals and naturally preserving multimodal structure. ETD can operate entirely score-free, requiring only pointwise evaluations of the unnormalized target density. Experiments on variance-collapse diagnostics, Bayesian logistic regression, neural networks, and molecular Boltzmann distributions show that ETD matches or outperforms SVGD, AGF-SVGD, and SGLD, with the largest gains in high-dimensional and multimodal settings. 1 Introduction Approximate sampling from an intractable target distribution \pi(x)\propto\exp(-V(x)) is a core computational task in Bayesian inference ( 19 ) , generative modeling ( 24 ) , and scientific simulation ( 41 ) . Particle-based variational inference (ParVI) methods are a non-parametric approach to variational inference (VI) that addresses this task by evolving an ensemble of N interacting particles such that their empirical distribution converges to \pi . Unlike parametric methods, ParVI methods make no restrictive assumptions on the form of the approximating distribution, and unlike Markov chain Monte Carlo (MCMC), they are typically parallelizable, do not require long burn-in periods on multimodal targets to mix, and yield an ensemble of samples. This paper introduces entropic transport descent (ETD), a family of ParVI algorithms whose inter-particle interaction is mediated by entropic optimal transport. Building on the JKO proximal scheme ( 23 ) , each iteration solves an entropic OT problem that assigns every particle a conditional distribution over a shared pool of target-weighted proposals. A single parameter \tau\geq 0 controls coupling fidelity, interpolating from a closed-form solution to one that enforces the target marginal exactly. ETD can operate score-free—requiring only pointwise evaluations of the unnormalized target—or incorporate score information through the proposal distribution. Contributions. ( i ) We introduce ETD, a family of ParVI algorithms derived from the JKO proximal scheme ( 23 ) by lifting the optimization to the space of couplings and relaxing via the KL chain rule, reducing each iteration to an entropic OT problem solvable by the Sinkhorn algorithm ( Section 4 ). ( ii ) We characterize the stationary distribution of balanced ETD exactly ( Theorem 2 ), showing the bias is independent of both the transport cost and the entropic regularization, and provide importance-corrected target weights that make \pi exactly stationary ( Theorem 3 ). ( iii ) Experiments on variance-collapse diagnostics, Bayesian logistic regression, Bayesian neural networks, and sampling from multimodal energy functions demonstrate that ETD matches or outperforms existing ParVI baselines, including both score-based and score-free methods, with the largest gains in high-dimensional and multimodal settings ( Section 6 ). 2 Background Definitions and Notation. Let \mathcal{P}(\mathbb{R}^{n}) denote the space of probability distributions on \mathbb{R}^{n} absolutely continuous with respect to Lebesgue measure; all distributions are assumed to have finite second moments. No distinction is made between a probability measure and its density. The Kullback–Leibler (KL) divergence between \mu,\pi\in\mathcal{P}(\mathbb{R}^{n}) (with \mu absolutely continuous with respect to \pi ) is denoted \mathcal{D}\quantity[\mu\, |\, \pi] . We write \delta_{x} for the Dirac measure centered at x , \Delta^{N} for the probability simplex in \mathbb{R}^{N} , \mu\otimes\nu for the product measure , and \oslash for elementwise division . The inner product between two matrices A,B\in\mathbb{R}^{n\times m} is denoted: \langle A,B\rangle=\sum_{ij}A_{ij}B_{ij} . The vector \mathbf{1}_{n}\in\mathbb{R}^{n} is the vector for which all entries are one. A coupling of distributions \mu,\nu\in\mathcal{P}(\mathbb{R}^{n}) is a joint distribution \gamma on \mathbb{R}^{n}\times\mathbb{R}^{n} with marginals \mu and \nu ; we denote the set of all such couplings \mathcal{C}[\mu,\nu] . When only the source marginal is fixed, we write \mathcal{C}(\mu)\coloneqq\bigcup_{\nu\in\mathcal{P}(\mathbb{R}^{n})}\mathcal{C}[\mu,\nu] . Optimal Transport. Optimal transport (OT) provides a geometry on probability measures by quantifying the cost of moving mass between distributions. Given a transport cost c:\mathbb{R}^{n}\times\mathbb{R}^{n}\to\mathbb{R}_{\geq 0} , the OT cost between \mu,\nu\in\mathcal{P}(\mathbb{R}^{n}) is \mathcal{T}_{c}[\mu,\nu]=\inf_{\gamma\in\mathcal{C}[\mu,\nu]}\int c\,\differential\gamma , where the infimum ranges over all couplings with marginals \mu and \nu . The 2-Wasserstein distance \mathcal{W}_{2}[\mu,\nu] corresponds to \sqrt{\mathcal{T}_{c}[\mu,\nu]} with c(x,y)=\tfrac{1}{2}\|x-y\|^{2} . When both marginal constraints are enforced, the problem is balanced ; when one or both are relaxed to KL penalties, it is unbalanced ; and when only the source marginal is constrained while the target enters as a reference measure, it is semi-relaxed . Adding an entropic regularization term \varepsilon\,\mathcal{D}\quantity[\gamma\, |\, \mu\otimes\nu] to the objective yields entropic OT (EOT), which is strictly convex and solvable via the Sinkhorn algorithm ( 13 ) , and whose value is denoted \mathcal{T}_{c}^{\varepsilon}[\mu,\nu] . Background on OT and EOT, its discrete specialization, and the Sinkhorn algorithm are provided in Appendix A . Variational Inference. Many statistical methods, e.g., Bayesian inference, require sampling from an intractable target distribution \pi(x)\propto\exp(-V(x)) . In such cases, variational inference (VI) is a popular approach to approximately sample from \pi by identifying a family of tractable distributions, \mathcal{Q}\subset\mathcal{P}(\mathbb{R}^{n}) , then identifying the closest member of the family to \pi as measured by the KL divergence ( 5 ) : \displaystyle\mu^{\star}\ \coloneqq\ \argmin_{\mu\in\mathcal{Q}}\quad\mathcal{D}\quantity[\mu\, |\, \pi]. (VI) Particle VI (ParVI) methods are a non-parametric approach that selects the tractable family \mathcal{Q} to be represented by a finite number of interacting particles \mathbf{z}=(z_{i})_{i=1}^{N} —thereby reducing an infinite-dimensional optimization problem over distributions to a finite one. Typically, as in the sequel, particles are treated as individual samples and the tractable family consists of the empirical distributions : \mu(x;\mathbf{z})=\frac{1}{N}\sum_{i=1}^{N}\delta_{z_{i}}(x) . Particles are then evolved according to an update law designed so that their mutual interactions drive them to be approximately distributed according to \pi . Thus, the choice of interaction mechanism defines many of a ParVI algorithm’s properties. The JKO Scheme. The probability density \mu_{t}\in\mathcal{P}(\mathbb{R}^{n}) of the overdamped Langevin equation (OLE) driven by a standard Wiener process W_{t} , \displaystyle\differential X_{t}=-\partial_{x}V(X_{t})\,\differential t+\sqrt{2}\,\differential W_{t}, (1) is the backbone of many sampling and inference algorithms. Under mild regularity conditions, \mu_{t} evolves according to the Fokker–Planck equation (FPE), \displaystyle\partial_{t}\mu_{t}=\partial_{x}\cdot(\mu_{t}\partial V)+\partial_{x}\cdot\partial_{x}\mu_{t}, (2) and converges to the target density \pi(x)\propto\exp(-V(x)) as t\to\infty . The score function is the gradient of the target log-density: \partial_{x}\log\pi(x)=-\partial_{x}V(x) . Jordan, Kinderlehrer, and Otto ( 23 ) showed that ( 2 ) is the gradient flow of \mathcal{D}\quantity[\cdot\, |\, \pi] with respect to \mathcal{W}_{2} on \mathcal{P}(\mathbb{R}^{n}) . This structure yields a natural implicit discretization—the JKO scheme : \displaystyle\mu_{k+1}=\argmin_{\mu\in\mathcal{P}(\mathbb{R}^{n})}\quad\mathcal{D}\quantity[\mu\, |\, \pi]\;+\;\frac{1}{2\tau}\,\mathcal{W}_{2}^{2}[\mu,\mu_{k}], (3) where \tau>0 is the step size. Each step advances \mu_{k+1} toward \pi while penalizing large displacements from \mu_{k} ; as \tau\to 0 , the iterates recover ( 2 ). The JKO scheme is elegant but computationally demanding—each iteration requires solving an optimal transport problem. Even in simple discrete settings, solving such problems has a complexity O(N^{3}\log N) . Section 4 develops a tractable upper bound on ( 3 ) by introducing entropic regularization, which produces a strictly convex transport problem that is solved via