A note on convergence of Wasserstein policy optimization
David Šiška 1, Yufei Zhang 2 · 2026
Read A note on convergence of Wasserstein policy optimization by David Šiška 1, Yufei Zhang 2 (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-22622.txt.
Introduction
Šiška and Zhang argue that Wasserstein policy optimization, a reinforcement learning method built on Wasserstein gradient flows, converges linearly under entropy regularized Markov decision processes. Their route uses mean field analysis of gradient flows with log Sobolev inequalities, assuming a sufficiently regular flow solution. From steady energy loss along the flow and a local log Sobolev inequality, they conclude the value function should reach the global optimum linearly. The note fills a theoretical gap behind a method whose empirical success had outrun its convergence proofs.
Catalog record
| CREATOR | David Šiška 1, Yufei Zhang 2 |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 34733 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.22622
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
A note on convergence of Wasserstein policy optimization David Šiška 1, Yufei Zhang 2 Abstract . Wasserstein Policy Optimization (WPO) is a recently proposed reinforcement learning algorithm that leverages Wasserstein gradient flows to optimize stochastic policies in continuous action spaces. Despite its empirical success, the theoretical convergence properties of WPO in environments with continuous state and action spaces have yet to be fully established. In this note, we argue that WPO within the framework of entropy-regularised Markov Decision Processes converges linearly. This is done by leveraging recent advances in mean-field analysis for convergence of gradient flows using log-Sobole inequalities. Assuming existence of sufficiently regular solution to the gradient flow equation we demonstrate monotonic energy dissipation along the flow and establish a local log-Sobolev inequality. Ultimately, these properties allow us to argue that the value function should converge linearly to the global optimum. 1. Introduction Reinforcement learning has achieved remarkable success in solving complex control problems with continuous action spaces. A significant driver of this progress has been policy optimization, a family of methods that directly updates a policy’s parameters via stochastic gradient descent to minimize expected long-term costs. Recently, Wasserstein Policy Optimization (WPO) was introduced as a novel actor-critic algorithm derived as an approximation to Wasserstein gradient flow over the space of all policies [ 14 ] . WPO bridges the gap between classic stochastic policy gradients and deterministic policy gradients: it exploits the gradient of the action-value function with respect to the action, yet it can be applied to arbitrary stochastic policies. Despite its promising empirical performance, the theoretical convergence properties of WPO require a rigorous mathematical foundation. The aim of this note is to answer the following question. Can one expect WPO to converge and at what rate? This note begins to answer by establishing the global linear convergence of WPO within the framework of entropy-regularised Markov Decision Processes (MDPs). Specifically, we consider an infinite horizon Markov decision model defined by a discrete and finite state space S , a continuous action space A=\mathbb{R}^{d} , a transition probability P , a bounded cost function c , and a discount factor \gamma\in[0,1) . In this work, we postulate that the policy evolves according to a Wasserstein gradient flow driven by the flat derivative of the regularised value function. We assume the gradient flow has solutions which are sufficiently well behaved so that, in particular, we can derive energy dissipation. Our analysis extends techniques from the convex analysis of mean-field Langevin dynamics [ 3 , 3 , 12 , 7 , 4 , 13 , 5 ] . We prove that energy dissipates monotonically along the flow. Furthermore, by demonstrating that a local log-Sobolev inequality holds along the gradient flow, we establish that the value function converges exponentially fast to the global optimal value. While we carried out the analysis for a continuous-time gradient it should also be possible to establish convergence for discrete stepping schemes using the techniques proposed in [ 9 ] . We believe that key ideas are easier to follow for the continuous time gradient flow. The remainder of this note is organized as follows. Section 2 formally introduces the problem formulation and key tools, defining the entropy-regularised MDPs for continuous state and action spaces. Section 2 formulates the Wasserstein gradient flow continuity equation and presents our main theoretical results regarding its convergence analysis. Finally, Appendix A provides classical results and Bellman equations for entropy-regularised MDPs utilized throughout our proofs. 2. Problem formulation and key tools 2.1. Entropy-regularised MDPs In this section, we formulate the entropy-regularised MDPs with continuous state and action spaces. Let S and A be Polish spaces, P\in\mathcal{P}(S|S\times A) , c\in B_{b}(S\times A) and \gamma\in[0,1) . The five-tuple (S,A,P,c,\gamma) determines an infinite horizon Markov decision model, where S and A represent the state and action spaces, respectively, P represents the transition probability, c represents the cost function and \gamma represents the discount factor. Let \Pi=\{\pi=\{\pi_{n}\}_{n\in\mathbb{N}_{0}}:\pi_{n}\in\mathcal{P}(A|H_{n})\} denote the set of (possibly non-Markovian) stochastic policies, where for each n\in\mathbb{N}_{0} , H_{n}\coloneqq(S\times A)^{n}\times S is the space of admissible histories. Let (\Omega:=(S\times A)^{\mathbb{N}_{0}},\mathcal{F}) denote the canonical sample space, where \mathcal{F}=\mathcal{B}(\Omega) is the corresponding Borel sigma-algebra. Elements of \Omega are of the form (s_{0},a_{0},s_{1},a_{1},\ldots) with s_{n}\in S and a_{n}\in A denoting the projections and called the state and action variables, at time n\in\mathbb{N}_{0} , respectively. By [ 1 , Proposition 7.28] , for any given initial distribution \rho\in\mathcal{P}(S) and policy \pi\in\Pi , there exists a unique product probability measure \mathbb{P}^{\pi}_{\rho} on (\Omega,\mathcal{F}) with expectation denoted \mathbb{E}^{\pi}_{\rho} such that for all n\in\mathbb{N}_{0} , B\in\mathcal{B}(S) and C\in\mathcal{B}(A) , \mathbb{P}_{\rho}^{\pi}(s_{0}\in B)=\rho(B) and (1) \mathbb{P}_{\rho}^{\pi}(a_{n}\in C|h_{n})=\pi_{n}(C|h_{n}),\quad\mathbb{P}_{\rho}^{\pi}(s_{n+1}\in B|h_{n},a_{n})=P(B|s_{n},a_{n})\,, where h_{n}=(s_{0},a_{0},\ldots,s_{n-1},a_{n-1},s_{n})\in H_{n} . In particular, if \pi is a Markov stochastic policy (i.e., \pi_{n}\in\mathcal{P}(A|S) for all n\in\mathbb{N}_{0} ), then \{s_{n}\}_{n\in\mathbb{N}_{0}} is a Markov process with kernel \{P_{\pi,n}\}_{n\in\mathbb{N}_{0}}\in\mathcal{P}(S|S) given by P_{\pi,n}(ds^{\prime}|s)=\int_{A}P(ds^{\prime}|s,a)\pi_{n}(da|s),\quad\forall s\in S,n\in\mathbb{N}_{0}\,. For s\in S , we denote \mathbb{E}^{\pi}_{s}=\mathbb{E}^{\pi}_{\delta_{s}} , where \delta_{s}\in\mathcal{P}(S) denotes the Dirac measure at s\in S . Let \mu\in\mathcal{\mathcal{P}}(A) denote a reference measure and \tau\in(0,\infty) denote a regularisation parameter. For each \pi=\{\pi_{n}\}_{n\in\mathbb{N}_{0}}\in\Pi and s\in S , define the following regularised value function: (2) V^{\pi}_{\tau}(s)=\mathbb{E}_{s}^{\pi}\left[\sum_{n=0}^{\infty}\gamma^{n}\Big(c(s_{n},a_{n})+\tau\operatorname{KL}(\pi_{n}(\cdot|h_{n})|\mu)\Big)\right]\in\mathbb{R}\cup\{\infty\}\,, which may be infinite if \pi_{n}\not\in\mathcal{P}_{\mu}(A|S) for some n\in\mathbb{N}_{0} , or if \mathbb{E}_{s}^{\pi}\left[\sum_{n=0}^{\infty}\gamma^{n}\operatorname{KL}(\pi_{n}(\cdot|h_{n})|\mu)\right] diverges. Since c is bounded and H_{n}\ni h_{n}\mapsto\operatorname{KL}(\pi_{n}(\cdot|h_{n})|\mu)\in[0,\infty] is non-negative and measurable, V^{\pi}_{\tau}:S\rightarrow\mathbb{R}\cup\{\infty\} is a well-defined measurable function. We define the optimal value function V^{*}_{\tau}:S\rightarrow\mathbb{R}\cup\{\infty\} by (3) V^{*}_{\tau}(s)=\inf_{\pi\in\Pi}V^{\pi}_{\tau}(s),\quad\forall s\in S\,, and refer to \pi^{*}\in\Pi as an optimal policy if V^{\pi^{*}}_{\tau}(s)=V^{*}_{\tau}(s) , for all s\in S . One can prove that V^{*}_{\tau} satisfies a dynamic programming principle (see Theorem A.1 for a precise statement), which implies that V^{*}_{\tau}\in B_{b}(S) and for all s\in S , V^{\ast}_{\tau}(s)=-\tau\ln\int_{A}\exp\left(-\frac{1}{\tau}Q^{\ast}_{\tau}(s,a)\right)\mu(da), where Q^{*}_{\tau}\in B_{b}(S\times A) is defined by Q^{*}_{\tau}(s,a)=c(s,a)+\gamma\int_{S}V_{\tau}^{*}(s^{\prime})P(ds^{\prime}|s,a)\,,\quad\forall(s,a)\in S\times A\,. Moreover, there is an optimal policy \pi^{*}_{\tau}\in\mathcal{P}_{\mu}(A|S) given by (4) \pi^{*}_{\tau}(da|s)=\exp\left(-\frac{1}{\tau}(Q^{\ast}_{\tau}(s,a)-V^{\ast}_{\tau}(s))\right)\mu(da)\,,\quad\forall s\in S. This suggests that, without loss of generality, it suffices to minimise ( 2 ) over the class of stationary Markov policies that are equivalent to the reference measure \mu . Definition 2.1 . Let \Pi_{\mu} denote the class of policies \pi=\{\pi_{n}\}_{n\in\mathbb{N}_{0}}\in\Pi such that \pi_{n}\in\mathcal{P}_{\mu}(A|S) for all n\in\mathbb{N}_{0} , and for which there exists f\in B_{b}(S\times A) such that \pi_{n}(da|s)=\frac{\exp\left(f(s,a)\right)}{\int_{A}\exp\left(f(s,a)\right)\mu(da)}\mu(da) for all s\in S and n\in\mathbb{N}_{0} . In the sequel, we identify \Pi_{\mu} with the set \{\boldsymbol{\pi}(f)\mid f\in B_{b}(S\times A)\}\subset\mathcal{P}_{\mu}(A|S) , where \boldsymbol{\pi}:B_{b}(S\times A)\to\mathcal{P}_{\mu}(A|S) is defined by (5) \boldsymbol{\pi}(f)(da|s)=\frac{e^{f(s,a)}}{\int_{A}e^{f(s,a^{\prime})}\mu(da^{\prime})}\mu(da),\quad\forall f\in B_{b}(S\times A)\,. For each \pi\in\Pi_{\mu} , we define the Q -function Q^{\pi}_{\tau}\in B_{b}(S\times A) by (6) Q^{\pi}_{\tau}(s,a)=c(s,a)+\gamma\int_{S}V_{\tau}^{\pi}(s^{\prime})P(ds^{\prime}|s,a)\,. Then due to the on-policy Bellman equation (see Lemma A.2 ), for all \pi\in\Pi_{\mu} and s\in S , (7) V^{\pi}_{\tau}(s)=\int_{A}\left(Q_{\tau}^{\pi}(s,a)+\tau\ln\frac{\mathrm{d}\pi}{\mathrm{d}\mu}(a|s)\right)\pi(da|s)\,. For each \pi\in\mathcal{P}(A|S) , we define the occupancy kernel d^{\pi}\in\mathcal{P}(S|S) by (8) d^{\pi}(ds^{\prime}|s)=(1-\gamma)\sum_{n=0}^{\infty}\gamma^{n}P^{n}_{\pi}(ds^{\prime}|s)\,, where P^{n}_{\pi} is the n -times product of the kernel P_{\pi} with P^{0}_{\pi}(ds^{\prime}|s)\coloneqq\delta_{s}(ds^{\prime}) and the convergence is understood in b\mathcal{M}(S|S) . For a given initial distribution \rho\in\mathcal{P}(S) , we define (9) V^{\pi}_{\tau}(\rho)=\int_{S}V^{\pi}_{\tau}(s)\rho(ds)\quad\textnormal{and}\quad d^{\pi}_{\rho}(ds)=\int_{S}d^{\pi}(ds|s^{\prime})\rho(ds^{\prime})\,. The flat derivative is given by (10) \frac{\delta V^{\pi}_{\tau}(\rho)}{\delta\pi}\bigg|_{\nu}(s,a)=\left(Q^{\pi}_{\tau}(s,a)+\tau\ln\frac{\mathrm{d}\pi}{\mathrm{d}\mu}(s,a)-V^{\pi}_{\tau}(s)\right)\frac{\mathrm{d}d^{\pi}_{\rho}}{\mathrm{d}\nu}(s)\,, where d^{\pi}_{\rho}\in\mathcal{P}(S) is the occupancy measure associated with \pi . The flat derivative ( 10 ) generalises the notation of the flat derivative applied to probability measures to encompass probability transition kernels. 3. Wasserstein Gradient Flow Throughout this section, we assume the action space A=\mathbb{R}^{d} for some d\in\mathbb{N} and S is discrete and finite. Let \lambda denote the Lebesgue measure on A=\mathbb{R}^{d} . We