Wasserstein Policy Optimization
David Pfau, Ian Davies, Diana Borsa, João G. M. Araújo, Brendan Tracey, Hado van Hasselt · 2025
Read Wasserstein Policy Optimization by David Pfau, Ian Davies, Diana Borsa, João G. M. Araújo, Brendan Tracey, Hado van Hasselt (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-00663.txt.
Introduction
David Pfau and co-authors present Wasserstein Policy Optimization in 2025, a new reinforcement learning method for continuous control. An agent choosing smooth actions, like torques or doses, needs updates that improve behavior without wild jumps. Their method views learning as flowing the whole policy through the space of possible policies, following a Wasserstein gradient flow.
Projecting that flow onto a neural network's finite weights yields a simple closed-form update. The algorithm is actor-critic in shape and fully general across tasks. The authors report it combines the stability of trust-region methods with the simplicity of plain gradient steps. Robotics and control teams get a principled update rule with fewer knobs to tune.
Catalog record
| CREATOR | David Pfau, Ian Davies, Diana Borsa, João G. M. Araújo, Brendan Tracey, Hado van Hasselt |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 67 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-00663
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Wasserstein Policy Optimization David Pfau, Ian Davies, Diana Borsa, João G. M. Araújo, Brendan Tracey, Hado van Hasselt Abstract We introduce Wasserstein Policy Optimization (WPO), an actor-critic algorithm for reinforcement learning in continuous action spaces. WPO can be derived as an approximation to Wasserstein gradient flow over the space of all policies projected into a finite-dimensional parameter space (e.g., the weights of a neural network), leading to a simple and completely general closed-form update. The resulting algorithm combines many properties of deterministic and classic policy gradient methods. Like deterministic policy gradients, it exploits knowledge of the gradient of the action-value function with respect to the action. Like classic policy gradients, it can be applied to stochastic policies with arbitrary distributions over actions – without using the reparameterization trick. We show results on the DeepMind Control Suite and a magnetic confinement fusion task which compare favorably with state-of-the-art continuous control methods. Figure 1: Conceptual illustration of how WPO combines elements of stochastic and deterministic policy gradient methods. Left: “classic” policy gradient. Samples are taken from a stochastic policy. Each sample contributes a scalar Q^{\pi}(\mathbf{s},\mathbf{a}) factor to the gradient. Middle: deterministic policy gradient (DPG). A deterministic action is chosen and the policy gradient depends on the gradient of Q^{\pi}(\mathbf{s},\mathbf{a}) . Right: Wasserstein policy optimization (WPO). Samples are taken from a stochastic policy, as in classic policy gradient, but depend on the gradient of Q^{\pi} with respect to the action, as in DPG. 1 Introduction Reinforcement learning has made impressive progress in controlling complex domains with continuous actions such as robotics ( Haarnoja et al. 2024 ) , magnetic confinement fusion ( Degrave et al. 2022 ) and game playing ( Farebrother & Castro 2024 ) . A significant portion of this progress can be attributed to policy optimization methods, which directly optimize the parameters of a policy by stochastic gradient ascent on the expected long term return ( Schulman et al. 2015a ; Schulman et al. 2017 ; Abdolmaleki et al. 2018 , e.g.,) . While unbiased estimates of the policy gradient can be computed directly from returns ( Williams 1992 ) , many practical deep reinforcement learning algorithms for continuous control employ an actor-critic approach ( Barto et al. 1983 ) , which also estimates a value function to diminish the variance of the policy update. The majority of these methods use a policy update derived from the classic policy gradient theorem for stochastic policies ( Sutton et al. 1999 ) , which applies to both discrete and continuous action spaces. One notable exception is the class of algorithms derived from the deterministic policy gradient (DPG) theorem ( Silver et al. 2014 ; Lillicrap 2015 ) . These use information about the gradient of the value in action space to update the policy, but are limited to deterministic policies, hindering exploration. Adding in stochasticity can be difficult to tune, and extensions that learn the variance ( Heess et al. 2015 ; Haarnoja et al. 2018 ) rely on the reparameterization trick, which limits the class of policy distributions that can be used. Here we present a new policy gradient method that uses gradients of the action value but can learn arbitrary stochastic policies. The method can be derived from the theory of Wasserstein gradient flows ( Ambrosio et al. 2008 ) , and projecting the nonparametric flow onto the space of parametric functions (e.g. neural networks) leads to a simple closed-form update that combines elements of stochastic and deterministic policy gradients. We give a high-level illustration of this in Fig. 1 . The update is natural to implement as an actor-critic method. We call the resulting method Wasserstein Policy Optimization (WPO). The paper is organized as follows. First, we review policy optimization for continuous control, and Wasserstein gradient flows. We then show how the latter can be used to derive WPO when applied to the former. We review prior work in this area and analyze the dynamics of WPO and how it relates to other policy gradients. We describe several extensions needed to convert WPO into a practical and competitive deep reinforcement learning algorithm. Finally, we compare its performance against several baseline methods on the DeepMind Control Suite ( Tassa et al. 2018 ; Tunyasuvunakool et al. 2020 ) and a task controlling magnetic coils in a simulated magnetic confinement fusion device ( Tracey et al. 2024 ) . On a task where the task dimension can be varied arbitrarily, we find that WPO learns faster than baseline methods by an amount that increases as the task dimension grows, suggesting it may work well in very high dimensional action spaces. An open-source implementation of WPO is available in Acme ( Hoffman et al. 2020 ) 1 1 1 https://github.com/google-deepmind/acme . Note that the implementation in Acme is not the version used for the experiments in this paper. However we have run this implementation on DeepMind Control Suite tasks and found qualitatively similar performance to that reported here. . 2 Method 2.1 Policy Gradient and Iteration Methods We aim to find a policy \pi(\mathbf{a}|\mathbf{s}) which is a distribution over a continuous space of actions \mathbf{a}\in\mathbb{R}^{n} conditioned on a state \mathbf{s}\in\mathbb{R}^{m} that maximizes the expected long-term discounted return \mathcal{J}[\pi]=\mathbb{E}_{\mathbf{a}_{t}\sim\pi,\mathbf{s}_{t}\sim\mathcal{P}}\left[\sum_{t=0}^{T}\gamma^{t}r_{t}\right] for a Markov decision process with transition distribution \mathcal{P}(\mathbf{s}^{\prime}|\mathbf{s},\mathbf{a}) and reward function r(\mathbf{s},\mathbf{a}) , where r_{t} is shorthand for r(\mathbf{s}_{t},\mathbf{a}_{t}) . While “classic” policy gradient methods come in many forms ( Schulman et al. 2015b ) , they are mostly variants of the basic update \nabla_{\theta}\mathcal{J}[\pi_{\theta}]=\mathbb{E}_{\mathbf{a}_{t}\sim\pi,\mathbf{s}_{t}\sim\mathcal{P}}\left[\sum_{t=0}^{T}\Psi_{t}\nabla_{\theta}\mathrm{log}\pi_{\theta}(\mathbf{a}_{t}|\mathbf{s}_{t})\right] and only differ in the choice of scalar \Psi_{t} . If \Psi_{t} is entirely a function of the returns like \sum_{t^{\prime}=t}^{T}\gamma^{t^{\prime}}r_{t^{\prime}} , then the policy can be optimized directly, as in REINFORCE ( Williams 1992 ) . If \Psi_{t} is the action-value function Q^{\pi}(\mathbf{s}_{t},\mathbf{a}_{t}) or some transformation of the action-value function like the advantage function, or softmax of the action-value, then both a policy and value function must be estimated simultaneously, which is standard in actor-critic methods ( Barto et al. 1983 ) . Additionally, it is common to add some form of trust region or regularization to prevent the policy update from changing too much on any step ( Schulman et al. 2015a ; Schulman et al. 2017 ; Abdolmaleki et al. 2018 ) . This is especially critical for achieving good performance with deep reinforcement learning. One notable exception to this is deterministic policy gradients (DPG) ( Silver et al. 2014 ; Lillicrap 2015 ; Barth-Maron et al. 2018 ) , which can be seen as the limit of the policy gradient update as the policy becomes deterministic. These algorithms were developed as early as the 1970s ( Werbos 1974 ) and were known under names such as ‘action-dependent heuristic dynamic programming’ ( Prokhorov & Wunsch 1997 ) or ‘gradient ascent on the value’ ( van Hasselt & Wiering 2007 ) . As the name suggests, this only applies to deterministic policies \bm{\pi} that map state vectors to a single action vector, so \mathbf{a}_{t}=\bm{\pi}(\mathbf{s}_{t}) . Then the deterministic policy gradient has the form \nabla_{\theta}\mathcal{J}[\bm{\pi}_{\theta}]=\mathbb{E}_{\mathbf{s}_{t}\sim\mathcal{P}}\left[\nabla_{\mathbf{a}_{t}}Q(\mathbf{s}_{t},\mathbf{a}_{t})\nabla_{\theta}\bm{\pi}(\mathbf{s}_{t})\right] , where \nabla_{\theta}\bm{\pi}(\mathbf{s}_{t}) is the Jacobian of the deterministic policy. The appearance of the gradient of the value in action space in the policy gradient is potentially useful in high-dimensional action spaces.