Policy Gradient Learning for Distributionally Robust Markov Decision Processes under Wasserstein Ambiguity
Yadh Hafsi, Samy Mekkaoui, Huyên Pham, Kaixin Yan · 2026
Read Policy Gradient Learning for Distributionally Robust Markov Decision Processes under Wasserstein Ambiguity by Yadh Hafsi, Samy Mekkaoui, Huyên Pham, Kaixin Yan (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-27610.txt.
Introduction
Decision problems under uncertainty are modelled as Markov decision processes. The transition rules may not be known exactly, though. Yadh Hafsi and three colleagues study robust control for this setting in 2026. Their ambiguity model uses Wasserstein balls around nominal transitions. A ball holds all transition laws within a set distance. The goal is a policy that does well even in the worst case inside the ball.
The worst case depends on the policy itself. That dependence breaks the usual policy-gradient argument. The authors work around it with a dual reformulation of the robust Bellman recursion. Studying its directional derivatives yields an explicit recursive formula for the robust policy gradient. They build a robust actor-critic algorithm on that formula. Benchmark examples illustrate how the algorithm behaves. Both discrete and continuous cases are included.
Catalog record
| CREATOR | Yadh Hafsi, Samy Mekkaoui, Huyên Pham, Kaixin Yan |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 117 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2606.27610
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Policy Gradient Learning for Distributionally Robust Markov Decision Processes under Wasserstein Ambiguity Yadh Hafsi, Samy Mekkaoui, Huyên Pham, Kaixin Yan Abstract We study finite-horizon Markov Decision Processes (MDPs) under distributional uncertainty in the transition kernels and develop a policy-gradient framework for Wasserstein distributionally robust control. Ambiguity is modeled by state-action dependent Wasserstein balls around nominal transition kernels, leading to a max-min control problem over randomized policies and admissible transition laws. Since the worst-case transition law depends implicitly on the policy parameters, the usual policy-gradient argument does not apply. We address this difficulty by using a Wasserstein dual reformulation of the robust Bellman recursion and analyzing its directional differentiability. This yields an explicit recursive characterization of the robust policy gradient. Building on this characterization, we propose a robust actor-critic algorithm and illustrate its behavior on discrete and continuous benchmark examples. MSC Classification : 90C40, 90C17, 93E20, 49L20, 68T05. Key words : Robust Markov decision processes; Robust reinforcement learning; Actor-Critic; Dynamic programming principle; Randomized policies; Policy gradient; Distributionally robust optimization. 1. Introduction In many reinforcement learning and control problems, the transition law of the system is not known exactly. Even when a nominal model is available, the system may later be tested or deployed in an environment that differs from the one used for calibration. Different data sources, market conditions, or operating regimes may induce different probability measures on trajectories. As a result, a policy that performs well under one reference model may perform poorly when the underlying dynamics change. This motivates robust reinforcement learning, whose goal is to construct policies that perform well under a family of plausible transition laws rather than under a single nominal model. This issue appears naturally in many applications. In inventory control and supply chain management, demand distributions are difficult to estimate precisely and may vary over time because of seasonality, structural change, or external shocks. Robust and distributionally robust methods have therefore been used to hedge against ambiguity in demand models; see for example 6 , 30 , 15 , 19 . In portfolio optimization, market dynamics shift across regimes, with changes in volatility, correlations, and return distributions that may occur at different time scales. A strategy calibrated to one regime may break down after a regime shift. This has led to distributionally robust formulations under Wasserstein ambiguity 8 , as well as robust continuous-time control under model uncertainty and adaptive learning 12 . In all these settings, uncertainty acts through the law of the controlled state process. In our framework, this uncertainty is represented by set-valued transition kernels {\cal P}_{t}(x,a) , and robust optimization over these sets leads to policies that are more stable under model misspecification. We study finite-horizon robust Markov decision processes with randomized policies and set-valued transition kernels. Our main focus is on ambiguity sets defined around reference transition laws. This setting is attractive because it models local distributional perturbations in a flexible and interpretable way. At the same time, it creates a basic difficulty for policy optimization. Under general ambiguity classes, such as Wasserstein or Kullback–Leibler balls, the worst-case transition kernel is typically not available in closed form. It is selected adversarially according to the value function generated by the current policy, so changing the policy can also change the worst-case model. This makes direct differentiation of the robust Bellman recursion difficult, and generally intractable without strong regularity assumptions on the optimizer (see Remark 3.3 ). Related work. There are two main lines of literature behind our approach: robust Markov decision processes (MDPs) and distributionally robust optimization (DRO). On the robust MDPs side, the foundational papers 17 and 27 proved robust dynamic programming under rectangular ambiguity sets, showing that the Bellman recursion remains tractable when uncertainty separates across state–action pairs. Since then, robust MDPs have been studied under broader forms of model uncertainty (see 23 for example). MDPs under general model uncertainty are studied in 25 . Policy gradient methods for robust MDPs have recently been developed in 31 , while robust Q-learning under Wasserstein ambiguity is studied in 26 . In a related direction, 13 propose an actor–critic algorithm for robust risk-aware reinforcement learning under Wasserstein ambiguity, deriving policy gradients via the quantile representation of distortion risk measures. On the distributionally robust optimization side, a general convex framework and tractable reformulations are given in 33 . For Wasserstein DRO, dual reformulations and performance guarantees were developed in 24 , while structural properties of worst-case distributions were analyzed in 16 . The duality result most relevant for our work is 10 , which gives a scalar dual representation for Wasserstein DRO through optimal transport. Related links between Wasserstein robustness and regularization in statistical learning are established in 9 . Sensitivity of robust values with respect to the Wasserstein radius is studied in 3 , with related results under financial constraints in 29 , and dynamic robustness on path space in 14 . The closest related work is 32 , which studies policy gradient methods for robust MDPs under R -contamination sets. In that setting, the robust Bellman operator associated with contamination sets admits a closed-form expression. This makes the policy gradient derivation relatively direct. Under more general types of ambiguity like Wasserstein or Kullback-Leibler ambiguity, however, the worst-case transition kernel depends on the value function through an optimization problem, and differentiating through it would require regularity conditions on the optimal kernel that do not hold in general (see Remark 3.3 ). We instead work with the dual formulation of 10 , which avoids differentiating the optimal kernel directly. Our framework also differs from the robust Q-learning algorithm of 26 , which operates in the infinite-horizon discounted setting with deterministic policies and tabular updates. By contrast, our actor–critic approach handles finite-horizon problems, optimizes over randomized parametric policies via gradient ascent and accommodates continuous state spaces through function approximation. Additionally, our numerical results show that the method remains effective even for large values of \varepsilon . Table 1 : Comparison with related robust reinforcement learning and Wasserstein control methods. Work Ambiguity set Finite horizon Actor–critic Wasserstein dual Sensitivity term 32 R -contamination yes policy gradient / actor–critic no closed form 31 rectangular transition uncertainty yes policy gradient no robust PG inner loop 20 rectangular transition uncertainty yes policy gradient no robust gradient 22 transition-kernel uncertainty average-cost policy mirror descent no robust sub-gradient 35 distributional shift contextual bandit policy gradient problem-specific no Bellman sensitivity 26 Wasserstein transition ambiguity discounted tabular Q-learning yes no actor gradient 34 , 18 Wasserstein LQ finite-horizon control no Riccati / dual special LQ structure This paper Wasserstein transition kernels yes yes yes directional Bellman sensitivity Our main contributions. We derive directional policy-gradient formulas for finite-horizon robust MDPs with state–action dependent Wasserstein ambiguity sets. Our approach combines the robust dynamic programming recursion with a Wasserstein dual representation, thereby avoiding differentiation of the implicit worst-case transition kernel. Since the robust Bellman operator is generally nonsmooth, our main result is a one-sided directional derivative formula, which reduces to a vector-valued policy-gradient recursion under uniqueness of the dual multiplier and inner transport selector. More precisely, our contributions are as follows. • Robust Bellman and Wasserstein dual formulation. We formulate the finite-horizon distributionally robust control problem under state–action dependent ambiguity sets and establish