Rectified Robust Policy Optimization for Model-Uncertain Constrained Reinforcement Learning without Strong Duality
Shaocong Ma [email protected], Ziyi Chen [email protected], Yi Zhou [email protected], Heng Huang [email protected] · 2025
Read Rectified Robust Policy Optimization for Model-Uncertain Constrained Reinforcement Learning without Strong Duality by Shaocong Ma [email protected], Ziyi Chen [email protected], Yi Zhou [email protected], Heng Huang [email protected] (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-17448.txt.
Introduction
Shaocong Ma, Ziyi Chen, Yi Zhou, and Heng Huang forge a safe learn craft for a world half known in a 2025 paper. The craft must do well in the worst world its doubt holds, while it keeps a set bar of safe acts.
Past ways lean on a twin-sum trick: solve the foe form of the task and trade marks back. The team shows that trick can have no sound end here, so those ways can miss the best safe plan. Their way works the plain task head-on, with a mend that keeps each step in the safe span. Proof shows it nears the best safe plan at the best known pace. In a grid world it held safe where a bold way broke the worst-case bar.
Catalog record
| CREATOR | Shaocong Ma [email protected], Ziyi Chen [email protected], Yi Zhou [email protected], Heng Huang [email protected] |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 110 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2508-17448
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Rectified Robust Policy Optimization for Model-Uncertain Constrained Reinforcement Learning without Strong Duality Shaocong Ma [email protected], Ziyi Chen [email protected], Yi Zhou [email protected], Heng Huang [email protected] Abstract The goal of robust constrained reinforcement learning (RL) is to optimize an agent’s performance under the worst-case model uncertainty while satisfying safety or resource constraints. In this paper, we demonstrate that strong duality does not generally hold in robust constrained RL, indicating that traditional primal-dual methods may fail to find optimal feasible policies. To overcome this limitation, we propose a novel primal-only algorithm called Rectified Robust Policy Optimization (RRPO), which operates directly on the primal problem without relying on dual formulations. We provide theoretical convergence guarantees under mild regularity assumptions, showing convergence to an approximately optimal feasible policy with iteration complexity matching the best-known lower bound when the uncertainty set diameter is controlled in a specific level. Empirical results in a grid-world environment validate the effectiveness of our approach, demonstrating that RRPO achieves robust and safe performance under model uncertainties while the non-robust method can violate the worst-case safety constraints. 1 Introduction In many practical reinforcement learning (RL) applications, it is critical for an agent to not only maximize expected cumulative rewards but also satisfy certain constraints, such as safety requirements ( Yao et al. 2024 ; Gu et al. 2024b ) or resource limitations ( Wang et al. 2023c ) . However, real-world environments often diverge from the training environment due to model mismatch ( Roy et al. 2017 ; Viano et al. 2021 ; Zhai et al. 2024 ; Wang et al. 2024 ) and environment uncertainty ( Lütjens et al. 2019 ; Wang & Zou 2021 ; Ma et al. 2023 ) . Such discrepancies can lead to significant performance degradation and, more severely, violations of constraints, which is unacceptable in safety-critical applications. For instance, an autonomous robot may encounter unforeseen transitions due to equipment aging or mechanical failures: In the environment navigation and the robust control tasks, the robot make take the action that perform better in the ideal simulated environment due to the ideal equipment condition but suffer high penalty from higher energy cost or unexpected operations in the worst-case scenario. Despite its practical importance, robust constrained RL has been relatively underexplored in the literature. Two closely related areas are robust RL ( Bagnell et al. 2001 ; Nilim & Ghaoui 2005 ; Iyengar 2005 ) and constrained RL ( Altman 1999 ; Wachi & Sui 2020 ) . Robust RL focuses on optimizing performance under model uncertainties but typically does not consider constraints. Constrained RL aims to optimize performance while satisfying certain constraints but often assumes a fixed environment without uncertainties. Seamlessly combining two fields presents inherent challenges. To address these challenges, we propose a framework for robust constrained RL under model uncertainty. Specifically, we consider Markov Decision Processes (MDPs) where the transition dynamics are not fixed but lie within an uncertainty set, which is commonly known as the robust MDPs ( Mannor et al. 2016 ; Ho et al. 2018 ; Tamar et al. 2013 ; Grand-Clément & Kroer 2021 ) . Our objective is to optimize the worst-case cumulative reward over this uncertainty set while ensuring that all constraints are also simultaneously satisfied in the worst-case scenario. This robust approach ensures that the agent’s policy remains effective and safe even when the environment deviates from the nominal model. A common approach to solving such constrained problems is the primal-dual method ( Altman 1999 ; Paternain et al. 2019 ; Bai et al. 2022 ; Liang et al. 2018 ; Chen & Wang 2016 ; Mahadevan et al. 2014 ; Chen et al. 2022 ) , which leverages the strong duality property to efficiently find optimal policies. Strong duality allows the original constrained problem to be solved by considering its dual problem, simplifying computations and enabling convergence guarantees. However, a crucial question arises: Q1: Does strong duality hold in robust constrained RL? In this paper, we address this question head-on. We first demonstrate that, unfortunately, strong duality does not generally hold in robust constrained RL . The presence of model uncertainties breaks the Fenchel-Moreau condition, the common routine of showing the strong duality in the non-robust constrained RL ( Altman 1999 ) . We construct a specific counterexample where the duality gap, the difference between the optimal values of the primal and dual problems, is strictly positive. This finding indicates that traditional constrained RL algorithm may fail to be directly generalized to the robust constrained setting; we also note that the non-zero duality gap does not exclude the potential global convergence of primal-dual algorithm. Recognizing this fundamental issue, we are motivated to ask the following question: Q2: Can we develop a non-primal-dual algorithm for solving robust constrained RL problems with provable convergence guarantees? To address this question, we introduce the Rectified Robust Policy Optimization (RRPO), a primal-only algorithm adapted from the CRPO ( Xu et al. 2021 ) . RRPO is specifically designed for robust constrained RL, which bypasses the issues associated with the duality gap. Our algorithm operates directly on the primal problem, ensuring constraint satisfaction and robustness without relying on dual formulations or strong duality assumptions. We summarize our key contributions as follows: 1. Counterexample—Non-Zero Duality Gap: In Section 3 , we provide a concrete example showing that strong duality does not hold in robust constrained RL. To the best of our knowledge, this negative result is the first theoretical example showing the non-zero duality gap in the existing literature, which resolves an open problem in constrained robust RL. We emphasize that establishing such a counterexample is nontrivial and crucial for understanding the theoretical boundaries of constrained robust RL. 2. Proposed Primal-Only Algorithm—RRPO: Motivated by the lack of strong duality of constrained robust RL problems, we introduce RRPO, a primal-only algorithm designed to solve robust constrained RL problems without relying on strong duality. Moreover, in Section 4 , we rigorously analyze the convergence properties of RRPO. Specifically, we prove that under appropriate conditions, RRPO converges to an approximately optimal feasible policy \pi^{*} within a specified tolerance \delta in the worst-case scenario. When the uncertainty set is sufficiently small, our derived convergence rate and iteration complexity also recover the best-possible lower bound for non-robust constrained RL problems ( Vaswani et al. 2022 ) . 3. Empirical Validation: We validate the effectiveness of RRPO through experiments in a grid-world environment and the classical mountain car environment. Our results show that RRPO achieves robust and safe performance under model uncertainties, outperforming the original CRPO method that may fail to maintain constraint satisfaction in the worst-case scenario. 1.1 Related Work Robust Constrained RL Here, we mainly explore the existing literature regarding robust constrained RL. In Section A.1 , we provide other related work. Robust constrained RL considers the problem of optimizing performance while satisfying constraints in the worst-case scenario. Although robust RL and constrained RL have each been extensively studied, fewer works address their intersection. In Russel & Petrik 2020 , the authors investigate robust constrained RL and propose a heuristic approach that estimates robust value functions and employs a standard policy gradient method ( Sutton et al. 1999 ) , substituting the nominal value function with the robust one.