On the Tension Between Optimality and Adversarial Robustness in Policy Optimization

Haoran Li, Jiayu Lv & Congying Han, Zicheng Zhang, Anqi Li, Yan Liu, Tiande Guo, Nan Jiang · 2025

Read On the Tension Between Optimality and Adversarial Robustness in Policy Optimization by Haoran Li, Jiayu Lv & Congying Han, Zicheng Zhang, Anqi Li, Yan Liu, Tiande Guo, Nan Jiang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2512-01228.txt.

Introduction

Haoran Li and co-authors probe a clash in deep reinforcement learning. A policy can aim for the best score. It can also aim to stay safe when foes try to trick it. For years those two aims were seen as foes as well.

Fresh theory hints the two aims may line up more than thought. The team asks how to gain both in real training. They test ways to keep score high while attacks fail. If both aims can be met, agents for games, robots, and trade could be both sharp and hard to fool.

Catalog record

CREATORHaoran Li, Jiayu Lv & Congying Han, Zicheng Zhang, Anqi Li, Yan Liu, Tiande Guo, Nan Jiang
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE162 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2512-01228

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

On the Tension Between Optimality and Adversarial Robustness in Policy Optimization Haoran Li, Jiayu Lv & Congying Han, Zicheng Zhang, Anqi Li, Yan Liu, Tiande Guo, Nan Jiang Abstract Achieving optimality and adversarial robustness in deep reinforcement learning has long been regarded as conflicting goals. Nonetheless, recent theoretical insights presented in CAR ( 49 ) suggest a potential alignment, raising the important question of how to realize this in practice. This paper first identifies a key gap between theory and practice by comparing standard policy optimization (SPO) and adversarially robust policy optimization (ARPO). Although they share theoretical consistency, a fundamental tension between robustness and optimality arises in practical policy gradient methods . SPO tends toward convergence to vulnerable first-order stationary policies (FOSPs) with strong natural performance, whereas ARPO typically favors more robust FOSPs at the expense of reduced returns. Furthermore, we attribute this tradeoff to the reshaping effect of the strongest adversaries in ARPO, which significantly complicates the global landscape by inducing deceptive sticky FOSPs . This improves robustness but makes navigation more challenging. To alleviate this, we develop the BARPO , a bilevel framework unifying SPO and ARPO by modulating adversary strength, thereby facilitating navigability while preserving global optima. Extensive empirical results demonstrate that BARPO consistently outperforms vanilla ARPO, providing a practical approach to reconcile theoretical and empirical performance. 1 Introduction Deep reinforcement learning (DRL) has demonstrated remarkable success across a variety of complex tasks ( 62 ; 53 ; 75 ) and real-world applications, including robotics ( 34 ; 57 ; 81 ) , autonomous driving ( 41 ; 15 ; 14 ) , and large model training ( 92 ; 45 ; 29 ) . Despite such substantial progress, DRL agents remain highly vulnerable to imperceptible attacks on their state observations, leading to severe performance degradation and even complete system failure ( 33 ; 7 ; 54 ; 43 ; 68 ; 86 ; 36 ; 35 ) . This vulnerability threatens the reliable deployment of DRL in safety-critical environments, highlighting the urgent need for robust agents capable of withstanding subtle perturbations and malicious attacks. Motivated by these challenges, 97 introduced the state-adversarial Markov decision process (SA-MDP) framework, providing a theoretical foundation for adversarial robustness in reinforcement learning. Building on this framework, they demonstrated that an optimal robust policy (ORP) may not always exist, indicating a potential conflict between the objectives of optimality and robustness. Recently, 49 ; 50 further explored the concept of ORP in practical settings and proposed the ISA-MDP formulation, a refinement that formally establishes the existence of an ORP that coincides with the Bellman optimality policy. These theoretical results suggest that the objectives of adversarial robustness and optimality are, in principle, aligned in most practical tasks. This insight drives the pursuit of DRL agents that not only maintain strong performance in clean environments but also exhibit resilience to adversarial attacks. This theoretical consistency is critical for deploying DRL agents in real-world applications, where strong malicious attacks are comparatively rare. However, in practice, achieving this alignment remains challenging. Despite sharing this theoretical consistency, conventional policy optimization methods struggle under adversarial settings ( 95 ; 79 ; 78 ) . It remains an open problem whether this theoretical alignment can be technically realized, leaving a critical gap between theory and practice. (a) SPO (b) ARPO (c) BARPO (d) Overall Figure 1: Schematic illustration of the optimization landscapes under SPO, ARPO, and BARPO. (a) SPO acsents along fragile directions, leading to vulnerable FOSPs with high natural value. (b) ARPO becomes trapped in robust regions but is limited to low-return solutions. (c) BARPO reshapes the landscape by lifting robust but low-return regions, enabling convergence to robust FOSPs with high returns. (d) Overall comparison of the three paradigms: contour lines represent natural returns, while background color indicates robustness, with darker red denoting lower robustness. In this paper, we delve into the optimization behavior and learning dynamics of two training paradigms: standard policy optimization (SPO) and adversarially robust policy optimization (ARPO) . SPO aims to maximize the standard value function, i.e., \max_{\pi}V^{\pi}(s),\ \forall\ s , while ARPO optimizes the worst-case adversarial value function, i.e., \max_{\pi}\min_{\nu}V^{\pi\circ\nu}(s),\ \forall\ s . By conducting a systematic comparison, we reveal a novel optimality-robustness tension in policy gradient methods. In our analysis, we first prove that both SPO and ARPO converge to first-order stationary policies (FOSPs) rather than global optima. Next, by examining the learning behaviors around these FOSPs, we uncover a key discrepancy: ARPO consistently produces policies with markedly higher robustness to state perturbations, whereas SPO-derived FOSPs remain vulnerable. These findings help explain the empirical vulnerability of SPO, despite the theoretical existence of an ORP, and emphasize the importance of ARPO in building robust DRL agents. Finally, we reveal that this robustness gain comes at a significant cost: ARPO-trained policies exhibit substantially lower natural returns compared to SPO. Altogether, these results expose a fundamental robustness-performance tradeoff in policy-gradient methods, highlighting a practical challenge for closing the theory-practice gap. To understand this tradeoff, we analyze the optimization landscape and value geometry induced by SPO and ARPO. We attribute the tension to the reshaping effect induced by the strongest adversaries: although they help guide the agent toward robust FOSPs, they often impede optimization by compromising the navigability of the landscape. As illustrated in Figure 1 , ARPO creates robust peaks but introduces additional valleys, distorting the global landscape to a more rugged terrain. These new valleys act as traps for the optimization path, hindering gradient-based methods from ascending towards the global or better local peaks. Instead, the optimization process stalls at these deceptive sticky FOSPs , which often result in lower natural returns compared to those achieved by SPO. This reshaping effect presents a core barrier to learning both optimality and robustness. To address this challenge, we propose a bilevel optimization framework that bridges SPO and ARPO, which adjusts the adversary strength to promote traversable optimization paths while maintaining robustness. This modulation prevents the optimization landscape from becoming overly rugged, helping to avoid local traps while still encouraging robustness. Importantly, it preserves the globally optimal robust policy shared by both SPO and ARPO. To enhance convergence and efficiency, we incorporate SPO dynamics into the bilevel framework, further smoothing the optimization landscape. The resulting approach, Bilevel ARPO (BARPO) , consistently achieves superior natural and robust returns, offering a promising approach to bridging the gap between theoretical alignment and practical performance. To summarize, this paper makes the following cohesive contributions: • We explain the nature of vulnerability in standard policy optimization (SPO) within the theoretically aligned framework. While an optimal robust policy exists, SPO typically converges to fragile first-order stationary policies (FOSPs), which offer strong natural performance but lack robustness. • We uncover a new form of optimality-robustness tension arising in policy optimization. Unlike prior notions of conflicting objectives, adversarially robust policy optimization (ARPO) shares the same global optima with SPO but prefers more robust FOSPs that typically yield lower returns. • We elucidate the underlying mechanism behind this trade-off. The strongest adversaries in ARPO reshape the optimization landscape by introducing numerous sticky FOSPs, which improve robustness but hinder effective policy improvement by making the landscape more difficult to navigate. • We propose a unified bilevel framework that

Read the complete text