Robust Adversarial Policy Optimization Under Dynamics Uncertainty
Mintae Kim & Koushil Sreenath · 2026
Read Robust Adversarial Policy Optimization Under Dynamics Uncertainty by Mintae Kim & Koushil Sreenath (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-10974.txt.
Introduction
Kim and Sreenath propose a robust adversarial policy method for reinforcement learning under dynamics uncertainty. Learned policies often fail when dynamics differ from training, a gap random sampling does not fully close. Robust methods help in theory but rely on stand-in foes that leave blind spots. The authors give a dual form showing the toughness-results trade-off plainly. A hostile network gives a temperature parameter for stable worst-case rollouts, while Boltzmann reweighting over dynamics groups focuses on adverse settings.
Catalog record
| CREATOR | Mintae Kim & Koushil Sreenath |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 118982 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.10974
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Robust Adversarial Policy Optimization Under Dynamics Uncertainty Mintae Kim & Koushil Sreenath Abstract Reinforcement learning (RL) policies often fail under dynamics that differ from training, a gap not fully addressed by domain randomization or existing adversarial RL methods. Distributionally robust RL provides a formal remedy but still relies on surrogate adversaries to approximate intractable primal problems, leaving blind spots that potentially cause instability and over-conservatism. We propose a dual formulation that directly exposes the robustness–performance trade-off. At the trajectory level, a temperature parameter from the dual problem is approximated with an adversarial network, yielding efficient and stable worst-case rollouts within a divergence bound. At the model level, we employ Boltzmann reweighting over dynamics ensembles, focusing on more adverse environments to the current policy rather than uniform sampling. The two components act independently and complement each other: trajectory-level steering ensures robust rollouts, while model-level sampling provides policy-sensitive coverage of adverse dynamics. The resulting framework, robust adversarial policy optimization (RAPO) outperforms robust RL baselines, improving resilience to uncertainty and generalization to out-of-distribution dynamics while maintaining dual tractability. 1 Introduction Reinforcement learning (RL) has made major progress in sequential decision-making, yet deployment in real domains is limited by dynamics mismatch 34 ; 12 ; 9 . Policies trained in simulation often fail when transferred to real systems due to aleatoric and epistemic uncertainty, where small deviations can accumulate into catastrophic failures 24 ; 43 . This motivates robust RL: agents that sustain reliable performance under uncertainty 28 ; 30 . While much of RL research prioritizes maximizing nominal returns, deployment inevitably faces unseen environments, making robustness essential. In safety-critical settings such as robotics, driving, and finance, brittle policies are unacceptable 19 ; 29 ; 20 ; 33 ; 11 . Robustness also improves generalization, enabling adaptation to perturbations without retraining 35 ; 3 ; 17 . Designing robust RL policies is difficult 38 ; 1 . Naïve methods often yield over-conservative policies or unstable training. The challenge lies in dynamics uncertainty: optimizing only for the nominal model gives fragile policies, while optimizing for all possible models is overly pessimistic 39 . The central question is: How can we design RL algorithms robust to dynamics uncertainty, without collapsing into conservatism or instability? Our approach begins from the robust MDP (RMDP) formulation with dynamics uncertain within a KL neighborhood of a nominal model 41 ; 13 . The dual problem converts this infinite-dimensional problem into a tractable scalar optimization, yielding closed-form adversarial distributions, entropic variance control, and contractive operators. This corresponds to evaluating returns under an exponentially tilted trajectory law with inverse temperature \eta . Direct sampling from this tilted law is infeasible in simulators that only roll out the nominal kernel 28 ; 47 . We therefore introduce an adversarial network (AdvNet) that predicts trajectory-level dual parameters. By steering rollouts toward low-return regions consistent with the KL budget, AdvNet effectively reweights trajectories to emphasize adverse outcomes, a practical realization of exponential tilting that exposes local vulnerabilities the policy might ignore. AdvNet alone cannot address global model misspecification. We therefore apply Boltzmann reweighting over an ensemble of environments with varied physical parameters, emphasizing models where the policy underperforms while remaining close to a prior. This provides structured coverage of adverse dynamics. Together, AdvNet and Boltzmann reweighting complement one another: the former enforces robustness at the trajectory-level , the latter at the model-level . Prior approaches capture only fragments of this picture. Domain randomization (model-level) spreads training across perturbations but treats all equally, producing inefficient or cautious policies 39 . Pure adversarial perturbations emphasize worst cases but lack distributional grounding and stability 28 ; 31 ; 40 . Ensemble methods capture variability but without adversarial reweighting underestimate risk 30 . Thus, our work introduces robust adversarial policy optimization (RAPO) , a practical realization of distributionally robust RL. RAPO combines trajectory-level exponential tilting via AdvNet with model-level Boltzmann reweighting, forming a two-layer structure that guards against local fragility and global misspecification. Finally, RAPO is practical and scalable: it amortizes trajectory tilting and ensemble biasing within a PPO-compatible loop, yielding policies that preserve in-distribution (ID) performance while substantially improving robustness to dynamics uncertainty and out-of-distribution (OOD) perturbations. 2 Related Work Robust MDPs model dynamics uncertainty by assuming the transition kernel lies in an ambiguity set, with policies optimized for worst-case return 24 ; 43 ; 23 . The power of this approach depends on how the set is defined. Early norm- or interval-bounded sets poorly captured mismatch, while distributional neighborhoods such as Wasserstein or f -divergence balls better align with statistical uncertainty and connect to distributionally robust optimization, yielding stronger guarantees 43 ; 1 . In practice, exact evaluation requires costly nested minimax problems, creating computational and sample inefficiency 41 ; 42 . Many works thus approximate the objective, trading nominal performance for conservative guarantees and exposing the core tension of robust RL: rigor versus scalability 7 ; 4 . An alternative is adversarial RL, formulating problem as a two-player game where an auxiliary agent perturbs states, actions, or dynamics to minimize return 28 ; 40 . This zero-sum view parallels adversarial training and H_{\infty} control 24 . Such adversaries act as disturbance generators or hard-example miners, but often yield instability, oscillatory dynamics, and narrow local robustness without distributional guarantees 38 ; 7 . Other variants perturb environment parameters, observations, or include explicit opponents in multi-agent settings 30 ; 22 ; 46 ; 45 ; 26 . Domain randomization instead samples environment parameters from broad distributions, enabling sim-to-real transfer but often producing inefficient or conservative policies 39 ; 27 ; 19 ; 5 ; 18 . Complementary research directions include verification methods with certified robustness to bounded noise 21 , adversarial input perturbations that enforce Lipschitz regularity 45 , and multi-agent opponents 10 ; 40 . These approaches extend robustness but target input or agent-level perturbations, remaining orthogonal to dynamics-level uncertainty. 3 Preliminaries Notations: For any finite set \mathcal{X} , we denote its cardinality by |\mathcal{X}| . The set [n] represents \{1,\dots,n\} . For a set \mathcal{X} , we write \Delta_{\mathcal{X}} for the set of probability distributions over \mathcal{X} (when \mathcal{X} is finite, it coincides with the probability simplex), and \mathrm{Unif}(\mathcal{X}) for the uniform distribution on \mathcal{X} . Markov decision process: A Markov decision process (MDP) is a tuple (\mathcal{S},\mathcal{A},p,r,\mu_{0},\gamma) , where \mathcal{S} is the state space, \mathcal{A} the action space, p:\mathcal{S}\times\mathcal{A}\to\Delta_{\mathcal{S}} the transition kernel, r:\mathcal{S}\times\mathcal{A}\to[0,1] a bounded reward function, \mu_{0}\in\Delta_{\mathcal{S}} the initial state distribution, and \gamma\in(0,1) the discount factor. A stochastic policy \pi:\mathcal{S}\to\Delta_{\mathcal{A}} induces the expected discounted return J(\pi,p)=\mathbb{E}_{s_{0}\sim\mu_{0},\,a_{t}\sim\pi,\,s_{t+1}\sim p}\big[\sum_{t=0}^{\infty}\gamma^{t}r(s_{t},a_{t})\big] . The state value, state–action value, and advantage are V^{\pi}(s)=\mathbb{E}_{a\sim\pi(\cdot|s),\,s^{\prime}\sim p(\cdot|s,a)}[r(s,a)+\gamma V^{\pi}(s^{\prime})] , Q^{\pi}(s,a)=r(s,a)+\gamma\,\mathbb{E}_{s^{\prime}\sim p(\cdot|s,a)}V^{\pi}(s^{\prime}) , and A^{\pi}(s,a)=Q^{\pi}(s,a)-V^{\pi}(s) , respectively. The Bellman operator w.r.t. \pi is \mathcal{T}^{\pi}V(s)=\mathbb{E}_{a\sim\pi(\cdot|s),\,s^{\prime}\sim p(\cdot|s,a)}[r(s,a)+\gamma V(s^{\prime})] 36 ; 2 . Robust RL: In many applications, the ground-truth kernel p^{\star} is unknown or subject to perturbations. To formalize robustness, we introduce a KL-ball ambiguity set around a nominal model \hat{p} , which represents the transition kernel of the training environment 37 ; 32 : \mathcal{P}_{\epsilon}(s,a)\;=\;\Big\{\,p(\cdot|s,a)\in\Delta_{\mathcal{S}}\ :\ D_{\mathrm{KL}}\!\big(p(\cdot|s,a)\,\|\,\hat{p}(\cdot|s,a)\big)\ \leq\ \epsilon\,\Big\}, (1) and \mathcal{P}_{\epsilon}=\bigotimes_{s,a}\mathcal{P}_{\epsilon}(s,a) under (s,a) -rectangularity 23 . RMDPs then pose the following control problem: \pi^{\star}\in\arg\max_{\pi}\ \inf_{p\in\mathcal{P}_{\epsilon}}J(\pi,p) . Rectangularity decouples uncertainty across state–action pairs and enables dynamic programming 14 ; 25 . Given \mathcal{P}_{\epsilon} , the