MADR: MPC-guided Adversarial DeepReach

Ryan Teoh, Sander Tonkens, William Sharpless, Aijia Yang, Zeyuan Feng, Somil Bansal, Sylvia Herbert, https://land-dev.github.io/madr/ · 2025

Read MADR: MPC-guided Adversarial DeepReach by Ryan Teoh, Sander Tonkens, William Sharpless, Aijia Yang, Zeyuan Feng, Somil Bansal, Sylvia Herbert, https://land-dev.github.io/madr/ (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-18845.txt.

Introduction

Ryan Teoh and his six co-authors extend a safety framework for robots facing an adversary. Hamilton-Jacobi reachability computes value functions that guarantee safety against worst-case disturbances, but exact computation collapses in high dimensions. Neural approximations scale better yet converge slowly and inaccurately, partly because the learning signal from the underlying equation is weak.

Their framework guides the neural training with model predictive control, an established technique that supplies regular supervision drawn from the structure of the control problem itself. Earlier guided approaches handled only single-player problems and simple games, while this one targets the full two-player, zero-sum setting, returning optimal strategies for both sides and safe policies under worst-case attack. Across many simulated and real robotic systems with varied dynamics, the method beat leading baselines in simulation and performed impressively on hardware.

Catalog record

CREATORRyan Teoh, Sander Tonkens, William Sharpless, Aijia Yang, Zeyuan Feng, Somil Bansal, Sylvia Herbert, https://land-dev.github.io/madr/
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE37 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2510-18845

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

MADR: MPC-guided Adversarial DeepReach Ryan Teoh, Sander Tonkens, William Sharpless, Aijia Yang, Zeyuan Feng, Somil Bansal, Sylvia Herbert, https://land-dev.github.io/madr/ Abstract Hamilton-Jacobi (HJ) Reachability offers a framework for generating safe value functions and policies in the face of adversarial disturbance, but is limited by the curse of dimensionality. Physics-informed deep learning is able to overcome this infeasibility, but itself suffers from slow and inaccurate convergence, primarily due to weak PDE gradients and the complexity of self-supervised learning. A few works, recently, have demonstrated that enriching the self-supervision process with regular supervision (based on the nature of the optimal control problem), greatly accelerates convergence and solution quality, however, these have been limited to single player problems and simple games. In this work, we introduce MADR: MPC-guided Adversarial DeepReach, a general framework to robustly approximate the two-player, zero-sum differential game value function. In doing so, MADR yields the corresponding optimal strategies for both players in zero-sum games as well as safe policies for worst-case robustness. We test MADR on a multitude of high-dimensional simulated and real robotic agents with varying dynamics and games, finding that our approach significantly out-performs state-of-the-art baselines in simulation and produces impressive results in hardware. I Introduction Zero-sum differential games provide a unifying framework for modeling interactions between an ego robot and an adversarial entity—whether that entity is a passive disturbance (e.g., wind, sensor noise) or an active agent (e.g., an opponent robot or attacker). To ensure robustness, zero-sum formulations assume that the other player or disturbance is adversarial, i.e. the ego robot aims to maximize a reward function (e.g., time to target, safety margin), while the disturbance or adversary attempts to minimize it. Hamilton-Jacobi (HJ) Reachability is a popular approach for solving zero-sum differential games with respect to target achievement and obstacle avoidance [ 1 , 2 ] . HJ reachability requires solving a partial differential equation (HJ-PDE) to compute a value function; this value function implicitly provides a) the optimal control policy for the robot and adversary, and b) the reachable set: the set of states from which a system can reach (or avoid) a target (or obstacle) over time using these optimal policies. HJ reachability is particularly well-suited to safety-critical applications due to its ability to model worst-case scenarios in a game-theoretic formulation [ 3 ] . Fig. 1 : A graphical abstract of MADR and the robotic experiments. In this work, we propose enriching self-supervised learning of HJ-PDE’s, i.e. DeepReach, with supervision given by the best sampled game rollout (top left), where the opponents policy is defined by the current value approximation (top right). We demonstrate this approach performs across games with general dynamics (bottom), namely with TurtleBots, Drones and Humanoid experiments. HJ reachability has been applied to a broad range of robotic systems, including collision avoidance for aerial vehicles, multi-agent coordination, and motion planning under uncertainty [ 1 , 4 ] . However, classical approaches rely on grid-based dynamic programming and scale poorly with state dimension due to the curse of dimensionality [ 5 ] . This limits their use to systems with fewer than six states in practice. To address this challenge, learning-based approaches have been explored for scalability. Methods such as DeepReach [ 6 ] replace the grid with a function approximator trained to satisfy the HJ-PDE in a physics-informed learning (PINN) framework. This approach significantly improves scalability and has been demonstrated on systems of 50 dimensions [ 7 ] . DeepReach has primarily focused on control-only cases at high dimensions, with only limited work addressing robustness or more complex dynamics [ 8 , 9 ] . A few works have demonstrated that introducing supervision greatly improves the learned approximation. In [ 7 ] , the Hopf formula is used to solve a linearization of the game for a supervision loss, however, this struggles with nonlinear or angular dimensions, which frequently appear in robotics. More recently, [ 10 ] employed sample-based Model Predictive Control (MPC) as supervisory signal, finding that this significantly improves performance in high-dimensions but is limited to the control-only case. I-A Contributions Our method, MADR, proposes solving zero-sum differential games using a value-only DeepReach approach combined with adversarial MPC roll-outs. Our key insight is to define the opponent’s policy through the current value gradient approximation, which allows robust sampling of improved ego policies and the associated optimal cost of their trajectories. Unlike actor-critic methods, where policy performance depends on simultaneous learning of both actor and critic, our value-only approach ensures that high-quality policies emerge directly from accurately-learned value functions. To achieve this, we mitigate co-learning issues by collecting separate datasets in which the sampling (and associated adversarial gameplay under the current value gradient) occurs for each agent individually. This enables robust learning of a single value function for both players, leading to effective policies in adversarial scenarios. We empirically demonstrate that this value-informed adversarial supervision improves both the fidelity and safety margins of learned reachable sets, leading to better robustness against disturbances and adversarial agents. The effectiveness of MADR is validated through extensive simulation experiments and a variety of hardware demonstrations, highlighting improved safety performance in complex, high-dimensional systems exposed to significant disturbances or adversarial players. In addition, we compare MADR against other current approaches in head-to-head policy matchup comparisons. Ultimately, our approach bridges the gap between the rigorous, theory-driven framework of HJ reachability and the practical use of optimal control to handle adversarial or worst-case scenarios in real-world environments. I-B Related Work Linear Quadratic Games (LQG) are a class of differential games where multiple players aim to minimize a quadratic cost function over time, subject to linear dynamics, either cooperatively or non-cooperatively. In the non-cooperative scenario, players often seek Nash equilibria, which can be computed by solving coupled Riccati equations [ 11 ] in limited scenarios, but practically in iteratively linearizations [ 12 ] . The cooperative case typically involves joint optimization to minimize a shared cost function [ 13 ] . Recent advancements have extended LQGs to incorporate uncertainties and stochastic disturbances, enhancing their applicability to real-world scenarios [ 14 ] . However, they remain relatively challenging in zero-sum settings and face limitations when applied to complex dynamical systems, where linearization of the dynamics can lead to significant inaccuracies. Model Predictive Control (MPC) is a model-based optimal control strategy. At each timestep, it solves a constrained optimization problem over a finite prediction horizon, a method known as receding horizon control. MPC is well-suited for safety-critical applications because it produces a control sequence that is guaranteed by the model to satisfy state and input constraints over the entire prediction horizon [ 15 ] . However, MPC has two significant drawbacks: its high computational burden and its sensitivity to model mismatch. Sampling-based variants, such as MPPI [ 16 ] , improve robustness to model inaccuracies by evaluating thousands of stochastic rollouts. Additionally, robust, stochastic, and distributed formulations further enhance performance and efficiency in uncertain environments [ 17 ] . Adversarial Reinforcement Learning (RL) formulates control in uncertain environments as a two-player game, where a controller learns to counteract worst-case disturbances or adversarial agents [ 18 , 19 ] . This framework improves robustness to unmodeled dynamics, environmental perturbations, and intelligent opponents. Modern algorithms, often based on actor-critic methods, enable

Read the complete text