Model Predictive Adversarial Imitation Learning for Planning from Observation

Tyler Han, Yanda Bao, Bhaumik Mehta, Gabriel Guo, Anubhav Vishwakarma, Emily Kang, Sanghun Jung, Rosario Scalise, Jason Zhou, Bryan Xu, Byron Boots, University of Washington · 2025

Read Model Predictive Adversarial Imitation Learning for Planning from Observation by Tyler Han, Yanda Bao, Bhaumik Mehta, Gabriel Guo, Anubhav Vishwakarma, Emily Kang, Sanghun Jung, Rosario Scalise, Jason Zhou, Bryan Xu, Byron Boots, University of Washington (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-21533.txt.

Introduction

Tyler Han and co-authors blend planning with learning from examples, in a 2025 robotics paper. Robots often learn by copying people, but human demos can be partial or unclear. A common fix learns a reward signal from the demos and then plans with it, in two separate steps.

The authors merge the steps by swapping the learned actor for a planner inside the training loop itself. Their method learns end to end from demonstrations that show only observations, with no recorded actions. Tests in simulated control tasks and real robot navigation showed better learning from few examples. The robots also handled new situations better and broke less often. The approach promises safer, clearer behavior from very little demonstration data.

Catalog record

CREATORTyler Han, Yanda Bao, Bhaumik Mehta, Gabriel Guo, Anubhav Vishwakarma, Emily Kang, Sanghun Jung, Rosario Scalise, Jason Zhou, Bryan Xu, Byron Boots, University of Washington
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE64 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2507-21533

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Model Predictive Adversarial Imitation Learning for Planning from Observation Tyler Han, Yanda Bao, Bhaumik Mehta, Gabriel Guo, Anubhav Vishwakarma, Emily Kang, Sanghun Jung, Rosario Scalise, Jason Zhou, Bryan Xu, Byron Boots, University of Washington Abstract Human demonstration data is often ambiguous and incomplete, motivating imitation learning approaches that also exhibit reliable planning behavior. A common paradigm to perform planning-from-demonstration involves learning a reward function via Inverse Reinforcement Learning (IRL) then deploying this reward via Model Predictive Control (MPC). Towards unifying these methods, we derive a replacement of the policy in IRL with a planning-based agent. With connections to Adversarial Imitation Learning, this formulation enables end-to-end interactive learning of planners from observation-only demonstrations. In addition to benefits in interpretability, complexity, and safety, we study and observe significant improvements on sample efficiency, out-of-distribution generalization, and robustness. The study includes evaluations in both simulated control benchmarks and real-world navigation experiments using few-to-single observation-only demonstrations. Keywords: Imitation Learning, Model Predictive Control, Inverse Reinforcement Learning 1 Introduction Figure 1: Model Predictive Adversarial Imitation Learning (MPAIL) embeds an MPC agent within Adversarial Imitation Learning (AIL) for learning costs from observation. On deployment, the MPC agent performs online optimization through short model rollouts, or plans, as they are evaluated by the learned costs and value networks. Inverse Reinforcement Learning (IRL) offers a principled approach to imitation learning by inferring the underlying reward function that explains expert behavior. A major advantage of IRL is its ability to generalize beyond demonstration data, enabling agents to generate new plans aligned with the expert’s objectives. This feature allows for improved interpretability in decision making, even when demonstrations are ambiguous, suboptimal, or complex [ 1 ] . As such, IRL is particularly well-suited for domains where understanding preferences and ensuring reliable planning are essential, such as routing on Google Maps [ 2 ] , socially aware navigation [ 3 ] , and autonomous driving [ 4 ] . For real-time systems, learned IRL and Inverse Optimal Control (IOC) rewards are typically deployed via Model Predictive Control (MPC) [ 5 , 6 , 7 , 8 , 9 , 10 , 11 ] . Here, the offline IRL algorithm iteratively solves a Reinforcement Learning (RL) problem in an inner loop, guided by the current reward estimate. An outer loop then updates this reward to minimize the discrepancy between the agent’s and the expert’s behavior. Once training is complete, the resulting reward is integrated with MPC for real-time planning and control. Adversarial Imitation Learning (AIL) has made significant improvements over IRL in algorithmic complexity and sample efficiency [ 12 , 13 ] . However, the reliance on an RL policy in AIL methods complicates their use in applications with safety constraints [ 5 , 7 , 8 ] . Further limited by partial observability, these deployments will often prioritize planning using a model for the sake of real-time performance, trustworthiness, and interpretability [ 14 , 15 , 16 ] . In this work, we propose to replace the standard policy network in AIL with an MPC agent , yielding key benefits: 1. Planning-from-Observation. Towards scalable Learning-from-Observation (LfO), a predictive model can be used to match the expert without requiring action data. In addition, MPC naturally admits interpretable model-based planning, which can provide insight into the agent’s decision making process as it learns from complex expert data. We further show that this can improve sample efficiency, out-of-distribution recovery, and enable navigation from a single partially observable demonstration. 2. Unification of IRL and MPC. These methods are otherwise considered as independent training and deployment procedures. By embedding the planner directly in the learning pipeline, online settings (e.g., dynamics, preferences, control constraints) can be brought into training and further allow for long-horizon reasoning through on-policy value function estimation. We find that this induces a more effective adversarial dynamic and enables MPC to robustly learn from observation beyond its planning horizon and data coverage. By choosing Model Predictive Path Integral control (MPPI) [ 17 ] as the embedded planner, we derive theoretical connections to the Adversarial Imitation Learning objective. Thus, we name this learning framework, Model Predictive Adversarial Imitation Learning (MPAIL) {pronounced: impale }. We implement MPAIL and empirically validate it as a planning-based AIL algorithm. From a single real-world partially observable demonstration, MPAIL enables interpretable real-time planning from observation on limited compute through a Real-Sim-Real pipeline. 2 Related Work IRL-MPC. High-dimensional continuous control applications often require an online planner for real-time control, trustworthiness, safety, or additional constraints. When using IRL to learn a reward, online deployments of these reward functions tend to rely on an independent online MPC procedure. To enable learning local costmaps across perception and control for off-road navigation, Lee et al. 2022 and Triest et al. 2023 similarly propose solving the forward RL problem by using MPPI but deploy the learned reward on a different configuration more suitable for real-time planning and control. This reward deployment framework of IRL-then-MPC is currently the dominant approach for planning in high-dimensional continuous control tasks from demonstration [ 5 , 6 , 7 , 8 , 9 , 10 , 11 ] . Model-Based IRL and Planning-Based RL. The proposed framework, MPAIL, can be viewed as a model-based AIL approach. Various other works have also explored model-based AIL [ 18 , 4 , 19 ] . However, scope is directed at training stabilization and policy performance rather than planning using the learned rewards. Notably, TD-MPC [ 20 ] , a model-based RL algorithm has demonstrated considerable improvement over existing state-of-the-art RL algorithms through latent state planning. MPAIL similarly performs local trajectory optimization and learns a terminal value function for long-horizon reasoning. This work does not include a policy-based prior to better isolate and study the effects of the planning-based agent [ 21 , 19 ] . 3 Model Predictive Adversarial Imitation Learning 3.1 The POMDP Setting and the Model Predictive Agent We adopt the Partially Observable Markov Decision Process (POMDP) to best consider highly desirable applications of IRL in which partial observability and model-based planning play crucial roles. In an unknown world state s_{w}\in\mathcal{S}_{w} , the agent makes an observation o\sim p(o|s_{w}) . From a history of observations \mathbf{o}_{0:t} , the Agent perceives its state s_{t}\sim p(s|\mathbf{o}_{0:t}) . Actions a\in\mathcal{A} and states s\in\mathcal{S} together allow the agent to self-predict forward in time using its predictive model f:\mathcal{S}\times\mathcal{A}\rightarrow\mathcal{S} . Note that these definitions crucially suggest the partial observability of s due to the implicit dependency on the observation history \mathbf{o}_{0:t} through the agent’s perception (e.g. mapping [ 22 ] ). However, partial observations in \mathcal{S} are desirably used for demonstrations to perform IRL and AIL, as full observation history would quickly become intractable [ 7 ] . The planner itself is a model predictive agent. It is capable of performing model rollouts \tau^{(H)}_{t}=\{s_{t^{\prime}},a_{t^{\prime}}\}_{t^{\prime}=t}^{t+H} such that s_{t^{\prime}+1}=f(s_{t^{\prime}},a_{t^{\prime}}) . Each rollout is then mapped to a cost C(\tau_{t}) . Finally, the agent’s objective is to create an H -step action sequence \mathbf{a}_{t:t+H} , or plan , that best minimizes its corresponding trajectory cost. 3.2 Adversarial Imitation Learning from Observation IRL algorithms aim to learn a cost function that minimizes the cost of expert trajectories while maximizing the cost of trajectories induced by other policies [ 23 , 12 ] . As

Read the complete text