Efficient Imitation Under Misspecification
Nicolas Espinosa-Dice, Sanjiban Choudhury, Wen Sun, Gokul Swamy · 2025
Read Efficient Imitation Under Misspecification by Nicolas Espinosa-Dice, Sanjiban Choudhury, Wen Sun, Gokul Swamy (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-13162.txt.
Introduction
Nicolas Espinosa-Dice, Sanjiban Choudhury, Wen Sun, and Gokul Swamy study imitation learning when a learner cannot copy an expert exactly in a 2025 paper. Such mismatch is common, since robots sense and move unlike people. Some errors are bound to occur, and practice in the real setting is needed to learn which errors compound.
Older efficient methods search locally for better behavior, with strong promises when copying is possible. The authors prove a new condition under which those methods still avoid compounding errors. They also show search should cover states that good learner policies can reach, not just expert states, and test sources of mismatch in experiments.
Catalog record
| CREATOR | Nicolas Espinosa-Dice, Sanjiban Choudhury, Wen Sun, Gokul Swamy |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 168 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-13162
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Efficient Imitation Under Misspecification Nicolas Espinosa-Dice, Sanjiban Choudhury, Wen Sun, Gokul Swamy Abstract We consider the problem of imitation learning under misspecification : settings where the learner is fundamentally unable to replicate expert behavior everywhere. This is often true in practice due to differences in observation space and action space expressiveness (e.g. perceptual or morphological differences between robots and humans). Given the learner must make some mistakes in the misspecified setting, interaction with the environment is fundamentally required to figure out which mistakes are particularly costly and lead to compounding errors . However, given the computational cost and safety concerns inherent in interaction, we would like to perform as little of it as possible while ensuring we have learned a strong policy. Accordingly, prior work has proposed a flavor of efficient inverse reinforcement learning algorithms that merely perform a computationally efficient local search procedure with strong guarantees in the realizable setting. We first prove that under a novel structural condition we term reward-agnostic policy completeness , these sorts of local-search based IRL algorithms are able to avoid compounding errors. We then consider the question of where we should perform local search in the first place, given the learner may not be able to “walk on a tightrope” as well as the expert in the misspecified setting. We prove that in the misspecified setting, it is beneficial to broaden the set of states on which local search is performed to include states reachable by good policies that the learner can actually play. We then experimentally explore a variety of sources of misspecification and how offline data can be used to effectively broaden where we perform local search from. 1 Introduction Interactive imitation learning (IL) is a powerful paradigm for learning to make sequences of decisions from an expert demonstrating how to perform a task. While offline imitation learning approaches suffer from covariate shift between the training distribution (i.e. the expert’s state distribution) and the test distribution (i.e. the learner’s state distribution)—and the associated compounding of errors—interactive approaches allow the learner to observe states from the test distribution during training via performing rollouts, unlocking the ability to recover from mistakes ( Ross et al., 2011 ) . Broadly speaking, the difference between the expert’s performance and the learned policy’s performance can be attributed to three forms of error: 1. Optimization error : The error resulting from imperfect search within a policy class (e.g. due to the difficulty of finding a global minima on non-convex problems). 2. Finite sample error : The statistical error arising from limited expert demonstrations. 3. Misspecification error : The irreducible error resulting from the learner’s policy class not containing the expert’s policy. Notably, misspecification error is a function of both the expert policy and the learner’s policy class; it cannot be improved by a better algorithm or more computation. Much of the prior work in imitation learning has focused on the first two sources of error, while avoiding the misspecification error by imposing expert realizability : the assumption that the expert policy is within the learner’s policy class ( Kidambi et al., 2021 ; Swamy et al., 2021a ; Swamy et al., 2022c ; Xu et al., 2023 ; Swamy et al., 2023 ; Ren et al., 2024 ) . In other words, expert realizability is the assumption that the learner can perfectly imitate the expert’s actions in all states: that more data and computation are all we need for optimality. However, in practice, it is often impossible to imitate the expert perfectly. For example, this is sometimes due to differences in observation spaces. In self-driving applications, autonomous vehicles often have distinct perception compared to human drivers (e.g. the ability to recognize hand gestures at intersections), giving the human expert privileged information over the learner ( Swamy et al., 2022b ) . Similarly, in legged locomotion, recent work has frequently used an expert policy trained with privileged information which is only known in simulation and therefore therefore unavailable to the learner that operates in the real world ( Kumar et al., 2021 ; Kumar et al., 2022 ; Liang et al., 2024 ) . Realizability can also be an inaccurate assumption due to a mismatch between learner and expert action spaces. In humanoid robotics, the morphological differences between robots and humans prevent perfect human-to-robot motion re-targeting ( Zhang et al., 2024 ; He et al., 2024 ; Al-Hafez et al., 2023 ) . More generally, physical robots face the problem of changing dynamics due to wear and tear, as well as manufacturing imperfections, that cause variations in link lengths and other physical properties over the course of their operation. Thus, even if demonstrations are collected via teleoperation of a robot of the same make and model, expert realizability doesn’t necessarily hold. In response, our paper considers the more general misspecified setting, where the learner is not necessarily capable of perfectly imitating the expert’s behavior. We analyze how the misspecification error interrelates with the optimization and finite sample errors. More specifically, we ask: Under what condition can interactive imitation learning in the misspecified setting avoid compounding errors while retaining computational efficiency? The last two words of the preceding question are central to our study. By reducing the problem of imitation learning to reinforcement learning against a learned reward, interactive imitation learning approaches like inverse reinforcement learning (IRL; Ziebart et al. (2008) ; Ho & Ermon (2016) ) face a similar global exploration problem to that of reinforcement learning— in the worst case, needing to explore all paths through the state space to find one reward (e.g. a tree-structured problem with sparse rewards) ( Kakade, 2003 ; Swamy et al., 2023 ) . Thus, in order to focus the exploration on useful states, efficient imitation algorithms leverage the expert’s state distribution. In particular, rather than resetting the learner to the true starting state distribution, the learner is instead reset to states from the expert’s demonstrations, resulting in an exponential decrease in interaction complexity ( Swamy et al., 2023 ) . We refer to this family of reset-based techniques as efficient IRL . At a high level, efficient IRL can be understood as replacing the hard problem of global exploration with a local exploration problem over the reset distribution —in this case, the expert states. In other words, the RL subroutine can be thought of as optimizing a policy with a similar state visitation distribution to the expert. However, prior work in efficient IRL makes the crucial assumption expert realizability ( Swamy et al., 2023 ) . In the realizable setting, the best policy in the learner’s policy class is the expert policy, so the goal of local policy search is to recover the expert policy, in which case it is natural to perform local policy optimization over the expert states. However, in the misspecified setting, there may be no policies in the policy class that match the expert’s state distribution, leading to two possible pitfalls of performing local policy search over expert states. First, while the learner may be able to optimize the policy at expert states (i.e. after being reset to them), the learner may not be able to reach the expert states. For example, even if a learned