Consistent Zero-Shot Imitation with Contrastive Goal Inference
Kathryn Wantlin, Chongyi Zheng, Ben Eysenbach · 2025
Read Consistent Zero-Shot Imitation with Contrastive Goal Inference by Kathryn Wantlin, Chongyi Zheng, Ben Eysenbach (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-17059.txt.
Introduction
Kathryn Wantlin, Chongyi Zheng, and Benjamin Eysenbach teach agents to copy demonstrations in a 2025 paper. Zero-shot imitation means copying an expert from one demonstration. The agent gets no practice runs or updates at test time. Success hinges on guessing what goal the expert pursued.
Their framework, CIRL, turns imitation into goal inference. Many useful tasks, they note, reduce to reaching one goal state. During pretraining, no rewards or demonstrations are used at all. A contrastive method learns goal-reaching policies for many goals. A goal proposal tool drives exploration toward interesting states. A probabilistic model then infers goals from observed trajectories. Proofs show the method recovers the demonstrator's intent consistently. Earlier similar methods can pick the wrong goal, they show.
Catalog record
| CREATOR | Kathryn Wantlin, Chongyi Zheng, Ben Eysenbach |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 51 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2510-17059
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Consistent Zero-Shot Imitation with Contrastive Goal Inference Kathryn Wantlin, Chongyi Zheng, Ben Eysenbach Abstract In the same way that generative models today conduct most of their training in a self-supervised fashion, how can agentic models conduct their training in a self-supervised fashion, interactively exploring, learning, and preparing to quickly adapt to new tasks? A prerequisite for embodied agents deployed in real world interactions ought to be training with interaction, yet today’s most successful AI models (e.g., VLMs, LLMs) are trained without an explicit notion of action. The problem of pure exploration (which assumes no data as input) is well studied in the reinforcement learning literature and provides agents with a wide array of experiences, yet it fails to prepare them for rapid adaptation to new tasks. Today’s language and vision models are trained on data provided by humans, which provides a strong inductive bias for the sorts of tasks that the model will have to solve (e.g., modeling chords in a song, phrases in a sonnet, sentences in a medical record). However, when they are prompted to solve a new task, there is a faulty tacit assumption that humans spend most of their time in the most rewarding states. The key contribution of our paper is a method for pre-training interactive agents in a self-supervised fashion, so that they can instantly mimic human demonstrations. Our method treats goals (i.e., observations) as the atomic construct. During training, our method automatically proposes goals and practices reaching them, building off prior work in reinforcement learning exploration. During evaluation, our method solves an (amortized) inverse reinforcement learning problem to explain demonstrations as optimal goal-reaching behavior. Experiments on standard benchmarks (not designed for goal-reaching) show that our approach outperforms prior methods for zero-shot imitation. Code: https://github.com/kwantlin/contrastive-irl 1 Introduction Figure 1: Zero shot imitation learning. Assuming access to a multi-task environment, our generalist agent must imagine and practice its own tasks to effectively imitate unknown task demonstrations at test time. Today’s AI agents, whether in language or robotics, are trained primarily by mimicking human demonstrations. But, in the same way that children conduct a large degree of learning in an unsupervised (adult-free) fashion ( Gweon & Schulz 2019 ; Gopnik 2020 ; Stahl & Feigenson 2015 ; Poli et al. 2025 ; Bonawitz et al. 2011 ) , how might AI agents develop a foundation of knowledge through exploration and play, rather than through mimicry? In this paper, we study the setting where agent pretraining is done with no demonstrations, no internet-scale data, and no rewards, but rather through self-supervised practice. The agent proposes goals, attempts to reach them, and learns from these self-collected data. After training, this agent is assessed by its ability to imitate: given a demonstration, the agent uses a (learned) inverse RL module to infer the demonstrator’s goal, and then uses the (learned) goal-conditioned policies to reach that goal. Our problem setting is thus zero-shot imitation learning (IL) , where we would like to infer behaviors from a single demonstration without additional updates ( Pirotta et al. 2024 ; Pathak et al. 2018 ; Jang et al. 2021 ) . It is unclear whether today’s recipe for building generative AI foundation models will be directly applicable to interactive settings. While generative models are primarily built by optimizing self-supervised objectives on input data ( Bommasani et al. 2021 ) , doing so requires that a human can supply these data and assumes that the goal of agents is to find patterns in data. The key premise of agents is that they act, and that their actions have consequences, yet this recipe for building generative AI models does not include an explicit notion of exploration or action. In robotics, policies are typically constructed by either mimicking human demonstrations ( Chi et al. 2023 ; Chi et al. 2024 ; Octo Model Team et al. 2024 ; Reed et al. 2022 ) or maximizing human-specified rewards ( Silver et al. 2016 ; Wurman et al. 2022 ) . These approaches do have an explicit notion of action, but agents typically practice on limited set of tasks, and those are not required to learn how to infer a human’s intention. The key idea in our paper is that self-supervised pretraining for agentic systems should involve interaction. Such pretraining involves exploration: agents should propose their own goals and learn to reach them via trial and error. Such pretraining also requires inverse RL, inferring the desired goal from a human demonstration. When inferring goals, there is an important yet subtle difference between intentions and outcomes: a person that takes a 6-hour flight to attend a 3-hour wedding does not enjoy flights 2\times more than weddings. Related work in inferring intentions projects a demonstration onto a hypothesis space of reward functions and then trains a general-purpose zero-shot RL policy to this space of rewards. We make the additional key observation that many tasks can be described in terms of goals, such as navigation or manipulation tasks ( Brockman et al. 2016 ) . In these settings, goals are described by the agent’s state, and we can imagine natural extensions of goals to more complex behaviors by including position, velocity, acceleration, etc. into the state space. Tasks where the necessary actions are more complex or hierarchical, such as cooking a recipe in a kitchen, could also be described by a high-dimensional observational state, demonstrating the expressivity of a goal-conditioned inductive bias. In addition, maintaining a prior that tasks can be described via goals allows us to define reward functions probabilistically in terms of whether we will reach the goal state in the future and apply state-of-art goal conditioned reinforcement learning methods (GCRL) to an even further reduced hypothesis space of reward functional forms ( Kaelbling 1993 ; Schaul et al. 2015 ; Andrychowicz et al. 2017 ) . Therefore, we re-imagine solving the zero-shot imitation learning task by first inferring the expert’s goal (thereby projecting onto the restricted space of goal-conditioned reward functions fully parameterized by goal vectors), and then commanding a zero-shot goal-conditioned RL policy to this inferred goal. We start by assessing our method on goal-reaching tasks, and then evaluate on reward-maximization tasks not tied to particular goal states. Our main contributions can be summarized as follows: • We propose a contrastive inverse reinforcement learning algorithm (CIRL) for self-supervised pretraining of interactive agents that extends contrastive reinforcement learning (CRL) methods to the MaxEnt RL setting and includes automatic goal sampling during pre-training. Training involves exploration and learning via trial and error, yet requires no demonstrations, no rewards, and no preferences. • Unlike some structurally similar methods, we prove that our method is consistent: it correctly infers the user’s goal using inverse RL, accounting for the relative difficulty of reaching different goals. • Empirically, we show that our method performs effective autonomous exploration and rapid adaptation in the standard URLB benchmark ( Laskin et al. 2021 ) , outperforming prior zero-shot imitation and zero-shot RL methods. Taken together, our results are a step towards the self-supervised pre-training of agents. 2 Related Work We turn to GCRL benchmarks to test our hypotheses for goal-conditioned zero-shot IL. Several state-of-art methods on goal-reaching RL use variants of temporally contrastive objectives to learn