A Smooth Sea Never Made a Skilled SAILOR : Robust Imitation via Learning to Search

Arnav Kumar JainMila- Quebec AI InstituteUniversité de Montréal, Vibhakar MohtaCarnegie Mellon UniversitySubin KimCornell UniversityAtiksh BhardwajCornell UniversityJuntao RenCornell UniversityYunhai FengCornell UniversitySanjiban ChoudhuryCornell UniversityGokul SwamyCarnegie Mellon University · 2025

Read A Smooth Sea Never Made a Skilled SAILOR : Robust Imitation via Learning to Search by Arnav Kumar JainMila- Quebec AI InstituteUniversité de Montréal, Vibhakar MohtaCarnegie Mellon UniversitySubin KimCornell UniversityAtiksh BhardwajCornell UniversityJuntao RenCornell UniversityYunhai FengCornell UniversitySanjiban ChoudhuryCornell UniversityGokul SwamyCarnegie Mellon University (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-05294.txt.

Introduction

Arnav Kumar Jain and seven co-authors built SAILOR in 2025, a robot learning system that recovers from its own mistakes instead of freezing. The catalog creator field garbles names with affiliations, but the hosted byline is clean. Standard cloning teaches a robot only what an expert did on the expert's own path. One slip takes the robot where its training never went. SAILOR instead learns to search: it learns a world model and a reward model from demonstrations.

At run time, the robot plans with those models to match the expert's outcome even after errors. Across a dozen visual manipulation tasks in three benchmarks, SAILOR beat strong cloning-based rivals trained on the same data. Even five to ten times more cloning data left a gap. The authors also show it spots subtle failures.

Catalog record

CREATORArnav Kumar JainMila- Quebec AI InstituteUniversité de Montréal, Vibhakar MohtaCarnegie Mellon UniversitySubin KimCornell UniversityAtiksh BhardwajCornell UniversityJuntao RenCornell UniversityYunhai FengCornell UniversitySanjiban ChoudhuryCornell UniversityGokul SwamyCarnegie Mellon University
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE51 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2506-05294

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

A Smooth Sea Never Made a Skilled SAILOR : Robust Imitation via Learning to Search Arnav Kumar JainMila- Quebec AI InstituteUniversité de Montréal, Vibhakar MohtaCarnegie Mellon UniversitySubin KimCornell UniversityAtiksh BhardwajCornell UniversityJuntao RenCornell UniversityYunhai FengCornell UniversitySanjiban ChoudhuryCornell UniversityGokul SwamyCarnegie Mellon University Abstract The fundamental limitation of the behavioral cloning (BC) approach to imitation learning is that it only teaches an agent what the expert did at states the expert visited. This means that when a BC agent makes a mistake which takes them out of the support of the demonstrations, they often don’t know how to recover from it. In this sense, BC is akin to giving the agent the fish – giving them dense supervision across a narrow set of states – rather than teaching them to fish : to be able to reason independently about achieving the expert’s outcome even when faced with unseen situations at test-time. In response, we explore learning to search (L2S) from expert demonstrations, i.e. learning the components required to, at test time, plan to match expert outcomes, even after making a mistake. These include (1) a world model and (2) a reward model. We carefully ablate the set of algorithmic and design decisions required to combine these and other components for stable and sample/interaction-efficient learning of recovery behavior without additional human corrections. Across a dozen visual manipulation tasks from three benchmarks, our approach SAILOR consistently out-performs state-of-the-art Diffusion Policies trained via BC on the same data. Furthermore, scaling up the amount of demonstrations used for BC by 5-10 × \times still leaves a performance gap. We find that SAILOR can identify nuanced failures and is robust to reward hacking. Our code is available at https://github.com/arnavkj1995/SAILOR . 1 Introduction The workhorse of modern imitation learning (IL) is behavioral cloning (BC, Pomerleau 1988 ). From training Diffusion Policies (DPs, Chi et al. 2023 ) on per-task expert demonstrations collected via a variety of teleoperation interfaces ( Zhao et al. 2023 ; Chi et al. 2024 ; Wu et al. 2024 ) to Visual-Language-Action models (VLAs, Team et al. 2024 ; Kim et al. 2024 ; Intelligence et al. 2025 ) trained on wider, multi-task datasets ( Khazatsky et al. 2024 ; ONeill et al. 2024 ) , we see the same recipe applied: collecting more data to train more expressive policy models. The latent hope here is that scaling will eventually lead to a “ChatGPT moment” for robotics ( Vemprala et al. 2023 ) . However, even for the simpler problem of language modeling where one doesn’t have to deal with the complexities of embodiment (e.g. grounding, stochastic dynamics, partial observability, safety), simply scaling next token prediction (i.e. BC) was insufficient: we needed interactive learning in the form of Reinforcement Learning from Human Feedback (RLHF, Stiennon et al. 2020 ; Ouyang et al. 2022 ) and more recently, Test-Time Scaling (TTS, Jaech et al. 2024 ; Guo et al. 2025 ), to build robust systems. If internet-scale offline pretraining was insufficient to solve language modeling, it stands to reason that we’ll need similar interactive learning algorithms to train (embodied) agents. Figure 1 : We introduce SAILOR , a method for learning to search from expert demonstrations. By learning world and reward models on a mixture of expert and base-policy data, we endow the agent with the ability to, at test time, reason about how to recover from mistakes that the base policy makes. At heart, this is because even when they are trained on large amounts of data and over expressive policy classes, agents still sometimes make mistakes that take them out of the support of the offline data. This is a fundamental property of sequential decision-making: one has to deal with the consequences of their own prior actions. When faced with the resulting unseen situation, we’d like our agents to attempt to recover and match the expert’s outcome (whenever it is possible to do so). The most direct approach to teach an agent this recovery behavior is to ask a human-in-the-loop to correct mistakes ( Ross et al. 2011 ; Kelly et al. 2019 ; Spencer et al. 2020 ) . While simple, such an approach can be difficult to scale as it fundamentally makes human time the bottleneck for robot learning. In an ideal world, we’d like our robots to be able to learn to recover from their own mistakes without additional human feedback. There are two fundamental capabilities an agent needs to reason at test-time about recovering from mistakes. The first is prediction : to understand the consequences of their proposed actions. The second is evaluation : to know which outcomes are preferable to others. We propose an algorithmic paradigm that allows us to acquire both of these capabilities without requiring any sources of human data beyond the standard imitation learning pipeline. In other words, a better recipe with the same ingredients . In particular, rather than merely learning a policy from expert demonstrations, we propose learning a local world model (WM) and a reward model (RM) from demonstration and base policy data ( Ren et al. 2024b ) . By combining these components with a planning algorithm, we have the capability to, at test time, reason about how to recover from mistakes that the base policy makes by planning against our learned RM inside our learned WM. Thus, rather than mere imitation, we’re learning to search (L2S, Ratliff et al. 2009 ) from expert demonstrations. Our key insight is that we can infer the latent search process required to recover from local mistakes from the same source of human data required for the standard behavioral cloning pipeline . Put differently, in contrast to approaches like BC and DAgger that give the agent the fish – i.e. relying on a human teacher to demonstrate desired or recovery behavior, we focus on teaching the agent to fish : to develop the reasoning process required to match the expert’s outcomes, even when faced with situations unseen in the training dataset , often as a result of the agent’s own earlier mistakes. In our work, we focus on long-horizon visual manipulation tasks and therefore instantiate the L2S paradigm by using base Diffusion Policies ( Chi et al. 2023 ) , Dreamer World Models ( Hafner et al. 2024 ) , and the Model-Predictive Path Integral Control (MPPI, Williams et al. 2017 ) Planner. Concretely, we learn a residual planner ( Silver et al. 2018 ) that performs a local search at test time to correct mistakes the base policy makes. We call our composite architecture SAILOR : Searching Across Imagined Latents Online for Recovery . More specifically, our contributions are three-fold: Figure 2 : Left: we see SAILOR consistently out-perform diffusion policies trained on the same demos across various visual manipulation problems at multiple dataset scales |\mathcal{D}| . Right : our learned reward model is able to detect shared prefixes (black dots and frames), base policy failures (purple dots, lines, and frames) and SAILOR ’s successes (orange dots, lines, and frames). 1. We demonstrate that across a dozen visual manipulation problems at three different dataset scales, SAILOR outperforms Diffusion Policies. Simply scaling up the number of demonstrations used for DP by \approx 5-10 \times still leaves

Read the complete text