SAIL: Test-Time Scaling for In-Context Imitation Learning with VLM

Makoto Sato, Yusuke Iwasawa, Yujin Tang, So Kuroki · 2026

Read SAIL: Test-Time Scaling for In-Context Imitation Learning with VLM by Makoto Sato, Yusuke Iwasawa, Yujin Tang, So Kuroki (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-08269.txt.

Introduction

Sato and colleagues present SAIL, a framework that reframes robot learning as an iterative polish problem that scales with test-time compute. In-context imitation learning lets robots learn skills from demonstrations, yet one-shot trajectory generation stays fragile under environmental variation. SAIL uses Monte Carlo Tree Search, where each node is a complete path and edges correspond to path refinements. Guidance comes from a stored archive of successful paths, a vision-language-model scoring mechanism for path evaluation, and step-level feedback with path-aligned scores.

Catalog record

CREATORMakoto Sato, Yusuke Iwasawa, Yujin Tang, So Kuroki
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE37959
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.08269

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

SAIL: Test-Time Scaling for In-Context Imitation Learning with VLM Makoto Sato, Yusuke Iwasawa, Yujin Tang, So Kuroki Abstract In-context imitation learning allows robots to acquire skills from demonstrations, yet one-shot trajectory generation remains fragile under environmental variation. We propose SAIL, a framework that reframes robot imitation as an iterative refinement problem capable of scaling with test-time compute. SAIL utilizes Monte Carlo Tree Search, where each node is a complete trajectory and edges correspond to trajectory refinements. The process is guided by three core components: an automated archive of successful trajectories for contextually relevant retrieval, a vision language model-based scoring mechanism for trajectory evaluation, and a step-level feedback that provides trajectory-aligned scores for iterative refinement. Experiments across six diverse manipulation tasks in simulation and real-world validation clearly demonstrate that increasing test-time compute consistently improves success rates, achieving up to 95% on complex tasks. Our results suggest that trajectory-level test-time scaling is a robust path toward more generalizable robotic agents. I Introduction In-context learning has emerged as a compelling paradigm for robots to acquire complex skills directly from a few visual hints and trajectory demonstrations [ 29 , 7 , 10 ] . However, a fundamental bottleneck remains because a single realized trajectory often fails to generalize when the robot encounters new initial conditions. While current vision language models (VLMs) can generate trajectories [ 38 , 24 ] , they typically treat this as a one-shot prediction task. This limitation means that the success of the system is capped by the initial guess of the model, which cannot adjust for environmental variations during the inference pass. In this work, we propose that robot imitation should instead be framed as an iterative search problem where performance can be improved by scaling test-time compute. By treating full trajectories as the objects of search, we enable the system to explore potential solutions in a trial-matched simulator before execution. This approach shifts the focus from static imitation to a dynamic paradigm where a robot can effectively think longer to resolve the ambiguities of a novel task. The reliance on foundation models’ one-shot inference is problematic because the high dimensional space of robot trajectories is extremely sensitive to small errors in initial state estimation or object localization. Even a slight misalignment in the predicted end effector pose can lead to cascading failure during execution which the model cannot correct without an external loop for evaluation. Recent efforts have attempted to mitigate this fragility by either distilling procedural reasoning into more robust visual representations [ 12 , 20 ] , or employing retrieval mechanisms to provide the model with richer context from successful historical demonstrations [ 15 ] . However, these methods typically focus on improving the quality of a single prediction rather than enabling the system to systematically explore and refine the continuous motion space itself. While some frameworks incorporate external validators to guide corrections [ 8 , 30 ] , they often remain limited to symbolic planning and do not address the fundamental need for test-time scaling at the level of full robot trajectories. To address these challenges we introduce SAIL ( S c a ling I n-context Imitation L earning), a novel framework that enables in-context imitation to scale through test-time compute (see Figure left). Rather than relying on a VLM’s single prediction, SAIL treats trajectory generation as a refinement problem, and introduces three key components to guide the refinement process toward success. First, we formulate trajectory generation in a Monte Carlo Tree Search (MCTS) setting where each node represents a complete robot trajectory. This enables the system to systematically explore the continuous motion space rather than depending on a single one-shot plan. Second, we maintain an automated archive of successful trajectories which are retrieved to provide contextually relevant demonstrations for new environmental configurations. This retrieval mechanism allows the system to bootstrap its search by leveraging past experiences in visually similar scenes. Finally, we propose a step-level feedback that uses trajectory-aligned scores to annotate trajectories. This allows the generator to identify specific points of failure and iteratively refine its motion based on granular critiques. We evaluate SAIL across six diverse manipulation tasks in simulation and validate the full pipeline on a real robot system. Our results demonstrate a clear scaling property where increasing the test-time compute budget consistently leads to higher success rates across all tasks (see Figure right). Notably, the system achieves a 95% success rate on the HandOverBanana task and successfully transfers to the physical world with a high success rate on a block moving task using a trial matched digital twin. These findings suggest that scaling search at the trajectory level is a viable path toward more robust and generalizable robotic agents. Our contributions can be summarized as follows: (1) We reformulate in-context robot imitation as a trajectory-level test-time refinement problem, enabling performance to scale beyond one-shot trajectory prediction. (2) To realize this formulation, we develop SAIL that combines trajectory-level MCTS, retrieval-augmented demonstrations, and step-level evaluation from a VLM for iterative refinement. (3) We demonstrate through extensive experiments in simulation and the real world that increasing test-time compute leads to consistent improvements in task success rates. II Related Work II-A Trajectory Generation Using LLMs and VLMs Recent work has demonstrated that large language models (LLMs) and VLMs can be used not only for high-level planning but also for direct generation of executable robot trajectories. Early approaches show that an off-the-shelf LLM can generate a complete trajectory in a single forward pass, producing dense pose sequences that are directly executable in simulation or on real robots [ 16 ] . This direct but unconstrained generation is often brittle, leading to subsequent work that introduces more structured representations to improve robustness, such as tokenizing observations and actions or converting demonstrations into structured textual templates for in-context trajectory or action prediction [ 7 , 35 ] . Related efforts further emphasize structural conditioning by moving from dense trajectory generation to waypoint-level representations explicitly constrained by object kinematics [ 34 ] . In parallel, trajectory editing methods address similar robustness issues by modifying an existing trajectory using language-derived waypoint updates or geometric constraints rather than generating a new trajectory from scratch [ 22 , 13 ] . Overall, these studies establish that direct trajectory generation with foundation models is feasible and can be stabilized through representation design and output-space constraints; however, trajectory generation is still typically performed as a one-shot or lightly iterative process for a single environment instance, and systematic test-time scaling across many initial conditions remains largely unexplored, which this work addresses. II-B Planning and Test-Time Scaling Recent work in robot planning increasingly combines LLMs with external evaluation modules and search processes, rather than treating LLMs as standalone plan generators. In this setting, planning is formulated as an iterative process in which generated plans are examined for feasibility and revised based on detected failures or inconsistencies, instead of being finalized in a single forward pass [ 6 , 33 ] . For long-horizon tasks, verifying plans through constraint satisfaction or motion planning and incorporating the resulting feedback into subsequent generations has been shown to substantially improve executability [ 6 ] . Beyond natural language plans, several studies propose generating plans in structured and verifiable

Read the complete text