Robotic Programmer: Video Instructed Policy Code Generation for Robotic Manipulation
Senwei Xie, Hongyu Wang, Zhanqi Xiao, Ruiping Wang, Xilin Chen · 2025
Read Robotic Programmer: Video Instructed Policy Code Generation for Robotic Manipulation by Senwei Xie, Hongyu Wang, Zhanqi Xiao, Ruiping Wang, Xilin Chen (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-04268.txt.
Introduction
Senwei Xie and co-authors build a robot brain that writes its own control code in a 2025 paper. Robots still struggle to handle new tasks, arms, and rooms with no fresh training. Code based control can bridge that gap, since large models write code well and skill libraries supply the base moves.
Their model, RoboPro, watches video and free form orders, then writes policy code on the spot. A side method turns wild video into training code with help from vision and code models, which cuts data costs. In sim and real tests, RoboPro beat top rivals on unseen tasks and held up when code formats or skill sets changed.
Catalog record
| CREATOR | Senwei Xie, Hongyu Wang, Zhanqi Xiao, Ruiping Wang, Xilin Chen |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 61 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2501-04268
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Robotic Programmer: Video Instructed Policy Code Generation for Robotic Manipulation Senwei Xie, Hongyu Wang, Zhanqi Xiao, Ruiping Wang, Xilin Chen Abstract Zero-shot generalization across various robots, tasks and environments remains a significant challenge in robotic manipulation. Policy code generation methods use executable code to connect high-level task descriptions and low-level action sequences, leveraging the generalization capabilities of large language models and atomic skill libraries. In this work, we propose Robo tic Pro grammer ( RoboPro ), a robotic foundation model, enabling the capability of perceiving visual information and following free-form instructions to perform robotic manipulation with policy code in a zero-shot manner. To address low efficiency and high cost in collecting runtime code data for robotic tasks, we devise Video2Code to synthesize executable code from extensive videos in-the-wild with off-the-shelf vision-language model and code-domain large language model. Extensive experiments show that RoboPro achieves the state-of-the-art zero-shot performance on robotic manipulation in both simulators and real-world environments. Specifically, the zero-shot success rate of RoboPro on RLBench surpasses the state-of-the-art model GPT-4o by 11.6%, which is even comparable to a strong supervised training baseline. Furthermore, RoboPro is robust to variations on API formats and skill sets. Our website can be found at https://video2code.github.io/RoboPro-website/ . I INTRODUCTION A long-term goal of embodied intelligence research is to develop a single model capable of solving any task defined by the user. Recent years have witnessed a trend towards large-scale foundation models on natural language processing tasks [ 1 , 2 ] . Scaling up these language models in terms of model size and training tokens significantly improves the few-shot performance on a range of end tasks, even achieving performance comparable to previous state-of-the-art fine-tuning methods. However, for robotic tasks, we have yet to see large-scale pre-trained models that can directly transfer across different robots, tasks and environments without additional fine-tuning. To improve the zero-shot generalization ability of robotic models, one common approach is to unify different tasks as the next action prediction. This paradigm requires the model to directly generate low-level action sequences. [ 3 , 4 , 5 ] collected large amount of trajectories across various robots, tasks and environments. They trained vision-language-action (VLA) models derived from LLMs to map images and task instructions into discrete action tokens. Despite these models achieve better performance and show the capacity to transfer on novel objects and different tasks, fine-tuning is still required when deploying on new robots and environments. Besides, it is extremely expensive to collect trajectories through real-world robots, while using human-built simulators often leads to lack of diversity and introduces additional gap between simulation platform and real-world usages. Another line of research aims to use code as compromise solution for bridging high-level instructions and low-level robot execution, leveraging the generalization capabilities of Large Language Models (LLMs) and atomic skill libraries. RoboCodeX [ 6 ] utilizes large vision-language model (VLM) to generate tree-of-thought plans and grasp preference. However, it also relies on manually-built simulation environment and human-annotated code for data curation, which is expensive and not friendly for scaling up in terms of training data. In this work, we introduce Robo tic Pro grammer ( RoboPro ), a robotic foundation model, enabling the capability of perceiving visual information and following free-form user instructions to perform manipulation tasks without additional fine-tuning. RoboPro generates the executable code to connect high-level instructions and low-level action sequences. To address low efficiency and high cost in collecting runtime code data for robotic tasks, we devise Video2Code, an automatic data curation pipeline for multimodal code generation. Fig. 1: Visualization of evaluation tasks and execution results. RoboPro shows impressive zero-shot performance on novel and compositional tasks in RLBench (a), long-termed manipulation tasks in LIBERO (b), and real-world tasks (c). Video demos can be found in our supplementary materials. We draw our inspiration from the extensive amount of operational videos in-the-wild that implicitly contain necessary procedural knowledge about how to finish operational tasks. Previous research has focused on utilizing videos for large-scale supervised learning [ 3 , 5 ] or extracting relevant knowledge (e.g., affordance [ 7 ] ), while extracting executable policy code from videos is still under-explored. Our data curation pipeline uses the off-the-shelf VLM and Code LLM to synthesize code execution data from videos, which is much more efficient and scalable compared with generating code data from manually-built simulation environments. With Video2Code, we synthesize 115k robot execution code data along with the corresponding scene information and task descriptions from DROID [ 8 ] . Extensive experiments (examples depicted in Fig. 1 ) show that RoboPro achieves the state-of-the-art zero-shot performance on robotic manipulation tasks in both simulators and real-world environments. Specifically, the zero-shot success rate of RoboPro on RLBench outperforms the state-of-the-art model GPT-4o by a gain of 11.6%. It is even comparable to a strong supervised training method PerAct [ 9 ] . Furthermore, we make an early attempt to discuss the adaptability of RoboPro across variations on API formats and unseen skill sets. II Related Works II-A Language-Guided Robot Manipulation. Language-conditioned robot manipulation refers to the use of natural language instructions to guide robotic actions. Natural language instructions allow non-experts to interact with robots through intuitive commands and enable robots to generalize to various tasks based on natural language input [ 10 ] . Recent advancements in language-conditioned embodied agents have leveraged Transformers [ 11 ] to enhance performance on multi-task settings. One category of recent approaches is language-conditioned behavior cloning (BC), where models learn to mimic demonstrated language-conditioned actions and output dense action sequences directly. 3D BC methods [ 9 , 12 ] trained from scratch perform well on specific environment, while lacking of generalization ability across environments. Vision-language-action (VLA) models [ 3 , 5 , 13 ] built on pre-trained large language models (LLMs) show capacity to transfer on novel objects and task settings, but need additional fine-tuning when being deployed on new environments and robots. Another line is to create high-level planners based on LLMs [ 14 , 15 , 16 ] , which output step-by-step natural language plans according to human instructions and environmental information. These methods show better generalization ability across environments, leveraging the reasoning and generalization ability of LLMs on language instructions and environments. However, there is still a gap between generated natural language plans and low-level robotic execution, requiring an extra step to score potential actions or decompose plans into relevant policies [ 17 ] . II-B Robot-Centric Policy Code Generation. Code-as-Policies [ 18 ] proposes that executable code can serve as a more expressive way to bridge high-level task descriptions and low-level execution. Atomic skills to perceive 3D environments and plan primitive tasks are provided in predefined API libraries. LLMs process textual inputs and generate executable policy code conditioned on the API libraries [ 17 , 18 , 19 , 20 ] . However, these methods rely solely on linguistic inputs, requiring detailed descriptions of environments and instructions as textual inputs, which limits their generalization and visual reasoning ability across environments. RoboCodeX [ 6 ] utilizes large vision-language model (VLM) to decompose multimodal information into object-centric units in a tree-of-thought format. Nevertheless, it relies on manually-built simulation environments and human-annotated data, which lacks environmental richness and is expensive for scaling