IGen: Scalable Data Generation for Robot Learning from Open-World Images
Chenghao Gu, Haolan Kang, Junchao Lin, Jinghe Wang, Duo Wu, Shuzhao Xie, Fanding Huang, Junchen Ge, Ziyang Gong, Letian Li, Hongying Zheng, Changwei Lv, Zhi Wang · 2025
Read IGen: Scalable Data Generation for Robot Learning from Open-World Images by Chenghao Gu, Haolan Kang, Junchao Lin, Jinghe Wang, Duo Wu, Shuzhao Xie, Fanding Huang, Junchen Ge, Ziyang Gong, Letian Li, Hongying Zheng, Changwei Lv, Zhi Wang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2512-01773.txt.
Introduction
Chenghao Gu and co-authors generate robot training data from open-world images. Generalist robot policies need vast training data. Collecting data on real robots is slow and tied to one setting. Open images hold huge scene variety that matches manipulation tasks well.
Their IGen frame turns such images into robot learning data at scale. Scenes from the wild supply objects, layouts, and contexts. Generated data can pretrain policies before costly robot time. That path could widen robot skills to homes and shops that no lab can rebuild. Image-derived data also spans rare scenes, like disaster zones, where robot trials cannot be staged but photos exist in bulk.
Catalog record
| CREATOR | Chenghao Gu, Haolan Kang, Junchao Lin, Jinghe Wang, Duo Wu, Shuzhao Xie, Fanding Huang, Junchen Ge, Ziyang Gong, Letian Li, Hongying Zheng, Changwei Lv, Zhi Wang |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 38 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2512-01773
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
IGen: Scalable Data Generation for Robot Learning from Open-World Images Chenghao Gu, Haolan Kang, Junchao Lin, Jinghe Wang, Duo Wu, Shuzhao Xie, Fanding Huang, Junchen Ge, Ziyang Gong, Letian Li, Hongying Zheng, Changwei Lv, Zhi Wang Abstract The rise of generalist robotic policies has created an exponential demand for large-scale training data. However, on-robot data collection is labor-intensive and often limited to specific environments. In contrast, open-world images capture a vast diversity of real-world scenes that naturally align with robotic manipulation tasks, offering a promising avenue for low-cost, large-scale robot data acquisition. Despite this potential, the lack of associated robot actions hinders the practical use of open-world images for robot learning, leaving this rich visual resource largely unexploited. To bridge this gap, we propose IGen , a framework that scalably generates realistic visual observations and executable actions from open-world images. IGen first converts unstructured 2D pixels into structured 3D scene representations suitable for scene understanding and manipulation. It then leverages the reasoning capabilities of vision-language models to transform scene-specific task instructions into high-level plans and generate low-level actions as S E ( 3 ) SE(3) end-effector pose sequences. From these poses, it synthesizes dynamic scene evolution and renders temporally coherent visual observations. Experiments validate the high quality of visuomotor data generated by IGen, and show that policies trained solely on IGen-synthesized data achieve performance comparable to those trained on real-world data. This highlights the potential of IGen to support scalable data generation from open-world images for generalist robotic policy training. Figure 1 : We propose IGen , a data generation framework that converts open-world images into grounded visuomotor data, enabling scalable data synthesis for robot learning. From a single image, IGen generates large-scale realistic observations and reliable actions. The policies trained solely on IGen-generated data can effectively generalize to real-world scenes and successfully perform manipulation tasks. † † * Equal contribution. \dagger Corresponding author. 1 Introduction Visuomotor policy learning 11 ; 68 ; 5 ; 4 ; 33 has shown great promise in enabling robots to perform open-world manipulation, yet it typically requires large-scale paired visual-action data for effective learning. Unfortunately, collecting such data on real robots is labor-intensive and often limited to specific environments. The high cost and environment-specific nature of robotic data collection remain a fundamental bottleneck to the generalization of visuomotor policies across diverse real-world scenarios 46 . In contrast to on-robot data collection, open-world images can be acquired at extremely low cost and encompass a vast diversity of scenarios that naturally align with real-world robotic tasks. Harnessing such rich visual resources offers a promising path toward building scalable, general-purpose robotic policies 73 ; 37 . Indeed, the remarkable success of large vision–language models (VLMs) 1 ; 3 ; 44 has demonstrated that open-world images provide a powerful foundation for training capable, generalist perception systems. However, robotic policy learning imposes unique demands: it requires not only semantically meaningful visual inputs but also physically grounded, executable action sequences 11 ; 75 ; 19 . This mismatch prevents the effective utilization of open-world images for robot learning. To address the complementary limitations of open-world images and robot-collected data, a growing line of work seeks to derive robot-relevant action representations from unstructured visual data, yet existing approaches remain limited in their ability to harness in-the-wild images for scalable robot learning. For instance, previous real-to-sim-to-real methods 58 ; 40 ; 52 ; 43 ; 30 require explicit reconstruction of physical workspaces to build simulation environments for data generation. This reliance on scene-specific acquisition prevents them from leveraging arbitrary open-world images, fundamentally limiting their scalability. Meanwhile, recent works 6 ; 63 ; 78 ; 72 ; 76 ; 29 leverage vision generative models to predict future visual observations of robot actions within real-world scenes. However, due to the limitations of existing video generation models, they cannot provide explicit robot actions and struggle to generate long-horizon or instruction-driven tasks of higher complexity. In this work, we introduce IGen , a novel framework that takes robotic data synthesis the extra mile by enabling scalable, diverse, and grounded visuomotor data generation from unannotated open-world images with minimal effort. It takes in-the-wild images as the sole visual input and leverages task and motion planning to automatically generate robot behaviors at scale without any human annotation. IGen further synthesizes large-scale visual observations and action trajectories that serve as high-quality training data for effective robot learning. To be more specific, IGen employs the following unified pipeline to generate visuomotor data from open-world images. First, rather than directly operating on unstructured 2D pixels , IGen leverages the strong visual capabilities of large vision models (LVM) to reconstruct scenes as 3D point clouds and spatial keypoints. Next, it utilizes the visual understanding capabilities of vision–language models (VLM) to perform high-level task planning in 3D pixel space and transform motions into low-level control functions. Meanwhile, to generate robot–environment interactions, IGen uses the end-effector SE(3) trajectory to perform rigid-motion–based synthesis of the scene point cloud sequence. Finally, frame-wise rendering is applied to the entire point cloud sequence, generating action-consistent visual observations of the target manipulation task. We evaluate IGen along three dimensions: (1) visual fidelity, measured by the consistency between the generated visual observations and real-world perception; (2) action quality, quantified by the scores of instruction following and physics alignment of the generated action videos; and (3) policy transferability, assessed through the performance of visuomotor policies trained on generated data and evaluated on real-world tasks. Experiments reveal that IGen generates visually realistic images and physically reliable robotic behaviors that align with specific task instructions, providing high-quality data for effective robot learning. Notably, results demonstrate that policies trained solely on IGen-generated data–without any human-annotated demonstrations–can outperform those trained on real-world trajectories. In summary, our contributions are as follows: • We propose an effective data generation framework that produces scalable visual-action datasets from open-world images, integrating cross-scene generalization, instruction diversity and long-horizon task applicability. • We transform unstructured open-world images into actionable 3D scene representations that enable robotic task reasoning and motion planning, generating task-consistent behaviors aligned with the physical world. Furthermore, we introduce a simulation-free point cloud synthesis approach that produces realistic visual observations and supports large-scale robot experience generation. • We demonstrate that robotic policies trained on IGen-generated data can successfully perform real-world manipulation tasks without any additional data collection. This suggests the promising potential of IGen to establish real-world images as an effective source for robot policy training, paving the way for scalable, annotation-free robot learning. 2 Related Work 2.1 Visuomotor Robot Learning Recent progress in visuomotor policy learning 11 ; 68 ; 75 ; 19 ; 77 ; 21 enables robots to perform real-world manipulation tasks through end-to-end imitation learning from paired visual and action demonstrations. Inspired by the development of large-scale vision–language models (VLM) 1 ; 3 ; 44 , recent efforts have focused on scaling up model size and data volume to build more generalist robotic models 56 ; 24 ; 5 ; 28 ; 4 ; 33 ; 55 ; 42 ; 8 . Meanwhile, many studies 26 ; 25 ; 50 leverage the planning capabilities of VLMs to enable open-world robotic manipulation. Existing datasets 57 ; 49 ; 16 ; 17 have collected large-scale