Learning in ImaginationLand: Omnidirectional Policies through 3D Generative Models (OP-Gen)

Yifei Ren, Edward Johns · 2025

Read Learning in ImaginationLand: Omnidirectional Policies through 3D Generative Models (OP-Gen) by Yifei Ren, Edward Johns (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-06191.txt.

Introduction

One real demonstration can become a whole imagined practice world for a robot, show Yifei Ren and Edward Johns in a 2025 paper. Modern three-dimensional generative models complete full object shapes from a few images. OP-Gen uses them to augment a single demonstration into many imagined scenes, then learns an omnidirectional policy inside that imagined dataset.

The learned policy works from starting states far from the demonstration, even from the object's opposite side, sharply cutting the demonstrations needed. Real-world experiments cover grasping objects, opening a drawer, and placing trash in a bin. Design choices are ablated, and the method beats recent data-augmentation baselines. The authors present generative imagination as a practical multiplier for robot teaching.

Catalog record

CREATORYifei Ren, Edward Johns
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE31 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2509-06191

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Learning in ImaginationLand: Omnidirectional Policies through 3D Generative Models (OP-Gen) Yifei Ren, Edward Johns Abstract Recent 3D generative models, which are capable of generating full object shapes from just a few images, now open up new opportunities in robotics. In this work, we show that 3D generative models can be used to augment a dataset from a single real-world demonstration, after which an omnidirectional policy can be learned within this imagined dataset. We found that this enables a robot to perform a task when initialised from states very far from those observed during the demonstration, including starting from the opposite side of the object relative to the real-world demonstration, significantly reducing the number of demonstrations required for policy learning. Through several real-world experiments across tasks such as grasping objects, opening a drawer, and placing trash into a bin, we study these omnidirectional policies by investigating the effect of various design choices on policy behaviour, and we show superior performance to recent baselines which use alternative methods for data augmentation. Videos of our experiments can be found on our webpage: https://www.robot-learning.uk/op-gen . Fig. 1: We utilise 3D generative models for data augmentation to enable one-shot imitation learning from a single demonstration, with end-to-end policies from a wrist-mounted RGB camera. Given a single real-world demonstration, a full 3D model of the object is generated through the generative model, which allows for 3D augmentation of an infinite number of “imagined” demonstrations via novel view rendering. The augmented dataset is then used to train an “omnidirectional” policy, enabling tasks to be executed from any initial state, even if that state is significantly different from any in the real-world demonstration. Leveraging the generalisation and high-quality novel view rendering capabilities of 3D generative models, our method can be applied to multiple everyday tasks. † † footnotetext: 1 The Robot Learning Lab at Imperial College London. I Introduction Demonstrations are a powerful source of data for policy learning, and in recent years, this has led to impressive examples of reactive and dexterous policies [ 1 , 2 ] . However, typical behavioural cloning methods would likely require hundreds of demonstrations to learn a policy for the tasks in Figure 1 . This is because to learn robust omnidirectional policies (OP) — which can succeed from any initial state — the demonstrations must densely cover a very large distribution of states. In this work, we ask whether these omnidirectional policies can instead be learned from only a single demonstration with a wrist-mounted RGB camera, by bringing in recent 3D generative models [ 3 ] . Having been trained on huge datasets of objects, these models enable images to be imagined from any viewpoint of a new object, when given only a small number of images of that object. For example, they can produce the “back” of an object when only its “front” is observed. Given real images observed during a single demonstration, we therefore propose that images from novel viewpoints can be imagined and automatically labelled with actions. A policy trained on this “imagined” dataset would then be in-distribution for any view of the object, even if that view is from the opposite side of the object from where the demonstration was collected. This can be seen as a form of dataset augmentation. Prior works such as DemoGen [ 4 ] reconstruct partial point clouds from demonstrations for this purpose, but conventional point cloud methods cannot recover unseen geometry accurately, limiting generalisation when those regions are relevant at test time. Fortunately, with the arrival of 3D generative AI, we now show how full 3D augmentation can be achieved to enable generalisation to viewpoints that were previously entirely unobserved. We introduce OP-Gen : omnidirectional policies through 3D generative models. Similar to [ 4 ] , we first decompose the single demonstration into motion and skill phases. Then, we generate a full 3D model of the target object by using wrist-camera images captured during a demonstration. Based on the 3D model, collision-free end-effector (EEF) trajectories as new motions could be generated that guide the robot to return to, and then follow, the original skill . By rendering synthetic images using the generated object model, we then create a large “imagined” dataset of image-action pairs and use these to train a diffusion policy [ 5 ] which is then robust to any initial EEF pose where the object is within the camera’s view. To the best of our knowledge, this is the first method for 6-DoF policy learning that uses 3D generative models for single-demonstration augmentation from a wrist-mounted RGB camera. Since current 3D generative models are typically designed for single-object generation, in our real-world experiments, we focus on single-object tasks such as grasping objects, opening a drawer, and placing trash into a bin. However, as generative models improve, this method could be extended to full scenes. We show that our omnidirectional policies outperform alternative methods for 3D augmentation, such as rendering images from a point cloud or a NeRF reconstruction. Our ablation studies then explore how the quality of the generated 3D model affects task performance, and we find what the optimal strategy is to generate these imagined datasets when given a single demonstration. II Related Work Efficient Imitation Learning. In robotics, real-world data is sparse and expensive to collect. This makes data-efficient imitation learning crucial, yet current approaches typically require around a hundred demonstrations to train robust policies [ 5 , 6 , 7 , 8 ] . Replay-based approaches mitigate this by positioning the robot at a “bottleneck” pose relative to the target object before replaying the demonstrated interactions [ 9 , 10 , 11 , 12 , 13 ] , which enables one-shot imitation learning. With additional self-supervised data collection, [ 14 ] achieves high performance on contact-rich tasks from a single demonstration. Novel in-context learning frameworks further enhance efficiency [ 15 , 16 ] , and recent works even push the boundaries toward zero-shot manipulation [ 17 , 18 , 19 ] . 2D Data Augmentation. Data augmentation has traditionally supported 2D domain randomisation through image cropping, flipping, and lighting variations to improve policy robustness [ 20 , 21 , 22 ] . Additionally, works such as [ 23 , 24 , 25 ] focus on augmenting data to reduce the sim-to-real gap. ROSIE [ 26 ] extends this by modifying backgrounds and introducing distractors, leveraging text-to-image models [ 27 , 28 , 29 , 30 ] . However, none of these techniques enable generalisation to unseen object parts. Our method seamlessly integrates with these approaches, enabling a more comprehensive form of augmentation. 3D Data Augmentation. Given that most robotic tasks operate in 6-DoF spaces, 3D data augmentation provides substantial benefits. SPARTN [ 31 ] employs NeRFs [ 32 ] to perturb end-effector poses for data augmentation but is limited to nearby views. In contrast, our method uses 3D generative models to fully generate the target object, allowing us to sample arbitrary augmented trajectories. While [ 33 ] achieves object-level generalisation with NeRFs and [ 34 ] uses diffusion models for future observation prediction even with dynamic interaction, both lack coverage of unseen regions. MimicGen [ 35 ] and its variations [ 36 , 37 ]

Read the complete text