Constraint-Preserving Data Generation for Visuomotor Policy Learning

Kevin Lin, Varun Ragunath, Andrew McAlinden, Aaditya Prasad, Jimmy Wu, Yuke Zhu, Jeannette Bohg · 2025

Read Constraint-Preserving Data Generation for Visuomotor Policy Learning by Kevin Lin, Varun Ragunath, Andrew McAlinden, Aaditya Prasad, Jimmy Wu, Yuke Zhu, Jeannette Bohg (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-03944.txt.

Introduction

Kevin Lin and co-authors show in 2025 how one expert robot demonstration can spawn many new ones that respect physical constraints. Big demonstration sets power robot learning, but gathering them by hand is slow and costly. Their method stretches a single trajectory to cover new object shapes and poses.

The recipe splits the demo into free space moves and contact skills. Unlike past pose only tricks, it reshapes the contact parts with the new geometry in mind, so grasps stay valid. Policies trained on the synthetic set moved to real robots with no further tuning. They also held up across varied objects. Cheap data that still respects physics could unlock skills that now need huge teleoperation farms.

Catalog record

CREATORKevin Lin, Varun Ragunath, Andrew McAlinden, Aaditya Prasad, Jimmy Wu, Yuke Zhu, Jeannette Bohg
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE35 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2508-03944

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Constraint-Preserving Data Generation for Visuomotor Policy Learning Kevin Lin, Varun Ragunath, Andrew McAlinden, Aaditya Prasad, Jimmy Wu, Yuke Zhu, Jeannette Bohg Abstract Large-scale demonstration data has powered key breakthroughs in robot manipulation, but collecting that data remains costly and time-consuming. We present Constraint-Preserving Data Generation (CP-Gen), a method that uses a single expert trajectory to generate robot demonstrations containing novel object geometries and poses. These generated demonstrations are used to train closed-loop visuomotor policies that transfer zero-shot to the real world and generalize across variations in object geometries and poses. Similar to prior work using pose variations for data generation, CP-Gen first decomposes expert demonstrations into free-space motions and robot skills. But unlike those works, we achieve geometry-aware data generation by formulating robot skills as keypoint-trajectory constraints: keypoints on the robot or grasped object must track a reference trajectory defined relative to a task-relevant object. To generate a new demonstration, CP-Gen samples pose and geometry transforms for each task-relevant object, then applies these transforms to the object and its associated keypoints or keypoint trajectories. We optimize robot joint configurations so that the keypoints on the robot or grasped object track the transformed keypoint trajectory, and then motion plan a collision-free path to the first optimized joint configuration. Experiments on 16 simulation tasks and four real-world tasks, featuring multi-stage, non-prehensile and tight-tolerance manipulation, show that policies trained using our method achieve an average success rate of 77%, outperforming the best baseline which achieves an average of 50%. cp-gen.github.io 1 Introduction Teaching robots to act by imitating humans is one of the most powerful, yet costly, ideas in robotics. At its best, large-scale imitation learning has enabled robots to tie shoelaces, fix broken grippers, and even cook shrimp [ 1 , 2 , 3 , 4 ] . Further scaling promises to unlock broad generalization across complex manipulation tasks. Behind these successes, however, lies a steep cost: human labor, months of continuous robot operation, and heavy infrastructure demands. For example, ALOHA Unleashed [ 3 ] collected 26,000 demonstrations over 8 months across 10 robots and 35 operators, while DROID [ 5 ] gathered 76,000 real-world demonstrations over a year. As the field pushes forward, reducing the burden of demonstration collection will be a key challenge. Automated data generation promises to reduce the manual effort required in data collection. One line of work [ 6 , 7 , 8 , 9 , 10 ] generates demonstrations for scenes with diverse object poses by collecting a handful of teleoperated demonstrations and algorithmically multiplying them using pose transformations and action replay. Conceptually, these works aim to enable data efficient spatial generalization by exploiting \mathrm{SE}(3) equivariance of robot actions with respect to object poses [ 9 ] . However, because they rely on \mathrm{SE}(3) transformations, these works cannot generate demonstrations that adapt to geometric variations (such as different aspect ratios) with a given object instance. For example, robot actions that succeed in hanging a short, wide wine glass onto a rack may fail entirely for a tall, narrow one, even if both are aligned under the same pose transformation. A complementary strategy to ease the burden on manual data collection is to increase learning efficiency by embedding structure—specifically, equivariance—into policy architectures [ 11 , 12 , 13 , 14 , 15 , 16 , 17 , 18 , 19 , 20 , 21 , 22 , 23 ] . One line of work focuses on open-loop manipulation, exploiting symmetry to generalize across different object poses [ 24 , 25 , 26 , 27 ] . More recent work has moved toward closed-loop policy learning, embedding equivariances directly into the policy architecture and directly predicting robot actions [ 28 , 11 , 29 , 30 , 31 , 32 ] . However, embedding equivariance bottlenecks can limit policy expressiveness and be insufficient when dealing with scenes that involve multiple interacting objects or symmetry-breaking variations. In this work, we propose Constraint-Preserving Data Generation (CP-Gen ), a geometry-aware data generation method that takes as few as a single expert demonstration and generates new demonstrations with both novel object poses and novel geometries. With CP-Gen, we aim to achieve the benefits of equivariant policy architectures without the bottleneck of embedding structure directly into policies. Our key insight is to bring geometry awareness into the data generation process through a keypoint-trajectory constraint formulation. Given an expert demonstration, we segment the trajectory into free-space motions (segments replaceable with point-to-point collision-free motion planning) and robot skills (segments requiring some interaction with task-relevant objects). We then formulate each skill as a keypoint-trajectory constraint: selected robot or grasped-object keypoints must track a reference trajectory defined in the frame of a task-relevant object. For example, in the Wine Glass Spiral Hanging task (Figure 1 ), grasping is formulated as a trajectory-tracking constraint between keypoints on the gripper tips and a target pre-grasp trajectory in the wine-glass frame. The spiral insertion introduces a second constraint: keypoints on the glass stem must track a spiral-like trajectory defined relative to the rack. By anchoring keypoints to the reference frame of task-relevant objects, we can sample new object geometries and poses, adapt the keypoint-trajectory constraints according to the transforms applied to the original object geometry, and generate thousands of demonstrations covering diverse shapes and geometries. Using these demonstrations, we can train a visuomotor policy that naturally generalizes robustly across changes in object pose and geometry, without requiring additional human demonstrations. Figure 1: CP-Gen uses one expert demonstration and keypoint-trajectory constraints to generate diverse demonstrations in simulation involving novel object geometries and poses, enabling large-scale policy training and zero-shot sim-to-real transfer. For the Wine Glass Spiral Hanging task, the first keypoint-trajectory is anchored to the wine glass and constrains the motion of points on the end effector. The second keypoint-trajectory is anchored relative to the spiral-shaped wine glass rack, and constrains a keypoint on the wine glass stem to track a complex spiral motion. Our contributions are threefold. First, we propose CP-Gen, a data generation method that produces robot demonstrations from a single expert demonstration using a keypoint-trajectory constraint formulation. Unlike prior work, and due to this formulation, CP-Gen generates data in a geometry aware manner. Second, we validate our keypoint-trajectory data generation formulation on the MimicGen simulation benchmark. We achieve state-of-the-art performance (85% vs. 63%) on the default, pose-variation only task settings, as well as on our proposed simulation benchmark featuring novel object geometries (70% vs. 37%). Third, we demonstrate successful zero-shot sim-to-real on a set of four real-world tasks featuring multi-stage, non-prehensile and tight-tolerance manipulation. 2 Related Work Data Generation for Robotics . One approach to addressing data scarcity in robotics is to generate new demonstrations from teleoperated source data [ 6 , 33 , 7 , 8 , 10 ] . MimicGen [ 6 ] generates demonstrations by resetting object poses and replaying pose-transformed versions of the source trajectory. IntervenGen [ 33 ] extends MimicGen with human interventions mid-trajectory to correct failures, while DexMimicGen [ 8 ] adapts MimicGen for bimanual and dexterous settings. SkillGen [ 7 ] segments demonstrations into skills and motions, alternating between learned policies and motion planning to improve robustness. At the core of these methods is a relative-pose formulation and the exploitation of

Read the complete text