AffordGen: Generating Diverse Demonstrations for Generalizable Object Manipulation with Affordance Correspondence
Jiawei Zhang, Kaizhe Hu, Yingqian Huang, Yuanchen Ju, Zhengrong Xue, Huazhe Xu · 2026
Read AffordGen: Generating Diverse Demonstrations for Generalizable Object Manipulation with Affordance Correspondence by Jiawei Zhang, Kaizhe Hu, Yingqian Huang, Yuanchen Ju, Zhengrong Xue, Huazhe Xu (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-10579.txt.
Introduction
Zhang and five colleagues grow robot demo variety from 3D shape matches. Imitation learners stall when training shapes vary too little. Their AffordGen frame links matching keypoints across huge 3D mesh sets to spawn new paths. The affordance aware set then trains a closed loop vision motor policy. Tests in sim and real rooms show high wins and zero shot jumps to unseen objects.
Catalog record
| CREATOR | Jiawei Zhang, Kaizhe Hu, Yingqian Huang, Yuanchen Ju, Zhengrong Xue, Huazhe Xu |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 44040 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.10579
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
AffordGen: Generating Diverse Demonstrations for Generalizable Object Manipulation with Affordance Correspondence Jiawei Zhang, Kaizhe Hu, Yingqian Huang, Yuanchen Ju, Zhengrong Xue, Huazhe Xu Abstract Despite the recent success of modern imitation learning methods in robot manipulation, their performance is often constrained by geometric variations due to limited data diversity. Leveraging powerful 3D generative models and vision foundation models (VFMs), the proposed AffordGen framework overcomes this limitation by utilizing the semantic correspondence of meaningful keypoints across large-scale 3D meshes to generate new robot manipulation trajectories. This large-scale, affordance-aware dataset is then used to train a robust, closed-loop visuomotor policy, combining the semantic generalizability of affordances with the reactive robustness of end-to-end learning. Experiments in simulation and the real world show that policies trained with AffordGen achieve high success rates and enable zero-shot generalization to truly unseen objects, significantly improving data efficiency in robot learning. Figure 1 : AffordGen overview. (a) Diverse trajectory generation for novel objects via one-shot demonstration. (b) Superior performance against powerful baselines. (c) Real-world generalization to unseen objects from a single source. 1 Introduction Visuomotor imitation learning has achieved impressive progress in robotic manipulation ( 2 ; 36 ; 12 ; 29 ; 10 ; 28 ) . However, its practical deployment is hindered by two fundamental challenges: the prohibitive cost of collecting large-scale, high-quality human demonstrations, and the poor generalization of learned policies to novel objects and scenarios not encountered during training 35 . This reliance on extensive demonstration constrains the applicability of imitation learning in diverse real-world applications. Synthetic data generation offers one promising remedy. Recent works like DemoGen ( 33 ) can expand a single demonstration into hundreds of spatially diverse trajectories, greatly improving data efficiency. Nevertheless, these approaches face a critical limitation: they primarily augment spatial relationships for a single object instance. As a result, they inherit the restricted semantic scope of the source demonstration and generalize poorly beyond the source object. They also focus more on the translation invariance of the given task, exhibiting a limited ability to deal with different orientations. Affordance-based methods, such as Robo-ABC ( 8 ) and DenseMatcher ( 39 ) , leverage semantic correspondence to transfer affordance knowledge to unseen objects. They can transfer the manipulation method of one instance of an object to the other, enabling one-shot imitation learning methods with cross-category generalization, such as FUNCTO ( 23 ) possible. However, these approaches are typically planning-centric, and rely heavily on the accuracy of the mapped affordance point and the planning algorithm. The execution of the policy just follows the pre-computed, open-loop trajectories, lacking the reactive ability of learning-based closed-loop policies. Thus, while affordance research is highly active, a systematic approach to effectively integrate this semantic knowledge into learning-based pipelines is still missing. In this work, we introduce AffordGen, a novel framework that repurposes affordance information as a generative prior for policy learning. AffordGen produces feasible training data on unseen objects that may come from a different category. Starting from a few human demonstrations, we establish correspondences between the demonstrated object’s keypoints and a large set of unseen 3D models. Then, we synthesize diverse yet semantically grounded trajectories in the style of DemoGen, but crucially over novel object instances and full 6D spatial relations. In this way, AffordGen scales a handful of demonstrations into thousands of high-quality, affordance-aware trajectories that cover a diverse realm of objects. The key innovation of AffordGen lies in its novel use of affordance knowledge to generate trajectories. We argue that affordance knowledge is best utilized not through online planning, but as a powerful guide of data generation. By leveraging affordances to generate a large, diverse dataset of plausible trajectories, we can then train a reactive, closed-loop visuomotor policy that inherits both the semantic generalizability of affordances and the robustness of end-to-end learning. AffordGen demonstrates impressive performance on both simulation and real-world tasks, achieving an average performance boost of 24.1% and 24.3% over the best baseline on simulation and real tasks. It can generate thousands of meaningful trajectories over hundreds of different meshes, achieving a high success rate from as few as one source demonstration. AffordGen boosts the model performance on both generated “seen” objects and truly unseen objects that are not in the generated dataset. Beyond these abilities, it also has the potential to generate new data on completely different objects, as long as they share the same manipulation type. We also find that the object-level generation ability increases first and then decreases as we extend the generation to more unseen objects, guiding future cross-instance generation works. The contribution of AffordGen can be summarized below: • We introduce AffordGen, a novel framework that repurposes affordance correspondence as a generative source, enabling the synthesis of diverse and semantically meaningful demonstrations. • We demonstrate that our method can scale a minimal number of human demonstrations into thousands of trajectories across novel object categories and full 6D poses, overcoming the semantic and geometric limitations of prior data augmentation techniques. • We validate through extensive experiments that policies trained with AffordGen achieve significant zero-shot generalization to unseen objects, presenting a new paradigm for data-efficient robot learning. 2 Related Works 2.1 Data Generation for Robot Manipulation Automated data generation techniques for robot manipulation have great potential to alleviate the data problem of embodied AI. These methods primarily fall into two categories: adapting existing demonstrations and generating new data from scratch. The first paradigm, trajectory adaptation, is exemplified by MimicGen 14 and its variants 7 ; 4 ; 16 , which segment a few source demonstrations into object-centric skills and replay them in new spatial configurations. While powerful, these methods rely on costly on-robot or in-simulation rollouts to capture the final interaction data. DemoGen 33 addresses this bottleneck by introducing a fully synthetic pipeline that generates paired actions and observations from direct 3D point cloud editing. More related to our approach, CPGen 11 extends DemoGen by stretching and transforming the source mesh to improve the diversity of generated data and the generalization ability of the policy. These methods, however, are still limited to the source object and have poor generalization ability to unseen objects even within the same category. The second paradigm leverages large-scale generative models to create data from scratch. Systems like GenSim 30 , GenSim2 6 , and RoboGen 31 use Large Language Models (LLMs) to propose novel tasks, generate corresponding scenes, and script automated solvers to create demonstrations. This approach excels at generating task-level diversity but is limited by the capability of the underlying automated solvers, which may not match the quality of human demonstrations. Orthogonal to this, another line of work 1 ; 34 ; 13 ; 5 uses diffusion models to augment the visual appearance of existing data, enhancing a policy’s visual robustness but not its ability to generalize to new spatial configurations or object types. Figure 2 : 1. AffordGen takes in a source expert demonstration and splits it into different functioning segments. 2. We extract keypoints on the source object and establish correspondences between them and many target objects in 3D space through visual foundation models. 3. By transferring the task-related segments and planning the transition segments, AffordGen can generate diverse and large-scale trajectories on both in-category and cross-category objects. 2.2 Semantic