AffordSim: A Scalable Data Generator and Benchmark for Affordance-Aware Robotic Manipulation

Mingyang Li, Haofan Xu, Haowen Sun, Xinzhe Chen, Sihua Ren, Liqi Huang, Xinyang Sui, Chenyang Miao, Qiongjie Cui, Zeyang Liu, Xingyu Chen, Xuguang Lan[1em] School of Artificial Intelligence, Xi’an Jiaotong University · 2026

Read AffordSim: A Scalable Data Generator and Benchmark for Affordance-Aware Robotic Manipulation by Mingyang Li, Haofan Xu, Haowen Sun, Xinzhe Chen, Sihua Ren, Liqi Huang, Xinyang Sui, Chenyang Miao, Qiongjie Cui, Zeyang Liu, Xingyu Chen, Xuguang Lan[1em] School of Artificial Intelligence, Xi’an Jiaotong University (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-11674.txt.

Introduction

Li and twelve colleagues present AffordSim, a data generator and benchmark for affordance-aware robot tasks. Robot skills fail when contact misses the key region an action needs, and generic tools ignore task semantics. Given a task description, AffordSim builds a scene, grounds affordance queries on surfaces, samples region grasps, and picks executable candidates by motion planning. A 50-task benchmark spans five robots and over 500 objects, and AffordSim reaches 93 percent of manual labels' success on affordance-critical tasks and 89 percent on hard composite tasks.

Catalog record

CREATORMingyang Li, Haofan Xu, Haowen Sun, Xinzhe Chen, Sihua Ren, Liqi Huang, Xinyang Sui, Chenyang Miao, Qiongjie Cui, Zeyang Liu, Xingyu Chen, Xuguang Lan[1em] School of Artificial Intelligence, Xi’an Jiaotong University
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE32324
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.11674

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

AffordSim: A Scalable Data Generator and Benchmark for Affordance-Aware Robotic Manipulation Mingyang Li, Haofan Xu, Haowen Sun, Xinzhe Chen, Sihua Ren, Liqi Huang, Xinyang Sui, Chenyang Miao, Qiongjie Cui, Zeyang Liu, Xingyu Chen, Xuguang Lan[1em] School of Artificial Intelligence, Xi’an Jiaotong University Abstract Simulation-based data generation has become a dominant paradigm for training robotic manipulation policies, yet existing platforms do not incorporate object affordance information into trajectory generation. As a result, tasks requiring precise interaction with specific functional regions—grasping a mug by its handle, pouring from a cup’s rim, or hanging a mug on a hook—cannot be automatically generated with semantically correct trajectories. We introduce AffordSim , the first simulation framework that integrates open-vocabulary 3D affordance prediction into the manipulation data generation pipeline. AffordSim uses our VoxAfford model, an open-vocabulary 3D affordance detector that enhances MLLM output tokens with multi-scale geometric features, to predict affordance maps on object point clouds, guiding grasp pose estimation toward task-relevant functional regions. Built on NVIDIA Isaac Sim with cross-embodiment support (Franka FR3, Panda, UR5e, Kinova), VLM-powered task generation, and novel domain randomization using DA3-based 3D Gaussian reconstruction from real photographs, AffordSim enables automated, scalable generation of affordance-aware manipulation data. We establish a benchmark of 50 tasks across 7 categories (grasping, placing, stacking, pushing/pulling, pouring, mug hanging, long-horizon composite) and evaluate 4 imitation learning baselines (BC, Diffusion Policy, ACT, Pi 0.5). Our results reveal that while grasping is largely solved (53–93% success), affordance-demanding tasks such as pouring into narrow containers (1–43%) and mug hanging (0–47%) remain significantly more challenging for current imitation learning methods, highlighting the need for affordance-aware data generation. Zero-shot sim-to-real experiments on a real Franka FR3 validate the transferability of the generated data. 1 1 footnotetext: Equal contribution. 2 2 footnotetext: Corresponding author. Figure 1 : Overview of AffordSim. Natural language task descriptions are processed by a VLM to generate simulation scenes in Isaac Sim. VoxAfford predicts 3D affordance maps on object point clouds, guiding grasp pose estimation and motion planning to produce semantically correct manipulation trajectories. Comprehensive domain randomization, including DA3-reconstructed real backgrounds, enables zero-shot sim-to-real transfer. 1 Introduction Simulation-based data generation has become a dominant paradigm for scaling robotic manipulation training data ( 12 ; 13 ; 7 ; 16 ) . By collecting demonstrations in simulation, researchers can generate thousands of diverse trajectories at low cost, train visuomotor policies via imitation learning, and deploy them on real robots through sim-to-real transfer. However, a fundamental limitation persists across existing simulation platforms: none of them incorporates object affordance information into the trajectory generation pipeline. Object affordances—the functional regions through which objects are meant to be manipulated—are critical for tasks requiring precise, semantically correct interactions. Pouring from a cup requires grasping its handle and tilting from the rim; hanging a mug demands locating the hook-compatible opening. Current simulation platforms handle grasp pose estimation through one of two approaches, both of which fail for affordance-demanding tasks. Manual grasp design requires human engineers to hand-specify grasp poses for each object and task, which is labor-intensive and does not scale. Generic grasp estimation methods such as AnyGrasp ( 4 ) predict stable grasps without considering task semantics, often selecting poses that are physically stable but functionally incorrect (e.g., grasping a mug by its body when the task requires pouring). In our experiments, affordance-demanding tasks such as mug hanging achieve at most 10% policy success when trajectories are generated with generic grasp methods, compared to meaningful improvements with affordance guidance. We propose AffordSim , the first simulation framework that integrates affordance prediction into the robotic manipulation data generation pipeline. AffordSim uses our VoxAfford model ( 21 ) , an open-vocabulary 3D affordance detection method that enhances MLLM output tokens with multi-scale 3D geometric features, to predict affordance maps indicating where and how a robot should interact with objects. These affordance predictions guide grasp pose estimation and motion planning, producing trajectories that are both physically plausible and semantically aligned with task objectives. Built on NVIDIA Isaac Sim ( 14 ) , AffordSim features: • Affordance-aware trajectory generation. VoxAfford predicts 3D affordance heatmaps on object point clouds, guiding grasp pose selection toward task-relevant functional regions. • Automated task-scene generation. A VLM-powered pipeline converts natural language task descriptions into simulation scenes, supporting 4 robot embodiments (Franka FR3, Panda, UR5e, Kinova) without task-specific engineering. • Photorealistic domain randomization. Five randomization axes—including DA3-based ( 20 ) 3D Gaussian reconstruction from real photographs—reduce the visual domain gap for sim-to-real transfer. • A 50-task benchmark across 7 manipulation categories (grasping, placing, stacking, pushing/pulling, pouring, mug hanging, long-horizon composite) with evaluations of 4 imitation learning baselines (BC, Diffusion Policy ( 2 ) , ACT ( 23 ) , Pi 0.5 ( 1 ) ). Our benchmark reveals a key finding: while grasping is largely solved by current imitation learning methods (53–93% success), affordance-demanding tasks such as pouring into narrow containers (1–43%) and mug hanging (0–47%) remain significantly more challenging. This performance gap highlights the need for affordance-aware data generation to address these challenging manipulation tasks. Zero-shot sim-to-real experiments on a real Franka FR3 validate that policies trained exclusively on AffordSim data transfer to the real world, with success rates ranging from 60% on grasping to 10% on mug hanging, mirroring the affordance-complexity gradient observed in simulation. 2 Related Work 2.1 Simulation Platforms for Robotic Manipulation A growing number of simulation platforms provide environments for generating robotic manipulation data. RoboTwin ( 12 ) combines digital twin environments with LLM-driven task generation and multi-embodiment support. RoboCasa ( 13 ) focuses on household environments with large-scale scene generation. ManiSkill2 ( 7 ) provides GPU-parallelized environments for contact-rich manipulation. Meta-World ( 22 ) and CALVIN ( 10 ) offer standardized multi-task benchmarks for policy learning. LIBERO ( 9 ) introduces lifelong learning benchmarks with procedural task generation. RoboVerse ( 16 ) unifies multiple simulators under a common interface. GenManip ( 5 ) uses LLM-driven simulation for generalizable instruction-following manipulation, focusing on language grounding rather than affordance-aware trajectory generation. However, none of these platforms integrates object affordance information into the trajectory generation process . Grasp poses are either manually designed or estimated by generic methods, limiting the range of tasks that can be automatically generated with semantically correct interactions. 2.2 3D Affordance Prediction Affordance prediction has been studied extensively as a standalone perception problem. Where2Act ( 11 ) predicts per-point actionability for articulated objects. 3D AffordanceNet ( 3 ) learns affordance representations from 3D point clouds. GAPartNet ( 6 ) detects generalizable and actionable parts across object categories. Our concurrent work, VoxAfford ( 21 ) , advances this line by injecting multi-scale geometric features from a frozen 3D VQVAE encoder into MLLM output tokens via cross-attention, achieving state-of-the-art open-vocabulary 3D affordance detection ( {\sim}8\% mIoU improvement) with zero-shot transfer to real robots. Despite progress in affordance prediction, these models have been studied in isolation from simulation data generation pipelines. AffordSim bridges this gap by integrating VoxAfford directly into the trajectory generation loop, enabling affordance-aware data collection at scale. 2.3 Automated Data Generation and Domain Randomization LLM- and VLM-driven approaches have enabled automated task and scene generation. GenSim2 ( 8 ) uses LLMs to generate task code and reward functions. RoboGen ( 19 ) automates the generation of diverse manipulation

Read the complete text