GenDexHand: Generative Simulation for Dexterous Hands
Feng Chen, Zhuxiu Xu, Tianzhe Chu, Xunzhe Zhou, Li Sun, Zewen Wu, Shenghua Gao, Zhongyu Li, Yanchao Yang, Yi Ma · 2025
Read GenDexHand: Generative Simulation for Dexterous Hands by Feng Chen, Zhuxiu Xu, Tianzhe Chu, Xunzhe Zhou, Li Sun, Zewen Wu, Shenghua Gao, Zhongyu Li, Yanchao Yang, Yi Ma (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-01791.txt.
Introduction
Feng Chen and nine colleagues generate training worlds for robotic hands. Dexterous manipulation, using many-fingered hands, is starved of data. Language models can auto-generate simulation tasks for simple grippers. But hands with many degrees of freedom need far more specialized environments. Hand-writing them does not scale.
Their pipeline, GenDexHand, produces diverse dexterous tasks and environments on its own. A closed loop refines each scene. It adjusts object placements and scales based on feedback from a vision-language model. This substantially raised the average quality of the generated environments. Each task is then decomposed into sub-tasks so reinforcement learning can proceed sequentially, cutting training time and raising success rates. The authors offer the system as a scalable source of synthetic training data for embodied intelligence.
Catalog record
| CREATOR | Feng Chen, Zhuxiu Xu, Tianzhe Chu, Xunzhe Zhou, Li Sun, Zewen Wu, Shenghua Gao, Zhongyu Li, Yanchao Yang, Yi Ma |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 66 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2511-01791
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
GenDexHand: Generative Simulation for Dexterous Hands Feng Chen, Zhuxiu Xu, Tianzhe Chu, Xunzhe Zhou, Li Sun, Zewen Wu, Shenghua Gao, Zhongyu Li, Yanchao Yang, Yi Ma Abstract Data scarcity remains a fundamental bottleneck for embodied intelligence. Existing approaches use large language models (LLMs) to automate gripper‑based simulation generation, but they transfer poorly to dexterous manipulation, which demands more specialized environment design. Meanwhile, dexterous manipulation tasks are inherently more difficult due to their higher degrees of freedom. Massively generating feasible and trainable dexterous hand tasks remains an open challenge. To this end, we present GenDexHand , a generative simulation pipeline that autonomously produces diverse robotic tasks and environments for dexterous manipulation. GenDexHand introduces a closed‑loop refinement process that adjusts object placements and scales based on vision‑language model (VLM) feedback, substantially improving the average quality of generated environments. Each task is further decomposed into sub‑tasks to enable sequential reinforcement learning, reducing training time and increasing success rates. Our work provides a viable path toward scalable training of diverse dexterous hand behaviors in embodied intelligence by offering a simulation-based solution to synthetic data generation. Our website: https://winniechen2002.github.io/GenDexHand/ . * * footnotetext: Denote equal contributions. ${}^{\ddagger}$ ${}^{\ddagger}$ footnotetext: Corresponding authors. 1 Introduction A long-term goal of artificial general intelligence lies in the development of embodied agents capable of interacting with the real world under autonomous control. Robot learning at scale is particularly promising, as it holds the potential to endow agents with the breadth of skills and robustness necessary for complex real-world deployment ( Intelligence et al. 2025 ; Black et al. 2024 ; Liu et al. 2025 ) . Large-scale, high-quality data emerges as a cornerstone for effective robot learning, particularly in manipulation, where diverse datasets drive improvements in policy robustness and generalization ( Torne et al. 2024 ; Lin et al. 2025 ; Ai et al. 2025 ) . However, constructing complex and diverse simulation environments or collecting data with real-world robotic platforms is both costly and technically challenging, particularly in the case of dexterous hand manipulation tasks ( Chen et al. 2022 ; Lin et al. 2024b ) . In parallel, foundation models ( Anthropic 2025 ; OpenAI 2025 ; Zhuo et al. 2025 ; Team 2023 ) demonstrate strong capabilities in generating formalized code ( Jain et al. 2024 ; Jimenez et al. 2023 ) , making the controllable synthesis of simulation environments through code generation a promising approach to reducing construction costs. Building on the capability of foundation models, prior studies have made initial progress in generative simulation for robotics, particularly in domains such as robotic grippers and locomotion ( Wang et al. 2023 ; Wang et al. 2024a ) . Figure 1: A showcase of 15 diverse and realistic task scenes automatically generated by GenDexHand. RoboGen ( Wang et al. 2023 ) leverages foundation models to generate diverse tasks, but its environments remain confined to gripper manipulation and locomotion. GenSim ( Wang et al. 2024a ) and GenSim2 ( Hua et al. 2024 ) narrow the focus further to simpler manipulation regimes—suction and parallel-jaw gripping—with GenSim2 additionally demonstrating sim-to-real transfer. Yet across these approaches, a consistent gap persists: none address the generation of dexterous hand tasks. This omission raises a central research question: why has the generation of dexterous hand tasks been systematically avoided? Dexterous hands, by virtue of their anatomical structure, possess the capability to execute complex tasks and exhibit greater generalization in manipulation compared to grippers or suction grippers ( Ma & Dollar 2011 ) . However, this potential comes with substantial challenges. To accomplish intricate tasks, dexterous hands require precise coordination among multiple fingers, and achieving such coordinated control has long been recognized as one of the primary difficulties in distinguishing dexterous hand manipulation from gripper or suction-based manipulation. A further source of difficulty arises from the high degrees of freedom (DoFs) inherent to dexterous hands. This substantial increase in controllable dimensions expands the exploration space in reinforcement learning and motion planning, necessitating more precise and fine-grained guidance for effective policy learning. Consequently, imposing constraints or structure on the exploration space is critical for improving both the accuracy and efficiency of learning complex dexterous-hand policies. In this work, we introduce GenDexHand, a generative agent for producing dexterous hand simulation data and corresponding control policies. The pipeline is structured into three stages: (i) task proposal and environment generation, (ii) multimodal large language model refinement, and (iii) policy generation. In the first stage, the system leverages our robotic asset library and object set to propose feasible tasks and synthesize the corresponding scene configurations, objects, and guidance components such as reward functions or goal poses. In the second stage, the generated environments are iteratively refined with the assistance of multimodal large language models to ensure semantic coherence and physical plausibility. Finally, in the third stage, an LLM determines whether a given task should be addressed through motion planning or reinforcement learning; for reinforcement learning, it specifies which finger joints are required, whereas for motion planning, it identifies the appropriate hand position and joint configuration. To further address the intrinsic difficulty of dexterous hand learning, we decompose long-horizon tasks into a sequence of shorter-horizon subtasks and introduce constraints on the action space for specific task categories. For example, in object rotation tasks, the wrist joint is fixed at its initial pose, thereby reducing the effective action dimension and focusing exploration on finger coordination. This structured generation and refinement process enables the creation of high-quality simulation environments and facilitates the learning of effective policies for dexterous manipulation. Our experiments demonstrate that GenDexHand is capable of robustly generating a diverse set of dexterous hand manipulation tasks (see Figure 1 ). Compared to directly generating scenes and policy guidance in a single step, our iterative refinement procedure yields policies with an average improvement of 53.4% on the target tasks. The datasets produced by GenDexHand also exhibit greater diversity than existing dexterous hand datasets, encompassing a broader range of long-horizon and complex tasks. In summary, our work takes a step toward transforming the latent behavioral knowledge embedded in foundation models into dexterous hand data within simulators. By doing so, GenDexHand not only expands the diversity of available dexterous hand data but also lays the groundwork for scaling up simulation-driven training. In contrast to prior generative simulation approaches, our contributions can be summarized as follows: • We introduce GenDexHand , the first generative pipeline specifically targeting dexterous hand manipulation, a domain largely overlooked in prior generative simulation work. • Our framework employs a generator–verifier refinement process, where scenes are rendered, analyzed by multimodal LLMs, and iteratively corrected to ensure semantic plausibility and physical consistency. • We design policy learning strategies tailored for dexterous hands, including degree-of-freedom constraints, motion planning integration, and subtask decomposition, which together enable a 53.4\% average improvement in task success rate over existing baselines. 2 Related Works 2.1 Foundation Models for Robotics Learning With the rapid advancement of language, vision, and multimodal models ( OpenAI 2023 ; Zhang et al. 2023 ) —such as GPT-4o ( OpenAI 2024 ) , GPT-5 ( OpenAI 2025 ) , Claude 4.0 ( Anthropic 2025 ) , and Gemini 2.5 Pro ( Team 2023 ; Comanici et al. 2025 ) —remarkable progress has been achieved