DexMachina: Functional Retargeting for Bimanual Dexterous Manipulation
Zhao Mandi, Yifan Hou, Dieter Fox, Yashraj Narang, Ajay Mandlekar, Shuran Song · 2025
Read DexMachina: Functional Retargeting for Bimanual Dexterous Manipulation by Zhao Mandi, Yifan Hou, Dieter Fox, Yashraj Narang, Ajay Mandlekar, Shuran Song (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-24853.txt.
Introduction
Zhao Mandi and co-authors teach robot hands to copy human hand work in a 2025 paper. The task is hard when both hands are used over long stretches on objects with moving parts.
Their method, DexMachina, uses virtual controllers that drive the object toward its target states at first. The help fades over time, so the learned policy takes over bit by bit under guidance on motion and contact. The team also releases a test set with varied tasks and robot hands. Their method beat the baselines by a clear margin. Results from the tests also point to which traits matter most in the design of dexterous hands.
Catalog record
| CREATOR | Zhao Mandi, Yifan Hou, Dieter Fox, Yashraj Narang, Ajay Mandlekar, Shuran Song |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 47 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-24853
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
DexMachina: Functional Retargeting for Bimanual Dexterous Manipulation Zhao Mandi, Yifan Hou, Dieter Fox, Yashraj Narang, Ajay Mandlekar, Shuran Song Abstract We study the problem of functional retargeting: learning dexterous manipulation policies to track object states from human hand-object demonstrations. We focus on long-horizon, bimanual tasks with articulated objects, which is challenging due to large action space, spatiotemporal discontinuities, and embodiment gap between human and robot hands. We propose DexMachina, a novel curriculum-based algorithm: the key idea is to use virtual object controllers with decaying strength: an object is first driven automatically towards its target states, such that the policy can gradually learn to take over under motion and contact guidance. We release a simulation benchmark with a diverse set of tasks and dexterous hands, and show that DexMachina significantly outperforms baseline methods. Our algorithm and benchmark enable a functional comparison for hardware designs, and we present key findings informed by quantitative and qualitative results. With the recent surge in dexterous hand development, we hope this work will provide a useful platform for identifying desirable hardware capabilities and lower the barrier for contributing to future research. Videos and more at project-dexmachina.github.io Keywords: Dexterous Manipulation, Reinforcement Learning, Simulation-based Learning Figure 1: Functional Retargeting. We study the problem of functional retargeting, where the goal is to retarget human hand demonstrations into functional dexterous robot policies that manipulate an object to follow the demonstrated trajectory. Our proposed algorithm, DexMachina, achieves functional retargeting from one human demonstration to a variety of existing dexterous hand embodiments over a range of articulated objects. 1 Introduction Dexterous robot hands, with their resemblance to human hands, spark the expectation for achieving human-level dexterity. Yet the reality presents many hardware and algorithmic challenges that bottleneck the progress in dexterous manipulation. Prior learning-based methods have seen success in relatively simple and short-horizon tasks, but are often limited by manual reward-engineering [ 1 , 2 ] or costly data-collection [ 3 , 1 ] due to the embodiment gap between human hands and dexterous hands. Human hands are hence a natural source for learning guidance. In this work, we formulate learning from human with an emphasis on task capability. We denote the problem as functional retargeting : given a human demonstration, the goal is to learn dexterous hand policies that can manipulate the object to follow the demonstrated trajectory (see Fig. 1 ). This is distinguished from kinematic retargeting [ 3 ] , which produces human-like motions without ensuring feasibility. The problem is even more compelling for long-horizon, bimanual demonstrations with articulated objects, which encompass a significant portion of daily human activities, but pose several key challenges: exploration is difficult under the high-dimensional action space, the intricate contact sequences demand stable and precise hand movements; due to the embodiment gap, human hand motion cannot be directly mapped to feasible robot actions, which limits the scalability of imitation data collection. To address these challenges, we propose DexMachina 1 1 1 Deus ex machina (“god from the machine”), is when a seemingly unsolvable problem is conveniently solved by an external force — much like how our algorithm moves an object by itself before the policy gradually learns to take over, hence the name DexMachina. , a novel curriculum-based RL algorithm for functional retargeting. Precise bimanual coordination is often required to manipulate an object successfully (e.g. opening a waffle iron mid-air, see Fig. 1 ), but naive approaches often get stuck in early failures or suboptimal actions. This motivates us to design a curriculum to allow the policy to explore in a less fragile setting. Our key idea is to use virtual object controllers —they apply control forces that drive an object towards its demonstrated trajectory—and auxiliary motion and contact rewards , which guide the policy to learn task strategies as the virtual controller strength decays. The policy first learns to mimic the human motion without worrying about failing the task, then learns to take over manipulation as the virtual controllers fade away. Despite continuous effort in developing new hands and sensing capabilities [ 4 , 5 , 6 , 7 , 8 , 9 ] , there is a lack for standardized and accessible evaluation benchmarks. To address this, we build a simulation benchmark with a diverse set of 6 dexterous hands and 5 articulated objects [ 10 ] , and provides a unified testbed where new hands and tasks can be easily added and quickly evaluated. On this benchmark, we empirically show that DexMachina significantly outperforms baseline methods, and applies successfully to a wide variety of hands, articulated objects, and long-horizon demonstrations. With an effective algorithm and evaluation benchmark for functional retargeting, it is now possible to make functional comparisons across different hardware: informed by the policy learning performance, we obtain a meaningful measure for both the hands’ functionality and readiness to learn from human guidance. This comparison is generalizable and accessible: our algorithm requires no hand-specific adaptations, and our task environments are fast to run and easy to customize. With the recent surge in the development of robotic hand hardware, we hope this functional comparison will be helpful for making informed decisions for both acquiring and designing new hands. Our contributions are summarized as follows: \bullet We study Functional Retargeting , where we learn feasible dexterous manipulation policies from human hand-object demonstrations. We propose DexMachina , a novel algorithm for functional retargeting based on a curriculum over virtual object controllers and motion and contact guidance. \bullet We introduce the DexMachina benchmark with 6 curated dexterous hand assets and 5 articulated objects, for evaluating both different functional retargeting algorithms and robotic hand designs. \bullet We demonstrate DexMachina achieves state-of-the-art learning performance across a variety of robotic hands and tasks. Our simulation environments and learning algorithms will be open-sourced to facilitate future research. 2 Related Work Reinforcement Learning for Dexterous Manipulation. Reinforcement Learning (RL) has been used for dexterous manipulation tasks such as in-hand object orientation [ 1 , 11 , 12 , 13 , 14 ] and single-hand grasping [ 2 , 15 , 16 , 17 , 18 , 19 ] , but achieving more complex, longer-horizon manipulation remains challenging due to the burden of designing rewards to guide exploration for such tasks. Model-based methods have been applied to tasks such as ball dribbling [ 8 ] and Rubik’s cube turning [ 9 ] , but they require careful engineering for each object and task. In our work, we seek to study bimanual long-horizon tasks where it can be difficult to specify concrete goals or design RL rewards to guide exploration. This motivates our use of human demonstrations, which both act as a goal specification and provide guidance for how to solve the task. Simulation is a common tool to train dexterous hand policies [ 20 ] due to the high exploration cost of running RL on real hardware [ 21 ] . Our simulation benchmark supports evaluation across several dexterous hands and diverse tasks defined by human demonstration data, in contrast to existing RL benchmarks [ 22 , 23 ] for dexterous manipulation. Imitation Learning for Dexterous Manipulation. Imitation learning (IL) is a compelling alternative to RL, since the the use of demonstrations can mitigate or eliminate the burden of exploration,