DextrAH-RGB: Visuomotor Policies to Grasp Anything with Dexterous Hands
Ritvik Singh, Arthur Allshire, Ankur Handa, Nathan Ratliff, Karl Van Wyk, Ritvik Singh 1 , Arthur Allshire 2 , Ankur Handa 1 , Nathan Ratliff 1 , Karl Van Wyk 1 · 2024
Read DextrAH-RGB: Visuomotor Policies to Grasp Anything with Dexterous Hands by Ritvik Singh, Arthur Allshire, Ankur Handa, Nathan Ratliff, Karl Van Wyk, Ritvik Singh 1 , Arthur Allshire 2 , Ankur Handa 1 , Nathan Ratliff 1 , Karl Van Wyk 1 (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-01791.txt.
Introduction
Ritvik Singh and co-authors build in 2024 a robot hand system that grasps varied objects from color images alone. Dexterous grasping is a core hard skill, and past work leaned on depth maps or object poses. Their policy trains first in simulation with full state access.
A fabric-guided teacher learns strong grasps by reward in the fake world. That skill then distills into a color-only student through photo-real rendering, still in simulation. Real tests show firm transfer to unseen objects with new shape, texture, and light. Scores rival depth-based grasp tools on the same tasks.
Catalog record
| CREATOR | Ritvik Singh, Arthur Allshire, Ankur Handa, Nathan Ratliff, Karl Van Wyk, Ritvik Singh 1 , Arthur Allshire 2 , Ankur Handa 1 , Nathan Ratliff 1 , Karl Van Wyk 1 |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 44 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2412-01791
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
DextrAH-RGB: Visuomotor Policies to Grasp Anything with Dexterous Hands Ritvik Singh, Arthur Allshire, Ankur Handa, Nathan Ratliff, Karl Van Wyk, Ritvik Singh 1 , Arthur Allshire 2 , Ankur Handa 1 , Nathan Ratliff 1 , Karl Van Wyk 1 Abstract One of the most important, yet challenging, skills for a dexterous robot is grasping a diverse range of objects. Much of the prior work has been limited by speed, generality, or reliance on depth maps and object poses. In this paper, we introduce DextrAH-RGB, a system that can perform dexterous arm-hand grasping end-to-end from RGB image input. We train a privileged fabric-guided policy (FGP) in simulation through reinforcement learning that acts on a geometric fabric controller to dexterously grasp a wide variety of objects. We then distill this privileged FGP into a RGB-based FGP strictly in simulation using photorealistic tiled rendering. To our knowledge, this is the first work that is able to demonstrate robust sim2real transfer of an end2end RGB-based policy for complex, dynamic, contact-rich tasks such as dexterous grasping. DextrAH-RGB is competitive with depth-based dexterous grasping policies, and generalizes to novel objects with unseen geometry, texture, and lighting conditions in the real world. Videos of our system grasping a diverse range of unseen objects are available at https://dextrah-rgb.github.io/ . Time Fig. 1 : DextrAH-RGB ( Dext erous A rm- H and RGB ) is an end-to-end RGB-based policy that can dexterously grasp a wide variety of objects. I Introduction Controlling multi-fingered robotic hands to naturally and quickly grasp objects has remained a longstanding challenge in robotics. Ideally, techniques must leverage visual, tactile, and proprioceptive sensory streams to generalize to novel objects and quickly react in dynamic, anthropocentric environments. With such capabilities, multi-fingered robots would become a pertinent technology to society with many relevant applications. Recently, significant progress has been made in leveraging reinforcement learning in simulation for manipulation and locomotion. Vectorized physics and sensor simulation have allowed practitioners to easily scale robot experience in dynamic environments, enabling reactive behavior informed by both proprioceptive and vision-based inputs. Moreover, techniques like domain randomization facilitate learning robust and adaptive policies that successfully transfer from simulation to the real world. Despite these powerful tools, the efficacy of many current strategies for learning manipulation skills remains limited. Existing systems almost always factorize the problem of creating a grasping policy which avoids training direct end-to-end RGB-to-action visuomotor policies. One common approach is to cast the problem as a kinematic one of finding a static grasp configuration and leveraging motion planning tools to acquire the targeted grasp. These methods are not continuously reactive and struggle with disturbances, partial observability, and unseen objects. Other methods that do continuous vision-based grasping via simulation are almost always restricted to depth cameras as most simulators are unable to render high-fidelity images efficiently at scale. Although methods relying on depth readings are fairly successful, IR-based depth cameras have issues with translucent and reflective objects and direct sunlight. They are also a source of various geometric distractors that are not easily simulated at scale. To address these challenges, we present DextrAH-RGB, a dexterous arm-hand grasping policy that computes high-rate actions directly from RGB input. Our main contributions are: 1) simulation-only training of an RGB-based FGP, 2) including significant scaling of the perceptual architecture to handle the substantially more challenging (stereo) RGB processing, and 3) successful deployment of our RGB policy in the real world to achieve safe, reliable, and reactive grasping behaviors. To our knowledge, this is the first work to demonstrate successful sim2real transfer of RGB policies for a robot hand-arm system for such complex, contact-rich tasks. Our method allows for training of both monocular and stereo RGB policies and can straightforwardly be extended to an arbitrary number of cameras in the scene. II Related Work Dexterous grasping has been studied extensively with a rich history of prior work. Classical methods typically involve optimizing analytic grasp metrics [ 16 , 5 , 7 ] . These works are typically limited to synthesizing precision grasps. They also rely on groundtruth object models and their performance suffers when accurate models are unavailable [ 13 ] . Data driven methods to dexterous grasping involve leveraging grasp datasets to train grasp planning networks. Some examples of these datasets include MultiDex [ 11 ] , DexGraspNet [ 27 ] , Grasp’D-1M [ 25 ] , and Get a Grip [ 13 ] . A lot of past work involves grasp synthesis based on pointcloud/depth information. Not only do they capture the geometry of the objects, but are also more feasible to simulate compared to high-fidelity RGB rendering. UniDexGrasp learns dexterous grasping policies from full pointclouds [ 31 ] and UniDexGrasp++ improved upon this method by introducing a geometry-aware curriculum [ 26 ] . Both of these works only demonstrate results in simulation. DexRepNet [ 12 ] learns a representation that combines spatial geometric features of the hand and object and demonstrate successful sim2real results; however, they require access to CAD models of their objects in order to register pointclouds for pose estimation, thereby limiting the generalizability of their method. DexPoint is able to demonstrate successful sim2real transfer of their pointcloud policies. Their key observation is that when deploying in the real world, there are typically many points on the finger that are missing due to occlusions. They use the proprioceptive states of the robot to perform forward kinematics on the hand and sample various points on the mesh of the fingers to fill in the missing points and feed this combined pointcloud into their policy [ 19 ] . Agarwal et al. [ 1 ] predict a pre-grasp pose by matching DINO-ViT features with previous instances of objects. They then have a proprioceptive-only policy execute the motion to realize the grasp. The policy predicts weights for different eigengrasps that they calculate by performing PCA on a set of grasp poses gathered from motion capture data. Singh et al. pretrain their policies on human videos [ 22 ] by reconstructing objects in simulation and retargeting human hand motions to robot hand actions. After pre-training, they are able to demonstrate impressive sim2real results by finetuning with PPO in simulation to train a policy conditioned on the depth image. The work that is most similar to ours is DextrAH-G [ 14 ] . This work also uses a student-teacher distillation framework, but distill instead to a simpler depth processing policy. They first train a state-based teacher policy to pick up objects from the Visual Dexterity [ 3 ] dataset. In order to allow for more functional grasps, they transform the action space for the hand into a PCA subspace of the robot hand finger joint motions derived from retargeting human grasping data. This action space is also embedded into a geometric fabric [ 20 ] to ensure safe and reactive controllers. The teacher is then distilled into the depth-based student and exhibits remarkable sim2real transfer, being able to grasp objects unseen in the training set. III DextrAH-RGB In this section, we detail the training of our RGB-based policy. To ensure safety of the robot, we use geometric fabrics. This also has the benefit of exposing an action space that lends itself well to the task of dexterous