HOGraspFlow: Exploring Vision-based Generative Grasp Synthesis with Hand-Object Priors and Taxonomy Awareness
Yitian Shi, Zicheng Guo, Rosa Wolf, Edgar Welte, Rania Rayyes · 2025
Read HOGraspFlow: Exploring Vision-based Generative Grasp Synthesis with Hand-Object Priors and Taxonomy Awareness by Yitian Shi, Zicheng Guo, Rosa Wolf, Edgar Welte, Rania Rayyes (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-16871.txt.
Introduction
Yitian Shi and co-authors teach robots grasps from human hands in a 2025 paper. A single photo of a hand holding an object can show how to grasp it. Their method turns that cue into grasps for a two-finger robot gripper. It needs no exact shape model of the target object.
Three cues guide the synthesis: image features, hand-object contact, and a taxonomy of grasp types. A flow-based generator then proposes grip poses. Real-world tests topped a high success rate on varied objects. Object-agnostic grasps could help warehouse and home robots handle new items without a fresh scan each time. Code is shared in public.
Catalog record
| CREATOR | Yitian Shi, Zicheng Guo, Rosa Wolf, Edgar Welte, Rania Rayyes |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 40 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2509-16871
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
HOGraspFlow: Exploring Vision-based Generative Grasp Synthesis with Hand-Object Priors and Taxonomy Awareness Thanks: Karlsruhe Institute of Technology, Karlsruhe, Germany. Email: {name}.{surname}@kit.edu . This work is sponsored by the DFG SFB-1574 Circular Factory project. Yitian Shi, Zicheng Guo, Rosa Wolf, Edgar Welte, Rania Rayyes Abstract We propose Hand-Object (HO)GraspFlow , an affordance-centric approach that retargets a single RGB with hand-object interaction (HOI) into multi-modal executable parallel jaw grasps without explicit geometric priors on target objects. Building on foundation models for hand reconstruction and vision, we synthesize S E ( 3 ) SE(3) grasp poses with denoising flow matching (FM), conditioned on the following three complementary cues: RGB foundation features as visual semantics, HOI contact reconstruction, and taxonomy-aware prior on grasp types. Our approach demonstrates high fidelity in grasp synthesis without explicit HOI contact input or object geometry, while maintaining strong contact and taxonomy recognition. Another controlled comparison shows that HOGraspFlow consistently outperforms diffusion-based variants ( HOGraspDiff ), achieving high distributional fidelity and more stable optimization in S E ( 3 ) SE(3) . We demonstrate a reliable, object-agnostic grasp synthesis from human demonstrations in real-world experiments, where an average success rate of over 83 % 83\% is achieved. I INTRODUCTION Recent years have witnessed a growing body of approaches in learning robotic manipulation behaviors directly from human demonstrations, ranging from teleoperation to internet-scale video corpora [ 1 , 2 , 3 ] . Specifically, learning from human-object interaction (HOI) demonstrations is increasingly framed as a retargeting problem, where a robotic end–effector (EE) is mapped from anthropometric hand motion discovered in the wild to the EE kinematics. Towards the challenge of kinematic mismatch between parallel jaw (PJ) EE and anthropometric hand in the video observations, a popular abstraction in vision-based imitation learning aligns the EE with the human thumb–index pair [ 1 , 4 , 5 ] , enabling simple pinch retargeting. While appealingly concise, this proxy collapses the diversity of human grasp taxonomy [ 6 ] and neglects contact–dependent antipodal force closure [ 7 ] . Hence, this approach is limited to pinch-like grasps and cannot be robustly applied to dynamic demonstrations that exhibit diverse grasp types. Moreover, recent research on multi-embodiment grasp generations [ 8 , 9 ] has demonstrated that object-conditioned grasp priors can be learned and modulated by geometric embeddings (e.g., Signed Distance Function (SDF), point graph features) that capture EE morphology. While such approaches enable transfer across different EEs, they are fundamentally object–centric and typically assume access to reliable 3D geometries and pose estimation at test time. Critically, they do not parse human intent or contact semantics from HOI, and thus cannot directly exploit in-the-wild video data, where hand pose, contact detection, and occlusions introduce significant noise. Targeting these limitations, in this work, we develop a vision-based generative hand-to–EE retargeting framework that supports adaptable transfer to dynamic, diverse anthropometric grasp demonstrations, while explicitly recognizing grasp taxonomy and contact with a single RGB crop of the hand. Our design is motivated by the asymmetry between intent and object variability, where objects vary widely while the set of underlying grasp intents remains relatively limited. On the object side, geometric diversity (shape, scale, etc.) and sensing brittleness (missing depth, specular failures) induce large, unstructured test-time shifts. In contrast, conditioned on a demonstrated grasp, the space of physically valid human hand poses with contact is constrained by anatomy, kinematics, compact grasp taxonomy, and affordance-driven intent, thus lying on a low-dimensional, structured manifold. Moreover, building on recent advances in multi-modal grasp synthesis with diffusion models [ 10 , 11 , 12 ] , we adopt denoising generative modeling on SE(3) for intent-conditioned retargeting given the following advantages: (i) Due to the kinematic mismatch between the human hand and PJ grippers, a single human grasp type often corresponds to multiple valid gripper approaches realizing the same affordance. Denoising-based models naturally preserve this multi-modality by sampling diverse modes in SE(3) ; (ii) The iterative denoising process admits controllable guidance [ 13 ] , allowing generated poses to be steered by differentiable constraints (e.g., collision avoidance [ 14 ] or curated affordance priors [ 12 ] ); (iii) In contrast to end-to-end methods that rely on post hoc filtering [ 15 , 16 , 17 ] , where feasibility check and affordance matching are performed after grasp generation and invalid candidates are discarded, our in-the-loop guidance enforces constraints during generation, thereby reducing rejections and improving sample efficiency—a critical factor given the scarcity and sampling cost of force-closure, affordance-consistent ground-truth datasets [ 18 ] . In summary, our contributions are: (i) We designed a vision-based affordance-centric HOI retargeting framework that produces multi-modal 6-DoF PJ grasps from a single RGB frame. This is achieved by conditioning on foundational features on HOI, without requiring explicit object models or pose estimation, while accounting for the diversity of human grasp taxonomy; (ii) We introduce two generative retargeting frameworks, HOGraspDiff and HOGraspFlow , inspired by SOTA SE(3) generative approaches to a vision-based setting. By integrating state-of-the-art visual foundation models’ features as contact priors, our method operates without explicit 3D geometric conditions or pose estimation of objects; (iii) Through extensive ablations and real-world deployment, we demonstrate consistent improvements in grasp-type prediction, contact accuracy, and distributional fidelity relative to the baselines, including a contact-oracle variant. With a minor translational correction with depths information at deployment, we achieved over 83\% success in grasp transfer in our real-world robot experiments. II Related Works II-A Generative 6-DoF grasp synthesis Learning-based grasp generators model a distribution over executable gripper poses conditioned on scene observations, enabling sampling-based exploration rather than hand-crafted proposal scoring. For instance, GraspNet-1B [ 19 ] and Contact-GraspNet [ 15 ] have established the effectiveness of learning from local contact geometry for grasp generation. In contrast, denoising-based grasp generators learn a latent density field through an iterative denoising process [ 20 ] , representing a recent trend in robotic manipulation learning. Among these, SE(3) diffusion fields [ 10 ] adapt denoising diffusion to the Lie group [ 21 ] , which learn grasp densities and refine samples directly on the SE(3) manifold, coupling pose synthesis with motion optimization for grasp execution. To handle symmetries and improve consistency under object rigid motions, EquiGraspFlow [ 11 ] enforces SE(3) equivariance while adopting flow-based denoising to handle symmetries and rigid object motions. To incorporate task constraints and human intent, HGDiffuser [ 12 ] augments the diffusion process with hand-intent cues, producing task-oriented 6-DoF grasps via guidance [ 22 ] . While effective, these generators typically assume accurate object geometry at inference, and thus usually fail under sensing artifacts and large out-of-distribution (OOD) variability in object shape and appearance. Fig. 2 : Pipeline for HOGraspFlow II-B Hand-object interaction (HOI) and reconstruction Recent progress in HOI recovery has been driven by the MANO parametric hand model [ 23 ] and modern monocular reconstructions that deliver accurate 3D hand pose and shape from RGB [ 24 , 25 ] . As the foundation, rich HOI datasets provide contact supervision and cross-object variability such as DexYCB [ 26 ] and OakInk [ 27 ] , which comprises affordances and human interactions over diverse household objects. Meanwhile, HOGraspNet [ 28 ] contributes dense HOI annotations with grasp taxonomy in