Retrieval Dexterity: Efficient Object Retrieval in Clutters with Dexterous Hand
Fengshuo Bai, Yu Li, Jie Chu, Tawei Chou, Runchuan Zhu, Ying Wen, Yaodong Yang, Yuanpei Chen · 2025
Read Retrieval Dexterity: Efficient Object Retrieval in Clutters with Dexterous Hand by Fengshuo Bai, Yu Li, Jie Chu, Tawei Chou, Runchuan Zhu, Ying Wen, Yaodong Yang, Yuanpei Chen (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-18423.txt.
Introduction
Fengshuo Bai, Yu Li, Jie Chu, and a wider team wrote this research paper, published in 2025. It teaches a robot arm with a many-fingered hand to fish a target object out of clutter, digging through the pile itself instead of clearing it piece by piece.
Objects buried under others are slow to retrieve because items rest on one another in tangled ways. Older systems either reason about supports and grasp items one by one, or nudge the pile first as a separate step. RetrDex learns in massive parallel simulation across varied piles. Its spatial encoding captures how the target, hand, and clutter occlude and support each other. From that, the hand learns to push, stir, and poke as part of one smooth skill. Tests on sixteen household objects, seen and new, went well, and the skill transferred to a real robot hand with no extra training.
Catalog record
| CREATOR | Fengshuo Bai, Yu Li, Jie Chu, Tawei Chou, Runchuan Zhu, Ying Wen, Yaodong Yang, Yuanpei Chen |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 53 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2502-18423
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Retrieval Dexterity: Efficient Object Retrieval in Clutters with Dexterous Hand Fengshuo Bai, Yu Li, Jie Chu, Tawei Chou, Runchuan Zhu, Ying Wen, Yaodong Yang, Yuanpei Chen Abstract Retrieving objects buried beneath multiple objects is not only challenging but also time-consuming. Performing manipulation in such environments presents significant difficulty due to complex contact relationships. Existing methods typically address this task by sequentially grasping and removing each occluding object, resulting in lengthy execution times and requiring impractical grasping capabilities for every occluding object. In this paper, we present a dexterous arm-hand system for efficient object retrieval in multi-object stacked environments. Our approach leverages large-scale parallel reinforcement learning within diverse and carefully designed cluttered environments to train policies. These policies demonstrate emergent manipulation skills (e.g., pushing, stirring, and poking) that efficiently clear occluding objects to expose sufficient surface area of the target object. We conduct extensive evaluations across a set of over 10 household objects in diverse clutter configurations, demonstrating superior retrieval performance and efficiency for both trained and unseen objects. Furthermore, we successfully transfer the learned policies to a real-world dexterous multi-fingered robot system, validating their practical applicability in real-world scenarios. Videos can be found on our project website https://ChangWinde.github.io/RetrDex . 1 Introduction Imagine having a box full of miscellaneous items with your desired book at the bottom. Having to remove each overlying object one by one to reach your book is undoubtedly a tedious task. Similar cluttered environments, such as messy drawers, disorganized office desks, or packed warehouses, present significant challenges for object retrieval in robotics. Previous approaches often relied on sequential removal of occluding objects using parallel-jaw grippers before attempting to grasp the target object. However, this strategy is not only time-consuming but also presents significant challenges in achieving reliable grasping manipulation. In such scenarios, humans typically employ their hands to efficiently move other objects aside, retrieving the desired object in a relatively short time. This is because the hand, as a high-degree-of-freedom manipulator, offers numerous configurations for object manipulation. To enable efficient object retrieval across diverse objects and scenarios, we propose using a multi-finger dexterous hand as the end-effector. Learning such skills brings multiple challenges: (i) Time efficiency : When objects are only partially visible or completely occluded in the perception system, sequentially grasping and placing objects is highly time-consuming. Determining how to quickly locate and expose sufficient pixels for grasping is a challenging yet important task. Hand-designed strategies often require unacceptably long execution times. (ii) Diverse objects : Training a policy that succeeds across various diverse object settings is even more challenging. Previous methods requiring pose estimation of surrounding objects can introduce new errors and struggle to generalize to unseen objects with limited training data. Some prior research typically assumes the existence of a universally successful grasping policy, which may be an even more difficult proposition. (iii) High-dimensional action space and contact-rich environment : Dexterous multi-finger hands introduce a high-dimensional state and action space, which increases the difficulty for optimization. Additionally, in cluttered stacking scenarios, object collisions are extremely rich, making environmental dynamics more challenging to model. In this paper, we propose Retrieval Dexterity , a new system for retrieving objects in clutter with one dexterous hand in a super-efficient manner. Our hardware system includes a Realman RM-75 robot equipped with an Inspired Hand (with 6 degrees of freedom) at the end, as shown in Figure . To solve this problem, we propose an approach leveraging Reinforcement Learning to train a policy in simulation and then perform Sim2Real transfer on real robots. Specifically, to solve the above challenges, we introduce two key technical contributions: • We have developed a realistic scene generation pipeline for stacked environments. The system simulates the natural accumulation of common household objects through gravitational dropping, while strategically placing occluded target objects at specified positions. Physical parameters are carefully calibrated to ensure realistic stacking environment. • We present a reinforcement learning framework for training object retrieval policies. The reward function is based on the pixel visibility of target objects captured by simulated cameras after a fixed time horizon. Using efficient reinforcement learning algorithms, we optimize the policy to discover emergent retrieval strategies in complex stacked environments. We conduct both simulation and real-world experiments. The results demonstrate that our method enables the hand to successfully retrieve objects and surpass baselines by a large margin in efficiency. To our knowledge, this is the first work that enables efficient object retrieval with dexterous multi-finger hands. 2 Related Work 2.1 Cluttered Objects Manipulation Interacting with objects in a cluttered environment is of significant importance for real-world applications [ 1 , 2 , 3 ] . Prior research has extensively explored robotic manipulation in these environments, aiming to equip robots with the ability to master diverse and complex skills. For instance, Murali et al. 2020 , Pang et al. 2023 have focused on improving robust object grasping techniques, while Li et al. 2024 , Zhao et al. 2024 investigate retrieval tasks. Additionally, studies such as Goyal et al. 2022 , Tang and Sukhatme 2023 , Jia and Chen 2024 address challenges in rearrangement, as well as grasping and throwing [ 10 ] . Visual-based approaches have also been widely adopted to enhance manipulation strategies in cluttered environments. For example, Huang et al. 2022a leverage visual prediction and planning to forecast the future states of objects after pushing actions, thereby optimizing grasping paths. In a related effort, Kurenkov et al. 2020 propose a continuous pushing strategy driven by real-time visual signals to improve object graspability. 2.2 Object Retrieval Retrieving a target object from complex clutter is a fundamental robotic skill with broad applications, ranging from domestic services to manufacturing. To address this challenge, various studies have proposed solutions from multiple perspectives. For instance, some works focus on planning strategies, such as object search optimization [ 13 ] , teacher-aided exploration [ 12 ] , and human-guided planning [ 14 ] , while others emphasize action-based methods, including push-grasping synergy policies [ 15 ] and learning pushing and grasping without visual foresight [ 16 ] . Additionally, approaches like analyzing support relations among cluttered objects have shown promise for improving retrieval efficiency [ 5 ] . In terms of perception, researchers have explored both tactile sensing [ 17 , 6 ] and visual or language-based modalities [ 18 , 8 ] . The choice of end-effector has also been a key focus, with methods employing rod-like pushers [ 19 ] , parallel grippers [ 20 , 8 , 6 , 21 ] , and dexterous hands [ 22 ] to address the challenges posed by cluttered environments. Moreover, task scenarios vary widely, from granular media [ 17 ] to confined spaces [ 6 ] , requiring tailored approaches to accommodate environmental constraints. Our approach differs fundamentally from most previous methods by actively manipulating occluding objects to expose the target object, enabling efficient retrieval while introducing more challenging control requirements. Figure 2 : Illustration of the Retrieval Skill System Design. (a) Constructs diverse cluttered scenes using a drop-from-above strategy. (b) Utilizes large-scale parallel RL with well-designed rewards to train policies. (c) Generates trajectories from the RL expert policy, selects useful ones based on our principle, and trains the distilled policy for deployment on