Object Pose Estimation through Dexterous Touch
Amir Hossein Shahidzadeh, Jiyue Zhu, Kezhou Chen, Sha Yi, Cornelia Fermüller, Yiannis Aloimonos, Xiaolong Wang ¡-this · 2025
Read Object Pose Estimation through Dexterous Touch by Amir Hossein Shahidzadeh, Jiyue Zhu, Kezhou Chen, Sha Yi, Cornelia Fermüller, Yiannis Aloimonos, Xiaolong Wang ¡-this (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-13591.txt.
Introduction
Amir-Hossein Shahidzadeh and six co-authors present a robotics method in 2025 for finding an object's pose, its position and orientation, by touch alone. Vision fails when lighting is poor or objects block the view, and touch sensors only report small local patches, so a robot must actively explore to piece the shape together.
Their setup uses two hands: one holds the object steady while the other probes its surface. A reinforcement learning policy, trained by trial and error, guides the probing hand to gather tactile point clouds. Those clouds then refine estimates of the object's shape and pose step by step. Tests show the method finds key pose features with no prior knowledge of the object's geometry.
Catalog record
| CREATOR | Amir Hossein Shahidzadeh, Jiyue Zhu, Kezhou Chen, Sha Yi, Cornelia Fermüller, Yiannis Aloimonos, Xiaolong Wang ¡-this |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 36 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2509-13591
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Object Pose Estimation through Dexterous Touch Amir Hossein Shahidzadeh, Jiyue Zhu, Kezhou Chen, Sha Yi, Cornelia Fermüller, Yiannis Aloimonos, Xiaolong Wang ¡-this Abstract Robust object pose estimation is essential for manipulation and interaction tasks in robotics, particularly in scenarios where visual data is limited or sensitive to lighting, occlusions, and appearances. Tactile sensors often offer limited and local contact information, making it challenging to reconstruct the pose from partial data. Our approach uses sensorimotor exploration to actively control a robot hand to interact with the object. We train with Reinforcement Learning (RL) to explore and collect tactile data. The collected 3D point clouds are used to iteratively refine the object’s shape and pose. In our setup, one hand holds the object steady while the other performs active exploration. We show that our method can actively explore an object’s surface to identify critical pose features without prior knowledge of the object’s geometry. Supplementary material and more demonstrations will be provided at https://amirshahid.github.io/BimanualTactilePose . 0 0 footnotetext: Indicates equal contribution. 0 0 footnotetext: University of Maryland, College Park 0 0 footnotetext: UC San Diego I Introduction Accurate object pose estimation is essential for dexterous robotic manipulation, enabling reliable interactions across a range of environments [ 1 , 2 ] . Traditional vision-based approaches have been effective but are limited by their sensitivity to lighting, occlusions, and material properties [ 3 , 4 ] . External cameras often introduce bulk and complexity when integrated with dexterous robotic hands. In contrast, tactile sensors, such as force-sensing resistors (FSRs), offer a compact and low-cost alternative suitable for embedding directly into robot hand fingertips [ 5 ] . However, tactile sensors [ 6 ] provide only sparse, local data, giving only isolated contact points rather than dense spatial information. To achieve the level of geometric detail required for accurate pose estimation, tactile approaches need active, long-horizon exploration to build up a comprehensive object model through interaction. Using a dexterous hand offers the benefits of multiple fingers with concurrent actions, which can be helpful for tactile exploration and pose estimation. However, tactile exploration with dexterous hands is challenging. The high-DOF configuration requires precise control to maintain effective and continuous contact with the object, ensuring that sufficient data is gathered throughout the process. Achieving this while maintaining sample efficiency–gathering valuable data without excessive interactions–is difficult. Due to these complexities, many previous works rely on template-based methods [ 7 ] to simplify the problem. Without object templates, accurate pose estimation requires more geometric-informed and sample-efficient policies to collect data. Inspired by the human ability to reconstruct object shapes through touch and interactions alone, we propose a reinforcement learning (RL) based framework that enables robot hands to actively explore an object and refine its pose estimate with each new contact. Our approach uses the series of contact positions to iteratively build a 3D representation of the object. To enhance the effectiveness of this long-horizon exploration, we design the reward function to guide the robot toward contact points that maximize coverage and minimize pose uncertainty. This encourages the robot to develop a touch-based exploration policy that converges to accurate pose estimates over time. Additionally, we incorporate demonstration trajectories to initialize the system with a better heuristic. This accelerates training and helps the robot achieve efficient exploration faster. In summary, our contributions include: • A bimanual exploration pipeline that endows tactile-only object pose estimation with simple FSR sensors, enabling adaptability to objects of different sizes. • Designing novel state representation and reward function that promotes pose estimation while exploring the object within a limited number of actions. We show that we can achieve 87% accuracy on the ADD-S pose estimation metric [ 8 ] on unseen objects only through tactile exploration in 100 steps. Fig. 1 : We present a tactile-only pose estimation framework using simple Force Sensing Resistors. We demonstrate that bimanual robot arms with dexterous hands can effectively explore [top row] and partially reconstruct [bottom row] the object’s surface to estimate its pose. II Related Work Pose estimation Accurate object pose estimation is crucial for robot manipulation tasks [ 9 , 10 , 11 ] . Vision-based approaches have proven successful in pose estimation [ 12 , 13 ] , yet these methods are sensitive to environmental factors such as lighting conditions, shadows, occlusions, and reflectivity or transparency [ 3 , 4 , 14 ] . In contrast, tactile sensors provide contact-based feedback by measuring forces or deformations [ 15 , 16 , 17 , 18 ] , making them especially valuable in scenarios where vision systems face limitations [ 5 , 19 , 20 ] . Furthermore, compact tactile sensors can be embedded into end-effectors, offering a more integrated and self-contained sensing system compared with external cameras. Recent works [ 21 , 19 , 22 ] have investigated pose estimation with vision-based tactile sensors, demonstrating promising results for object recognition and manipulation. however, these approaches typically rely on passive contact or pre-defined interactions, lacking active tactile exploration. This limitation restricts the range of objects that can be effectively explored, particularly those with complex geometries or varying sizes, and reduces the ability to gather rich, discriminative tactile information necessary for accurate pose estimation. We overcome this challenge by addressing both active exploration and pose estimation in a bimanual setting, enabling the exploration of objects with varying geometric properties. Active perception Traditional active sensing with vision-based methods primarily involves optimizing camera pose for better viewpoints [ 23 , 24 ] , which give better visual coverage and depth information, providing geometric information for tasks such as object recognition, mapping, and navigation [ 25 ] . In recent years, tactile-based active sensing has gained a broader impact. By planning where to make contact, tactile sensors provide detailed local surface information about the object. Such coordinated sensing approaches enable robots to gather information effectively optimized for a given task [ 26 , 27 ] . However, due to the sparse tactile sensor input, high accuracy regarding geometric information may require object templates [ 28 ] . Sensorimotor based exploration By leveraging sensorimotor coordination, robots can gather valuable sensory data more efficiently. Sensorimotor-based exploration has been extensively applied in active SLAM in object manipulation [ 29 , 30 ] . Earlier approaches to coordinating visual and tactile sensing in robotic hands predominantly relied on methods including information gain optimization [ 31 ] , uncertainty [ 32 , 29 ] , potential fields [ 33 ] , Gaussian processes [ 34 , 35 , 36 ] . Recent advancements in deep learning have introduced more sample-efficient approaches [ 37 ] , enabling optimization for specific tasks such as object reconstruction [ 38 , 39 , 40 ] , recognition and classification [ 28 ] , shape completion [ 41 , 42 , 43 ] , and pose estimation [ 44 ] . However, exploration tasks are inherently long-horizon problems, and leveraging temporal multi-fingered data over time is essential. To highlight the importance of pose estimation feedback during training, we compare our method against a multi-fingered extension of a recent tactile exploration approach [ 38 ] in Sec. V . Furthermore, most prior methods assume the object is fixed in the global frame by external forces;