A Helping (Human) Hand in Kinematic Structure Estimation
Adrian Pfisterer, Xing Li, Vito Mengers, Oliver Brock · 2025
Read A Helping (Human) Hand in Kinematic Structure Estimation by Adrian Pfisterer, Xing Li, Vito Mengers, Oliver Brock (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-05301.txt.
Introduction
Adrian Pfisterer, Xing Li, Vito Mengers, and Oliver Brock use the human hand as a prior for robot perception. Robots need the joint structure of objects they handle, but sight alone is noisy. Hands move under tight constraints while they work an object. Tracking that motion can reveal the object's hidden joints.
Their method runs in real time and models doubt in each sight read. A new data set holds hard cases with blocked views and few visible joints. Estimates beat two recent baselines by wide margins, near double or more. The fits were exact enough for a robot to handle small objects safely. The work shows a human hand can teach robots object structure.
Catalog record
| CREATOR | Adrian Pfisterer, Xing Li, Vito Mengers, Oliver Brock |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 34 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-05301
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
A Helping (Human) Hand in Kinematic Structure Estimation Adrian Pfisterer, Xing Li, Vito Mengers, Oliver Brock Abstract Visual uncertainties such as occlusions, lack of texture, and noise present significant challenges in obtaining accurate kinematic models for safe robotic manipulation. We introduce a probabilistic real-time approach that leverages the human hand as a prior to mitigate these uncertainties. By tracking the constrained motion of the human hand during manipulation and explicitly modeling uncertainties in visual observations, our method reliably estimates an object’s kinematic model online. We validate our approach on a novel dataset featuring challenging objects that are occluded during manipulation and offer limited articulations for perception. The results demonstrate that by incorporating an appropriate prior and explicitly accounting for uncertainties, our method produces accurate estimates, outperforming two recent baselines by 195 % 195\text{\,}\mathrm{\%} and 140 % 140\text{\,}\mathrm{\%} , respectively. Furthermore, we demonstrate that our approach’s estimates are precise enough to allow a robot to manipulate even small objects safely. I Introduction Safe and reliable robotic manipulation with articulated objects requires knowledge of their kinematic models. A common approach for obtaining kinematic models involves visually tracking the motion of individual object parts relative to one another. However, this method is susceptible to uncertainties in the visual input due to factors like a lack of texture and occlusions, which hinder accurate observation and tracking of object-part motion. Additionally, noise in measurements can lead to outliers that significantly deviate from the object’s true motion, skewing model estimates. To address these challenges, we propose a probabilistic real-time approach that leverages the human hand as a perceptual prior to mitigate uncertainties in visual observations for accurate kinematic model estimation. The key idea is that when humans manipulate objects, their hand motions are partially constrained by the object’s kinematic model. This allows us to infer object motion by tracking the hand, even when the object is fully occluded. In addition, the human hand’s consistent structure—particularly its landmarks such as fingers and joints—facilitates reliable motion tracking. Moreover, we introduce novel uncertainty models based on hand properties, which quantify the uncertainty of individual landmarks and reject outliers unrelated to object motion, significantly improving the estimation, as shown in Fig. 1 . To validate our method, we collected a novel benchmark dataset of ten objects. In particular, we include challenging objects like keys and slide locks, which have small articulations and are often heavily occluded during manipulation. To the best of our knowledge, visually estimating kinematic models of these small objects has not been explored yet. Our experiments demonstrate that the proposed method consistently produces accurate estimates for these challenging objects in an online manner, outperforming two recent baselines by 195\text{\,}\mathrm{\%} and 140\text{\,}\mathrm{\%} . These improvements stem from leveraging the human hand as a perceptual prior to enhance motion tracking and deliberately modeling uncertainties. We also demonstrate that our method’s estimations are precise enough to allow a robot to reliably manipulate small objects, where minor estimation errors could cause significant forces that prevent successful manipulation. Fig. 1 : Our approach estimates kinematic models from human hand motions for challenging objects where tracking object-part motions is difficult due to occlusions and a lack of texture (left). By deliberately modeling uncertainties based on hand properties, our approach weights observations and rejects outliers resulting from uncertainties and noisy measurements (center), significantly improving the estimation results (right). II Related Work Our work leverages the human hand as a prior for robust Kinematic Structure Estimation (KSE). Therefore, this section reviews prior works on KSE, followed by approaches that learn manipulation policies from human hand motions. II-A Kinematic Structure Estimation Prior works on KSE often estimate the kinematic models of articulated objects based on the relative motions of object parts. Some approaches assume known trajectories of part poses [ 1 , 2 , 3 , 4 ] , while others track salient image features [ 5 , 6 , 7 ] or use learned motion maps clustered into pose trajectories [ 8 ] . Combining image features with shape-based segmentation can also assist in tracking object parts [ 9 , 10 ] . Additionally, some methods learn kinematic structures end-to-end from image sequences [ 11 ] . Most approaches assume complete visibility of the object in order to estimate object-part motions. However, when humans manipulate objects—especially small ones as shown in Fig. 1 —they can often be largely occluded during the manipulation. Our approach circumvents this problem by tracking human hand motions, which are constrained by the object’s kinematics, to infer the kinematic model. Additionally, the distinct landmark features of human hands allow for reliable motion tracking. Many KSE methods provide deterministic estimates [ 1 , 12 , 6 , 8 , 11 ] , or are only partially probabilistic [ 13 , 14 ] , making it difficult to account for uncertainty. This limitation is particularly problematic for small objects, where noise and occlusions can introduce significant uncertainty, compromising the estimation results. Additionally, few methods operate online [ 15 , 5 ] , limiting their applicability in real-time tasks. Our work introduces a probabilistic real-time approach that effectively quantifies uncertainty based on hand properties, leading to enhanced estimation quality under noisy measurements. II-B Leanring Manipulation Policies from Hand Motions Learning robot manipulation policies from human hand motions has recently gained significant interest [ 16 ] . Our work is related to approaches that directly track human hands to learn object affordances [ 17 , 18 , 19 , 20 , 21 , 22 ] . However, these approaches do not provide kinematic models of the object, which is crucial for safe interaction, especially for small delicate objects. Our approach builds on existing methods that track human hand motions during manipulation to infer object kinematics for robotic manipulations. Existing approaches, however, track only a single point [ 23 ] or treat the hand as a rigid body [ 24 ] , which often results in noisy estimations and require extensive fine-tuning on the robot to enhance estimation quality. In contrast, our approach demonstrates that by tracking individual hand landmarks and modeling their uncertainties using hand-specific properties, we can achieve accurate estimations that enable direct robot interaction without the need for additional interactive fine-tuning. III Recursive Kinematic Model Estimation Our method estimates kinematic models by tracking human hand motions. Inspired by the Online Multimodal Interactive Perception (OMIP) system [ 5 ] , we decompose the estimation problem into three hierarchical levels: hand landmark motion, hand rigid body motion, and kinematic model estimation. At each level, we employ recursive Bayes filters. This hierarchical system structure allows us to model uncertainties and reject outliers at different levels of abstraction, as illustrated in Fig. 2 . While our approach builds upon the OMIP system, it addresses a crucial limitation: the assumption that objects are not occluded and offer sufficient image features for motion tracking. The following section will explain how to address this limitation by incorporating human hand motions into the estimation process. Fig. 2 : We decompose the estimation problem into three hierarchical levels, similar to the OMIP system [ 5 ] . This hierarchical structure allows us to model uncertainties and reject outliers at two different levels, each grounded in physical priors related to hand properties. III-A