ScaleHP: Estimating Hand Pose in Metric Space

Ruitao Jing, Xingyu Chen ∗, Hongyang Li, Qing Jiang, Yukai Shi, Lei Zhang 🖂 · 2026

Read ScaleHP: Estimating Hand Pose in Metric Space by Ruitao Jing, Xingyu Chen ∗, Hongyang Li, Qing Jiang, Yukai Shi, Lei Zhang 🖂 (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-25619.txt.

Introduction

Augmented reality needs a hand's true size and position in the room, not just its shape relative to the wrist, and most trackers deliver only the shape. Ruitao Jing and colleagues present ScaleHP in 2026, built on a simple bodily fact: the proportions among hand bones are stable across people and quietly encode the hand's overall metric size. A transformer decoder with a dedicated scale token fuses shape and appearance cues across resolutions, then a perspective constrained least squares step solves for coordinates in real camera space.

No separate depth module is needed, removing a fragile link in earlier pipelines. On standard benchmarks the system set new marks for metric error, including the best reported scores on two widely used hand datasets. The authors argue that built in anatomy, used as a prior, beats bolted on range sensing for everyday hand tracking.

Catalog record

CREATORRuitao Jing, Xingyu Chen ∗, Hongyang Li, Qing Jiang, Yukai Shi, Lei Zhang 🖂
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE43 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2606.25619

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

ScaleHP: Estimating Hand Pose in Metric Space Ruitao Jing, Xingyu Chen ∗, Hongyang Li, Qing Jiang, Yukai Shi, Lei Zhang 🖂 Abstract Accurate metric-space hand pose estimation (HPE) is essential for immersive HCI and robotics. However, most existing methods predict poses in a root-relative coordinate system and cannot estimate the hand in absolute metric scale. In this work, We observe that the intrinsic proportional relationships among human hand bones encode stable anthropometric priors that implicitly correlate with the overall metric size of the hand. Leveraging this insight, We present ScaleHP, an end-to-end one-stage hand pose estimation framework that bypasses fragile extrinsic depth modules to recover the hand in metric-space.ScaleHP employs a transformer-based decoder with a novel scale token to fuse multi-scale morphological and appearance features. By solving for metric coordinates through a perspective-constrained least-squares approach, we achieve high-precision pose estimation in the camera coordinate system. ScaleHP delivers state-of-the-art performance, including 35.8 CS-MPJPE on FreiHand and 4.6/5.9 PA-MPJPE on DexYCB and HO3Dv3. These results demonstrate that internal biological constraints significantly reduce relative geometry and absolute metric errors, offering a robust solution for generalized, real-world hand tracking. Figure 1 : We present ScaleHP , an one-stage framework for hand pose estimation in Metric Space. This method is the first to explicitly predict the global scale of the hand, enabling accurate recovery of the hand pose in the true metric space, which previous approaches failed to achieve. 1 Introduction Hand pose estimation (HPE) serves as a fundamental building block for a vast array of emerging technologies that require seamless human-computer interaction (HCI) [ 25 ] . In the realm of Virtual and Augmented Reality (VR/AR), precise hand tracking is the primary vehicle for immersive manipulation, allowing users to interact with digital twins [ 24 ] as they would with physical objects. Similarly, in the fields of Embodied AI and teleoperation [ 39 , 52 ] , the hand acts as the bridge between human intent and robotic execution. For a robot to perform dexterous manipulation—such as picking up a fragile tool or performing remote surgery [ 73 ] —it must understand not just the relative posture of the hand, but its precise location and dimensions within the physical environment [ 61 ] . This necessitates a shift from traditional root-relative 3D pose estimation toward accurate metric-space estimation within the camera coordinate system. Despite its importance, the majority of contemporary research remains confined to root-relative coordinates 1 1 1 Here, we refer to a coordinate system defined with respect to an intrinsic hand reference point (e.g., the base joint of the middle finger) as the origin, while its global orientation remains unchanged. [ 11 , 3 ] or or alternatively performing direct regression of the parameters of a parametric model, largely not considering the absolute scale of the hand. While these methods achieve impressive results in laboratory settings, they falter in real-world applications where the hand must interact with external objects and diverse environments. The core challenge of metric-space estimation lies in the inherent depth-scale ambiguity of monocular images. To resolve this problem, existing frameworks [ 17 , 37 , 55 , 28 ] often rely on auxiliary depth estimation modules or pre-computed depth maps to align predicted coordinates with a real-world scale. However, these external modules are notoriously fragile; when the scene consists solely of a hand or when the background deviates from the training distribution (out-of-domain), depth estimation becomes highly inaccurate. Consequently, these models struggle to provide stable and precise metric-space predictions across generalized scenarios. Drawing upon forensic and anatomical research [ 54 , 44 , 36 ] which establishes that skeletal bone proportions are intrinsically correlated with the overall metric dimensions of the human hand, we posit that the physical scale of a hand can be inferred from these proportional relationships, which are implicitly encoded in the hand keypoints configuration observed in an image. By capturing these anthropometric regularities, we can establish a deterministic mapping between hands in pixels and their corresponding real-world lengths, thereby enabling the reconstruction of hand geometry within the absolute camera-coordinate metric space. Crucially, as this scale is derived solely from the internal biological priors of the hand itself, it remains invariant to extrinsic environmental factors, such as scene context and depth fluctuations, ensuring robust generalization and effectively bypassing the fundamental scale-depth ambiguity inherent in monocular 3D reconstruction. Based on this insight, we present ScaleHP, an end-to-end one-stage framework designed for high-precision metric-space hand pose estimation. Our model utilizes a pre-trained, frozen detector as a backbone to maintain robust feature extraction while focusing on the geometric mapping task. We introduce a novel transformer-based decoder architecture that incorporates a dedicated scale token. This token engages in global self-attention with 2D and 3D joint queries and interacts with image tokens through a multi-scale deformable attention mechanism. This design allows the model to fuse features from multiple dimensions—ranging from local joint morphology to global hand appearance—to predict a unified global scale scalar. Finally, we integrate the predicted 2D coordinates, canonical 3D poses, and the global scale through a system of linear equations. By solving this via a least-squares approach under perspective projection constraints, we successfully reconstruct the hand pose in full metric space. The primary contributions of this work are summarized as follows: We introduce the concept of metric-aware hand pose estimation, a framework that explicitly models the global scale of the hand by leveraging intrinsic anatomical bone proportions. By incorporating this anatomical prior, the model can mitigate the inherent depth ambiguity in monocular images and directly predict metrically consistent hand poses in the camera coordinate system. We propose ScaleHP, the first one-stage end-to-end framework for hand pose estimation in metric space. Central to this design is a scale token, which interacts with 2D and 3D pose queries within the metric decoder, enabling the network to learn a direct mapping from image features to absolute physical scale and thereby produce robust metric-space hand pose prediction. Through extensive evaluation, our model achieves superior performance in metric-space pose estimation, yielding a 35.8 CS-MPJPE on the FreiHand benchmark, and 4.6/5.9 PA-MPJPE on the DexYCB and HO3Dv3 benchmarks respectively. Furthermore, our ablation studies demonstrate that the global scale factor can significantly reducing the relative geometry prediction error along the depth axis. 2 Related Works 2.1 Root-Relative 3D Hand Pose Estimation Monocular root-relative 3D hand reconstruction methods generally fall into three categories. Parametric approaches [ 1 , 10 , 18 , 21 , 33 , 63 , 64 , 67 , 71 , 70 , 72 , 74 ] regress MANO [ 47 ] pose and shape coefficients to recover anatomically constrained hand meshes. Heatmap-based methods [ 20 , 37 , 38 , 62 ] represent 3D structure through volumetric or 2.5D heatmaps, decoupling spatial localization and depth estimation but often relying on intermediate representations and multi-stage refinement. Vertex regression approaches [ 7 , 6 , 13 , 27 ] directly predict 3D vertex coordinates in an end-to-end manner, avoiding discretization but requiring stronger structural priors to ensure geometric consistency. More recently, Transformer architectures have reshaped this landscape by enhancing global context modeling and enabling unified prediction paradigms. Several works [ 12 , 26 , 32 , 31 , 42 , 66 , 75 ]

Read the complete text