M3DHMR: Monocular 3D Hand Mesh Recovery

Yihong Lin, Xianjia Wu, Xilai Wang, Jianqiao Hu, Songju Lei, Xiandong Li †, Wenxiong Kang † · 2025

Read M3DHMR: Monocular 3D Hand Mesh Recovery by Yihong Lin, Xianjia Wu, Xilai Wang, Jianqiao Hu, Songju Lei, Xiandong Li †, Wenxiong Kang † (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-20058.txt.

Introduction

Yihong Lin and co-authors present a pipeline to rebuild a 3D hand mesh from one photo in a 2025 paper. Hands bend in many ways, depth is ambiguous, and fingers hide each other. Past methods were slow or awkward at placing mesh points in space. The new pipeline predicts point positions directly.

It first draws 2D cues from the image to guide the 3D task. A spiral decoder then refines the mesh in layers that adapt weights to point positions. A region layer polishes each hand zone by its physical traits. On the FreiHAND benchmark, the method beats prior real-time systems by a clear margin.

Catalog record

CREATORYihong Lin, Xianjia Wu, Xilai Wang, Jianqiao Hu, Songju Lei, Xiandong Li †, Wenxiong Kang †
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE25 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-20058

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

M3DHMR: Monocular 3D Hand Mesh Recovery Yihong Lin, Xianjia Wu, Xilai Wang, Jianqiao Hu, Songju Lei, Xiandong Li †, Wenxiong Kang † Abstract Monocular 3D hand mesh recovery is challenging due to high degrees of freedom of hands, 2D-to-3D ambiguity and self-occlusion. Most existing methods are either inefficient or less straightforward for predicting the position of 3D mesh vertices. Thus, we propose a new pipeline called Monocular 3D Hand Mesh Recovery (M3DHMR) to directly estimate the positions of hand mesh vertices. M3DHMR provides 2D cues for 3D tasks from a single image and uses a new spiral decoder consist of several Dynamic Spiral Convolution (DSC) Layers and a Region of Interest (ROI) Layer. On the one hand, DSC Layers adaptively adjust the weights based on the vertex positions and extract the vertex features in both spatial and channel dimensions. On the other hand, ROI Layer utilizes the physical information and refines mesh vertices in each predefined hand region separately. Extensive experiments on popular dataset FreiHAND demonstrate that M3DHMR significantly outperforms state-of-the-art real-time methods. The code is available at https://github.com/Jackson-coder/M3DHMR. 1 Introduction Monocular 3D hand mesh recovery has attracted great attention in VR/AR, human-computer interaction, etc [ 30 , 16 ] . This task aims to estimate the 3D spatial positions of hand mesh vertices from a single RGB image. It improves the accuracy and security of hand gesture authentication by restoring biometric features. On the one hand, it repairs low-quality samples caused by factors such as wear, contamination, or limitations of capture devices, which reduces false recognition rates. On the other hand, it reconstructs incomplete data, thereby boosting performance of multimodal hand gesture authentication system. However, high degrees of hand freedom, 2D to 3D ambiguity and self-occlusion make significant challenges for hand reconstruction [ 1 , 9 , 5 ] . To overcome these challenges, parametric model-based approaches [ 34 , 6 , 29 , 27 ] usually use MANO [ 28 ] which obtains a hand mesh from coefficients of shape and pose to facilitately generate diverse hand poses for various applications with flexibility and high-precision. Though these methods are flexible, their 3D structure is not intuitive enough, especially the shape parameters are too difficult to optimize. Conversely, vertex-based methods can directly estimate the coordinates of the 3D mesh. Specifically, these models usually define the wrist of the hand as the root and calculate the root-relative 3D mesh coordinates. Some latest researches [ 22 , 23 , 18 ] are unable to implement in real-time due to their exclusive focus on improving the reconstruction accuracy, without considering the scale of models. Others [ 13 , 20 , 7 , 33 , 2 , 30 , 15 , 31 ] can achieve real-time performance but are limited in reconstruction accuracy. Additionally, all methods previously mentioned have to provide the absolute root coordinate in camera space to get absolute camera coordinates of the 3D mesh, which limits the application to many high-level tasks. Though CMR [ 5 ] and MobRecon [ 4 ] provide pipelines to directly acquire the absolute 3D coordinates of the mesh in real-time, their reconstruction accuracy still needs to be improved. Figure 1 : Qualitative results of our M3DHMR. We show the 2D pose, mesh projection, camera space mesh and pose in metres. The red rectangle indicates the camera. Herein, we propose a Monocular 3D Hand Mesh Recovery (M3DHMR) pipeline to further improve the performance of real-time camera-space 3D hand mesh recovery. The pipeline reconstructs the 3D hand through multiple stages and facilitates the 3D reconstruction by regressing 2D joint landmarks as a subtask. Novel Dynamic Spiral Convolution (DSC) Layers and Region of Interest (ROI) Layer are used to decode the 3D vertex features to obtain mesh 3D coordinates. Instead of employing stacked spiral convolution layers or increasing the number of channels in these layers to enhance the model capacity, which is often accompanied with expensive computation, DSC Layers explore a more reasonable spiral convolution architecture to further enhance the model’s representation capability with minimal increase in number of parameters and computational cost. More concretely, two convolution layers of DSC are used to adaptively adjust the normalized weights of the K spiral convolutions to extract the spatial-channel information. Furthermore, ROI Layer introduces the spatial information of the hand in the physical world by dividing the hand into multiple physical regions and refining the hand mesh vertices in each region separately. Extensive experiments indicate that M3DHMR surpasses current state-of-the-art real-time models in 3D hand mesh reconstruction. Some reconstruction results of M3DHMR in situations of complex pose, self-occlusion and object occlusion are shown in Fig. 1 . Our contributions are summarized as follows: • We design DSC Layers to adapt the weights of spiral convolutions based on the vertex positions and explore the mesh features in both space and channels. • We propose a ROI Layer that divides the mesh vertices of the hand into several sets based on the physical region to separately refine the hand mesh features. • Extensive experiments demonstrate that our method achieves superior performance in terms of reconstruction accuracy over existing state-of-the-art real-time methods. 2 Related Work 2.1 Hand Mesh Estimation Hand mesh estimation has become an important task in computer vision and augmented reality, which aims to reconstruct detailed 3D hand structures from monocular RGB images. Existing mainstream methods can be categorized into model-based and vertex-based approaches. Model-based approaches commonly rely on parametric hand models, such as MANO [ 28 ] , which provides a low-dimensional representation of hand poses and shapes. Building upon this foundation, Zhou et al. [ 34 ] proposed the first real-time monocular hand capture framework, integrating multi-modal data (RGB, keypoints, and contours) to achieve highly accurate 3D reconstructions. Furthermore, Chen et al. [ 6 ] developed a self-supervised model-based 3D hand reconstruction method that eliminates the need for 3D annotation by leveraging geometric constraints and 2D-3D consistency for training. Meanwhile, RealisticHands [ 29 ] combined parametric mesh priors with non-parametric details, enabling high-quality hand pose and shape estimation from monocular RGB inputs. HaMeR [ 27 ] used a fully transformer-based architecture to analyze hands with significantly increased accuracy and robustness. Despite their efficiency, these methods are inherently limited by the expressiveness of parametric models. Vertex-based methods directly predict the 3D coordinates of mesh vertices, offering flexibility beyond parametric models. Among these, Mesh Graphormer [ 23 ] and PointHMR [ 18 ] pioneered the use of transformer architectures for monocular 3D hand mesh recovery, achieving notable improvements in accuracy and robustness. However, their computational complexity hinders real-time deployment. To address this limitation, several efficient alternatives have emerged. MobileHand [ 13 ] introduced a lightweight network design, while YoutubeHand [ 20 ] and Pose2Mesh [ 7 ] leveraged graph convolutions for efficient mesh reconstruction. Further explorations include HIU-DMTL [ 33 ] , which employed deep multi-task learning, and I2UV-HandNet [ 2 ] , which adopted an image-to-UV mapping strategy. Tang et al. [ 30 ] enhanced real-time performance and significantly improved pose and shape accuracy through a novel alignment optimization module. CMR [ 5 ] and MobRecon [ 4 ] proposed end-to-end pipelines to directly estimate absolute 3D coordinates. More recently, CLIP-Hand3D [ 15 ] integrated vision-language pretraining with context-aware

Read the complete text