SiMHand: Mining Similar Hands for Large-Scale 3D Hand Pose Pre-training

Nie Lin, Takehiko Ohkawa, Yifei Huang, Mingfang Zhang, Minjie Cai, Ming Li, Ryosuke Furuta & Yoichi Sato · 2025

Read SiMHand: Mining Similar Hands for Large-Scale 3D Hand Pose Pre-training by Nie Lin, Takehiko Ohkawa, Yifei Huang, Mingfang Zhang, Minjie Cai, Ming Li, Ryosuke Furuta & Yoichi Sato (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-15251.txt.

Introduction

Lin and co-authors pre-train a hand pose estimator on millions of wild photos and videos. Judging a hand's 3D pose from one picture is hard, and earlier training rounds have not made full use of the varied hand footage online. The team gathers over two million hand images from human-activity videos, then pairs up different images that show similar hand shapes.

Their contrastive method pulls those look-alike pairs closer in the model's feature space, weighting pairs by how alike they are. It beats training that only compares altered copies of one image, gaining 15 percent on one test set and solid gains on two others. The code is public.

Catalog record

CREATORNie Lin, Takehiko Ohkawa, Yifei Huang, Mingfang Zhang, Minjie Cai, Ming Li, Ryosuke Furuta & Yoichi Sato
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE47 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2502-15251

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

SiMHand: Mining Similar Hands for Large-Scale 3D Hand Pose Pre-training Nie Lin, Takehiko Ohkawa, Yifei Huang, Mingfang Zhang, Minjie Cai, Ming Li, Ryosuke Furuta & Yoichi Sato Abstract We present a framework for pre-training of 3D hand pose estimation from in-the-wild hand images sharing with similar hand characteristics, dubbed SiMHand . Pre-training with large-scale images achieves promising results in various tasks, but prior methods for 3D hand pose pre-training have not fully utilized the potential of diverse hand images accessible from in-the-wild videos. To facilitate scalable pre-training, we first prepare an extensive pool of hand images from in-the-wild videos and design our pre-training method with contrastive learning. Specifically, we collect over 2.0M hand images from recent human-centric videos, such as 100DOH and Ego4D . To extract discriminative information from these images, we focus on the similarity of hands: pairs of non-identical samples with similar hand poses. We then propose a novel contrastive learning method that embeds similar hand pairs closer in the feature space. Our method not only learns from similar samples but also adaptively weights the contrastive learning loss based on inter-sample distance, leading to additional performance gains. Our experiments demonstrate that our method outperforms conventional contrastive learning approaches that produce positive pairs solely from a single image with data augmentation. We achieve significant improvements over the state-of-the-art method (PeCLR) in various datasets, with gains of 15% on FreiHand, 10% on DexYCB, and 4% on AssemblyHands. Our code is available at https://github.com/ut-vision/SiMHand . 1 1 footnotetext: ∗ Equal contribution. † Corresponding author. 1 Introduction Hands serve as a trigger for us to interact with the world, as seen in various human-centric videos. The precise tracking of hand states, such as 3D keypoints, is crucial for video understanding ( Sener et al., 2022 ; Wen et al., 2023 ) , AR/VR interfaces ( Han et al., 2022 ; Wu et al., 2020 ) , and robot learning ( Chao et al., 2021 ; Qin et al., 2022 ) . To this end, 3D hand pose estimation has been studied through constructing labeled datasets ( Ohkawa et al., 2023a ; Zimmermann et al., 2019 ; Chao et al., 2021 ; Ohkawa et al., 2023b ) and advancing supervised pose estimators ( Cai et al., 2018 ; Ge et al., 2019 ; Park et al., 2022 ; Liu et al., 2024 ; Fan et al., 2024 ) . However, utilizing large-scale, unannotated hand videos for pre-training remains underexplored, while collections of human-centric videos, like 3,670 hours of videos from Ego4D ( Grauman et al., 2022 ) and 131-day videos from 100DOH ( Shan et al., 2020 ) , are readily available. In pre-training, contrastive learning has been utilized to learn from unlabeled images like SimCLR ( Chen et al., 2020 ) , which maximizes agreement between positive pairs while repelling negatives. Spurr et al . ( Spurr et al., 2021 ) introduce pose equivariant contrastive learning (PeCLR) for 3D hand pose estimation, which aligns the geometry of features encoded from augmented images with affine transformations. However, both SimCLR and PeCLR create positive pairs from a single sample by applying data augmentation, limiting the gains from positive pairs as their hand appearance and backgrounds are identical. Ziani et al . ( Ziani et al., 2022 ) extend the contrastive learning framework to video sequences by treating temporally adjacent hand crops as positive pairs. However, in-the-wild videos can challenge tracking hands across frames, especially in egocentric views where hands are often unobservable due to camera motion. Meanwhile, this temporal positive sample mining remains the limited appearance variation of hands and backgrounds. Figure 1: The pipeline of pre-training and fine-tuning. (Left) Previous pre-training methods ( e.g . , PeCLR ( Spurr et al., 2021 ) ) learn from positive pairs originating from the different augmentations and fine-tune the network on a dataset. (Right) Our method is designed to learn from positive pairs with similar foreground hands, sampled from a pool of hand images in the wild. In this work, we introduce SiMHand, a novel contrastive learning framework for 3D hand pose pre-training, which leverages diverse hand images in the wild, with the largest 3D hand pose pre-training set to date. We specifically collect 2.0M hand images from human-centric videos, from Ego4D ( Grauman et al., 2022 ) and 100DOH ( Shan et al., 2020 ) , using an off-the-shelf hand detector ( Shan et al., 2020 ) . Our pre-training set significantly exceeds the scale of prior works by two orders of magnitude, such as over 32-47K images in ( Spurr et al., 2021 ) and 86K images from 100DOH in ( Ziani et al., 2022 ) . Our method focuses on learning discriminative information by mining hands with similar characteristics from various video domains. Based on our observations, contrastive learning can further benefit from discriminating the foreground of hands in varying backgrounds. As shown in Fig. 1 , our positive pairs are sourced from different images, offering additional information gains from different types of object interactions, backgrounds, and hand appearances. Specifically, we use an off-the-shelf 2D hand pose estimator ( Lugaresi et al., 2019 ) to identify similar hands from the pre-training set. Using the identified similar hands as positive pairs, we further propose adaptive weighting, to dynamically find informative pairs during training. A naive adaptation of the similar hands is to replace the original positive pairs in contrastive learning, but this scheme struggles to exploit how similar the paired hands are . To tackle this, we assign weights based on the similarity scores within the mini-batch in the contrastive learning loss. The weights are designed to have higher values as the similarity of the pairs increases. This allows the optimization of contrastive learning to explicitly consider the proximity of samples, beyond binary discrimination between positives and negatives. We validate the effectiveness of the pre-trained networks by fine-tuning on several datasets for 3D hand pose estimation, namely FreiHand ( Zimmermann et al., 2019 ) , DexYCB ( Chao et al., 2021 ) , and AssemblyHands ( Ohkawa et al., 2023b ) . Our proposed method consistently outperforms conventional contrastive learning methods, SimCLR and PeCLR. Additionally, we conduct extensive ablation experiments to analyze: 1) performance with varying pre-training and fine-tuning data sizes, 2) the effect of adaptive weighting, and 3) the improvement with different levels of similarity. In summary, the main contribution of this paper is threefold: • We propose SiMHand, a contrastive learning method for 3D hand pose pre-training, leveraging positive samples with similar hands mined from 2.0M in-the-wild hand images. • We introduce a parameter-free adaptive weighting mechanism in the contrastive learning loss, enabling optimization guidance according to the calculated similarity. • Our experiments demonstrate that our approach surpasses prior pre-training methods and achieves robust performances across different hand pose datasets. 2 Related Work 3D hand pose estimation: The task of 3D hand pose estimation aims to regress 3D hand joints. Since annotating 3D hand poses is challenging, only limited labeled datasets are available ( Ohkawa et al., 2023a ) , and most of which are constructed in controlled laboratory settings ( Zimmermann et al., 2019 ; Chao et al., 2021 ; Moon et al., 2020 ; Ohkawa et al., 2023b ) .

Read the complete text