EgoHandICL: Egocentric 3D Hand Reconstruction with In-Context Learning
Binzhu Xie, Shi Qiu, Sicheng Zhang, Yinqiao Wang, Hao Xu, Muzammal Naseer, Chi-Wing Fu, Pheng-Ann Heng · 2026
Read EgoHandICL: Egocentric 3D Hand Reconstruction with In-Context Learning by Binzhu Xie, Shi Qiu, Sicheng Zhang, Yinqiao Wang, Hao Xu, Muzammal Naseer, Chi-Wing Fu, Pheng-Ann Heng (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-19850.txt.
Introduction
Binzhu Xie and a large team of colleagues rebuild hands in three dimensions from head-mounted cameras. Egocentric video shows hands from the wearer's own view. Reconstructing their pose is hard: depth is ambiguous, fingers hide each other, and objects get in the way. Earlier systems scaled up training data or added extra cues, yet still struggled with new contexts.
Their method, EgoHandICL, uses in-context learning. The model adapts from examples shown at run time rather than only from fixed training. That flexibility targets unseen situations directly. Reliable hand tracking underpins augmented reality and robot learning from human video. The work aims at hands that stay accurate in the wild.
Catalog record
| CREATOR | Binzhu Xie, Shi Qiu, Sicheng Zhang, Yinqiao Wang, Hao Xu, Muzammal Naseer, Chi-Wing Fu, Pheng-Ann Heng |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 52 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2601-19850
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
EgoHandICL: Egocentric 3D Hand Reconstruction with In-Context Learning Binzhu Xie, Shi Qiu, Sicheng Zhang, Yinqiao Wang, Hao Xu, Muzammal Naseer, Chi-Wing Fu, Pheng-Ann Heng Abstract Robust 3D hand reconstruction is challenging in egocentric vision due to depth ambiguity, self-occlusion, and complex hand-object interactions. Prior works attempt to mitigate the challenges by scaling up training data or incorporating auxiliary cues, often falling short of effectively handling unseen contexts. In this paper, we introduce EgoHandICL , the first in-context learning (ICL) framework for 3D hand reconstruction that achieves strong semantic alignment, visual consistency, and robustness under challenging egocentric conditions. Specifically, we develop (i) complementary exemplar retrieval strategies guided by vision–language models (VLMs), (ii) an ICL-tailored tokenizer that integrates multimodal context, and (iii) a Masked Autoencoders (MAE)-based architecture trained with 3D hand–guided geometric and perceptual objectives. By conducting comprehensive experiments on the ARCTIC and EgoExo4D benchmarks, our EgoHandICL consistently demonstrates significant improvements over state-of-the-art 3D hand reconstruction methods. We further show EgoHandICL’s applicability by testing it on real-world egocentric cases and integrating it with EgoVLMs to enhance their hand–object interaction reasoning. Our code and data are available at: https://github.com/Nicous20/EgoHandICL . Figure 1: Comparing EgoHandICL with previous methods. (i) Prior works improve egocentric hand reconstruction by exploiting auxiliary cues ( 48 ; 24 ) , thereby having limited capabilities in handling challenging scenarios with severe occlusions. (ii) EgoHandICL improves the precision through two key steps. Step A: We prompt vision-language models with egocentric cues to retrieve template images for the input query image. Step B: We construct ICL demonstrations by aligning the input-target pairs of both template and query images. 1 Introduction Reconstructing 3D hands from monocular RGB images has long been a core computer vision task, with a wide range of applications in extended reality (XR), human–computer interaction (HCI), robotics, etc . By leveraging transformer-based backbones trained on large-scale datasets to extract rich visual representations ( 11 ) , recent methods, such as HaMeR ( 45 ) and WiLoR ( 47 ) , achieve strong 3D hand reconstruction performance across various benchmarks. While effective, these models can still encounter practical challenges, such as depth ambiguity, self-occlusion, and domain shift, which often affect their robustness and generalization in real-world applications ( 26 ; 69 ; 12 ) . These challenges are further aggravated in egocentric settings, where severe occlusions, perspective distortions, and complex hand–object interactions are commonly present. Previous works ( 48 ; 24 ) have also explored specialized solutions tailored for egocentric 3D hand reconstruction: as illustrated in Fig. 1 -(i), current methods utilize auxiliary supervision cues that require additional annotations, but still fail to resolve severe occlusions or ambiguous hand-object interactions in egocentric views. These limitations highlight the urgent need for a flexible and generalizable framework that can adapt to diverse egocentric contexts with robustness. Although each egocentric scenario presents unique visual ambiguities, humans naturally resolve them by drawing on prior experience, multimodal context, and task-specific knowledge. This remarkable ability inherently aligns with the core concept of in-context learning (ICL) ( 6 ) , which adapts to solve new problems by conditioning on a few relevant contextual examples. While ICL has achieved remarkable progress in language modeling ( 14 ; 19 ) , recent studies have begun to apply this paradigm for vision tasks, where rapid problem adaptation and example-guided reasoning are equally critica ( 70 ) . Given that ICL mirrors how humans exploit relevant cues to overcome ambiguity, it provides a natural paradigm for egocentric vision. Motivated by this inherent connection, we formulate a new framework, namely EgoHandICL, to advance egocentric 3D hand reconstruction by exploiting the strong reasoning capabilities of the in-context learning paradigm. As the first ICL-based approach for 3D hand reconstruction, we address the following two critical issues. First, ICL’s effectiveness highly depends on selecting relevant examples ( 38 ; 66 ; 52 ) , yet egocentric scenarios make this selection particularly difficult. As shown in Fig. 1 -(ii), we propose complementary retrieval strategies for exemplar template selection: (i) using four hand-involvement modes (left-hand, right-hand, two-hand, and non-hand) that broadly cover typical egocentric hand-related activities, we retrieve examples with similar hand involvement to ensure visual consistency between the query image and the matched templates; and (ii) we design interaction-specific prompts with a vision–language model (VLM) to retrieve adaptive templates conditioned on semantic context. Together, these strategies ensure that exemplars with strong semantic alignment and visual fidelity are exploited during in-context learning. Second, unlike conventional ICL methods addressing single-modal mappings, 3D hand reconstruction requires bridging 2D visual inputs and 3D parametric outputs. To ensure structural consistency between queries and templates, we represent both inputs and outputs using a unified MANO parameterization ( 51 ) , which yield semantically aligned input–output pairs that facilitate effective ICL. Moreover, we incorporate egocentric hand priors and multimodal cues to encode visual, textual, and structural context into a learnable token space, and adopt a Mask Autoencoders (MAE)-based architecture 25 trained with 3D geometric and perceptual objectives. By conducting extensive experiments, we demonstrate that these designs enable our EgoHandICL framework to achieve more robust and generalizable egocentric hand reconstruction performance. In summary, our main contributions are threefold: • We present the first in-context learning approach to 3D hand reconstruction, showcasing a promising way to tackle severe occlusions and diverse interactions in egocentric vision. • We develop the EgoHandICL framework, which retrieves effective exemplars, tokenizes multimodal context, and learns through an MAE-style architecture for robust and generalizable 3D hand reconstruction. • We present extensive experiments, demonstrating that our EgoHandICL achieves state-of-the-art performance on egocentric benchmarks and exhibits strong practicality in real-world scenarios, including self-captured data and enhancing hand–object interaction reasoning in VLMs. 2 Related Work 3D Hand Reconstruction. Hand reconstruction has been extensively studied for a long time, where early works used to employ depth cameras to recover 3D hand joints and meshes ( 21 ; 43 ; 56 ; 50 ; 53 ; 54 ; 55 ; 57 ) . A milestone is the introduction of MANO ( 51 ) , a low-dimensional parametric hand model, which enables single-image 3D hand reconstruction by regressing associated posed shapes. 5 then demonstrates the applicability of MANO by using a CNN model and an articulated hand mesh. This learning-based approach has inspired following methods that either directly regress MANO parameters ( 1 ; 46 ; 2 ) or predict hand vertices for improved image alignment ( 33 ; 10 ; 22 ; 32 ) . Beyond parametric regression, alternative strategies have been explored, including voxel-based ( 28 ; 40 ) and vertex regression methods ( 9 ) , with a few work further emphasizing efficiency and robustness of reconstruction ( 8 ; 44 ; 42 ; 29 ) . More recently, large-scale vision transformers pretrained on millions of images demonstrate that scaling both model capacity and data volume significantly enhances the generalization of hand reconstruction models to in-the-wild scenarios ( 13 ; 31 ; 37 ) . In particular, HaMeR ( 45 ) presents a powerful vision transformer-based ( 58 ) pipeline that utilizes patched image tokens to reconstruct the MANO parametric model, achieving state-of-the-art accuracy in both egocentric and in-the-wild settings. However, existing methods mainly focus on general reconstruction and lack robustness under challenging interactions and