EvHand-FPV: Efficient Event-Based 3D Hand Tracking from First-Person View
*Zhen Xu, *Guorui Lu, Chang Gao, Qinyu Chen · 2025
Read EvHand-FPV: Efficient Event-Based 3D Hand Tracking from First-Person View by *Zhen Xu, *Guorui Lu, Chang Gao, Qinyu Chen (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-13883.txt.
Introduction
Zhen Xu and three co-authors present EvHand-FPV in a 2025 computer vision paper. It is a lightweight system that tracks a hand in three dimensions from a first-person view. It uses a single event camera. Event cameras report brightness changes with microsecond timing at milliwatt power. That suits extended reality headsets, where ordinary cameras are too slow or hungry.
To fix the shortage of egocentric benchmarks, they built a new dataset. It pairs synthetic training data with three-dimensional labels. It also pairs real event data with two-dimensional labels for evaluation. The tracker locates a wrist-based region of interest from geometric cues. It folds the region's offset into the network to save computation. An auxiliary geometry head improves the learning at no test-time cost. Accuracy rose while parameters and compute each fell by nearly ninety percent.
Catalog record
| CREATOR | *Zhen Xu, *Guorui Lu, Chang Gao, Qinyu Chen |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 32 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2509-13883
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
EvHand-FPV: Efficient Event-Based 3D Hand Tracking from First-Person View *Zhen Xu, *Guorui Lu, Chang Gao, Qinyu Chen Abstract Hand tracking holds great promise for intuitive interaction paradigms, but frame-based methods often struggle to meet the requirements of accuracy, low latency, and energy efficiency, especially in resource-constrained settings such as Extended Reality (XR) devices. Event cameras provide μ \mu s-level temporal resolution at mW-level power by asynchronously sensing brightness changes. In this work, we present EvHand-FPV, a lightweight framework for egocentric ( F irst- P erson- V iew) 3D hand tracking from a single event camera. We construct an event-based FPV dataset that couples synthetic training data with 3D labels and real event data with 2D labels for evaluation to address the scarcity of egocentric benchmarks. EvHand-FPV also introduces a wrist-based region of interest (ROI) that localizes the hand region via geometric cues, combined with an end-to-end mapping strategy that embeds ROI offsets into the network to reduce computation without explicit reconstruction, and a multi-task learning strategy with an auxiliary geometric feature head that improves representations without test-time overhead. On our real FPV test set, EvHand-FPV improves 2D-AUCp from 0.77 to 0.85 while reducing parameters from 11.2 M to 1.2 M by 89% and FLOPs per inference from 1.648 G to 0.185 G by 89%. It also maintains a competitive 3D-AUCp of 0.84 on synthetic data. These results demonstrate accurate and efficient egocentric event-based hand tracking suitable for on-device XR applications. The dataset and code are available at https://github.com/zen5x5/EvHand-FPV . I Introduction Hand tracking has emerged as a cornerstone technology for natural human-computer interaction, transforming how users engage with digital environments. In Extended Reality (XR) applications, accurate hand tracking eliminates the need for physical controllers, enabling intuitive manipulation of virtual objects through natural gestures. This capability has already been integrated into commercial AR/VR devices such as Meta Quest [ 1 ] , and is also being extended to domains including robotic teleoperation [ 2 ] , sign language recognition [ 3 ] , and medical rehabilitation [ 4 ] . These applications demand not only precise estimation of hand poses with high degrees of freedom but also real-time tracking with minimal latency requirements that become particularly challenging, particularly under rapid hand motions, where low latency and high temporal fidelity are critical. Current hand tracking solutions predominantly rely on conventional frame-based cameras [ 5 , 6 , 7 , 8 ] , which capture entire images at fixed frame rates regardless of scene dynamics. This approach presents limitations for resource-constrained devices. Typical 2D RGB sensors consume about 200 mW at 25–30 FPS, whereas 3D depth systems require 3–5 W at comparable frame rates. [ 9 ] . Attempts to reduce power consumption by lowering frame rates [ 10 ] severely compromise tracking responsiveness, resulting in motion blur and loss of temporal information during rapid hand movements precisely when accurate tracking is most critical. Event cameras alleviate these limitations by asynchronously capturing brightness changes with \mu s-level temporal resolution and mW-level power. Unlike conventional cameras, these bio-inspired sensors asynchronously capture only pixel-level brightness changes, achieving microsecond temporal resolution while consuming as little as 10 mW [ 11 , 12 , 13 , 14 ] . This sparse, event-driven sensing mechanism eliminates motion blur and provides exceptional temporal fidelity for tracking fast movements. Recent event-based hand tracking systems have demonstrated promising results [ 15 , 16 , 17 , 18 ] , validating the potential of this technology. However, existing event-based approaches fail to fully exploit the efficiency advantages of event cameras, employing deep learning models with over 10 M parameters that impose significant computational burdens. This mismatch impedes deployment on AR/VR headsets and glasses, where compute and power budgets are highly constrained. Moreover, these methods have focused exclusively on third-person perspectives, overlooking the critical importance of first-person (egocentric) hand tracking for interaction applications such as virtual keyboard typing, AR interface manipulation, and robotic teleoperation [ 19 , 20 , 21 ] . To address these challenges, we propose EvHand-FPV : an E fficient e v ent-based 3D Hand tracking framework from F irst- P erson V iew. Our approach rethinks event-based hand tracking for practical deployment, achieving superior accuracy while reducing both model parameters and computational load by 89% compared to the state-of-the-art (SOTA) methods. We demonstrate that efficient, accurate egocentric hand tracking is achievable through careful co-design of data representation, network architecture, and training strategies. The main contributions of this work are as follows: • We propose a lightweight and accurate event-based framework for 3D hand tracking from first-person view. • We construct an event-based first-person view hand tracking dataset, comprising synthetic data with 3D labels for training and, for the first time, real event data with 2D labels for testing, addressing the critical data gap for egocentric perspectives [ 22 ] . • We introduce a wrist-based ROI method with an end-to-end mapping strategy that embeds ROI offsets into the network, reducing input size and enabling efficient alignment without explicit reconstruction. • We design a multi-task learning strategy with an auxiliary geometric feature prediction task that guides the network toward more discriminative representations, improving accuracy while adding zero overhead during inference. II Related Works II-A 3D Hand Reconstruction 3D hand reconstruction from visual data has been extensively studied, with approaches broadly categorized into two paradigms. The first category employs end-to-end deep learning to directly map from image space to hand pose space [ 5 , 6 , 7 , 23 ] . These methods leverage the power of deep neural networks for efficient inference and have demonstrated promising performance. However, they often lack geometric constraints and require large-scale annotated datasets for training, limiting their generalization capabilities. The second category incorporates predefined parametric hand models as prior knowledge to ensure physiologically plausible reconstructions. Models such as MANO [ 24 ] and SMPL [ 25 ] provide strong structural constraints that improve the realism of reconstructed hands, particularly in scenarios with limited training data or occlusions. Current methods predominantly rely on conventional imaging modalities, including RGB cameras [ 5 , 6 , 7 , 23 , 26 , 27 , 28 , 29 , 30 , 31 ] and depth sensors [ 32 , 33 , 34 , 35 , 36 , 37 ] . While these technologies benefit from mature hardware and extensive datasets, they face fundamental limitations: low temporal resolution (typically 30-60 FPS), inadequate for capturing rapid hand movements, significant motion blur during fast motions, and high power consumption that increases substantially with frame rate, making them unsuitable for resource-constrained edge devices and high-speed tracking applications. Our task focuses on hand tracking, i.e., estimating hand pose continuously in real time. Since we adopt the MANO model to represent the hand, our formulation is also connected to 3D hand reconstruction methods. II-B Event-Based Hand Tracking Event cameras represent a paradigm shift in visual sensing, capturing pixel-level brightness changes asynchronously rather than full frames [ 14 ] . This bio-inspired approach offers microsecond-level temporal resolution, inherent motion blur reduction, high dynamic range (140+ dB), and ultra-low power consumption (mW-level), making them particularly suitable for high-speed motion capture and edge computing applications. II-B 1 Third-Person View Methods Recent advances have successfully applied event cameras