The Invisible EgoHand: 3D Hand Forecasting through EgoBody Pose Estimation
Masashi Hatano, Zhifan Zhu, Hideo Saito, Dima Damen · 2025
Read The Invisible EgoHand: 3D Hand Forecasting through EgoBody Pose Estimation by Masashi Hatano, Zhifan Zhu, Hideo Saito, Dima Damen (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-08654.txt.
Introduction
Masashi Hatano, Zhifan Zhu, Hideo Saito, and Dima Damen tackle the task of guessing where a person's hands will move next in first-person video. Their paper appeared in twenty twenty-five. Past tools guess hand paths only when hands are in view, and skip finger pose. Yet a viewer can still guess rough hand spots when hands leave the frame, from the rest of the body.
Their model, EgoH4, reads past video and camera moves and predicts full hand paths and poses. It uses whole-body pose to bound where hands can be. Tests on a large first-person data set beat the base method on both path and pose scores. Gains were a few units on each score.
Catalog record
| CREATOR | Masashi Hatano, Zhifan Zhu, Hideo Saito, Dima Damen |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 43 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2504-08654
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
The Invisible EgoHand: 3D Hand Forecasting through EgoBody Pose Estimation Masashi Hatano, Zhifan Zhu, Hideo Saito, Dima Damen Abstract Forecasting hand motion and pose from an egocentric perspective is essential for understanding human intention. However, existing methods focus solely on predicting positions without considering articulation, and only when the hands are visible in the field of view. This limitation overlooks the fact that approximate hand positions can still be inferred even when they are outside the camera’s view. In this paper, we propose a method to forecast the 3D trajectories and poses of both hands from an egocentric video, both in and out of the field of view. We propose a diffusion-based transformer architecture for Egocentric Hand Forecasting, EgoH4, which takes as input the observation sequence and camera poses, then predicts future 3D motion and poses for both hands of the camera wearer. We leverage full-body pose information, allowing other joints to provide constraints on hand motion. We denoise the hand and body joints along with a visibility predictor for hand joints and a 3D-to-2D reprojection loss that minimizes the error when hands are in-view. We evaluate EgoH4 on the Ego-Exo4D dataset, combining subsets with body and hand annotations. We train on 156K sequences and evaluate on 34K sequences, respectively. EgoH4 improves the performance by 3.4cm and 5.1cm over the baseline in terms of ADE for hand trajectory forecasting and MPJPE for hand pose forecasting. Figure 1 : Given signals during observation: camera poses, images, and visible hand locations in 2D, our proposed method EgoH4 forecasts future 3D hand pose. EgoH4 can forecast hand joints even when hands are out of view during observation. We show visible 2D hand positions overlaid on the observation frames t_{1} and t_{2} , and the corresponding camera poses attached on the heads. At t_{2} , the right hand is invisible. In the forecasting frame, the right hand is back in view while the left hand is now out of view. 1 Introduction Understanding human motion from an egocentric video is critical for a variety of applications, including AR/VR, human-robot interaction, and assistive technology. Unlike category-level discrete predictions, such as action recognition Pirsiavash and Ramanan 2012 ; Poleg et al. 2016 ; Plizzari et al. 2023 ; Tekin et al. 2019 ; Hatano et al. 2024a or action anticipation Roy et al. 2024 ; Thakur et al. 2024 ; Furnari and Farinella 2019 ; Girdhar and Grauman 2021 ; Wu et al. 2020 , motion provides fine-grained, continuous predictions. Prior works predicting future hand positions are 2D-based, estimating the location within a moving camera frame. These works focus on hand-object interactions Liu et al. 2020 ; Liu et al. 2022 or ego-motion awareness Hatano et al. 2024b ; Ma et al. 2024b ; Ma et al. 2024a . However, 2D methods are unable to give persistent predictions of the hand in the environment due to the camera motion. Instead, recent works have targeted predicting in a world coordinate frame, with successful methods targeting body pose Khirodkar et al. 2023 ; Müller et al. 2024 ; Goel et al. 2023 ; Ye et al. 2023a ; Rempe et al. 2021 , ego body pose Akada et al. 2022 ; Wang et al. 2021 ; Yuan and Kitani 2018 ; Cuevas-Velasquez et al. 2024 ; Zhang et al. 2023 ; Li et al. 2023 ; Castillo et al. 2023 and recently 3D hand trajectories Bao et al. 2023 . Works that focus on 3D forecasting body pose Yuan and Kitani 2019 ; Choudhury et al. 2023 ; Escobar et al. 2025 , hand location or pose Bao et al. 2023 ; Tang et al. 2024 remain scarce. The current egocentric hand forecasting task has three significant limitations we address here: Out-of-view . Previous works Bao et al. 2023 ; Liu et al. 2022 ; Liu et al. 2020 have not considered scenarios where hands move out of the field of view during observations, yet this is a common occurrence in egocentric videos (as shown in Fig. 1 ). Hands are visible primarily when interacting with objects or just before intentional contact. For example, when reaching for distant objects, we initially move our body closer, only extending our hand once we are within reach. Consequently, relying on visible observations for forecasting can result in delayed predictions, limiting early-stage forecasting capabilities. Body Movements Awareness . Given the natural coordination between the hands and body, incorporating body movements could enhance the accuracy of hand forecasting. Predicting body poses alongside hand poses helps prevent unrealistic hand motions as body joints serve as the constraints. Hand Articulation . Prior egocentric hand forecasting works Bao et al. 2023 ; Liu et al. 2022 ; Liu et al. 2020 predict a hand position without finger articulation. While HoloAssist Wang et al. 2023 recently established a benchmark for egocentric hand pose forecasting without introducing a specific method, the task of hand pose forecasting in egocentric videos is still largely underexplored. To address these limitations, we propose EgoH4 , a 3D hand forecasting method that leverages body pose estimation to predict 3D hand motion and pose. To our knowledge, this is the first work to attempt 3D hand trajectory and pose forecasting when hands are out of the field of view, i.e., invisible . We achieve this by (1) jointly optimizing the hands and body joints. Leveraging the body pose knowledge helps locate the hand joints when they are out of frame by constraining hand joints relative to the body pose. Additionally, we (2) incorporate a classifier that estimates hand visibility. This improves the capability of dealing with invisible hands and enhancing hand forecasting accuracy. We evaluate EgoH4 on the Ego-Exo4D Grauman et al. 2024 dataset, which offers 3D annotations even when hands are outside the camera’s field of view thanks to the multiple exocentric cameras. In summary, our contributions are: • We are the first to address egocentric 3D hand forecasting when hands are in- or out-of-view, during both the observation and the forecasting timesteps. • We also extend the task of egocentric hand trajectory forecasting to hand pose forecasting for a fine-grained understanding of human intention. • We propose EgoH4, a diffusion-based transformer model jointly denoising body pose and hands along with a visibility predictor and 3D-to-2D reprojection regularization. • We evaluate EgoH4 on the Ego-Exo4D dataset, a large-scale egocentric dataset. From available annotations, we curate a 3D hand forecasting task, resulting in 156K training sequences and 34K testing sequences. • We improved the hand trajectory forecasting accuracy in ADE by 5.5cm, 1.9cm, and 3.4cm, and hand pose forecasting in MPJPE by 5.0 cm, 5.9 cm, and 5.1 cm for in-view, out-of-view, and overall sequences, respectively. 2 Related Work We review related works on hands in egocentric videos, egocentric hand forecasting, motion forecasting, and ego-body pose estimation. 2.1 Understanding Hands in Egocentric Videos Hand-object interactions are best studied in egocentric videos. Prior works have addressed 2D hand detection and side classification Shan et al. 2020 ; Cheng et al. 2023 ; Contributors 2020 , hand segmentation Darkhalil et al. 2022 ; Zhang et al. 2022a ; Jia et al. 2022 , grasp type classification Goyal et al. 2022 and hand pose estimation Pavlakos et