WHOLE: World-Grounded Hand-Object Lifted from Egocentric Videos

Yufei Ye, Jiaman Li, Ryan Rong, C. Karen Liu · 2026

Read WHOLE: World-Grounded Hand-Object Lifted from Egocentric Videos by Yufei Ye, Jiaman Li, Ryan Rong, C. Karen Liu (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2602-22209.txt.

Description

First-person videos of people handling objects are brutally hard to reconstruct: hands hide objects, objects leave the frame, and separate estimates of hand and object often disagree. Yufei Ye and three co-authors present WHOLE, a method that rebuilds hand and object motion together in world space from egocentric video, given object templates. Its core is a learned generative prior over hand-object motion.

At test time, the prior is guided to produce trajectories that match what the video shows, reasoning jointly about interaction instead of patching separate estimates. The joint approach substantially beats separate hand-then-object pipelines. WHOLE sets a new state of the art on hand-object benchmarks, offering robots and AR systems a truer picture of manipulation.

Catalog record

CREATORYufei Ye, Jiaman Li, Ryan Rong, C. Karen Liu
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE39 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2602-22209

Browse the full library · About the Tower of Babel