WHOLE: World-Grounded Hand-Object Lifted from Egocentric Videos
Yufei Ye, Jiaman Li, Ryan Rong, C. Karen Liu · 2026
Read WHOLE: World-Grounded Hand-Object Lifted from Egocentric Videos by Yufei Ye, Jiaman Li, Ryan Rong, C. Karen Liu (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2602-22209.txt.
Description
First-person videos of people handling objects are brutally hard to reconstruct: hands hide objects, objects leave the frame, and separate estimates of hand and object often disagree. Yufei Ye and three co-authors present WHOLE, a method that rebuilds hand and object motion together in world space from egocentric video, given object templates. Its core is a learned generative prior over hand-object motion.
At test time, the prior is guided to produce trajectories that match what the video shows, reasoning jointly about interaction instead of patching separate estimates. The joint approach substantially beats separate hand-then-object pipelines. WHOLE sets a new state of the art on hand-object benchmarks, offering robots and AR systems a truer picture of manipulation.
Catalog record
| CREATOR | Yufei Ye, Jiaman Li, Ryan Rong, C. Karen Liu |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 39 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2602-22209