Interaction-aware Representation Modeling with Co-occurrence Consistency for Egocentric Hand-Object Parsing
Yuejiao Su, Yi Wang, Lei Yao, Yawen Cui, Lap-Pui ChauThe Hong Kong Polytechnic University {yuejiao.su, rayyoh.yao}@connect.polyu.hk{yi-eie.wang, yawen.cui, lap-pui.chau}@polyu.edu.hk · 2026
Read Interaction-aware Representation Modeling with Co-occurrence Consistency for Egocentric Hand-Object Parsing by Yuejiao Su, Yi Wang, Lei Yao, Yawen Cui, Lap-Pui ChauThe Hong Kong Polytechnic University {yuejiao.su, rayyoh.yao}@connect.polyu.hk{yi-eie.wang, yawen.cui, lap-pui.chau}@polyu.edu.hk (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2602-20597.txt.
Description
Parsing hands and active objects in first-person video matters for next-generation embodied agents, yet current methods stumble. Yuejiao Su and four co-authors note three flaws. Query starts lean on semantic cues or learned settings and adapt poorly when the active object changes. Pixel-level features can smuggle interaction-irrelevant content into masks. Models can suffer interaction illusion, predicting physically inconsistent scenes. Their end-to-end method, InterFormer, tackles all three flaws.
InterFormer grounds query starts in the spatial dynamics of hand-object contact through a Dynamic Query Generator. A Dual-context Feature Selector fuses coarse interactive cues with semantic features, cutting irrelevant noise and stressing interactive links. A Conditional Co-occurrence loss adds hand-object relationship rules to enforce physical consistency. The model leads methods on the EgoHOS set and the hard out-of-distribution mini-HOI4D set, showing generalization. Code and models are public.
Catalog record
| CREATOR | Yuejiao Su, Yi Wang, Lei Yao, Yawen Cui, Lap-Pui ChauThe Hong Kong Polytechnic University {yuejiao.su, rayyoh.yao}@connect.polyu.hk{yi-eie.wang, yawen.cui, lap-pui.chau}@polyu.edu.hk |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 59 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2602-20597