Interaction-aware Representation Modeling with Co-occurrence Consistency for Egocentric Hand-Object Parsing

Yuejiao Su, Yi Wang, Lei Yao, Yawen Cui, Lap-Pui ChauThe Hong Kong Polytechnic University {yuejiao.su, rayyoh.yao}@connect.polyu.hk{yi-eie.wang, yawen.cui, lap-pui.chau}@polyu.edu.hk · 2026

Read Interaction-aware Representation Modeling with Co-occurrence Consistency for Egocentric Hand-Object Parsing by Yuejiao Su, Yi Wang, Lei Yao, Yawen Cui, Lap-Pui ChauThe Hong Kong Polytechnic University {yuejiao.su, rayyoh.yao}@connect.polyu.hk{yi-eie.wang, yawen.cui, lap-pui.chau}@polyu.edu.hk (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2602-20597.txt.

Description

Parsing hands and active objects in first-person video matters for next-generation embodied agents, yet current methods stumble. Yuejiao Su and four co-authors note three flaws. Query starts lean on semantic cues or learned settings and adapt poorly when the active object changes. Pixel-level features can smuggle interaction-irrelevant content into masks. Models can suffer interaction illusion, predicting physically inconsistent scenes. Their end-to-end method, InterFormer, tackles all three flaws.

InterFormer grounds query starts in the spatial dynamics of hand-object contact through a Dynamic Query Generator. A Dual-context Feature Selector fuses coarse interactive cues with semantic features, cutting irrelevant noise and stressing interactive links. A Conditional Co-occurrence loss adds hand-object relationship rules to enforce physical consistency. The model leads methods on the EgoHOS set and the hard out-of-distribution mini-HOI4D set, showing generalization. Code and models are public.

Catalog record

CREATORYuejiao Su, Yi Wang, Lei Yao, Yawen Cui, Lap-Pui ChauThe Hong Kong Polytechnic University {yuejiao.su, rayyoh.yao}@connect.polyu.hk{yi-eie.wang, yawen.cui, lap-pui.chau}@polyu.edu.hk
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE59 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2602-20597

Browse the full library · About the Tower of Babel