Closed-Loop Transfer for Weakly-supervised Affordance Grounding
Jiajin Tang, Zhengxuan Wei, Ge Zheng, Sibei Yang · 2025
Read Closed-Loop Transfer for Weakly-supervised Affordance Grounding by Jiajin Tang, Zhengxuan Wei, Ge Zheng, Sibei Yang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-17384.txt.
Introduction
People learn where to grip a new object by watching someone else use it, and Jiajin Tang with three co-authors gives computers a similar skill. Their task, called affordance grounding, asks a model to find the part of an object that allows an action, learning from photos of people interacting with objects. Earlier methods passed knowledge one way only, from those interaction photos to plain object views.
Tang and his co-authors close the loop, letting the plain views also sharpen what is drawn from the interaction photos. Their LoopTrans system adds shared location tools and a cleaning step for the transferred knowledge. It improved scores on every measure across photo and video tests, even when a person body hid the contact region completely.
Catalog record
| CREATOR | Jiajin Tang, Zhengxuan Wei, Ge Zheng, Sibei Yang |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 51 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2510-17384
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Closed-Loop Transfer for Weakly-supervised Affordance Grounding Jiajin Tang, Zhengxuan Wei, Ge Zheng, Sibei Yang Abstract Figure 2 : Comparison of (a) one-way exo-to-ego transfer framework [ 29 , 52 ] , (b) one-way transfer with feature selection [ 27 ] , and (c) our closed-loop transfer framework, LoopTrans. KE refers to extracting human-object interaction knowledge from exocentric images, while AL refers to localizing affordance regions of objects in egocentric images. Humans can perform previously unexperienced interactions with novel objects simply by observing others engage with them. Weakly-supervised affordance grounding mimics this process by learning to locate object regions that enable actions on egocentric images, using exocentric interaction images with image-level annotations. However, extracting affordance knowledge solely from exocentric images and transferring it one-way to egocentric images limits the applicability of previous works in complex interaction scenarios. Instead, this study introduces LoopTrans, a novel closed-loop framework that not only transfers knowledge from exocentric to egocentric but also transfers back to enhance exocentric knowledge extraction. Within LoopTrans, several innovative mechanisms are introduced, including unified cross-modal localization and denoising knowledge distillation, to bridge domain gaps between object-centered egocentric and interaction-centered exocentric images while enhancing knowledge transfer. Experiments show that LoopTrans achieves consistent improvements across all metrics on image and video benchmarks, even handling challenging scenarios where object interaction regions are fully occluded by the human body. Code is available at https://github.com/nagara214/LoopTrans . Figure 1 : Visualization samples for lie on and ride affordances on egocentric images, comparing state-of-the-art methods with ours. † † footnotetext: ∗ Equal contribution. † Corresponding author is Sibei Yang. 1 Introduction The term “affordance,” first introduced by J.Gibson [ 16 ] , is later formalized in computer vision and robotics to typically describe the “action possibilities” offered by objects [ 2 , 56 , 34 , 38 ] , such as a knife affords cutting or a bicycle affords riding. Affordance grounding [ 8 , 12 , 13 , 35 , 37 , 49 ] further refines this by not only predicting the actions an object can afford but also pinpointing the specific regions that enable those actions, e.g . , a bicycle’s handlebars for pushing and both its handlebars and seat for riding (see Fig 1 ). In contrast to most vision perception systems [ 57 , 6 , 20 , 47 , 10 ] that primarily focus on how objects appear [ 54 , 46 , 48 ] , such as instance and part segmentation, affordance grounding emphasizes how objects function. This is essential for embodied intelligent agents to actively interact with and use objects in the real world [ 18 , 44 , 14 , 50 , 1 ] , while also facilitating downstream tasks such as object manipulation [ 19 , 3 ] and human-object interaction [ 17 , 5 ] . Humans can infer precise affordance grounding on objects across diverse actions and environments, even performing unfamiliar interactions with novel objects, simply by observing others interact with them. To mimic this learning process, we follow a practical weakly-supervised affordance grounding setting [ 31 , 29 , 36 , 27 ] , learning object affordances from human-object interaction images or videos without using any pixel-level affordance annotations. As shown in Fig 1 , given exocentric human-object interaction images and object images with corresponding interaction labels ( e.g . , lie on) during training, the goal in inference is to ground the affordances of each interaction label on the target egocentric object image. Two essential yet challenging cores of affordance grounding are (1) extracting affordance knowledge from exocentric interactions and (2) transferring it to egocentric localization. On one hand, for affordance knowledge extraction , current methods [ 27 , 52 , 22 ] primarily rely solely on exocentric interaction images, using CAM [ 55 ] to generate activation maps. CAM is a classic approach for localizing category-relevant regions using image-level labels by highlighting the most discriminative areas for classification. These methods extract and represent interaction knowledge using image features from the generated activation maps, as shown in the “exocentric+KE” branch in Fig 2 a-b . However, the diversity and complexity of interaction scenes make exocentric images alone insufficient for precise affordance activation, leading to vague, broad regions that may include background or human body parts in simple scenarios, and scattered attention in more complex interactions (see Appendix). On the other hand, transferring knowledge to egocentric localization presents notable challenges due to the significant domain gap between exocentric and egocentric images, as well as the increased difficulty of transferring knowledge from occluded objects in interaction regions. First, exocentric images are often cluttered with small and potentially occluded interaction regions, while egocentric images are clear and object-centered. Most methods [ 29 , 52 ] that constrain appearance similarity of affordance regions between views fail to address knowledge transfer in complex interaction scenes with large domain gaps, often resulting in mislocalization of non-affordance object parts, as illustrated in the “egocentric+AL” branch in Fig 2 a . Second, while recent advances [ 27 ] propose part selection (Fig 2 b ) on egocentric images to mitigate partial occlusion issues, they still rely on exocentric interaction appearance for guidance, limiting applicability in fully occluded interactions, such as “lie on” and “ride,” as shown in Fig 1 . In this paper, we propose a novel closed-loop knowledge transfer framework, LoopTrans, to address these challenges. Unlike the conventional one-way, non-closed-loop pipeline from exocentric interaction to activation and then to egocentric localization (see the arrow flow in Fig 2 a and 2b ), LoopTrans introduces an innovative closed-loop mechanism that enables egocentric localization to feed back into knowledge attention (Fig 2 c ). LoopTrans’s dual design naturally addresses key challenges: (1) Egocentric localization on simple, object-centered egocentric images provides clear localization without interference from the background or human body. It can naturally be used to refine knowledge activation in complex exocentric images (see yellow arrow flow in Fig 2 c ), focusing more precisely on potential affordance regions and resolving coarse and scattered activation issues. (2) In turn, the shared and unified affordance knowledge activation learning across exocentric and egocentric modalities (see green arrow flow in Fig 2 c ) not only reduces the domain gap but also enables direct use of egocentric activation to transfer affordance knowledge (see red arrow flow in Fig 2 c ), bypassing exocentric-to-egocentric appearance transfer and naturally overcoming transfer difficulties caused by occlusions in egocentric interactions. Specifically, LoopTrans works as follows in a closed-loop process. First, interaction → activation: Based on image-level affordance labels, LoopTrans learns a unified classifier for both egocentric and exocentric images, using a shared CAM to highlight both their affordance knowledge activation. Exocentric images provide interaction knowledge, while egocentric images help CAM activation focus more on the object, reducing background and human body interference. More importantly, this shared CAM can effectively identify affordance regions in egocentric objects even when interactions in exocentric images are fully occluded. Second, activation → localization: Thanks to the shared knowledge attention, LoopTrans refines egocentric localization directly through egocentric activation, rather than relying on exocentric-to-egocentric appearance transfer as in previous methods, thereby reducing the challenges of cross-domain knowledge transfer. LoopTrans directly selects clustered object parts in egocentric images based on