AFUN: Towards an Affordance Foundation Model for Functionality Understanding
Zhaoning Wang Yi Zhong Jiawei Fu Henrik I. Christensen Jun Gao · 2026
Read AFUN: Towards an Affordance Foundation Model for Functionality Understanding by Zhaoning Wang Yi Zhong Jiawei Fu Henrik I. Christensen Jun Gao (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-02551.txt.
Introduction
Wang and co-authors present AFUN, a step toward a foundation model for affordance understanding. Affordance links what a robot sees to what it can do. A model must say where to interact and how to move after contact. Past methods do one or the other. Scaling has also lagged. AFUN takes a single RGB-D view plus a language task text.
It predicts a task-conditional functional mask. It also predicts a 3D post-contact motion curve. A large data pipeline feeds the model. Robot, human, simulation, and scan data are converted into one schema. The schema holds language, masks, and object-centric 3D motion labels. On affordance segmentation, AFUN beats all baselines across 8 test sets. Contact-point hit rates gain 12.7 to 61.3 percent over the best baseline. Motion scores lead on all three test sets. The model deploys on real robots with no fine-tuning.
Catalog record
| CREATOR | Zhaoning Wang Yi Zhong Jiawei Fu Henrik I. Christensen Jun Gao |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 62 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2606.02551
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
AFUN: Towards an Affordance Foundation Model for Functionality Understanding Zhaoning Wang Yi Zhong Jiawei Fu Henrik I. Christensen Jun Gao Abstract Affordance understanding bridges visual perception and physical action, serving as an explainable interface for robot manipulation in open and unstructured real-world environments. Yet, building an affordance foundation model that not only understands where and how the interaction should happen, but also generalizes across diverse environments, objects, and tasks, remains a long-standing research challenge. Existing methods typically address only part of this challenge, either localizing task-relevant regions without specifying executable motion, or predicting motion but with limited scalability. In this paper, we present AFUN , a step towards an affordance foundation model for functionality understanding. From a single RGB-D observation and a language task description, AFUN predicts a task-conditional functional mask ( where to interact) and a 3D post-contact motion curve ( how to interact). To support open-world generalization, we build a large-scale standardized data pipeline that converts heterogeneous robot, human, simulation, and real-world scan data into a shared affordance schema with language, masks, and object-centric 3D motion labels. We evaluate AFUN from three aspects: for affordance segmentation, AFUN outperforms all baselines by a large margin across 8 test sets from 4 benchmarks, improving mean gIoU/cIoU by +23.9/+26.3 ; for contact-point prediction, it predicts substantially more accurate points, with a 12.7–61.3% hit-rate gain over the best baseline; and for 3D motion, it achieves the best performance on all three test sets. AFUN can be deployed for real-world robot manipulation without finetuning for robot embodiment or using task-specific heuristics, demonstrating the ability to adapt to open-world affordance tasks. Project page: https://www.zhaoningwang.com/AFUN † † footnotetext: * Equal contribution. 1 Introduction Imagine stepping into a brand-new bedroom; a human can immediately understand which object can do what , and how to do it. For instance, a drawer can be opened or closed from its handle, and a human can identify the exact grasping location before the actual action. This concept of visual affordance [ 20 ] to understand objects’ functionalities underpins human’s capability to perform daily tasks in unstructured real-world environments [ 70 , 63 ] . In robotics and embodied AI, affordance understanding serves as a crucial and explainable interface between visual understanding and physical action. Yet, building a foundation model for affordance understanding that can scale across diverse environments, objects, and tasks is a long-standing research challenge. There are three interconnected requirements when building such an affordance foundation model. (I) The dataset used to train the model must reflect the diversity of real-world manipulation tasks to enable generalization, rather than being collected from narrow domains or a closed set of object categories. (II) The model needs to accurately produce instruction-conditioned segmentation masks: not only locating where robots can interact with, but also adapting to the instruction, since the same object affords different regions under different tasks. (III) To make the interaction actionable for robots, the model must further predict how the interaction should be performed, with a 3D motion representation that a robot can follow. The 3D motion should remain expressive enough to capture diverse behaviors and structured enough for stable supervision and robot execution. In practice, however, existing affordance methods focus mainly on the second requirement alone, formulating the problem as static segmentation [ 73 , 47 ] , keypoint detection [ 88 ] , or reasoning-based grounding [ 70 ] . These approaches can localize interaction regions, but they do not characterize how the object should move after interaction. For the methods focusing on the third requirement, some predict motion in 2D [ 79 , 2 ] , leaving robot execution ambiguous when lifting into 3D, while others [ 87 ] require heuristic localization of actionable objects. Beyond limitations on each modality, most current deep-learning models for affordance understanding [ 79 , 2 , 7 , 87 ] still fall short of open-world generalization due to small-scale datasets with limited diversity. To address these gaps, we present AFUN , a step toward an open-world affordance foundation model. First, we build a large-scale standardized data pipeline that converts public robot, human, and simulation datasets into coherent affordance data with task descriptions, functional masks and 3D motions, extending the current affordance dataset towards one of the largest public affordance datasets to date (Figure 1 (a)). Then, we introduce a unified affordance foundation model that jointly predicts where to act (functional segmentation) and how the interaction should happen (3D motion, represented as a Bézier spline curve), conditioned on text instructions from users and robot-native RGB-D observations, as shown in Figure 1 (b). The mask can then be unprojected into 3D points for robots to perform action leveraging downstream grasping modules such as AnyGrasp [ 18 ] . We evaluate AFUN on 8 segmentation-based affordance benchmarks and 3 motion-based benchmarks. AFUN reaches 69.3 mean segmentation gIoU, compared with 45.4 for the strongest segmentation baseline [ 70 ] ; for motion prediction, it surpasses standalone baselines [ 79 , 2 , 7 ] by a substantial margin. AFUN also demonstrates strong generalization capability when qualitatively evaluated on open-world images (Fig. 1 (c)). Furthermore, we deploy AFUN on a real robot for manipulation. Without any robot-specific finetuning, AFUN can predict precise mask and motion for robot to plan and execute a successful path for manipulation, as illustrated in Fig. 1 (d) and Fig. 7 . Figure 1: Overview of AFUN. (a) We first build a data pipeline to gather a large-scale diverse dataset for affordance understanding. (b) With such a dataset, we then train AFUN to predict a task-conditional functional segmentation mask and a 3D motion trajectory, conditioned on an RGB-D observation and a language task phrase. (c) AFUN can generalize to open-world images for functionality understanding and (d) is directly deployable to the real robot for manipulation. 2 Related Work Affordance localization. Affordance localization mainly asks where a task-specified interaction is possible with various grounding representation. Dense 2D methods cover classical instance- and part-level segmentation [ 55 , 14 ] , weakly supervised cross-view grounding [ 46 , 37 , 29 , 78 ] , egocentric- and human-video mask supervision [ 38 , 24 , 47 ] , LISA-style SAM grounding from LLM hidden states [ 36 , 58 ] , two-stage VLM-to-segmenter pipelines using LLM-emitted coordinates or boxes [ 70 , 40 , 26 , 6 ] , and language-conditioned SAM-style benchmarks and decoders [ 73 , 68 , 30 , 27 ] . Sparse alternatives predict contact points or keypoints, including language-conditioned image keypoints [ 88 ] , 3D keypoint quadruplets encoding contact location and direction [ 44 ] , and cross-category contact transfer by semantic correspondence or retrieval [ 32 , 35 ] . 3D approaches use point clouds, from foundational benchmarks and 2D-to-3D interaction grounding [ 13 , 80 ] to open-vocabulary and language-conditioned 3D-MLLM grounding [ 56 , 60 , 45 , 93 , 9 , 85 , 71 ] , cross-instance or object-to-object transfer [ 67 , 15 ] , and video-driven MLLM learning from human-object interaction [ 69 , 48 ] . Hand-centric work localizes functional grasps and dexterous contacts, from category-level grasp generation [ 10 ] and language-guided task-oriented grasping [ 65 , 91 , 86