Affordance Benchmark for MLLMs DOI: XXXXXXX.XXXXXXX Conference: Make sure to enter the correct conference title from your rights confirmation email; June 03–05, 2018; Woodstock, NY ISBN: 978-1-4503-XXXX-X/2018/06 CCS: Computing methodologies Computer vision CCS: Computing methodologies Spatial and p

Junying Wang, Wenzhe Li, Yalun Wu, Yingji Liang, Yijin Guo, Chunyi Li, Haodong Duan, Zicheng Zhang, Guangtao Zhai · 2025

Read Affordance Benchmark for MLLMs DOI: XXXXXXX.XXXXXXX Conference: Make sure to enter the correct conference title from your rights confirmation email; June 03–05, 2018; Woodstock, NY ISBN: 978-1-4503-XXXX-X/2018/06 CCS: Computing methodologies Computer vision CCS: Computing methodologies Spatial and p by Junying Wang, Wenzhe Li, Yalun Wu, Yingji Liang, Yijin Guo, Chunyi Li, Haodong Duan, Zicheng Zhang, Guangtao Zhai (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-00893.txt.

Introduction

Wang and eight co-authors grade big image-tongue models on how well they read what a thing can do in this twenty twenty-five paper. A cup can be drunk from and a chair sat on, and such use sense guides safe acts. Top models ace most sight-word tests, yet this use sense went unchecked.

Inside, their test holds two banks of ask-and-call pairs on use sense. One bank checks fixed use traits of things in nine sub-fields of the theme. One bank checks use that shifts with place, time, or the user at hand. Near a score of models took the test next to human grades for scale. Paid models led free ones, yet all fell far short of the human marks.

Catalog record

CREATORJunying Wang, Wenzhe Li, Yalun Wu, Yingji Liang, Yijin Guo, Chunyi Li, Haodong Duan, Zicheng Zhang, Guangtao Zhai
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE28 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2506-00893

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Affordance Benchmark for MLLMs DOI: XXXXXXX.XXXXXXX Conference: Make sure to enter the correct conference title from your rights confirmation email; June 03–05, 2018; Woodstock, NY ISBN: 978-1-4503-XXXX-X/2018/06 CCS: Computing methodologies Computer vision CCS: Computing methodologies Spatial and physical reasoning Junying Wang, Wenzhe Li, Yalun Wu, Yingji Liang, Yijin Guo, Chunyi Li, Haodong Duan, Zicheng Zhang, Guangtao Zhai Abstract . Affordance theory posits that environments inherently offer action possibilities that shape perception and behavior. While Multimodal Large Language Models (MLLMs) excel in vision-language tasks, their ability to perceive affordance, which is crucial for intuitive and safe interactions, remains underexplored. To address this, we introduce A4Bench , a novel benchmark designed to evaluate the affordance perception abilities of MLLMs across two dimensions: 1) Constitutive Affordance , assessing understanding of inherent object properties through 1,282 question-answer pairs spanning nine sub-disciplines, and 2) Transformative Affordance , probing dynamic and contextual nuances (e.g., misleading, time-dependent, cultural, or individual-specific affordance) with 718 challenging question-answer pairs. Evaluating 17 MLLMs (nine proprietary and eight open-source) against human performance, we find that proprietary models generally outperform open-source counterparts, but all exhibit limited capabilities, particularly in transformative affordance perception. Furthermore, even top-performing models, such as Gemini-2.0-Pro (18.05% overall exact match accuracy), significantly lag behind human performance (best: 85.34%, worst: 81.25%). These findings highlight critical gaps in environmental understanding of MLLMs and provide a foundation for advancing AI systems toward more robust, context-aware interactions. The dataset is available in https://github.com/JunyingWang959/A4Bench/ . Figure 1. The motivation of the A4Bench. The affordance theory proposed by James J. Gibson ( Gibson 2014 ) defines the action possibilities provided by the object for organisms. Evaluating the affordance perception abilities of MLLMs can help enable more efficient task execution and improved adaptability to diverse environments for AI systems. 1. Introduction ‘ What we perceive when we look at objects are their affordances, not their qualities ’ . This bold assertion by James J. Gibson encapsulates the essence of affordance theory , which posits that environments inherently offer animals a spectrum of action possibilities that shape perception and behavior ( Gibson 2014 ) . Unlike traditional theories that focus on discerning physical attributes such as shape or color, this framework suggests that organisms directly apprehend the action possibilities embedded within environmental features. For instance, a door is perceived not merely as a structure but as an entity affording passage: it can be unlocked with a key or forcefully opened with a kick. Defined as the inherent values and meanings of objects, affordance form a vital link between ecological features and behavioral responses, challenging dualistic paradigms and promoting a unified ecological approach to perception. Why affordance is important? Understanding affordance is vital for intelligent agents, whether biological or artificial, to engage meaningfully with their environments . For artificial intelligence systems navigating complex settings, understanding affordance ensures robust and intuitive interactions while enhancing safety. For example, a robot perceiving a surface as affording support can navigate terrain securely, while one identifying an object as graspable can manipulate it effectively. Moreover, effective affordance perception enables industrial robots to execute tasks with greater efficiency and allows rehabilitation robots to enhance human-machine interaction by adapting to user needs. The capacity to discern affordance, whether beneficial or harmful, underpins behaviors ranging from survival to complex social interactions. Gibson underscored this by noting how humans modify environments ’to change what it affords’, emphasizing the profound link between affordance and intentional action. What Affordance Perception Should MLLMs Possess? Accurate perception of an object affordance enables recognition of its true utility, as Gibson asserts, ‘ If the affordance of a thing are perceived correctly, we say that it looks like what it is ’. Yet discerning genuine affordance often demands experiential learning, since a leaf appearing benign may conceal a nettle sting or a politician seeming helpful may mask deceptive demagoguery. To evaluate this capability rigorously, we introduce a novel benchmark ( A4Bench ) to assess affordance perception across two primary dimensions. First, constitutive affordance examines how MLLMs apprehend inherent object and environmental properties such as shape, size, or material that determine whether a surface affords walking or an object affords grasping, using 1282 question-answer pairs across nine sub-disciplines. Second, transformative affordance probes comprehension of dynamic affordance, including misleading affordance (like a glass barrier appearing as open air but affording collision), time-independent affordance (e.g., fruit ripening changes its food value affordance, specified by color), cross-cluture affordance (e.g., a postbox affording letter-mailing only within a postal system), and individual affordance (e.g., a ledge being sit-on-able depending on leg length). A4Bench pioneers comprehensive affordance perception evaluation, highlighting contextual and dynamic nuances to reveal MLLM capabilities and steer future development. Testing 17 leading MLLMs, both open-source and closed-source, on A4Bench reveals that even the top-performing model significantly trails human performance, leading to a critical conclusion: MLLMs are still poor at affordance perception. MLLMs exhibit significant limitations in perceiving affordance with human-like proficiency, with marked disparities between open-source and closed-source variants. This deficiency spans constitutive and transformative affordance. Challenges arise from limited contextual understanding, notably in medical disciplines within constitutive affordance and agent-specific dimensions within transformative affordance. Conversely, human observers consistently excel across all dimensions, highlighting inadequate comprehension by MLLMs of object-human-environment interactions essential for robust affordance perception (poor at affordance perception). Evaluating affordance perception in MLLMs shapes AI advancements by driving A4Bench development, as shown in Fig. 1 . And our contributions can be summarized as follows: • We extend affordance theory to the context of MLLMs and introduce a systematic evaluation framework for affordance that rigorously tests the understanding of MLLMs on understanding constitutive and transformative affordance. • We propose A4Bench , the first comprehensive benchmark specifically designed to assess the affordance perception capabilities of MLLMs. It encompasses 2,000 multimodal question-answer pairs, covering both Constitutive Affordance (1,282 pairs across nine sub-disciplines) and Transformative Affordance (718 pairs addressing misleading, time-dependent, cross-cultural, and individual-specific affordance). • By evaluating 17 MLLMs (9 proprietary and 8 open-source) against human performance, we provide a detailed analysis of their affordance perception capabilities. Our findings reveal significant limitations in existing models, with even top performers lagging far behind human benchmarks, thus offering a roadmap for future improvements in context-aware AI systems. 2. Related Works Multimodal Large Language Models. Multimodal Large Language Models (MLLMs) have demonstrated remarkable capabilities in diverse vision-language tasks. Among them, diverse proprietary ( close-source ) models ( OpenAI 2024 ; Anthropic 2024 ; Google 2024a ; step-1o 2024 ; BailingMM-Pro-0120 2025 ; MUG-U-7B 2025 ) and representative open-source models ( Wu et al. 2024b ; Team et al. 2025 ; Dai et al. 2023 ; Zhu et al. 2023 ; Liu et al. 2023 ; Liu et al. 2024b ; Li et al. 2024b ; Lu et al. 2024 ; Guo et al. 2025 ; Dong et al. 2025 ; Yao et al. 2024 ; Dai et al. 2024 ; Agrawal et al. 2024 ; Liu et al. 2024c ; Zhang et al. 2025 ; Lin et al. 2023 ; Chen et al. 2025 ) have exhibited impressive superiority from embodied agents ( Song et al. 2023 ) to other real-world applications ( Huynh et al. 2025 ) . However, whether these MLLMs are masters at percepting the affordance of object is

Read the complete text