Assessing VLM-Driven Semantic-Affordance Inference for Non-Humanoid Robot Morphologies Conference: Proc. of the 25th International Conference on Autonomous Agents and Multiagent Systems (AAMAS 2026); May 25 – 29, 2026; Paphos, Cyprus 1076

Jess Jones, Sabine Hauert, Raul Santos-Rodriguez · 2026

Read Assessing VLM-Driven Semantic-Affordance Inference for Non-Humanoid Robot Morphologies Conference: Proc. of the 25th International Conference on Autonomous Agents and Multiagent Systems (AAMAS 2026); May 25 – 29, 2026; Paphos, Cyprus 1076 by Jess Jones, Sabine Hauert, Raul Santos-Rodriguez (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-19509.txt.

Introduction

Jones, Santos-Rodriguez, and Hauert test whether vision-language models can infer affordances for non-humanoid robots. Such models handle human-object interaction well, but robots with different bodies remain largely untested. The authors build a hybrid dataset mixing real annotated affordance relations with model-generated synthetic scenarios. Tests across object types and robot shapes show wide variation in inference skill. The goal is robots that handle daily objects with bodies unlike our own.

Catalog record

CREATORJess Jones, Sabine Hauert, Raul Santos-Rodriguez
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE38623
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.19509

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Assessing VLM-Driven Semantic-Affordance Inference for Non-Humanoid Robot Morphologies Conference: Proc. of the 25th International Conference on Autonomous Agents and Multiagent Systems (AAMAS 2026); May 25 – 29, 2026; Paphos, Cyprus 1076 Jess Jones, Sabine Hauert, Raul Santos-Rodriguez Abstract . Vision-language models (VLMs) have demonstrated remarkable capabilities in understanding human-object interactions, but their application to robotic systems with non-humanoid morphologies remains largely unexplored. This work investigates whether VLMs can effectively infer affordances for robots with fundamentally different embodiments than humans, addressing a critical gap in the deployment of these models for diverse robotic applications. We introduce a novel hybrid dataset that combines annotated real-world robotic affordance-object relations with VLM-generated synthetic scenarios, and perform an empirical analysis of VLM performance across multiple object categories and robot morphologies, revealing significant variations in affordance inference capabilities. Our experiments demonstrate that while VLMs show promising generalisation to non-humanoid robot forms, their performance is notably inconsistent across different object domains. Critically, we identify a consistent pattern of low false positive rates but high false negative rates across all morphologies and object categories, indicating that VLMs tend toward conservative affordance predictions. Our analysis reveals that this pattern is particularly pronounced for novel tool use scenarios and unconventional object manipulations, suggesting that effective integration of VLMs in robotic systems requires complementary approaches to mitigate over-conservative behaviour while preserving the inherent safety benefits of low false positive rates. \doi WTKR8312 C. Amato, L. Dennis, V. Mascardi, J. Thangarajah (eds.) 1. Introduction Bringing robotic systems out of controlled laboratory settings into dynamic, real-world environments remains a fundamental challenge on the path toward a future where smart machines are ubiquitous within human society. Achieving robust and adaptive autonomy requires robots to not only perceive their surroundings but also intuitively understand how they can interact with objects and other robots. This is particularly true for future embodied AI, likely heterogeneous in both form and function, with diverse robots developed for specific tasks, often by different commercial entities ( 7 ) . Enabling adaptive, collaborative behaviours within such heterogeneous multi-robot systems necessitates a foundational approach to grounding robot interactions. We hypothesise that grounding these interactions through the concept of affordances ( 12 ) can substantially improve their capacity to learn and generalise collaboration strategies. Affordances, defined as the actionable possibilities an environment offers an agent, intrinsically link perception to potential actions. For a robot, understanding affordances means recognising not just what an object is but how it can be interacted with given the robot’s unique physical capabilities and the current task. This shifts the paradigm from basic object classification and localisation to dynamic, context-aware interaction, paving the way for more flexible and intelligent robotic systems. Our broader research proposes to leverage the semantic encoding power of VLMs to enhance the scene-representation of each robot with semantic-affordance descriptions of the objects it encounters, linking its unique capabilities to its environment, and providing a framework from which collaborative interactions can be derived in multi-robot settings in the future. While VLMs show promise, their application to robotics presents a critical challenge, rooted in the fact that their training data is predominantly human-centric ( 20 ) . This raises a fundamental question, which this paper aims to address through empirical analysis: Can VLMs effectively infer affordances for robots with fundamentally different embodiments than humans? We use a novel hybrid dataset, curated for non-humanoid robotic affordance inference, to investigate the performance of VLMs in supporting the zero-shot affordance characterisation of objects across a breadth of diverse contexts, including household items, food products, environmental clean-up, and construction materials. Performance is evaluated on multiple non-humanoid robot morphologies to understand the generalisability and limitations of these models. Our research reveals variations in affordance inference capabilities, identifying biases where VLMs tend toward conservative predictions, characterised by low false positive rates but notably high false negative rates. This conservative bias, while minimising erroneous actions, means robots may underutilise their true capabilities, missing valid interaction and collaboration opportunities, particularly for novel tool use or unconventional manipulations. Our findings provide critical insights into the practical deployment of VLMs in real-world multi-robot systems. By quantifying the performance variations and identifying the nature of VLM biases in affordance understanding for non-humanoid forms, this work lays the groundwork for developing more robust and reliable embodied AI. The enriched scene-representations, which can be derived from the affordance characterisations of this approach, capture not only what and where objects are, but how the local environment can be interacted with given an arbitrary robot morphology. This will be leveraged in future work to inform action priors that ground learning task decomposition and decision-making strategies in collaborative multi-robot settings, integrating both intrinsic robot abilities and situational demands. 2. Related Work The ability of smart machines to perceive and interact with their environment is fundamental to achieving robust autonomous behavior. A cornerstone concept in this regard is that of affordances ( 12 ) . Affordances refer to the actionable possibilities that the environment offers to an individual, which Gibson argues, is not exclusive to humans, but extends to all animals based on a fundamental animal-environment interaction system ( 11 ) . The framing of affordances provides a model which intrinsically links perception and morphology to action. For robotic systems, understanding affordances is crucial for tasks ranging from object manipulation and navigation, to collaborative multi-robot, and human-robot interaction ( 2 ) . Rather than merely identifying objects, robots must infer how they can interact with those objects given their own physical capabilities and the task at hand. Traditionally, affordance perception in robotics has relied on hand-engineered features, geometric reasoning, or supervised learning on curated datasets specific to particular objects, robots, and tasks ( 6 ; 14 ) . While effective in constrained environments, these approaches often struggle with generalisation to novel objects, unseen environments, or diverse robot morphologies. The high cost of data annotation and the limited scalability of these methods present significant barriers to deploying robots in unstructured, dynamic real-world settings. The recent emergence of large-scale VLMs has impacted various fields within artificial intelligence, including computer vision and natural language processing ( 10 ) . Trained on vast amounts of internet-scale image-text data, VLMs have shown capabilities in zero-shot generalisation, common-sense reasoning, and cross-modal understanding ( 23 ; 21 ) . Their ability to connect visual input with high-level semantic concepts, often expressed in natural language, presents a compelling opportunity to address the limitations of traditional affordance perception methods in robotics. Initial research has begun to explore the application of VLMs to robotic affordance prediction, primarily by leveraging their inherent understanding of human-object interactions ( 8 ; 3 ; 22 ; 24 ; 18 ; 17 ) . These works often frame affordance as a segmentation, localisation or trajectory modelling problem, identifying regions of objects that afford certain human actions (e.g. "grasp," "cut," "pour"). However, a critical gap remains in understanding how these human-centric affordance concepts translate to non-humanoid robotic embodiments. Robots possess diverse manipulators, grippers, and end-effectors, with physical capabilities that may differ significantly from human hands. This disparity raises fundamental questions about the direct applicability and generalisation capabilities of VLMs across the broader spectrum of existing and future robotic morphologies, and their unique interaction possibilities. This work represents an initial exploration

Read the complete text