Affordance Representation and Recognition for Autonomous Agents
Habtom Kahsay Gidey, Niklas Huber, Alexander Lenz, Alois Knoll · 2025
Read Affordance Representation and Recognition for Autonomous Agents by Habtom Kahsay Gidey, Niklas Huber, Alexander Lenz, Alois Knoll (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-24459.txt.
Introduction
Habtom Kahsay Gidey and three co-authors address how autonomous software agents build an actionable model of their digital environment. Agents read structured data such as a web page's document object model and service descriptions, but raw HTML is too verbose for foundation models to use directly. Hardcoded integrations, meanwhile, cannot adapt when services evolve.
The authors introduce a pattern language for world modeling from structured data, presenting two complementary architectural patterns. The patterns focus on affordances: what actions the environment offers the agent and how to recognize them. The goal is a compact, adaptable internal model that lets agents act reliably across changing web services.
Catalog record
| CREATOR | Habtom Kahsay Gidey, Niklas Huber, Alexander Lenz, Alois Knoll |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 22 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2510-24459
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Affordance Representation and Recognition for Autonomous Agents Habtom Kahsay Gidey, Niklas Huber, Alexander Lenz, Alois Knoll Abstract The autonomy of software agents is fundamentally dependent on their ability to construct an actionable internal world model from the structured data that defines their digital environment, such as the Document Object Model (DOM) of web pages and the semantic descriptions of web services. However, constructing this world model from raw structured data presents two critical challenges: the verbosity of raw HTML makes it computationally intractable for direct use by foundation models, while the static nature of hardcoded API integrations prevents agents from adapting to evolving services. This paper introduces a pattern language for world modeling from structured data, presenting two complementary architectural patterns. The DOM Transduction Pattern addresses the challenge of web page complexity by distilling a verbose, raw DOM into a compact, task-relevant representation or world model optimized for an agent’s reasoning core. Concurrently, the Hypermedia Affordances Recognition Pattern enables the agent to dynamically enrich its world model by parsing standardized semantic descriptions to discover and integrate the capabilities of unknown web services at runtime. Together, these patterns provide a robust framework for engineering agents that can efficiently construct and maintain an accurate world model, enabling scalable, adaptive, and interoperable automation across the web and its extended resources. [orcid=0000-0001-5802-2606, [email protected], ] *1 [[email protected], ] [[email protected], ] [orcid=0000-0003-4840-076X, [email protected], ] keywords Autonomous Agents ,Affordances ,World Modeling ,Design Patterns ,Web Automation. † † copyrightyear: 2025 † † copyright: Copyright for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). † † venue: The Second International Workshop on Hypermedia Multi-Agent Systems (HyperAgents 2025), in conjunction with the 28th European Conference on Artificial Intelligence (ECAI 2025); October 26, 2025, Bologna, Italy † † address: Technische Universität München, München, Germany † † address: Jessy Works, München, Germany † † corresp: Corresponding author. 1 Introduction Recent advances and the abundance of foundation models are driving a new generation of software agents performing complex cognitive tasks in dynamic, mixed-reality ecosystems Gur et al. 2023 ; Ning et al. 2025 ; Macedo et al. 2024 . For these agents to operate effectively, they must perceive and understand their environment to build an internal world model , an actionable representation that guides their reasoning and planning Ha and Schmidhuber 2018 ; LeCun 2022 ; Richens et al. 2025 . Although significant research has focused on visual perception from pixels, many digital environments are built upon rich, structured data sources like the hypermedia of the Document Object Model (DOM) of web pages and the descriptive interfaces of web services Yang et al. 2024 ; He et al. 2024 . To this end, leveraging this explicit structure offers a more efficient and deterministic path to building a world model Tan et al. 2025 ; Liu et al. 2025 . However, directly consuming this structured data presents two critical architectural challenges that hinder the development of truly autonomous agents Assouel et al. 2023 ; Ning et al. 2025 . First, the verbosity of raw HTML is a major bottleneck for agents using foundation models for reasoning and planning Gur et al. 2023 . Empirical studies of DOM pruning show that the majority, often 80 - 90%, of tokens in raw HTML consist of non-semantic markup such as scripts, styles, and trackers Tan et al. 2025 ; Yang et al. 2024 , which can overwhelm the model’s, i.e., LLMs, context window, degrade reasoning quality, and incur high computational costs Lù et al. 2025 . This is the fundamental challenge of representation , where the raw state of the world is too complex for the agent’s cognitive core to process efficiently Liu et al. 2025 ; Kim et al. 2025 . Second, agents must interact with a dynamic ecosystem of services and devices in the increasingly interconnected Web of Things (WoT) and microservice architectures Gandon 2021 ; Castellucci1 et al. 2025 ; Gidey et al. 2022 . In such cases, the traditional approach of hardcoding binary interfaces, such as API endpoints and interaction logic, creates brittle, tightly coupled systems that cannot adapt when a service changes or a new device is introduced in the web microcosm Gidey et al. 2023 ; Assouel et al. 2023 . This is the challenge of interoperability, adaptability, and discovery , i.e., recognition of affordances, where the agent’s world model is static and cannot be updated to reflect a changing digital environment Lù et al. 2025 ; Liu et al. 2025 . This paper introduces preliminary work on a pattern language Gidey et al. 2017 that addresses these challenges by framing them as problems of world model construction Ha and Schmidhuber 2018 ; LeCun 2022 ; Richens et al. 2025 . We present two architectural design patterns that provide reusable solutions for building and enriching an agent’s world model from structured data: 1. The DOM Transduction Pattern: A pattern for distilling a complex, raw DOM into a compact, task-relevant world model optimized for an agent’s reasoning core. 2. The Hypermedia Affordances Recognition Pattern: A pattern for dynamically enriching the world model by parsing standardized semantic descriptions of web services to discover and integrate their capabilities at runtime. Combined with other percepts, these patterns provide a robust framework for engineering agents that can efficiently and adaptively interact with the structured web and its connected extended resources and the web of things Castellucci1 et al. 2025 ; LeCun 2022 . 2 Background This research work covers several domains; in particular, it is situated at the intersection of hypermedia multi-agent systems, world modeling, and cognitive automation Gidey et al. 2023 ; Gandon 2021 ; Ha and Schmidhuber 2018 ; Castellucci1 et al. 2025 ; LeCun 2022 ; Richens et al. 2025 . 2.1 Web Agents Observation Space The challenge of processing verbose HTML for LLM-based agents has become a significant area of research Assouel et al. 2023 ; Ning et al. 2025 . The core problem is that the agent’s observation space, when represented by a raw DOM, is misaligned with the LLM’s processing capabilities. As a result, this has led to the development of various techniques for DOM cleaning and simplification Tan et al. 2025 ; Yang et al. 2024 ; Liu et al. 2025 . Recent work on LLM-based web agents such as WebVoyager, Agent-E, and AgentOccam has highlighted the critical importance of managing the agent’s observation space Abuelsaad et al. 2024 ; He et al. 2024 ; Yang et al. 2024 . The primary challenge lies in handling complex and verbose HTML, which motivates approaches such as DOM distillation and HTML pruning. Several approaches have emerged, collectively known as DOM distillation or HTML pruning, which aim to simplify the DOM to make it more tractable for LLMs. For example, AgentOccam focuses on refining the observation space to better align with the LLM’s capabilities Yang et al. 2024 . More recent work, such as HtmlRAG, has introduced block-tree-based pruning methods and concrete methods to clean and compress HTML while preserving its structure Tan et al. 2025 . This concept of webpage segmentation has a long history in information extraction, aiming to break down a page into semantically related parts to improve downstream tasks