Grounding Agent Reasoning in Image Schemas: A Neurosymbolic Approach to Embodied Cognition

François Olivier, Zied Bouraoui · 2025

Read Grounding Agent Reasoning in Image Schemas: A Neurosymbolic Approach to Embodied Cognition by François Olivier, Zied Bouraoui (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-24110.txt.

Introduction

François Olivier and Zied Bouraoui propose a way to ground machine reasoning in the patterns of body experience. People grasp the world through image schemas, stock patterns built from sensing and moving, such as inside and outside or up and down. Machines lack that base, so their reasoning can feel hollow. Their 2025 plan joins language models with formal logic built on these patterns.

A tuned language model would turn plain descriptions into formal forms based on the schemas. The result is a system that reasons over body-born structures rather than words alone. The authors argue this should make agents faster to steer and easier to read. It may also make talk between people and agents feel more natural, since both sides share the same basic patterns.

Catalog record

CREATORFrançois Olivier, Zied Bouraoui
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE25 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-24110

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Grounding Agent Reasoning in Image Schemas: A Neurosymbolic Approach to Embodied Cognition François Olivier, Zied Bouraoui Abstract . Despite advances in embodied AI, agent reasoning systems still struggle to capture the fundamental conceptual structures that humans naturally use to understand and interact with their environment. To address this, we propose a novel framework that bridges embodied cognition theory and agent systems by leveraging a formal characterization of image schemas, which are defined as recurring patterns of sensorimotor experience that structure human cognition. By customizing LLMs to translate natural language descriptions into formal representations based on these sensorimotor patterns, we will be able to create a neurosymbolic system that grounds the agent’s understanding in fundamental conceptual structures. We argue that such an approach enhances both efficiency and interpretability while enabling more intuitive human-agent interactions through shared embodied understanding. Y. Vorobeychik, S. Das, A. Nowé (eds.) Conference: Proc. of the 24th International Conference on Autonomous Agents and Multiagent Systems (AAMAS 2025); May 19 – 23, 2025; Detroit, Michigan, USA bs080 Blue Sky Ideas Track 1. Introduction By the end of the 20th century, the classical paradigm of cognitive science was fundamentally challenged as evidence mounted that our minds do not operate like isolated symbol processing computers, but rather are inextricably linked to our bodily experiences in the world. This became particularly evident in how we understand and use language, as Lakoff and Johnson’s groundbreaking work in ‘ Metaphors We Live By ’ ( Lakoff and Johnson, 1980 ) demonstrated that we comprehend abstract concepts (the target domain) by relying on our physical experiences as a source domain – we understand time through location ("the future is ahead of us"), importance through size ("this is a big deal"), and emotional states through spatial orientation ("I’m feeling down"). To bridge the gap between bodily experience and thought, Johnson ( Johnson, 1987 ) introduced image schemas - recurring patterns abstracted from our sensorimotor interactions - and showed their pervasive role in structuring human thought across both concrete and abstract domains. Over the years, the theory has received robust experimental confirmation across multiple studies ( Richardson et al., 2001 ; Mandler and Cánovas, 2014 ) and has proven fruitful even in non-linguistic domains such as mathematics ( Lakoff and Núñez, 2000 ) . A common example of image schema is OBJECT_INTO_CONTAINER which arises from our early physical experiences of putting objects into containers (e.g., cups and buckets), and later serves as a source domain to understand literal sentences like "Bill is in the house", more abstract ones such as "Berlin is in Germany" or "to be in love", and mathematical expressions such as " 2\in\mathbb{N} ". More recent work has explored how these image schemas can be decomposed into even more basic constituents called conceptual primitives ( Mandler and Cánovas, 2014 ) . For instance, our comprehension of the concept of SUPPORT requires one to have the conceptual primitives of UP/DOWN and CONTACT . Just as cognitive science had to move beyond purely computational models to explain human cognition and linguistic ability, there is an ongoing debate about whether AI systems need similar grounding to achieve genuine language understanding and commonsense reasoning ( Bender and Koller, 2020 ; Søgaard, 2023 ) . While some recent work suggests that Large Language Models (LLMs) can grasp physical concepts through text alone ( Patel and Pavlick, 2022 ) , there are reasons to be skeptical about whether this statistical learning can capture the full depth of human conceptual understanding ( McCoy et al., 2023 ; Mahowald et al., 2024 ) . For instance, ( Peng et al., 2023 ) highlights that LLMs employing in-context learning face significant challenges with tasks that require extensive specification, particularly those where even human annotators must carefully review a complex set of annotation guidelines to perform the task correctly. Using a simulation task, ( Tamari et al., 2020 ) also demonstrates fundamental conceptual limitations of statistical methods - limitations that persist regardless of the scale of the data. Equipping artificial agents with such conceptual embodied structures therefore becomes a crucial goal, as it would not only enable more intuitive and explainable human-agent interactions through shared embodied understanding, but also possibly represent, as suggested by ( Bisk et al., 2020 ) , the necessary step to move AI into its next major paradigm beyond current multimodal systems. However, the primary challenge in achieving such agents is to formalize these psychological theories and deeply embodied structures, and to intertwine the resulting symbolic language with neural recognition and metaphorical mapping techniques in a promising way. In this work, we discuss the main challenges of such an endeavor and propose a promising approach that combines symbolic languages with neural architectures to create an integrated neurosymbolic framework. The main strengths of our approach compared to the existing work are the fully formal characterizations of conceptual structures, the use of existing symbolic solvers to reason with these characterizations, and the deep integration in a neural network in order to create a neurosymblic architecture. The rest of the paper is organized as follows. Section 2 presents some related work from a symbolic and machine learning perspective. Section 3 discusses some of the main properties that the expected formalism should satisfy in order to, as shown in Section 4 , effectively capture the different conceptual primitives that compose image-schematic structures. Section 5 presents how the formalism can be combined with neural networks in a meaningful way in order to enable fully embodied agents. Section 6 discusses the advantages gained in reasoning and natural language understanding with such embodied agents. Section 7 concludes the paper. 2. Related Work The formalization of image schemas is not a new endeavor - by the end of the twentieth century, Frank and Raubal ( Frank and Raubal, 1999 ) had already surveyed the existing formalisms. Among the formalisms that followed, notable approaches include bigraph-based representations ( Amant et al., 2006 ) , methods leveraging the WordNet lexical database ( Kuhn, 2007 ) , and approaches based on qualitative calculi ( Bennett and Cialone, 2014 ; Hedblom, 2020 ) . Qualitative calculi, which generally correspond to relation algebras ( Dylla et al., 2017 ) , appeared particularly well-suited for this formalization task, as they abstract away from precise numerical measurements similarly to how human cognitive processing focuses on relative relationships. Significantly advancing the field, Hedblom’s work made extensive use of the adequacy of qualitative calculi by combining the Region Connection Calculus, Qualitative Trajectory Calculus, Cardinal Directions, and Linear Temporal Logic in order to represent both spatial and temporal dimensions of image schemas ( Hedblom, 2020 ) . Recently, Hedblom et al. proposed the Diagrammatic Image Schema Language (DISL) ( Hedblom et al., 2024 ) , a systematic diagrammatic representation language for image schemas that provides a structured visual framework. Regarding the study of image schemas and embodied approaches in the machine learning community, the work of Wachowiak et al. explores how artificial agents capture implicit human intuitions underlying language ( Wicke and Wachowiak, 2024 ) and introduces systematic methods for classifying natural language expressions into image schemas ( Wachowiak and Gromann, 2022 ) . Recent advances in LLMs have also been leveraged to enhance performance in embodied learning

Read the complete text