Cognitive Science-Inspired Evaluation of Core Capabilities for Object Understanding in AI
Danaja Rutar, Alva Markelius, Konstantinos Voudouris, José Hernández-Orallo, Lucy Cheke · 2025
Read Cognitive Science-Inspired Evaluation of Core Capabilities for Object Understanding in AI by Danaja Rutar, Alva Markelius, Konstantinos Voudouris, José Hernández-Orallo, Lucy Cheke (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-21668.txt.
Introduction
Rutar and co-authors borrow from cognitive science to grade how well AI systems understand objects. Human world models rest on intuitive physics, a felt sense of objects, space, and cause. The authors review three research traditions, Gestalt psychology, enactive cognition, and developmental psychology, and list the core object skills each one credits to the mind.
Set against that list, current AI tests look narrow: benchmarks can catch a system modeling one isolated skill, but not its failure to weave the skills together. The authors sketch evaluation methods built around integration, aiming at systems whose object understanding works as a whole. They argue this is a step toward general-purpose AI that grasps the physical world.
Catalog record
| CREATOR | Danaja Rutar, Alva Markelius, Konstantinos Voudouris, José Hernández-Orallo, Lucy Cheke |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 90 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-21668
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Cognitive Science-Inspired Evaluation of Core Capabilities for Object Understanding in AI Danaja Rutar, Alva Markelius, Konstantinos Voudouris, José Hernández-Orallo, Lucy Cheke Abstract One of the core components of our world models is ‘intuitive physics’—an understanding of objects, space, and causality. This capability enables us to predict events, plan action and navigate environments, all of which rely on a composite sense of objecthood. Despite its importance, there is no single, unified account of objecthood, though multiple theoretical frameworks provide insights. In the first part of this paper, we present a comprehensive overview of the main theoretical frameworks in objecthood research—Gestalt psychology, enactive cognition, and developmental psychology—and identify the core capabilities each framework attributes to object understanding, as well as what functional roles they play in shaping world models in biological agents. Given the foundational role of objecthood in world modelling, understanding objecthood is also essential in AI. In the second part of the paper, we evaluate how current AI paradigms approach and test objecthood capabilities compared to those in cognitive science. We define an AI paradigm as a combination of how objecthood is conceptualised, the methods used for studying objecthood, the data utilised, and the evaluation techniques. We find that, whilst benchmarks can detect that AI systems model isolated aspects of objecthood, the benchmarks cannot detect when AI systems lack functional integration across these capabilities, not solving the objecthood challenge fully. Finally, we explore novel evaluation approaches that align with the integrated vision of objecthood outlined in this paper. These methods are promising candidates for advancing from isolated object capabilities toward general-purpose AI with genuine object understanding in real-world contexts. keywords Objecthood, World Models, Core Objecthood Capabilities, Cognitive Science Paradigms, AI Evaluation 1 Introduction 1.1 Modelling the World One key premise of our existence is that we live in fundamentally open-ended and complex environments. This means that oftentimes we do not know in advance how to plan, act and what inferences to make about the events that have transpired. Despite this foregrounding challenge of human existence, adults, children and animals more generally appear to function seamlessly in their environments most of the time. Even when faced with unforeseen challenges or observing new statistical patterns we respond to them reasonably well. Consider the everyday task of going to the grocery store. When mentally planning the visit to the store, one must continuously switch between different levels of granularity. For instance, whilst it makes sense to consider specific details like the appearance of an avocado (high granularity), it makes little sense to plan each individual step required to reach the store (low granularity). Determining the appropriate level of granularity depends on both the context and the agent’s level of knowledge about the situation (e.g., making more general predictions when highly uncertain, and more specific ones when certain). Putting a glove on to place a couple of avocados in a bag and going to the scale to weigh them requires the integration and coordination of multiple sensorimotor and cognitive skills about objects. Simultaneously, one must filter out irrelevant details, such as background music or other people’s conversations, whilst directing attention to relevant information. Additionally, each trip to the store may vary depending on factors like the layout, time of day, or how crowded it is (context). These contextual changes impact navigation choices, pacing, and even the selection of substitute items when certain products are unavailable. Finally, effective grocery shopping also involves interpreting social cues and adapting to others’ intentions, such as gauging when someone is about to turn into an aisle or reaching for the same product. The ability to track and take advantage of complex environments is not limited to humans. For example, primates, dolphins and birds of the crow and parrot families have demonstrated impressive ability to spontaneously manipulate their physical environment to achieve complex goals Bastos and Taylor 2020 ; Boesch and Boesch 1990 ; Taylor et al. 2009 ; Visalberghi et al. 2009 even among species that did not evolve to use tools Cheke et al. 2011 . Various species have also been shown to keep track of others’ knowledge, desires and social relationships Cheney and Seyfarth 1999 ; Ostojić et al. 2013 . It is thus clear that humans and other animals are masters at solving every-day, multifaceted challenges. Adaptive and context-sensitive behaviours like the one presented in the grocery shopping example in both humans and animals are underpinned by internal models of the world. These world models represent objects, space, causality, and the intentions of others, enabling agents to navigate novel circumstances with flexibility and precision Lake et al. 2017 ; Tenenbaum et al. 2011 . Such models allow natural agents to predict future outcomes, plan actions, and respond efficiently to unexpected challenges Shanahan et al. 2020 . World models are not merely static representations but are dynamic, continuously updated frameworks that support the agent’s ability to generalise across different contexts Hassabis et al. 2017 . By using these models, natural agents can interpret new events, make decisions in real time, and solve everyday challenges seamlessly, even in unfamiliar scenarios. Through these processes, world models allow for effective adaptation and problem-solving in a wide variety of physical and social environments Bottou 2014 . Whilst an easy task for a human, tasks described in the grocery shopping example are exactly the kinds of tasks that AI systems are still struggling with. Namely, flexibly switching between the relevant levels of granularity that monitor internal degree of uncertainty and contextual demands and making instantaneous, almost intuitive decisions. And similarly, deciding what information is relevant to attend to and what not presents a significant challenge. Possessing a robust world model allows humans to do all this by understanding the underlying concepts, rules and relationships that govern different situations in general. However, this does not imply that AI systems universally lack world models; many artificial systems do incorporate such models to varying degrees Hafner et al. 2023 . World models encompass various dimensions of understanding, including social, psychological, physical, and biological aspects of the world Lake et al. 2017 . But perhaps most fundamentally, human and animal intelligent behaviour is grounded in an implicit knowledge of objects, space, and causality—often referred to as intuitive physics Lake et al. 2017 ; Shanahan et al. 2020 . This tacit understanding serves as the foundation for much of our interaction with the environment, underpinning virtually every aspect of intelligent behaviour. Given its foundational role, we will focus on intuitive physics in this paper, exploring how understanding objects and their interactions shapes adaptive behaviour in both natural and artificial agents. 1.2 Object Understanding is Central to Robust Models of the World As humans, we comprehend the world in terms of objects that behave in predictable ways. We use this information to predict the unfolding of future events Battaglia et al. 2013 ; Smith and Vul 2013 , to plan actions and navigate the environment Kaplan and Friston 2018 , and to skillfully use tools Osiurak and Badets 2016 . All these activities assume some understanding of what objects are, what properties they have, and how they behave - in short, these activities assume a sense of objecthood. But what exactly does this objecthood sense entail, and from