Decoupling the components of geometric understanding in Vision Language Models

Eliza Kosoy, Annya Dahmani, Andrew K. Lampinen, Iulia M. Com s , a, Soojin Jeong, Ishita Dasgupta, Kelsey Allen · 2025

Read Decoupling the components of geometric understanding in Vision Language Models by Eliza Kosoy, Annya Dahmani, Andrew K. Lampinen, Iulia M. Com s , a, Soojin Jeong, Ishita Dasgupta, Kelsey Allen (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-03840.txt.

Introduction

Eliza Kosoy and co-authors test whether vision language models grasp basic geometry in a 2025 study. Seeing shapes is central to geometric thought, yet model scores often mix vision with reasoning and world knowledge. A paradigm from cognitive science strips the task down to pure visual understanding.

Top models were compared with adults in the United States and with earlier results from Amazonian adults without formal schooling. The models trailed both human groups overall. They handled some concepts well and failed others, in a patchy pattern unlike human results. Their grip also proved brittle when tasks called for mental rotation of a shape. The authors link the gap to how each side learns: print and schooling for models, hands-on world contact for people, or a blend of both.

Catalog record

CREATOREliza Kosoy, Annya Dahmani, Andrew K. Lampinen, Iulia M. Com s , a, Soojin Jeong, Ishita Dasgupta, Kelsey Allen
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE13 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-03840

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Decoupling the components of geometric understanding in Vision Language Models Eliza Kosoy, Annya Dahmani, Andrew K. Lampinen, Iulia M. Com s , a, Soojin Jeong, Ishita Dasgupta, Kelsey Allen Abstract Understanding geometry relies heavily on vision. In this work, we evaluate whether state-of-the-art vision language models (VLMs) can understand simple geometric concepts. We use a paradigm from cognitive science that isolates visual understanding of simple geometry from the many other capabilities it is often conflated with such as reasoning and world knowledge. We compare model performance with human adults from the USA, as well as with prior research on human adults without formal education from an Amazonian indigenous group. We find that VLMs consistently underperform both groups of human adults, although they succeed with some concepts more than others. We also find that VLM geometric understanding is more brittle than human understanding, and is not robust when tasks require mental rotation. This work highlights interesting differences in the origin of geometric understanding in humans and machines – e.g. from printed materials used in formal education vs. interactions with the physical world or a combination of the two – and a small step toward understanding these differences. 1 Introduction A substantial amount of recent work has tried to understand how well vision-language models understand visual concepts ( Patraucean et al. 2024 , Mouselinos et al. 2024 , Kamoi et al. 2024 , Ollikka et al. 2025 ) . Here, we focus on geometry. Geometry is complex and relies heavily on visual understanding in humans (though AI approaches often sidestep the visual component, e.g. Trinh et al. 2024 ). However, the visual-understanding component of geometry task performance is often entangled with other capabilities – for example, geometric stimuli often occur as a figure in a math textbook, where "understanding" the figure might include the ability to read the labels in the image (OCR), to recognize the mathematical concepts conveyed (world knowledge), and to use this information to then answer some math question (multi-step reasoning). Several geometry or geometry-adjacent benchmarks ( Kazemi et al. 2023 , e.g.) conflate these different capabilities. Kamoi et al. 2024 attempt to remove reasoning components, but still incorporate some aspects of reading, understanding formal mathematics, etc. In this work, we are interested in isolating whether VLMs understand basic geometric concepts visually – separately from knowledge that might arise from a formal education in mathematics. To this end, we build on a long history of work in cognitive science that attempts to assess the origins of geometric understanding ( Piaget et al. 1948 , Spelke 2011 , Spelke & Lee 2012 ) . Specifically, we adapt stimuli from Dehaene et al. 2006 , who investigated the geometric knowledge of the Munduruku, an indigenous Amazonian group, to understand whether fundamental geometric concepts are present across cultural and educational differences. We also adapt their experimental design where rather than asking participants to verbalize specific geometric concepts, participants are presented with 6 images, 5 of which satisfy some geometric concept (e.g. symmetry, or connectedness) and one that violates this concept. Participants are then asked to select the odd one out. We also test adult participants from the USA on the same concepts using Prolific ( Palan & Schitter 2018 ) , and compare to the Munduruku’s performance as reported in Dehaene et al. 2006 . Our overall findings show that even adult humans do not uniformly agree at some of these tasks, and that there is an overall trend that US-adults perform most consistently, followed by the Munduruku, followed by the VLMs. Here, we focus our investigations on a few concepts that showed the maximum deviation or variability in VLM performance. Based on these exploratory analyses, we found that there was still another capability entangled with geometric understanding implicit in the stimuli – that of mental rotation. Many experiments in cognitive science show that humans engage in mental rotation to understand visual stimuli ( Shepard & Metzler 1971 , e.g.) . Several of the geometry stimuli from Dehaene et al. 2006 implicitly assume this capability 2 A. This assumption seems valid for the Munduruku – on a different set of stimuli also from Dehaene et al. 2006 , they find "Performance with allocentric, egocentric, and rotated maps did not differ and was always significantly above chance, suggesting that participants either extracted the geometrical relationships directly or performed a mental rotation so as to align the map with the environment." However, it remains to be seen if this assumption is valid for VLMs; recent results suggest that VLMs’ perception is less robust to rotation ( Ollikka et al. 2025 ) . We therefore generated rotation-controlled versions for 4 of the stimulus sets from the broader geometry set, and conducted experiments on both US adults and VLMs. In these stimulus sets, all 6 images are presented with the same global angle of orientation, therefore eliminating the need for mental rotation. We found that human performance was relatively unaffected (both by the need to realign stimuli with mental rotation, as well as the final rotation angle of stimuli presented to the participants), while VLM performance varied dramatically across rotation angle, and was consistently lowest in the condition that required mental rotation. Figure 1: Overview. A. Adapted from Dehaene et al. 2006 to illustrate the geometric concepts we explore in this work. For the full set of stimuli refer to Appendix 3 . B. An example of the experimental protocol. This is what human participants saw, VLMs received a similar prompt. See main text for details. C. Overview of results; see main text for discussion. 2 Methods 2.1 Adapting stimuli The geometric stimuli were copied from Dehaene et al. 2006 and Franco & Sperry 1977 and created in the free software program Inkscape by tracing the stimuli from the images in the original papers. The images were all black and white. To investigate the effects of mental rotation, we also generated rotation-controlled versions of the original stimuli. The stimuli represent various forms of geometrical shapes from 6 concepts, these were: points, lines, parallelism, figures, congruence, and symmetry. See Figure 3 for actual examples. Within each category there were various numbers of trials. These were originally selected by Dehaene et al. 2006 to test intuitive comprehension of basic geometrical concepts. For each trial, there were six images, one of which violated the core concept. 2.2 VLM experiments We evaluated the Gemini Pro 1.5 model ( Team et al. 2024 ) with default parameters. We used the following prompt: I will give you 6 images, you have to choose which one is the odd one out. Respond with the index (between 1 and 6) of the odd one out image, followed by the word ‘EXPLANATION:’,followed by an explanation of why this is the odd one out image. . We ran this 20 times per trial, randomizing the order of the images presented to the model each time. Error bars are calculated over these 20 runs. We used the explanations to debug our process, but did not use them in our final analyses. 2.3 Human experiments We recruited N = 15 U.S. adults (mean age = 39.53, \sigma=9.56 ) on Prolific Palan & Schitter 2018 to complete a similar

Read the complete text