Can Vision-Language Models See Squares? Text-Recognition Mediates Spatial Reasoning Across Three Model Families

Yuval Levental · 2026

Read Can Vision-Language Models See Squares? Text-Recognition Mediates Spatial Reasoning Across Three Model Families by Yuval Levental (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2602-15950.txt.

Description

Levental runs a deliberately simple test on vision language models: copy a fifteen by fifteen grid of filled and empty cells. When cells are shown as text symbols, the models do well, but when the same grids appear as plain filled squares with no gridlines, their accuracy falls sharply. The catch is that both versions pass through the same visual encoder, since the symbols are images rather than tokenized text.

Across three frontier models, the gap in F1 score between text symbols and squares runs from roughly thirty to fifty points. Each model fails in its own way, undercounting, overcounting, or inventing a template, yet all share the same weakness in spatial localization for non textual marks. The result suggests these models lean on a strong text reading pathway for spatial tasks, while their native visual pathway for plain shapes lags far behind.

Catalog record

CREATORYuval Levental
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE30 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2602-15950

Browse the full library · About the Tower of Babel