NoReGeo: Non-Reasoning Geometry Benchmark

Irina Abdullaeva, Anton Vasiliuk, Elizaveta Goncharova, Temurbek Rahmatullaev, Zagorulko Ivan, Maxim Kurkin, Andrey Kuznetsov · 2026

Read NoReGeo: Non-Reasoning Geometry Benchmark by Irina Abdullaeva, Anton Vasiliuk, Elizaveta Goncharova, Temurbek Rahmatullaev, Zagorulko Ivan, Maxim Kurkin, Andrey Kuznetsov (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-10254.txt.

Introduction

Irina Abdullaeva and co authors test if AI has a raw feel for shape. Most shape tests let a model work it out by steps. That tests algebra more than sight. A model may pass yet lack a true sense of space. The team made a new test set named NoReGeo.

The set asks for direct shape calls. Can a model tell if lines cross, if a point sits in a shape, or if forms match? No step work should be needed. The set thus tests if shape sense is built in. It gives a clean check that past sets did not.

Catalog record

CREATORIrina Abdullaeva, Anton Vasiliuk, Elizaveta Goncharova, Temurbek Rahmatullaev, Zagorulko Ivan, Maxim Kurkin, Andrey Kuznetsov
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE33 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2601-10254

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

NoReGeo: Non-Reasoning Geometry Benchmark Irina Abdullaeva, Anton Vasiliuk, Elizaveta Goncharova, Temurbek Rahmatullaev, Zagorulko Ivan, Maxim Kurkin, Andrey Kuznetsov Abstract We present NoReGeo, a novel benchmark designed to evaluate the intrinsic geometric understanding of large language models (LLMs) without relying on reasoning or algebraic computation. Unlike existing benchmarks that primarily assess models’ proficiency in reasoning-based geometry-where solutions are derived using algebraic methods-NoReGeo focuses on evaluating whether LLMs can inherently encode spatial relationships and recognize geometric properties directly. Our benchmark comprises 2,500 trivial geometric problems spanning 25 categories, each carefully crafted to be solvable purely through native geometric understanding, assuming known object locations. We assess a range of state-of-the-art models on NoReGeo, including frontier models like GPT-4, observing that even the most advanced systems achieve an overall maximum of 65% accuracy in binary classification tasks. Further, our ablation experiments demonstrate that such geometric understanding does not emerge through fine-tuning alone, indicating that effective training for geometric comprehension requires a specialized approach from the outset. Our findings highlight a significant gap in current LLMs’ ability to natively grasp geometric concepts, providing a foundation for future research toward models with true geometric cognition. Code — https://github.com/FusionBrainLab/NoReGeo Figure 1: Evaluation samples from NoReGeo benchmark. Each problem is shown in three formats – (a) text‑only, (b) text with dotted-image (points only), and (c) text with full-image (points plus connecting lines) – together with the golden answer (yellow) and the model’s prediction. Introduction Although modern LLMs excel at symbolic reasoning, they still treat even the simplest spatial relations as formal reasoning problems, producing multi-step chains of thought instead of relying on an intuitive sense of geometry. Ask whether two line segments intersect, and many models unfold a miniature proof rather than making a direct geometric judgment. This gap between symbolic reasoning and intuitive spatial understanding becomes especially limiting in time-critical settings — CAD engines updating thousands of vertices per frame, robots refining grasp trajectories in milliseconds, or geospatial systems performing rapid visibility checks. In such environments, approximate geometric intuition is far more valuable than multi-line chains of thought. Yet current models lack native geometric understanding and must simulate reasoning even for the simplest relations. Existing geometry benchmarks primarily emphasize complex proofs or multi-step algebraic reasoning ( Lu et al. 2021 ; Kazemi et al. 2024 ) , conflating two skills: (i) identifying relevant spatial facts and (ii) executing symbolic derivations. To isolate the first skill, we introduce NoReGeo , a benchmark of 2,500 trivially solvable geometry problems across 25 categories. Each item can be answered directly from point locations, without auxiliary constructions, theorems, or lengthy CoT. Every problem appears in both text-only form and a paired diagram, enabling controlled comparisons between text-based LLMs and multimodal vision-language models (VLMs). We evaluate more than 45 frontier and open-source models under both modalities. Even the strongest open-source VLM achieves only \sim\!55\% accuracy (Phi3.5-Vision), and the best proprietary model reaches \sim\!65\% — well below human performance of \sim\!74.5\% on the same multiple-choice tasks. These results expose a substantial gap in basic spatial intuition. Further analysis shows that fine-tuning alone does not confer geometric competence, whereas a simple linear probe on a frozen vision encoder solves the tasks almost perfectly — suggesting that geometric features are present in embeddings but are not accessed by current LLM architectures or training regimes. Our contributions are as follows: 1. We motivate and formalize the concept of native geometric understanding as a core capability required for spatially intensive applications. 2. We introduce NoReGeo , the first benchmark explicitly designed to test this ability without chain-of-thought or algebraic computation, comprising 2 500 items across 25 categories in both text and image formats. 3. We provide a comprehensive evaluation of over 45 state-of-the-art LLMs and VLMs, showing that all fall short of human-level performance on elementary geometric tasks. 4. Through fine-tuning and linear-probing studies, we demonstrate that geometric knowledge exists latently in vision encoders but fails to naturally emerge in current LLM training paradigms. Overall, our findings point to an urgent open problem: bridging the gap between symbolic reasoning and true geometric cognition in future foundation models. Related Work Spatial reasoning in vision–language models. Vision–language systems must ground textual concepts such as left of or bigger than in a spatial frame of reference. Synthetic VQA benchmarks like CLEVR isolate this ability by rendering scenes of coloured shapes and asking queries that combine Boolean logic with coarse spatial predicates ( Abraham, Alirezaie, and Raedt 2024 ) . Subsequent datasets move toward natural imagery: SpatialSense adversarially mines object pairs (e.g. person–bench ) to evaluate relative positioning ( Yang, Russakovsky, and Deng 2019 ) , while 3DSRBench embeds similar relations in RGB-D scans of indoor environments ( Ma et al. 2025 ) . Interactive navigation tasks extend spatial grounding to embodied settings: BabyAI and ALFRED require locating, manipulating, and placing objects in simulated rooms ( Chevalier-Boisvert et al. 2019 ; Shridhar et al. 2020 ) , and Room-to-Room (R2R) tests natural-language navigation in realistic 3-D reconstructions ( Hong et al. 2025 ) . Together, these datasets show that VLMs handle qualitative spatial relations reasonably well, yet they provide little evidence that models encode the fine-grained geometric attributes — distances, angles, midpoints, coordinate relations — needed for the intuitive geometric understanding. Geometry word-problem benchmarks. A parallel line of work examines whether models can solve textbook geometry questions that pair a diagram with natural language. Benchmarks such as Geometry3K ( Lu et al. 2021 ) , PGPS-9K ( Zhang, Yin, and Liu 2023 ) , MathVista ( Lu et al. 2024 ) , MathVerse ( Zhang et al. 2024b ) , and GeomVerse ( Kazemi et al. 2024 ) collect thousands of K–12 geometry problems, while GeoEval adds a calibrated difficulty ladder and reports that even math-tuned LLMs plateau at \approx 55\% accuracy on “regular” items and <10\% on Olympiad-level ones ( Zhang et al. 2024a ) . Solving these tasks typically requires multi-step reasoning : interpreting diagrams, identifying theorems, and chaining symbolic deductions. As a result, existing evaluations emphasize algebraic manipulation and proof-style solution pipelines — often supported by chain-of-thought prompting, external tools, or full symbolic proof synthesis (e.g. AlphaGeometry ( Trinh et al. 2024 ) ) — making it difficult to determine whether models possess any native geometric perception independent of reasoning. Skill-specific probes. To obtain a more targeted diagnostic signal, several recent efforts isolate individual geometric skills. PlanQA presents floor-plan layouts as JSON and asks questions about visibility or shortest paths ( Rodionov et al. 2025 ) . GeomRel focuses on detecting equal segments and angles before any numeric computation ( Wang et al. 2025 ) , and GeoGramBench converts Asymptote-style code into natural-language questions of varying abstraction ( Luo et al. 2025b ) . These benchmarks remove some confounds but still require models to parse code-like input formats or engage in subtle inference — leaving open the question of whether models can make direct, perception-level geometric judgments. The proposed cross-modal NoReGeo benchmark targets single-step, school-level geometry questions (midpoints, area comparisons, collinearity, symmetry tests, and similar micro-skills) that can be answered instantaneously by anyone with basic geometric intuition with no auxiliary constructions, theorem recall, or multi-hop reasoning. By providing both text-only and vision‑augmented variants under a shared output format, NoReGeo

Read the complete text