GeoBuildBench: A Benchmark for Interactive and Executable Geometry Construction from Natural Language
Jinwoong Kim, Rui Yang, Huishuai Zhang · 2026
Read GeoBuildBench: A Benchmark for Interactive and Executable Geometry Construction from Natural Language by Jinwoong Kim, Rui Yang, Huishuai Zhang (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-13167.txt.
Introduction
Kim, Yang and Zhang introduce GeoBuildBench, a benchmark testing whether language models and multimodal agents can turn informal plane geometry problems into executable constructions. Given a textual problem, an agent must write a domain-specific language program producing a diagram meeting objects and verifiable constraints. The set holds 489 Chinese textbook problems, filtered by machine and validated by humans to ensure complete, constructible specs. Tested models succeed often yet frequently hallucinate structure, omit objects and miss constraints, with limited use of visual and constraint feedback for self-correction.
Catalog record
| CREATOR | Jinwoong Kim, Rui Yang, Huishuai Zhang |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 62055 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.13167
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
GeoBuildBench: A Benchmark for Interactive and Executable Geometry Construction from Natural Language Jinwoong Kim, Rui Yang, Huishuai Zhang Abstract We introduce GeoBuildBench , a benchmark designed to evaluate whether large language models and multimodal agents can ground informal natural-language plane geometry problems into executable geometric constructions. Unlike existing geometry benchmarks that focus on answer correctness or static diagram interpretation, GeoBuildBench treats geometry diagram as an interactive construction task: given a textual problem, an agent must generate a domain-specific language (DSL) program to produce a diagram satisfying explicitly specified geometric objects and verifiable constraints. The benchmark features 489 Chinese textbook-style problems, curated through automated filtering and human validation to ensure text-complete, constructible problem specifications. We evaluate several state-of-the-art multimodal models in a bounded iterative setting and show that, despite reasonable success rates, models frequently exhibit structural hallucinations, missing objects, and failures to satisfy geometric constraints, with limited ability to exploit visual and constraint-based feedback for self-correction. These results highlight geometry construction as a rigorous testbed for grounded, executable reasoning beyond textual or visual plausibility. Our benchmark and code are released at https://github.com/ooongs/GeoBuildBench . 1 Introduction Large language models (LLMs) and multimodal LLMs (MLLMs) have shown strong performance on geometry-focused benchmarks such as GeoQA and Geometry3K, which evaluate multimodal reasoning over textbook-style geometry problems and diagrams 2 ; 8 . In parallel, neuro-symbolic systems such as AlphaGeometry achieve near–Olympiad-level performance by combining language models with formal deduction engines 13 . Despite this progress, existing approaches largely assume formalized representations or given diagrams, leaving open a fundamental question: can these models transform informal, natural-language descriptions into consistent, executable geometric structures? Geometry provides a uniquely stringent testbed for grounded reasoning. Textbook problems describe points, lines, and geometric relations in natural language, implicitly requiring the construction of a consistent diagram before symbolic reasoning. Errors in this process are rarely ambiguous: incorrect or missing constructions typically lead to impossible or visibly invalid diagrams, making geometry well suited for studying structural hallucination. Prior work has often treated diagram construction as a one-shot preprocessing step, using relation-extraction to map text into solver-specific representations 4 or numerical optimization 7 to produce coordinates. More recent text-to-diagram methods emphasize coordinate accuracy and formal constraints 6 ; 14 ; 3 , but their evaluations rely on visual similarity or parser-dependent checks, and do not assess whether agents can incrementally build, inspect, and repair geometric structures under explicit semantic constraints. Although agentic frameworks like ReAct and Reflexion 15 ; 12 have shown promise in interleaving reasoning with actions, they have primarily been tested in textual environments. Geometry differs fundamentally: every action must satisfy strict spatial constraints, and failures, such as intersecting parallel lines or failing a tangency requirement, are immediately observable through the environment. To address this gap, we introduce GeoBuildBench , a benchmark and interactive environment for evaluating agents that translate natural-language plane geometry problems into executable geometric constructions. We define a compact, execution-oriented geometry Domain Specific Language (DSL), together with a Python interpreter and Matplotlib renderer. Instead of evaluating against a single “gold” diagram, for each problem GeoBuildBench specifies a set of required objects and explicit verifiable geometric constraints. Agents iteratively generate DSL programs, execute them, and use visual and constraint-based feedback to correct errors, forming a closed agent–environment loop. Figure 1 illustrates this agent–environment loop, including the geometry DSL action space, execution and rendering, and constraint-based verification. This setting enables us to evaluate models beyond textual correctness or visual plausibility. We measure (i) executability of constructions, (ii) object coverage , (iii) geometric constraint satisfaction , e.g., parallelism and tangency, and (iv) structural hallucinations , such as infeasible constructions or references to undefined objects. Our contributions are threefold: 1. An Interative Environment: An executable geometry construction environment with a minimal DSL and deterministic rendering for grounded agent interaction. 2. The GeoBuildBench Dataset: A benchmark of 489 Chinese textbook-style geometry problems, annotated with constraints to support evaluation without assuming a single gold construction. 3. Empirical Analysis of MLLMs: An evaluation of state-of-the-art models revealing that current MLLMs struggle with structural correctness and iterative self-repair despite their strong performance on existing geometry QA benchmarks. Figure 1: Overview of the GeoBuildBench environment. An agent translates a natural-language geometry problem into executable DSL programs, executes them to obtain rendered diagrams, and verifies the result against required object coverage, construction validity, and geometric constraints. Violation feedback is returned to the agent for iterative repair, while the interaction terminates once all required objects are present and all constraints are satisfied. 2 Related Work 2.1 Geometry Problem Solving Benchmarks and Formal Representations Geometry problem solving has long been studied as a multimodal reasoning task over natural-language descriptions, diagrams, and geometric rules. Early systems such as GEOS combine text parsing and diagram interpretation to answer standardized geometry questions 11 . More recent benchmarks significantly scale up supervision and emphasize structured intermediate representations. GeoQA introduces a large collection of geometry problems paired with executable programs, enabling interpretable multimodal reasoning 2 . Geometry3K further provides densely annotated geometry problems and supports symbolic reasoning through the Inter-GPS framework, which translates textual and diagrammatic inputs into a formal geometry language for theorem-guided inference 8 . In parallel, prior work has explored formalizing natural-language geometry problems by extracting relations and mapping them into solver-compatible representations 4 . While these datasets and systems advance geometry question answering and symbolic reasoning, they primarily focus on predicting answers or formal programs, and do not directly evaluate whether a model can construct a geometrically consistent diagram from natural language under strict executability and constraint satisfaction. 2.2 Text-to-Diagram Generation and Constraint-Based Geometry Construction A complementary line of work addresses diagram construction from textual descriptions. Geometry Model Builder (GMB) and its domain-specific language GMBL represent constructions and constraints explicitly, generating diagrams via numerical optimization over geometric constraints 7 . Hu and Zhong propose a training-free text-to-diagram method that optimizes point coordinates to produce accurate textbook-style diagrams 6 . More recently, MagicGeo and MagicGeoBench combine LLM-based text formalization with a formal geometry solver to generate constraint-consistent diagrams 14 . However, their evaluation relies primarily on CLIP-based visual similarity, which measures visual-textual alignment but does not verify whether geometric relations such as parallelism, perpendicularity, or incidence are semantically satisfied. GeoUni explores a unified neural model for diagram generation and geometric reasoning 3 , but its evaluation pipeline depends on a geometry parser with limited relational coverage, making it difficult to detect violations of unrecognized constraints. In contrast to these approaches, our work focuses on evaluating geometry construction as an executable, constraint-checked process, and on analyzing whether general-purpose LLM-based agents can iteratively build, inspect, and repair geometric structures from natural language. 3 GeoBuildBench Environment 3.1 Task Definition GeoBuildBench evaluates an agent’s ability to ground a natural-language plane geometry problem into an executable geometric construction. Each task consists of three components: (i) a natural-language problem description, (ii) a set of required geometric objects that must appear in the final construction, and (iii) a list of verification conditions encoding the geometric semantics of the problem. The agent’s goal is to generate a construction program that, when executed, produces a geometric configuration satisfying all required objects and verification conditions. Importantly, GeoBuildBench does not assume a single gold construction or canonical diagram. Multiple constructions are considered valid