GeoMathCode: Understanding Interleaved Math-Code Reasoning for Geometry Problem Solving

Yingji Zhang, Yong Dai, André Freitas · 2026

Read GeoMathCode: Understanding Interleaved Math-Code Reasoning for Geometry Problem Solving by Yingji Zhang, Yong Dai, André Freitas (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-25384.txt.

Introduction

Yingji Zhang, Yong Dai, and André Freitas study how geometry problem solving mixes reasoning with code. Their GeoMathCode setup lets program-like representations serve as intermediate visual steps, including added lines or points that clarify a figure. Experiments show reasoning steps and code steps separate cleanly in the model's hidden space. Supervised fine-tuning makes the reasoning landscape more structured and informative, while layered code syntax forms its own subspaces. Those code subspaces carry more mathematical symbol information than visual representations alone, revealing how the two reasoning modes divide labor.

Catalog record

CREATORYingji Zhang, Yong Dai, André Freitas
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE43909
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.25384

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

GeoMathCode: Understanding Interleaved Math-Code Reasoning for Geometry Problem Solving Yingji Zhang, Yong Dai, André Freitas Abstract Mathematical reasoning is a hallmark of human intelligence, requiring logical deduction, symbolic manipulation, and abstract thinking. Recent multimodal large language models (MLLMs) have demonstrated strong performance on geometry problems through multi-step reasoning. To better emulate human problem-solving, intermediate steps can incorporate auxiliary visual constructions, such as additional lines or points, which improve geometric interpretation and educational clarity. In this work, we introduce the GeoMathCode, where programmatic representations serve as intermediate visual outputs. We further conduct an in-depth analysis of the underlying reasoning geometry. Experimental results show that reasoning and code generation steps can be disentangled in the latent space, while supervised fine-tuning (SFT) makes the reasoning manifold more structured and informative. Moreover, hierarchical syntactic code structures emerge as disentangled latent subspaces, and contain more mathematical symbolic information than visual representations. 1 Introduction Mathematical reasoning is widely regarded as a pinnacle of human intelligence, requiring a sophisticated interplay of logical deduction, symbolic manipulation, and abstract thinking. Recent advances in MLLMs have shown strong potential for mathematical reasoning Guo et al. 2025 . Given a geometry problem, MLLMs can generate multi-step solutions. To further mirror human reasoning, each step can incorporate auxiliary visual constructions, such as additional lines or points, to provide clearer geometric interpretations and enhance educational effectiveness Shi et al. 2025 . In this scenario, traditional approaches to interleaved multimodal mathematical reasoning, such as BAGEL-Canvas Shi et al. 2025 , often rely on pixel-based image representations for auxiliary visual generation. However, this fine-grained cross-modal generation paradigm is inherently difficult to align and verify Chen et al. 2023 and increases system complexity by introducing additional vision-specific generator Deng et al. 2025 , increasing the potential of error propagation and inconsistency across modalities. To address these limitations, we investigate interleaved math-code reasoning, where executable code serves as an intermediate reasoning representation for geometry problem solving, as code-based representations are inherently precise and readily verifiable. However, in this task, an important question remains unclear: Does programmatic diagram generation fundamentally reshape the latent geometry of multimodal reasoning, or does it merely function as an auxiliary output representation? In particular, we seek to understand how interleaved math-code reasoning trajectories evolve across layers and reasoning steps, and whether symbolic code representations capture geometric concepts differently from visual representations. To study these questions, we introduce a systematic pipeline for constructing a public multimodal math–code reasoning dataset, GeoMathCode , incorporating automated solution generation, soft rule-based evaluation, and code verification mechanisms. As illustrated in Figure 1 , given a GeoMath question, we employ the representative model Gemini 3-Flash to generate interleaved math–code reasoning solutions, while GPT-5.1 is used to produce key checkpoints for textual reasoning evaluation as well as factual mathematical rules for automatic code verification. Building upon this dataset, we conduct a systematic analysis of the latent geometry underlying math-code reasoning in MLLMs from both step-wise and stage-wise perspectives, our experiments reveal several consistent findings. Reasoning-code disentanglement. First, reasoning and code-generation steps form disentangled latent subspaces across transformer layers, suggesting functional separation between mathematical reasoning and programmatic execution. Reasoning manifold regularisation. Second, SFT regularises the latent reasoning manifold by increasing global information capacity while simultaneously reducing local geometric complexity. Hierarchical programmatic space. Third, hierarchical programmatic syntactic structures emerge as structured latent subspaces, and latent code representations better capture fine-grained mathematical symbolic concepts than visual representations, revealing a progressive abstraction process from lexical syntax to execution-level semantics. Overall, this work introduces a new dataset for multimodal math-code reasoning and provides insights into the latent geometric structure of reasoning processes, motivating future research on mechanistic interpretability in LLMs. Figure 1: Pipeline overview. We construct GeoMathCode dataset for supervised fine-tuning. 2 Related Work In this section, we review three topics, multimodal GeoMath datasets , language geometry , and code reasoning , to further illustrate the motivation underlying our work. Multimodal GeoMath Datasets. Early benchmarks such as Geometry3K Lu et al. 2021 and ScienceQA Lu et al. 2022 established foundational tasks for visual mathematical reasoning, while later datasets including MMMU Yue et al. 2023 , MathVista Lu et al. 2023 , MathVision Wang et al. 2024 , and MathVerse Zhang et al. 2024a further advanced the reasoning capabilities of multimodal models. However, most existing benchmarks focus on static problem–solution pairs and lack step-wise reasoning processes. Although MathCanvas Shi et al. 2025 introduces interleaved visual–textual reasoning, its reliance on pixel-based visualisations complicates verification and increases system complexity. In contrast, we focus on math–code reasoning, where Python-generated geometric diagrams provide structured and verifiable visual demonstrations. Language Geometry. The geometric structure of language representations has been extensively studied Li et al. 2020 ; Chang et al. 2022 ; Jiang et al. 2024 ; Li et al. 2022 ; Geva et al. 2022 ; Nanda et al. 2023 ; Trager et al. 2023 ; Merullo et al. 2023 ; Turner et al. 2023 . Prior work shows that semantic representations often exhibit approximately linear structures in latent space, a phenomenon observed across modalities including vision Huh et al. 2024 , language Trager et al. 2023 , and audio Liu et al. 2025a , and across model architectures such as Transformers Ushio et al. 2021 ; Merullo et al. 2023 ; Hernandez et al. 2023 , VAEs Zhang et al. 2024b ; Zhang et al. 2025 ; Zhang et al. 2026b ; Carvalho et al. 2023 , and diffusion models Jing et al. 2022 . Theoretical perspectives such as NTK-based reasoning control Zhang et al. 2026a and weight-space disentanglement Ortiz-Jiménez et al. 2023 further support structured linear geometry in neural representations. Recent studies have examined geometric properties of reasoning in LLMs, including step-wise trajectories Sun et al. 2026 , layer-wise inference dynamics Ma et al. 2026 , and stage-wise training effects Li et al. 2026 ; Hoogland et al. 2024 . Building on these insights, we investigate the latent geometric structure underlying interleaved math–code reasoning processes. Code Reasoning. Previous studies have shown that incorporating code during pre-training can improve agentic behaviour Chen et al. 2025 , tool use Hong et al. 2024 and reasoning Liu et al. 2024 ; Aryabumi et al. 2025 , while other work reports contradictory findings on reasoning performance Zhao et al. 2026 . Other studies further explore logical reasoning through programmatic representations Zhao et al. 2025 , and Liang et al. 2025 demonstrate that grammar-based representations help LLMs distinguish subtle code differences. This work suggests that code functions primarily as an auxiliary diagram output rather than an active reasoning modality. Our experiments show that intermediate code representations do not improve textual reasoning performance and reasoning and code steps can be disentangled in the latent space, suggesting a functional separation from textual reasoning in latent space. 3 Data Construction First, we propose an automated, verification-driven pipeline for constructing a high-quality GeoMath reasoning dataset. The pipeline is designed to ensure both the correctness and diversity of the generated samples. An overview of the overall process is illustrated in Figure 1 . To ensure comprehensive coverage of high-quality GeoMath problems, we utilise MathCanvas Shi et al. 2025 as the primary resource, which encompasses eight topics: Algebra, Analytic Geometry, Calculus, Trigonometry, Plane Geometry, Solid Geometry, Statistics, and Transformational Geometry. For

Read the complete text