What Really Improves Mathematical Reasoning: Structured Reasoning Signals Beyond Pure Code
Yuze Zhao, Junpeng Fang, Lu Yu, Zhenya Huang, Kai Zhang, Qing Cui, Qi Liu, JUN ZHOU, Enhong Chen · 2026
Read What Really Improves Mathematical Reasoning: Structured Reasoning Signals Beyond Pure Code by Yuze Zhao, Junpeng Fang, Lu Yu, Zhenya Huang, Kai Zhang, Qing Cui, Qi Liu, JUN ZHOU, Enhong Chen (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-19762.txt.
Introduction
Zhao and colleagues ask what actually lifts mathematical reasoning in large language models: raw code, or structured reasoning traces. Training experiments on a huge corpus show standalone code boosts programming but crowds out knowledge-heavy maths, rather than sharpening general reasoning. Gains blamed on code instead track mixed code-text and maths-text traces. Packing more structured maths samples into a fixed maths budget improves hard maths scores while largely preserving coding skill, and routing patterns mirror those trade-offs.
Catalog record
| CREATOR | Yuze Zhao, Junpeng Fang, Lu Yu, Zhenya Huang, Kai Zhang, Qing Cui, Qi Liu, JUN ZHOU, Enhong Chen |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 68156 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.19762
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
What Really Improves Mathematical Reasoning: Structured Reasoning Signals Beyond Pure Code Yuze Zhao, Junpeng Fang, Lu Yu, Zhenya Huang, Kai Zhang, Qing Cui, Qi Liu, JUN ZHOU, Enhong Chen Abstract Code has become a standard component of modern foundation language model (LM) training, yet its role beyond programming remains unclear. We revisit the claim that code improves reasoning through controlled pretraining experiments on a 10T-token corpus with fine-grained domain separation. Our findings are threefold. First, when code is restricted to standalone executable programs and Code-NL data are controlled for, code substantially improves programming ability but does not act as a general reasoning enhancer; instead, it competes with knowledge-intensive tasks, especially complex mathematical reasoning. Second, the reasoning gains often attributed to code are better explained by cross-domain structured reasoning traces, such as code-text and math-text mixtures, rather than by executable code alone. Third, increasing the density of structured math-domain samples within a fixed math budget yields substantial gains on difficult mathematical reasoning while largely preserving programming performance, suggesting that cognitive scaffolds offer a targeted way to mitigate cross-domain trade-offs. Finally, routing analyses show that data-composition effects are reflected in expert-activation patterns, providing mechanism-level evidence for competitive and synergistic interactions across domains. Our results clarify which data characteristics transfer across capability dimensions and point to more precise data-centric optimization strategies. 1 Introduction Figure 1 : The impact of three data compositions on model performance across capability dimensions. Starting from the 10T-token corpus, we ablate either the code corpus (w/o code) or the math corpus (w/o math), and then evaluate the resulting models along five dimensions: general knowledge, coding ability, mathematical ability, comprehensive reasoning, and professional knowledge. The results suggest clear trade-offs under a fixed-token training budget: code data competes with knowledge-intensive tasks, especially mathematical reasoning, whereas math data competes with tasks requiring comprehensive cross-domain reasoning. In general-purpose large language models (LLMs), code typically accounts for approximately 10%-30% of the pretraining corpus, making it a central component of modern training pipelines. Since the success of Codex ( 6 ) , which demonstrated the effectiveness of fine-tuning language models on code, and InstructGPT ( 29 ) , which showed that even a modest proportion of code data can substantially improve general-purpose performance, code has become a standard ingredient in LLM development. As LLMs have rapidly evolved, training on code has been associated with improvements in agentic behavior ( 27 ; 18 ; 5 ) , code reasoning ( 7 ; 12 ; 45 ) , software-engineering ( 42 ; 37 ) , and tool use ( 35 ; 15 ; 40 ) . Collectively, these advances underscore the influential role of code corpora in shaping contemporary LLM capabilities. The benefits of incorporating code into training data are multifaceted. First, exposure to code substantially improves performance on programming-related tasks ( 43 ; 44 ) . Second, compared to heterogeneous web text, code is typically more structured, less noisy, and of higher average quality, often resulting in lower training loss at convergence (Figure 7 ). Third, the explicit syntactic and structural constraints inherent in code can encourage models to produce outputs that are more logically organized and syntactically well-formed ( 28 ; 8 ) . Beyond these well-established advantages for programming tasks, a growing line of work has investigated whether code data also benefits non-coding abilities, particularly reasoning. For example, 24 studied the role of code in both pretraining and supervised instruction tuning, finding that mixed code-text pretraining improves reasoning performance across logical, scientific, analogical, and legal domains without evident negative transfer. Similarly, 2 examined the effects of code quality and corpus composition on natural-language reasoning, reporting that allocating roughly 25% of the training corpus to code yields the best overall performance. They further showed that high-quality synthetic code is more effective than large quantities of web-scraped code. Despite differences in experimental setup, these studies converge on a seemingly counterintuitive conclusion: incorporating code data into the training corpus enhances the reasoning abilities of general-purpose LLMs. In this work, we revisit and critically reassess the claim that code enhances reasoning ability. We conduct large-scale, tightly controlled experiments by constructing a high-quality 10T-token corpus and pretraining Mixture-of-Experts (MoE) models of varying sizes from scratch. On this basis, we perform two systematic ablation studies: (i) disentangling algorithmic code, or pure code, from application-oriented code to isolate the contribution of the former, and (ii) ablating the mathematics portion of the corpus. Our results show that domain corpora do not contribute additively to model capabilities; instead, they exhibit pronounced competitive interactions. Specifically, pure code primarily competes with knowledge-intensive tasks, whereas math data mainly competes with comprehensive reasoning tasks. Incorporating pure code is associated with an overall performance drop of 14.38% on mathematical benchmarks. Conversely, including math data substantially improves performance on competitive programming tasks (up to 37.11%) but reduces code reasoning performance by as much as 17.30%. We attribute the discrepancy between our findings and prior work in part to differences in corpus definition and granularity. Unlike previous studies that treat code as a monolithic category, our corpus construction enforces stricter standards for quality control and classification precision, explicitly distinguishing pure code from cross-domain code-text mixtures. In our setting, code is defined strictly as executable functions or program segments, excluding explanatory text, comments, and instructional descriptions. This distinction allows us to disentangle programming ability from cross-domain reasoning signals at the data level, thereby revealing their divergent effects on model capabilities. Under this formulation, we find that pure code alone does not directly enhance reasoning performance; instead, the gains reported in prior work may be partly attributable to mixed-reasoning data that integrate code, natural language, and mathematical structure. Building on this insight, we further investigate whether a more targeted form of cognitive scaffolding can improve data efficiency. We identify a class of mathematically grounded samples that are highly structured and explicitly support multi-step reasoning. These samples improve complex reasoning performance without substantially degrading programming ability. By increasing their training proportion while holding the overall mathematical token budget constant, we observe significant gains on challenging mathematical reasoning benchmarks, while performance on code benchmarks remains largely unchanged, with a negligible drop of approximately 1%. These results suggest that structured reasoning signals can serve as a useful scaffold for high-difficulty reasoning tasks. Finally, we analyze expert-routing patterns associated with cooperative and adversarial interactions among domain-specific data sources. By examining expert-routing distributions in the MoE model under different corpus configurations, together with hook-based analyses, we find that the proposed cognitive scaffolding improves complex reasoning while maintaining stable expert allocation. This evidence is consistent with the role of structured reasoning data as a cross-domain signal that enhances reasoning performance without destabilizing other competencies. 2 Related Work 2.1 Data Influence on LLM Reasoning Data constitute the fundamental source of model capabilities ( 34 ; 33 ) . Understanding how data from different domains, particularly code and mathematics, interact, cooperate, or compete within LLMs has emerged as a central theme in recent research. 24 investigated the impact of introducing code corpora at different training stages and concluded that pretraining on mixed code and text substantially enhances reasoning abilities, including logical, scientific, analogical, and legal reasoning, with little evidence of negative transfer. Similarly, 2 demonstrated that incorporating high-quality synthetic code