On Code-Induced Reasoning in LLMs
Abdul Waheed Zhen Wu Carolyn Rosé Daphne Ippolito · 2025
Read On Code-Induced Reasoning in LLMs by Abdul Waheed Zhen Wu Carolyn Rosé Daphne Ippolito (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-21499.txt.
Introduction
Training on code seems to improve language model thinking, but which part of code helps is unclear. Abdul Waheed and 3 co-authors investigate with a data centric system in a 2025 paper. Parallel instruction data sets were built in ten programming languages. Controlled perturbations disrupted form or semantic properties of the code. Models from five families and eight scales were fine tuned on each variant.
Thousands of tests measured effects on language, maths, and code tasks. Models proved more vulnerable to form perturbations than semantic ones. That gap was sharpest on maths and code tasks. Pseudocode and flowcharts could match real code as training material. Encoding the same content with fewer tokens often retained or improved performance, a striking useful finding.
Catalog record
| CREATOR | Abdul Waheed Zhen Wu Carolyn Rosé Daphne Ippolito |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 63 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2509-21499
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
On Code-Induced Reasoning in LLMs Abdul Waheed Zhen Wu Carolyn Rosé Daphne Ippolito Abstract Code data has been shown to enhance the reasoning capabilities of large language models (LLMs), but it remains unclear which aspects of code are most responsible. We investigate this question with a systematic, data-centric framework. We construct parallel instruction datasets in ten programming languages and apply controlled perturbations that selectively disrupt structural or semantic properties of code. We then finetune LLMs from five model families and eight scales on each variant and evaluate their performance on natural language, math, and code tasks. Across 3,331 experiments, our results show that LLMs are more vulnerable to structural perturbations than semantic ones, particularly on math and code tasks. Appropriate abstractions like pseudocode and flowcharts can be as effective as code, while encoding the same information with fewer tokens without adhering to original syntax can often retain or even improve performance. Remarkably, even corrupted code with misleading signals remains competitive when surface-level regularities persist. Finally, syntactic styles also shape task-specific gains with Python favoring natural language reasoning and lower-level languages such as Java and Rust favoring math. Through our systematic framework, we aim to provide insight into how different properties of code influence reasoning and inform the design of training data for enhancing LLM reasoning capabilities. 1 Introduction There has been substantial interest in the last several years in engineering language models that can tackle challenging reasoning tasks ( Huang & Chang 2023 ) . Language reasoning tasks, such as math word problems or logic puzzles, tend to require multi-step, structured “thinking” in order to produce the correct answer. Recent work has found that training the language model on code, either during pre-training ( Fu & Khot 2022 ; Ma et al. 2023b ) or during post-training ( Zhang et al. 2024b ) , can improve its skill at reasoning tasks, even ones that are unrelated to programming. These prior works have hypothesized that the properties of code data, such as its logical consistency, compositional structure, and reduced ambiguity compared to natural language, provide effective signals that benefit reasoning. Despite the broad effectiveness of code data in training, we still lack a systematic understanding of which aspects of code drive these improvements: is it the its syntactic regularity, structural abstractions, or linguistic styles? In this work, we aim to provide such an account by systematically investigating which aspects of code serve as effective training signals. To this end, we construct parallel instruction datasets in both natural language and code, and further expand the code dataset into language-specific variants by generating responses in ten widely used programming languages. This design allows us to examine how structural differences across languages affect downstream reasoning. In addition, we introduce controlled perturbations to the code data to isolate contributing factors: (1) rule-based transformations such as whitespace removal or comment shuffling, and (2) generative transformations where GPT-4o-mini rewrites or reformats the code (e.g., with augmented comments, pseudocode, or flowcharts). We then fine-tune language models on each dataset variant, and evaluate them across natural language and general knowledge, math, as well as code understanding and generation tasks. Our contributions are: • We introduce a systematic framework to disentangle what aspects of code data improve reasoning, combining parallel instruction data construction, controlled perturbations, and large-scale evaluation across five model families and eight scales. • We design a comprehensive and controlled suite of perturbations spanning rule-based edits and generative rewritings. • We provide new insights into the role of code in reasoning to inspire guidance on leveraging its structural and linguistic properties in future training data design. 2 Related Work Code data for LLM reasoning Recent work has increasingly demonstrated that incorporating code data can substantially improve the reasoning abilities of LLMs. Prior studies show that adding code during pretraining or instruction tuning consistently improves model performance across reasoning tasks, domains, model scales and architectures ( Ma et al. 2023a ; Zhang et al. 2024a ; Yang et al. 2025b ; Aryabumi et al. 2024 ) . Several works further explore the synergy between code and reasoning and highlight how code’s structured and verifiable properties support logical decomposition and intermediate step generation ( Bi et al. ; Yang et al. 2024 ) . This effect has been observed in multilingual contexts as well, where code-augmented training improves structured reasoning in under-resourced languages ( Li et al. 2024 ) . Complementary research focuses on code’s impact for alignment and reward modeling, where pretraining with code-preference pairs or code-based intermediate steps can improve model calibration for reasoning-intensive tasks ( Yu et al. 2024 ) . The closest line of research to our work explores stress-testing LLMs with structural and semantic code perturbations ( Lam et al. 2025 ) , which shows that small corruptions can significantly reduce reasoning performance. Data impact on LLM performance The performance of LLMs are tied to the vast amounts of training data, but the quality, composition, and characteristics of this data greatly shape their abilities ( Wang et al. 2024 ; Li et al. 2023 ; Lee et al. 2022 ) . For example, extensive analyses by Longpre et al. 2024 have shown that pretraining data curation decisions for dataset age, composition, and content filtering have systematic impact on downstream performance, and that these effects persist even after fine-tuning steps. Zhang et al. 2024c demonstrate that poisoning as little as 0.1% (and even 0.001%) can produce persistent behavioral changes that survive instruction tuning and alignment. In addition, Havrilla & Iyer 2024 showed that LLMs are sensitive to global, accumulative errors in chain-of-thought-structured training data, and that it is critical to filter out documents containing large amounts of dynamic, global noise during both pretraining and fine-tuning. 3 Methodology We design a controlled experimental framework to understand what aspects of code improve reasoning in language models. Our methodology consists of three stages: constructing parallel natural language and code instruction datasets (Section 3.1 ); applying systematic modifications to code instruction data (Section 3.2 ); and fine-tuning various language models on each dataset variant and then conducting evaluation (Section 3.3 ). An overview of this framework is shown in Figure 1 . Figure 1: We construct parallel code and natural language instruction datasets, apply targeted modifications (rule-based and generative-based perturbations, single programming language ablations), and fine-tune a separate LLM on each modified dataset. We then evaluate the resulting models across general natural language, code, and math reasoning tasks. 3.1 Instruction Data Generation We construct two parallel instruction datasets: one in natural language and the other in code, each containing 120,000 instruction-response pairs. We collect instructions from publicly available datasets, carefully process and filter them through deduplication and language-agnostic filtering, and augment the code data in a controlled way. This construction enables a more controlled comparison of natural- and code-based instruction following under a unified training framework. Code instructions We aggregate code instructions from Codeforces-CoT ( Penedo et al. 2025 ) , Code-Instruction-122K ( TokenBender 2024 ) , Evol-Instruct-Code-80k-v1 ( nickrosh 2024 ) , CodeInstruction ( red1xe 2023 ) , Code-Instruct-Sets ( AtlasUnified 2023 ) , and Code-Instruct-Alpaca-Vicuna-WizardLM ( rombodawg 2024 ) . We aim to construct instruction data that is high-quality, diverse, and language-agnostic. To ensure generality and eliminate redundancy,