Which Data Attributes Stimulate Math and Code Reasoning? An Investigation via Influence Functions
Siqi Kou, Qingyuan Tian, Hanwen Xu, Zihao Zeng, Zhijie Deng · 2025
Read Which Data Attributes Stimulate Math and Code Reasoning? An Investigation via Influence Functions by Siqi Kou, Qingyuan Tian, Hanwen Xu, Zihao Zeng, Zhijie Deng (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-19949.txt.
Introduction
What makes a training example teach a language model to reason? Siqi Kou and co-authors answer with influence functions. The tool traces a model's skill back to the individual examples, sequences, and tokens it trained on. Their attribution study covers chains of thought for math and coding problems. Such chains are the usual fuel for sharpening reasoning after training.
The findings cross domains in surprising ways. Hard math examples improve both math and code reasoning. Easy code tasks help coding most. Simply reweighting a dataset by flipping these difficulty preferences doubled one benchmark score and lifted another. Zooming in further, exploratory moves inside a solution help everywhere. But math leans on plain-language logic connectors while code leans on structural syntax.
Catalog record
| CREATOR | Siqi Kou, Qingyuan Tian, Hanwen Xu, Zihao Zeng, Zhijie Deng |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 36 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-19949
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Which Data Attributes Stimulate Math and Code Reasoning? An Investigation via Influence Functions Siqi Kou, Qingyuan Tian, Hanwen Xu, Zihao Zeng, Zhijie Deng Abstract Large language models (LLMs) have demonstrated remarkable reasoning capabilities in math and coding, often bolstered by post-training on the chain-of-thoughts (CoTs) generated by stronger models. However, existing strategies for curating such training data predominantly rely on heuristics, limiting generalizability and failing to capture subtleties underlying in data. To address these limitations, we leverage influence functions to systematically attribute LLMs’ reasoning ability on math and coding to individual training examples, sequences, and tokens, enabling deeper insights into effective data characteristics. Our Influence-based Reasoning Attribution (Infra) uncovers nontrivial cross-domain effects across math and coding tasks: high-difficulty math examples improve both math and code reasoning, while low-difficulty code tasks most effectively benefit code reasoning. Based on these findings, we introduce a simple yet effective dataset reweighting strategy by flipping task difficulty, which doubles AIME24 accuracy from 10% to 20% and boosts LiveCodeBench accuracy from 33.8% to 35.3% for Qwen2.5-7B-Instruct. Moreover, our fine-grained attribution reveals that the sequence-level exploratory behaviors enhance reasoning performance in both math and code, and the token-level influence patterns are distinct for math and code reasoning: the former prefers natural language logic connectors and the latter emphasizes structural syntax. 1 Introduction Large language models (LLMs) for reasoning, with OpenAI-o1 [ 15 ] and DeepSeek-R1 [ 9 ] as popular examples, have shown great promise in solving complex math and coding problems. Recently, the community has witnessed the prevalence of reproducing such reasoning capacities on open-source small- to medium-sized LLMs [ 19 , 5 , 29 ] . An initial stage of the solutions often involves post-training the model on some chain-of-thought (CoT) reasoning traces curated by leading models (e.g., R1) for diverse problems [ 34 , 23 , 24 , 37 , 13 , 25 ] . As a data-centric paradigm, the core research question here is: which attributes of the training data are effective in stimulating reasoning capabilities? Pioneering studies addressing this question predominantly adopt heuristic approaches. Typically, they first establish quantitative data quality metrics based on human expertise or empirical preferences, then selectively retain high-quality data for model training to cultivate robust reasoning capabilities with minimal data inputs. For example, s1K [ 24 ] filters 1k (question, answer) pairs with well-structured formatting, longer CoT length, and broader domain coverage from an initial pool of 59k data for training math reasoning LLMs. Similarly, LIMO [ 37 ] suggests incorporating more challenging math questions with complex reasoning chains to enable better math reasoning. Beyond focusing exclusively on math, Sky-T1 [ 25 ] targets competitive reasoning performance across both math and coding tasks. It notices that the naive incorporation of code data from APPS [ 10 ] degrades math performance and advocates mitigating this by introducing more difficult math questions and code tasks for training. Nevertheless, the underlying mechanism of such cross-domain influence remains underexplored. Furthermore, these heuristic strategies suffer from unreliable generalization to other reasoning scenarios and cannot clearly explain how some fine-grained reasoning patterns in the training data (e.g., verification, backtracking, etc.) affect the learned models. To bridge the gap, we leverage influence functions [ 17 ] —a classical technique for tracing the impacts of individual training data on model behavior—to systematically identify which training examples, along with their internal patterns and tokens, most significantly enhance the reasoning capabilities on math and coding tasks. Following previous works on influence functions for LLMs [ 8 , 30 ] , we define an easy-to-implement and cost-effective influence function for reasoning-oriented supervised fine-tuning (SFT). We further extend the instance-wise influence function to more fine-grained variants at the sequence and token levels for an in-depth data attribution. We dub our approach as Infra. We begin by investigating cross-domain influence in basic math and code reasoning scenarios without long CoT. To this end, we fine-tune LLaMA3-8B-Base [ 7 ] on a mixture of MetaMathQA [ 39 ] and OSS-Instruct [ 33 ] datasets and compute the influence function on the accuracy of GSM8k [ 3 ] and MBPP [ 10 ] . We rank all training data by their influence scores and find that, while in-domain data yield the highest scores as expected, cross-domain data also contribute nontrivially. Furthermore, aggregating these scores by category and difficulty reveals that symbolic math examples and high-difficulty math problems are particularly effective in improving code reasoning. Figure 1: An illustration of our key findings towards the question: Which attributes of training data effectively stimulate reasoning capabilities? Mixing challenging math problems with easier coding tasks leads to the highest influence scores for mathematical and coding reasoning ( left ). Guided by this insight, we curate an improved dataset and observe enhanced performance ( right ). Extending Infra to complex long CoT reasoning, we fine-tune Qwen2.5-7B-Instruct [ 36 ] on Bespoke-Stratos-17k 1 1 1 https://huggingface.co/datasets/bespokelabs/Bespoke-Stratos-17k dataset and measure influence using AIME, MATH500 [ 11 ] , and LiveCodeBench [ 16 ] benchmarks. Consistent with earlier findings, we observe cross-domain gains, with harder math problems better helping code reasoning. Going a step further, we find that both high-difficulty math and code examples are more influential on math reasoning, whereas low-difficulty code tasks contribute most significantly to code reasoning (see Figure 1 ). Motivated by these insights, we flip easy math problems as hard and hard code tasks as easy in the training data. This reweighted dataset doubles AIME accuracy and improves LiveCodeBench accuracy from 33.8% to 35.3%. Furthermore, we perform attribution at sequence and token levels in long CoT. Sequence-level attribution shows that the exploration behavior of seeking alternative approaches after reaching correctness (refer to Figure 6 ), which is common in long CoTs, improves both math and code reasoning performance. Despite being seen as overthinking [ 2 , 32 ] , our studies suggest it is advantageous. Besides, we observe distinct token-level influence patterns for math and code reasoning. In math, the most influential tokens are natural language with logical connectors, whereas code CoTs rely more on syntax markers. This divergence explains why easier code problems with clearer structural solutions benefit code reasoning when combined with math CoT that already provides logical skills. 2 Related Work LLM reasoning. Reasoning is a cognitive process that involves using evidence, arguments, and logic to arrive at conclusions or make judgments. It is widely regarded as a foundational element of advanced Large Language Models (LLMs) and an essential milestone on the path toward Artificial General Intelligence (AGI) [ 12 , 28 , 29 , 15 , 35 ] . A very recent approach to achieve reasoning capacity in LLMs is through post-training, such as OpenAI-o1 [ 15 ] , and Deepseek-R1 [ 9 ] , which expose the model to large-scale curated reasoning examples after the initial pretraining phase to refine its inferential abilities [ 18 ] . These reasoning datasets predominantly fall into two categories: (1) Mathematical reasoning: In earlier work, the construction of high-quality mathematical datasets primarily relied on increasing the quantity of problems and enhancing their difficulty levels [ 20 , 40 ] . Nevertheless, LIMO dataset [ 37 ] demonstrated that complex reasoning capabilities can be elicited through surprisingly small datasets (hundreds of examples). In addition,