A Practical Two-Stage Recipe for Mathematical LLMs: Maximizing Accuracy with SFT and Efficiency with Reinforcement Learning
Hiroshi Yoshihara, Taiki Yamaguchi, Yuichi Inoue · 2025
Read A Practical Two-Stage Recipe for Mathematical LLMs: Maximizing Accuracy with SFT and Efficiency with Reinforcement Learning by Hiroshi Yoshihara, Taiki Yamaguchi, Yuichi Inoue (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-08267.txt.
Introduction
Hiroshi Yoshihara, Taiki Yamaguchi, and Yuichi Inoue share in 2025 a two-stage plan for training models that solve math problems. Strong math skill is a key test for large models. Two training styles lead the field, but how to blend them for both accuracy and speed was not clear.
The plan first trains with guided examples for many rounds to push accuracy to its peak. A second stage then uses reward-based learning from live answers to cut answer length while keeping that peak. The team did well in a strict math contest with more than two thousand teams. Code will be shared so others can repeat the work.
Catalog record
| CREATOR | Hiroshi Yoshihara, Taiki Yamaguchi, Yuichi Inoue |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 25 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2507-08267
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
A Practical Two-Stage Recipe for Mathematical LLMs: Maximizing Accuracy with SFT and Efficiency with Reinforcement Learning Hiroshi Yoshihara, Taiki Yamaguchi, Yuichi Inoue Abstract Enhancing the mathematical reasoning of Large Language Models (LLMs) is a pivotal challenge in advancing AI capabilities. While Supervised Fine-Tuning (SFT) and Reinforcement Learning (RL) are the dominant training paradigms, a systematic methodology for combining them to maximize both accuracy and efficiency remains largely unexplored. This paper introduces a practical and effective training recipe that strategically integrates extended SFT with RL from online inference (GRPO). We posit that these methods play complementary, not competing, roles: a prolonged SFT phase first pushes the model’s accuracy to its limits, after which a GRPO phase dramatically improves token efficiency while preserving this peak performance. Our experiments reveal that extending SFT for as many as 10 epochs is crucial for performance breakthroughs, and that the primary role of GRPO in this framework is to optimize solution length. The efficacy of our recipe is rigorously validated through top-tier performance on challenging benchmarks, including a high rank among over 2,200 2,200 teams in the strictly leak-free AI Mathematical Olympiad (AIMO). This work provides the community with a battle-tested blueprint for developing state-of-the-art mathematical reasoners that are both exceptionally accurate and practically efficient. To ensure full reproducibility and empower future research, we will open-source our entire framework, including all code, model checkpoints, and training configurations at https://github.com/analokmaus/kaggle-aimo2-fast-math-r1 . 1 Introduction The remarkable advancements of Large Language Models (LLMs) have demonstrated their potential across a vast spectrum of applications. Beyond their well-established capabilities in natural language understanding and conversation, the frontier of AI research is increasingly focused on enhancing their reasoning abilities, which are essential to solve complex and challenging problems ( Qwen-Team 2024 ; OpenAI 2024 ; OpenAI 2025 ; DeepSeek-AI 2025 ) . Among the diverse domains for evaluating reasoning, mathematical problem-solving stands out as an ideal testbed. It demands not only factual knowledge but also a complex interplay of strategic planning, step-by-step logical inference, and self-correction, thereby providing a comprehensive benchmark for the problem-solving capabilities of current models. To enhance the mathematical reasoning of LLMs, two primary paradigms have been explored: Supervised Fine-Tuning (SFT) and Reinforcement Learning (RL). SFT, which uses high-quality step-by-step solutions datasets, has been instrumental in the bootstrapping of model capabilities ( Chen et al. 2024 ; Qwen-Team 2024 ) . However, its effectiveness is intrinsically tied to the scale and quality of the demonstration data, potentially leading to a performance plateau. On the other hand, RL offers a promising avenue for models to learn from their own generated solutions, moving beyond the confines of static datasets. Recently, Group Relative Policy Optimization (GRPO) has emerged, showing promise in improving sampling efficiency ( Shao et al. 2024 ; DeepSeek-AI 2025 ) . Yet, it remains unclear whether GRPO alone is sufficient to maximize performance, and a principled, systematic methodology for combining it with SFT is notably absent. This paper bridges this gap by proposing a practical recipe that strategically integrates SFT and GRPO to unlock new levels of mathematical reasoning in LLMs. We posit that these two methods are not competing but rather play highly complementary roles. Our core idea is that an extended SFT phase is first employed to establish a strong performance baseline, which is then refined by a GRPO phase focused on enhancing efficiency without compromising accuracy. This sequential and synergistic approach provides a clear, reproducible pathway for developing high-performing mathematical reasoners. Our investigation yields several key insights into the effective training of mathematical LLMs. First, we experimentally demonstrate that prolonged SFT is crucial for performance breakthroughs. Contrary to the common practice of short-duration fine-tuning such as cold start, we find that although initial epochs may show a temporary dip in performance, extending the SFT process for as long as 10 epochs consistently and significantly boosts the model’s problem-solving accuracy. Second, we uncover a new primary role for GRPO in this combined framework. While prior work often associates preference optimization with direct accuracy gains, our findings indicate that GRPO excels at dramatically improving token efficiency. After our intensive SFT stage, GRPO maintains or slightly improves the high accuracy achieved, while substantially shortening the length of generated solutions. Third, we establish a clear synergistic relationship: SFT is responsible for pushing the performance ceiling of the model, while GRPO is responsible for optimizing the solution generation process, making the high-performing model more practical and efficient for real-world applications. To validate the efficacy of our proposed recipe, we conduct extensive evaluations on a suite of challenging benchmarks, including AIME 2024 and AIME 2025. Crucially, we also test our model on the AI Mathematical Olympiad (AIMO) ( Frieder et al. 2024 ) , one of the most competitive benchmarks with stringent measures against data leakage. Our method achieves top-tier performance, demonstrating its practical effectiveness and robustness in a highly competitive setting. These results not only underscore the power of our combined SFT-GRPO strategy but also offer the community a valuable, battle-tested recipe for advancing the frontiers of mathematical reasoning with LLMs. To support the reproducibility of our work, we will release key artifacts at https://github.com/analokmaus/kaggle-aimo2-fast-math-r1 . The release will include our final model weights, the complete source code for the SFT and GRPO training and evaluation procedures, all curated datasets, and the full set of checkpoints from the GRPO stage. We believe these resources will enable the community to rigorously verify and build upon our proposed training methodology. Table 1: Performance and the mean token usage across model sizes for AIME 2024 and 2025. Acc. represents mean Pass@1(%). Original means Deepseek-R1-Distill-Qwen 14B. AIME 2024 AIME 2025 1.5B 7B 14B 1.5B 7B 14B Method Acc. Tokens Acc. Tokens Acc. Tokens Acc. Tokens Acc. Tokens Acc. Tokens Original 27.8 11,235 51.0 10,136 63.3 9,590 22.3 11,154 38.1 10,612 46.7 10,602 Original + RL 29.5 10,702 53.6 7,735 60.9 7,255 23.2 10,354 39.0 8,176 41.8 8,246 + SFT (10 epochs) 26.0 13,014 52.0 10,647 65.2 10,268 22.1 12,826 38.3 11,507 49.7 11,264 + SFT (10 epochs) + RL 27.3 11,767 54.7 9,577 66.0 7,932 23.2 11,480 39.8 10,445 49.2 9,066 2 Related Work LLM Reasoning. The performance of Large Language Models (LLMs) is well-known to improve with the scaling of training compute budgets, as established by scaling laws ( Kaplan et al. 2020 ; Hoffmann et al. 2022 ) . More recently, a parallel research direction has focused on increasing the computational budget at inference time, a strategy often referred to as test-time scaling. Several studies have demonstrated that allocating more compute during the inference phase—for instance, by generating a larger number of tokens or candidate solutions—can significantly enhance LLM performance on complex reasoning tasks ( Li et al. 2022 ; Lewkowycz et al. 2022 ; Brown et al. 2024 ; Wu et al. 2025 ; Misaki et al. 2025 ) . This line of work underscores the value of trading inference-time resources for higher accuracy. Post-training for Reasoning LLMs. A significant body of work has focused on enhancing the reasoning abilities of LLMs through post-training refinement ( Qwen-Team 2024 ; OpenAI 2025 ; DeepSeek-AI 2025 ; Ye et al. 2025 ; Muennighoff et al. 2025 ; Team 2025 ) . Methods such as