Nemotron-Math: Efficient Long-Context Distillation of Mathematical Reasoning from Multi-Mode Supervision

Wei Du, Shubham Toshniwal, Branislav Kisacanin, Sadegh Mahdavi, Ivan Moshkov, George Armstrong, Stephen Ge, Edgar Minasyan, Feng Chen, Igor Gitman · 2025

Read Nemotron-Math: Efficient Long-Context Distillation of Mathematical Reasoning from Multi-Mode Supervision by Wei Du, Shubham Toshniwal, Branislav Kisacanin, Sadegh Mahdavi, Ivan Moshkov, George Armstrong, Stephen Ge, Edgar Minasyan, Feng Chen, Igor Gitman (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2512-15489.txt.

Introduction

Wei Du and co authors release Nemotron-Math, a large data set for teaching models to reason through math, in a 2025 paper. Good training data for math needs varied reasoning styles, long solution traces, and tool use, and older sets offer these only in part. The team builds on the multi mode output of the gpt-oss-120b model.

The set holds 7.5 million solution traces across high, medium, and low reasoning modes, each with and without Python tool use. Sources mix 85 thousand curated AoPS problems with 262 thousand StackExchange Math problems. Models trained on it beat the prior OpenMathReasoning set, and a bucketed training strategy speeds long context fine tuning by two to three times.

Catalog record

CREATORWei Du, Shubham Toshniwal, Branislav Kisacanin, Sadegh Mahdavi, Ivan Moshkov, George Armstrong, Stephen Ge, Edgar Minasyan, Feng Chen, Igor Gitman
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE44 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2512-15489

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Nemotron-Math: Efficient Long-Context Distillation of Mathematical Reasoning from Multi-Mode Supervision Wei Du, Shubham Toshniwal, Branislav Kisacanin, Sadegh Mahdavi, Ivan Moshkov, George Armstrong, Stephen Ge, Edgar Minasyan, Feng Chen, Igor Gitman Abstract Abstract: High-quality mathematical reasoning supervision requires diverse reasoning styles, long-form traces, and effective tool integration, capabilities that existing datasets provide only in limited form. Leveraging the multi-mode generation ability of gpt-oss-120b , we introduce Nemotron-Math , a large-scale mathematical reasoning dataset containing 7.5M solution traces across high, medium, and low reasoning modes, each available both with and without Python tool-integrated reasoning (TIR). The dataset integrates 85K curated AoPS problems with 262K community-sourced StackExchange-Math problems, combining structured competition tasks with diverse real-world mathematical queries. We conduct controlled evaluations to assess the dataset’s quality. Nemotron-Math consistently outperforms the original OpenMathReasoning on matched AoPS problems, and incorporating StackExchange-Math substantially improves robustness and generalization, especially on HLE-Math, while preserving accuracy on math-competition benchmarks. To support efficient long-context training, we develop a sequential bucketed strategy that accelerates 128K context-length fine-tuning by 2–3× without significant accuracy loss compared to the full-length training. To verify the scalability of our supervision, we further perform experiments on Qwen3-8B and Qwen3-30B-A3B , showing that both models converge to similar final accuracy under our full context training recipe. Overall, Nemotron-Math provides diverse, high-quality, and scalable reasoning supervision, enabling state-of-the-art performance, including 100% maj@16 accuracy on AIME 2024/2025 for both Qwen3-8B and Qwen3-30B-A3B with Python TIR. 1 Introduction Mathematical problem-solving remains a key benchmark for evaluating the reasoning capabilities of large language models (LLMs). Unlike general natural language processing tasks, mathematical problems often require multi-step logical deduction, symbolic manipulation, and long-context understanding. Recent large-scale datasets, such as OpenMathInstruct-2 [ 21 ] , Skywork-MathQA [ 24 ] , and NuminaMath [ 9 ] , have substantially advanced the study of mathematical reasoning in LLMs. Building on these efforts, subsequent datasets such as BackMATH [ 25 ] and OpenMathReasoning [ 13 ] extend this progress toward competition-level and olympiad-style reasoning. Despite these advances, most existing mathematical reasoning datasets are generated by single mode reasoning models, which produce relatively uniform solution styles and limited variation in reasoning depth or tool usage. At the same time, many recent efforts have focused primarily on increasing the difficulty of competition-style mathematical problems. While highly effective for constructing challenging reasoning data, these datasets tend to focus on formal, competition-style problems that cover a relatively narrow range of mathematical domains. As a result, current mathematical reasoning datasets capture correctness and complexity but only partially reflect the broader spectrum of reasoning behaviors encountered across diverse mathematical problems. The recently released GPT-OSS family of open-weight reasoning models, with gpt-oss-120b [ 1 ] as the flagship, offers a new opportunity to address these limitations. Unlike prior models, it provides three controllable reasoning modes, high, medium, and low, that produce solutions of varying depth and length, and it can generate exceptionally detailed tool-integrated reasoning traces through extensive Python calls. These capabilities make it possible to construct datasets that capture diverse reasoning styles, self-verification behaviors, and tool-usage patterns that were previously inaccessible. Building on this opportunity, we construct Nemotron-Math 1 1 1 https://huggingface.co/datasets/nvidia/Nemotron-Math-v2 , a large-scale mathematical reasoning dataset designed to combine rich reasoning diversity with long-context supervision. On the data side, we curate a high-quality mathematical problem set by combining structured, competition-style questions from OpenMathReasoning [ 13 ] with diverse, community-driven questions from Math Stack Exchange [ 19 ] and MathOverflow [ 20 ] , which together constitute the StackExchange-derived portion (StackExchange-Math) of our dataset. Following prior work, we remove proof-style items and further filter out problems that are too easy for gpt-oss-120b : for each question, the model generates 16 solutions with low reasoning mode (8 with and 8 without Python tool-integrated reasoning (TIR)), and any problem with a high pass rate (>= 0.8) is discarded as trivial. This filtering reduces the Art of Problem Solving (AoPS) [ 2 ] portion, corresponding to the AoPS problems included in [ 13 ] , from the original 175K problems to 85K, and reduces the StackExchange-Math portion from 651K to 262K, resulting in a balanced collection of challenging and nontrivial tasks. From this filtered problem pool, we retain the multi-mode trajectories generated by gpt-oss-120b under high, medium, and low reasoning modes, each both with and without Python TIR. After removing trajectories that fail to reach the reference answer, the final Nemotron-Math corpus contains 7.5M high-quality long-form reasoning traces up to 128K tokens in length, capturing diverse reasoning depths, self-verification styles, and tool-usage behaviors. These properties make Nemotron-Math a rich resource for studying long-context and tool-augmented mathematical reasoning. To further enhance the efficiency of long-context fine-tuning on such large-scale data, we propose a sequential bucketed training strategy that groups samples by sequence length and trains the model progressively from 16K to 128K tokens. This staged approach enables optimized parallelism configurations at each length scale, significantly improving training throughput and resource utilization. Although it may introduce minor accuracy trade-offs compared to full-length joint training, the method achieves 2–3× faster training while maintaining strong overall performance. Building on these developments, our main contributions are summarized as follows: • We present Nemotron-Math , a large-scale mathematical-reasoning dataset that contains 7.5M long-form solution traces produced by gpt-oss-120b under three distinct reasoning modes (high, medium, and low), both with and without Python TIR. By integrating 85K curated AoPS problems with 262K diverse StackExchange-Math questions, the dataset provides broad domain coverage and rich variability in reasoning depth, style, and tool usage. • Through controlled comparisons, we show that Nemotron-Math provides substantially higher-quality supervision than prior datasets. Under the high reasoning mode setting without Python TIR, it improves AIME25 pass@1 for Qwen3-30B-A3B by 13.1% over the baseline. Incorporating StackExchange-Math further enhances robustness, especially on HLE-Math, without sacrificing competition performance. • We propose a sequential bucketed training strategy that progressively expands model context from short to long (128K) windows. This approach improves training throughput by 2–3× while maintaining accuracy within 1–3% of full-length joint training, making ultra-long-context fine-tuning computationally practical. • We conduct scaling studies on Qwen3-8B and Qwen3-30B-A3B , showing that both architectures benefit from Nemotron-Math and converge to similar final performance. Under the high reasoning mode with Python TIR, both models achieve 100% maj@16 accuracy on AIME24 and AIME25, demonstrating the strength and generality of the dataset. 2 Nemotron-Math Overview The Nemotron-Math dataset integrates diverse mathematical problems from multiple sources, combining problems from the OpenMathReasoning dataset [ 13 ] , which is based on the AoPS community, with additional problems collected from the Stack Exchange-math [ 19 , 20 ] . Each problem is used as a prompt to the gpt-oss-120b model, which generates multiple solution trajectories under different reasoning modes, including both with and without Python TIR. In total, this process yields approximately 7.5M reasoning traces. Detailed descriptions of the data sources, generation pipeline, quality filtering, and dataset statistics are provided in the following subsections. 2.1 Problem Set The problem set of Nemotron-Math is constructed from two complementary sources, designed to balance curated mathematical rigor with real-world diversity. 2.1.1 AoPS Source The first source is adopted from the OpenMathReasoning dataset [ 13 ] , which was constructed from problems collected on the AoPS community forum [ 2 ] . For convenience,

Read the complete text