Nemotron-cc-math: A 133 Billion-Token-Scale High Quality Math Pretraining Dataset

Rabeeh Karimi Mahabadi, Sanjeev Satheesh, Shrimai Prabhumoye, Mostofa Patwary, Mohammad Shoeybi, Bryan Catanzaro · 2025

Read Nemotron-cc-math: A 133 Billion-Token-Scale High Quality Math Pretraining Dataset by Rabeeh Karimi Mahabadi, Sanjeev Satheesh, Shrimai Prabhumoye, Mostofa Patwary, Mohammad Shoeybi, Bryan Catanzaro (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-15096.txt.

Introduction

Rabeeh Karimi Mahabadi and co-authors build a huge clean maths corpus for model pre-train in a 2025 paper. Rich maths and code data lift the reason skill of big chat models in past work. Past maths sets drawn from web crawls lose form to crude pulls and flat text shifts.

Their pipe pulls maths in a few mark forms by a page-shape read with a text tool. A chat-led clean stage then scrubs the takes for the corpus at hand. The set runs to a vast count of clean maths bits.

Catalog record

CREATORRabeeh Karimi Mahabadi, Sanjeev Satheesh, Shrimai Prabhumoye, Mostofa Patwary, Mohammad Shoeybi, Bryan Catanzaro
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE119 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2508-15096

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Nemotron-cc-math: A 133 Billion-Token-Scale High Quality Math Pretraining Dataset Rabeeh Karimi Mahabadi, Sanjeev Satheesh, Shrimai Prabhumoye, Mostofa Patwary, Mohammad Shoeybi, Bryan Catanzaro Abstract Pretraining large language models (LLMs) on high-quality, structured data such as mathematics and code substantially enhances reasoning capabilities. However, existing math-focused datasets built from Common Crawl suffer from degraded quality due to brittle extraction heuristics, lossy HTML-to-text conversion, and the failure to reliably preserve mathematical structure. In this work, we introduce Nemotron-CC-Math , a large-scale, high-quality mathematical corpus constructed from Common Crawl using a novel, domain-agnostic pipeline specifically designed for robust scientific text extraction. Unlike previous efforts, our pipeline recovers math across various formats (e.g., MathJax, KaTeX, MathML) by leveraging layout-aware rendering with lynx and a targeted LLM-based cleaning stage. This approach preserves the structural integrity of equations and code blocks while removing boilerplate, standardizing notation into L a T e X representation, and correcting inconsistencies. We collected a large, high-quality math corpus, namely Nemotron-CC-Math-3+ (133B tokens) and Nemotron-CC-Math-4+ (52B tokens). Notably, Nemotron-CC-Math-4+ not only surpasses all prior open math datasets—including MegaMath, FineMath, and OpenWebMath—but also contains 5.5× more tokens than FineMath-4+, which was previously the highest-quality math pretraining dataset. When used to pretrain a Nemotron-T 8B model, our corpus yields +4.8 to +12.6 gains on MATH and +4.6 to +14.3 gains on MBPP+ over strong baselines, while also improving general-domain performance on MMLU and MMLU-Stem. We present the first pipeline to reliably extract scientific content—including math—from noisy web-scale data, yielding measurable gains in math, code, and general reasoning, and setting a new state of the art among open math pretraining corpora. To support open-source efforts, we release our code 1 1 1 Code will be released as a part of Nemo-curator: https://github.com/NVIDIA-NeMo/Curator and datasets 2 2 2 https://huggingface.co/datasets/nvidia/Nemotron-CC-Math-v1 . . 1 Introduction The rapid advancement of large language models (LLMs) has sparked a growing interest in improving their mathematical reasoning capabilities. Recent studies indicate that pretraining on carefully curated domain-specialized data—such as mathematics ( Paster et al. 2024 ; Han et al. 2024 ; Wang et al. 2024b ; Azerbayev et al. 2024 ) and code ( Kocetkov et al. 2022 ; Lozhkov et al. 2024 ; Li et al. 2023 ) —substantially improves domain-specific accuracy, general knowledge and reasoning abilities ( Muennighoff et al. 2023 ; Aryabumi et al. 2024 ; Lewkowycz et al. 2022 ; Shao et al. 2024 ) . This suggests that high-quality mathematical data plays a pivotal role not only in solving math problems but also in strengthening broader reasoning skills. Math capabilities of models like O1 ( OpenAI 2024 ) and DeepSeek-R1 ( Guo et al. 2025 ) critically depend on access to large-scale, high-quality mathematical pretraining data. Unfortunately, datasets used in pretraining SOTA models like DeepSeekMath ( Shao et al. 2024 ) , Minerva ( Lewkowycz et al. 2022 ) and Qwen-2.5-Math ( Yang et al. 2024 ) are not publicly released. Meanwhile, open-source alternatives such as OpenWebMath (OWM) ( Paster et al. 2024 ) , FineMath ( Allal et al. 2025 ) , InfiMMWebMath ( Han et al. 2024 ) and MathPile ( Wang et al. 2024b ) remain limited in both scale and fidelity—largely due to brittle extraction pipelines that degrade content quality and fail to preserve mathematical equations and structure (see Appendix A.2 ). While Common Crawl forms a primary source for large-scale pretraining ( Penedo et al. 2023 ; Tang et al. 2024 ; Su et al. 2025 ) , its value for mathematical pretraining remains underexploited. Existing math-specific extraction pipelines ( Paster et al. 2024 ; Zhou et al. 2025 ) are not well-suited to fully leverage this resource. In particular, current methods frequently fail to detect or accurately extract equations, either omitting them altogether or corrupting their structure ( Han et al. 2024 ; Allal et al. 2025 ) . This severely compromises content fidelity. Mathematical notation on the web appears in a wide range of formats—including MathML, L a T e X , and dynamically rendered scripts—whose representations continue to evolve over time (see Figure 2 ). Compounding this challenge, HTML pages in Common Crawl often lack associated stylesheets and JavaScript resources, preventing proper rendering and further obstructing reliable equation recovery. These limitations collectively hinder the construction of high-quality mathematical pretraining corpora that capture the breadth and variety of real-world mathematical content. To bridge this gap, we propose a modular, scalable, and domain-agnostic framework for reliably extracting mathematically rich content from raw web data, enabling the construction of a large-scale, diverse, and high-fidelity math corpus. Our multi-stage extraction and filtering pipeline ensures quality at scale (see Figure 1 ). In the first stage, HTML documents are rendered into text using the Lynx text-based browser 3 3 3 https://lynx.invisible-island.net/ , which preserves mathematical equations and symbols with high accuracy. In the second stage, a lightweight LLM normalizes heterogeneous math representations into L a T e X while discarding boilerplate and irrelevant content. This LLM-based approach allows us to avoid the brittle, heuristic-based rules employed in previous pipelines ( Paster et al. 2024 ) , resulting in more reliable and consistent extraction of mathematical content. Subsequently, we apply a quality classifier to retain the high-quality pages, followed by deduplication to eliminate redundancy. In addition, we perform thorough contamination detection against downstream benchmarks (see § 2.4 ), ensuring that any overlapping or duplicated samples are identified and removed from the corpus. Figure 1: Overview of the Nemotron-CC-Math construction pipeline. Starting from Common Crawl snapshots, we extract math-related URLs using curated datasets (e.g., MegaMath, FineMath). After fetching 229.54M webpages, we render pages through Lynx and apply LLM-based cleaning, quality filtering, and deduplication (see § 2.1 ). We visualize the topic distribution of our data (Right). By leveraging the scale of Common Crawl and the rigor of our pipeline, we present Nemotron-CC-Math —the highest quality open-source math corpus to date, comprising of 133B tokens, where its highest quality subset (Nemotron-CC-Math-4+ ) totals 53B tokens. Our pipeline is optimized for performance using Polars and Ray, enabling us to process terabytes of HTML content efficiently. To facilitate future research, we release both the dataset and our full pipeline implementation. Our contributions are as follows: • We reviewed prior data extraction pipelines, and show that they fail to accurately extract math and code content, often stripping math equations and code snippets (Appendix A.2 ). • We introduce a scalable and modular pipeline for extracting high-quality mathematical content from Common Crawl, explicitly addressing the longstanding challenge of HTML math variability—including LaTeX, MathML, Unicode, and inline or malformed equations. • To our knowledge, this is the first work to employ the text-based browser Lynx for HTML-to-plain-text conversion with preservation of math and code formatting, and to introduce LLM-based standardization of mathematical representations across the content. • We release Nemotron-CC-Math , a dataset of 133B tokens of high-quality and diverse math-rich web documents extracted from Common Crawl, whose 4+ subset contains 5.5× more tokens than FineMath-4+, the previous best math pretraining dataset. • We open-source our full pipeline (extraction, processing and scoring) to ensure reproducibility and support its application to other domains. • We thoroughly analyze Nemotron-CC-Math by examining its composition, including statistics on webpage types, subject areas, and the most frequent source

Read the complete text