How Can We Synthesize High-Quality Pretraining Data? A Systematic Study of Prompt Design, Generator Model, and Source Data
Joel Niklaus 1 Atsuki Yamaguchi 2 Michal Štefánik 3 Guilherme Penedo 1 Hynek Kydlíček 1 Elie Bakouch 1 Lewis Tunstall 1 Edward Emanuel Beeching 1 Thibaud Frere 1 Colin Raffel 1 Leandro von Werra 1 Thomas Wolf 1 1 Hugging Face 2 University of Sheffield, United Kingdom 3 National Institute of Informatics, Japan Pipeline code: https://github.com/huggingface/finephrase FinePhrase dataset: https://hf.co/datasets/HuggingFaceFW/finephrase · 2026
Read How Can We Synthesize High-Quality Pretraining Data? A Systematic Study of Prompt Design, Generator Model, and Source Data by Joel Niklaus 1 Atsuki Yamaguchi 2 Michal Štefánik 3 Guilherme Penedo 1 Hynek Kydlíček 1 Elie Bakouch 1 Lewis Tunstall 1 Edward Emanuel Beeching 1 Thibaud Frere 1 Colin Raffel 1 Leandro von Werra 1 Thomas Wolf 1 1 Hugging Face 2 University of Sheffield, United Kingdom 3 National Institute of Informatics, Japan Pipeline code: https://github.com/huggingface/finephrase FinePhrase dataset: https://hf.co/datasets/HuggingFaceFW/finephrase (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-13977.txt.
Introduction
Niklaus and colleagues ask how to synthesize high-quality pretraining data, testing prompt design, generator size, and source data in controlled trials over one trillion tokens. Structured formats like tables, math problems, FAQs, and tutorials beat curated web baselines, while generators larger than 1 billion parameters add nothing. Their FinePhrase set holds 486 billion tokens of rephrased web text and cuts generation costs up to 30 times.
Catalog record
| CREATOR | Joel Niklaus 1 Atsuki Yamaguchi 2 Michal Štefánik 3 Guilherme Penedo 1 Hynek Kydlíček 1 Elie Bakouch 1 Lewis Tunstall 1 Edward Emanuel Beeching 1 Thibaud Frere 1 Colin Raffel 1 Leandro von Werra 1 Thomas Wolf 1 1 Hugging Face 2 University of Sheffield, United Kingdom 3 National Institute of Informatics, Japan Pipeline code: https://github.com/huggingface/finephrase FinePhrase dataset: https://hf.co/datasets/HuggingFaceFW/finephrase |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 71008 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.13977
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
How Can We Synthesize High-Quality Pretraining Data? A Systematic Study of Prompt Design, Generator Model, and Source Data Joel Niklaus 1 Atsuki Yamaguchi 2 Michal Štefánik 3 Guilherme Penedo 1 Hynek Kydlíček 1 Elie Bakouch 1 Lewis Tunstall 1 Edward Emanuel Beeching 1 Thibaud Frere 1 Colin Raffel 1 Leandro von Werra 1 Thomas Wolf 1 1 Hugging Face 2 University of Sheffield, United Kingdom 3 National Institute of Informatics, Japan Pipeline code: https://github.com/huggingface/finephrase FinePhrase dataset: https://hf.co/datasets/HuggingFaceFW/finephrase Abstract Synthetic data is a standard component in training large language models, yet systematic comparisons across design dimensions, including rephrasing strategy, generator model, and source data, remain absent. We conduct extensive controlled experiments, generating over one trillion tokens, to identify critical factors in rephrasing web text into synthetic pretraining data. Our results reveal that structured output formats, such as tables, math problems, FAQs, and tutorials, consistently outperform both curated web baselines and prior synthetic methods. Notably, increasing the size of the generator model beyond 1B parameters provides no additional benefit. Our analysis also demonstrates that the selection of the original data used for mixing substantially influences performance. By applying our findings, we develop FinePhrase , a 486-billion-token open dataset of rephrased web text. We show that FinePhrase outperforms all existing synthetic data baselines while reducing generation costs by up to 30 times. We provide the dataset, all prompts, and the generation framework to the research community. 1 Introduction The pretraining data landscape of large language models (LLMs) has undergone three paradigm shifts. First, models like GPT ( Radford et al. 2018 ) and BERT ( Devlin et al. 2019 ) relied on curated corpora including BookCorpus ( Zhu et al. 2015 ) and Wikipedia. Scaling initiatives like C4 ( Raffel et al. 2020 ) and The Pile ( Gao et al. 2020 ) later expanded available datasets into hundreds of gigabytes of web text. This expansion followed scaling laws, which indicate that increased pretraining duration and data volume consistently improve downstream performance ( Kaplan et al. 2020 ; Hoffmann et al. 2022 ) . Efforts such as FineWeb ( Penedo et al. 2024a ) and DCLM ( Li et al. 2024 ) further brought pretraining data to the trillion-token scale, covering most of the crawlable web. As the volume of available web text plateaus, the focus of the field has moved from quantity to quality: modern development pipelines now employ neural classifiers to filter for educational content ( Gunasekar et al. 2023 ; Penedo et al. 2023 ) and analyze data repetition extensively ( Muennighoff et al. 2023 ) . Synthetic data constitutes the latest step in this progression, appearing as a solution to the exhaustion of usable real-world data. Rather than merely filtering web text, recent work uses LLMs to rephrase text into variants that preserve information while optimizing the presentation ( Maini et al. 2024 ; Kiyomaru et al. 2026 , inter alia. ) . The community has adopted this approach at scale: Su et al. 2025 rephrased two trillion tokens for Nemotron-CC, and several recent LLMs utilized hundreds of billions of synthetic tokens during pretraining ( Abdin et al. 2024 ; NVIDIA et al. 2025 ; Yang et al. 2025a ; Singh et al. 2026 ) . However, the design space for synthetic data generation is expansive and currently lacks comprehensive assessment. Prior work has explored individual approaches in isolation. For instance, WRAP ( Maini et al. 2024 ) focuses on stylistic rewriting; Nemotron-CC ( Su et al. 2025 ) extracts QA pairs and knowledge lists; BeyondWeb ( DatologyAI et al. 2025 ) proposes continuation and summarization rephrasing; and REWIRE ( Nguyen et al. 2025 ) targets the guided transformation of low-quality documents. While each study reports gains relative to a specific baseline, no systematic comparison exists across these methods. Furthermore, the interaction between key design choices, such as the choice of generator models and types of source data, remains unexplored, leaving researchers without a clear strategy to maximize the utility of synthetic data for pretraining. To address these limitations, we conduct a controlled analysis across three axes: the rephrasing strategy , the choice of generator models , and the selection of source and mix-in data . First, we investigate rephrasing strategy by comparing established rephrasing prompts from prior work against four new pedagogical structured formats: tutorials, FAQs, tables, and math reformulations. These formats leverage the benefits of structured signals and educational content for the learning process of the language model ( Gunasekar et al. 2023 ; Galke et al. 2024 ; Cheng et al. 2024 ; Yamaguchi et al. 2026 ) . Second, to determine the influence of generator models, we test six model families with sizes ranging from 135M to 27B parameters. Finally, we evaluate how the quality of the source text and the integration of original web data (i.e., mix-in data) affect outcomes. This clarifies whether the source quality or the mix-in dataset selection serves as the primary driver of performance. Our main contributions are as follows: • We present a systematic comparison of synthetic pretraining data across the strategy of rephrasing, the generator model, and the source data (§ 4 ). • We show that the choice of rephrasing prompt is the dominant factor for downstream performance; large models (10B+) are not always necessary to perform effective rephrasing; the mix-in dataset is more critical than the source data; and output diversity is more important than formatting consistency (§ 4 & § 5 ). • We create and release FinePhrase , a 486B token dataset and associated generation framework. Using SmolLM2 1.7B ( Allal et al. 2025 ) , it outperforms all existing synthetic data baselines at up to a 30-fold reduction in generation cost relative to previous methods (§ 6 ). 2 A Systematic Framework for Pretraining Data Synthesis Figure 1: Overview of our experimental methodology. 2.1 Problem Setting To address the scaling limits of crawlable web data, we systematically evaluate methods for generating synthetic pretraining data by rephrasing web corpora. Unlike prior work evaluating isolated rephrasing methodologies ( Maini et al. 2024 ; Su et al. 2025 ; Nguyen et al. 2025 , inter alia. ) , we conduct a controlled ablation of the key components within the generation pipeline (Figure 1 ). The goal is a reproducible experimental framework for synthetic data generation that reduces reliance on intuition and replaces ad hoc practice with a rigorously designed pipeline. By isolating prompt design, model scale, and data composition, this research identifies configurations that maximize performance while minimizing compute allocation. We construct the tested pretraining configurations from a rephrasing prompt \mathcal{P} (§ 2.2 ), an instruction-tuned generator model \mathcal{G} (§ 2.3 ), and both a mix-in \mathcal{D}_{\text{mix}} and a source dataset \mathcal{D}_{\text{source}} (§ 2.4 ). We validate the effectiveness of each configuration by training an LM \mathcal{M} from scratch on a mixture of rephrased samples and \mathcal{D}_{\text{mix}} for 21B tokens. Success is defined by the aggregate performance of the trained model across evaluation benchmarks spanning factual knowledge, reading comprehension, and reasoning (§ 3 ). This framework enables the identification of the optimal generation configuration, which we employ to create FinePhrase , a synthetic dataset consisting of 486B tokens. 2.2 Rephrasing Strategy The first axis of investigation