Large-Scale Diverse Synthesis for Mid-Training
Xuemiao Zhang, Chengying Tu, Can Ren, Rongxiang Weng, Hongfei Yan, Jingang Wang, Xunliang Cai · 2025
Read Large-Scale Diverse Synthesis for Mid-Training by Xuemiao Zhang, Chengying Tu, Can Ren, Rongxiang Weng, Hongfei Yan, Jingang Wang, Xunliang Cai (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-01326.txt.
Introduction
Xuemiao Zhang and co-authors present a 2025 method for making large scale training data for language models. Good, knowledge rich data are scarce, and past synthetic sets tied to one source lacked scale and range, mainly across fields. The team set out to build a far larger and more varied question and answer set.
Their pipeline, BoostQA, holds 100 billion tokens of such data. It gathers seed data from many sources, uses a strong model to build science focused questions at many school levels, and refines answers with a second model. Used in a mid training stage for an 8 billion parameter model, the data raised scores on broad knowledge tests by a wide margin. Gains held as model and data size grew.
Catalog record
| CREATOR | Xuemiao Zhang, Chengying Tu, Can Ren, Rongxiang Weng, Hongfei Yan, Jingang Wang, Xunliang Cai |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 65 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2508-01326
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Large-Scale Diverse Synthesis for Mid-Training Xuemiao Zhang, Chengying Tu, Can Ren, Rongxiang Weng, Hongfei Yan, Jingang Wang, Xunliang Cai Abstract The scarcity of high-quality, knowledge-intensive training data hinders the development of LLMs ( LLMs ), as traditional corpora provide limited information. Previous studies have synthesized and integrated corpora-dependent QA ( QA ) data to improve model performance but face challenges in QA data scalability and knowledge diversity, particularly in cross-domain contexts. Furthermore, leveraging our designed discipline and difficulty annotation system, we probe model deficiencies in STEM disciplines and high-difficulty data. To overcome these limitations, we propose a novel diversified pipeline to synthesize BoostQA, a 100B-token large-scale QA dataset. Our synthesis framework: (1) curates seed data from heterogeneous sources; (2) utilizes DeepSeek-R1 to implement STEM-focused multi-grade synthesis to boost data diversity and high-difficulty synthesis to mitigate difficulty degradation; (3) refines answers via DeepSeek-V3 to improve output quality. We utilize BoostQA in mid-training, a mid-stage between pre-training and post-training, to optimize domain-specific knowledge acquisition and enhance data quality. Our method enables Llama-3 8B, mid-trained on a 40B-token dataset, to achieve an average improvement of 12.74% on MMLU and CMMLU and establish SOTA average performance across 12 benchmarks. BoostQA also demonstrates robust scalability, with performance consistently improving as model size, data volume, and initial FLOPs scale. 1 1 1 The core code and dataset will be made available at link. 1 Introduction Mid-training has emerged as a pivotal phase in the development of LLMs , strategically positioned between pre-training and post-training stages ( Wang et al. 2025 ) . During this intermediate process, high-quality datasets prove crucial for enhancing the domain-specific capabilities of LLMs ( Dubey et al. 2024 ; OLMo et al. 2025 ) . Prior research has demonstrated that integrating QA data into corpora significantly improves model performance ( Parmar et al. 2024 ; Chen et al. 2025 ; Wang et al. 2025 ) , which is attributable to the high knowledge density and logical organization inherent in the structured QA format. As naturally collected QA data remains insufficient to meet training demands, synthetic methods have become pivotal and effective for QA data generation ( Maini et al. 2024 ; Cheng et al. 2024 ; Jiang et al. 2025 ; Su et al. 2025 ; Akter et al. 2025 ; Zhou et al. 2025 ) . Figure 1: Comparison between BoostQA and baselines. However, existing QA data synthesis methods exhibit critical limitations. Most pipelines operate by leveraging LLMs to rephrase or generate QA pairs from corpora, appending them as supplementary material ( Maini et al. 2024 ; Cheng et al. 2024 ; Su et al. 2025 ) . These strategies synthesize QA pairs tightly bound to source passages, which merely extend rather than enrich the original corpora, with text dependency severely constraining QA data reusability. Furthermore, while some open-source synthetic QA datasets are available, most focus on narrow domains, particularly mathematics ( Zhou et al. 2025 ; Akter et al. 2025 ) , and lack both diversity and scale. Consequently, large-scale cross-domain QA datasets remain scarce, limiting model generalization across diverse fields and high-difficulty scenarios that require deep knowledge mastery and reasoning. Indeed, our probe experiments, as shown in Figure 3 , demonstrate that the model exhibits differential mastery across QA tasks of varying disciplines and difficulty levels, e.g., achieving 33.78% accuracy in the highest difficulty level (H5) and 60.74% in the lowest difficulty level (H1) at the 10T-token checkpoint. This observation motivates a differentiated enhancement strategy, specifically prioritizing the synthesis of STEM disciplines and high-difficulty data. Building on these insights, we propose a novel pipeline leveraging DeepSeek-R1 ( DeepSeek-AI et al. 2025 ) to synthesize diverse, STEM-focused, and difficulty-enhanced QA data. We first curate seed data from heterogeneous sources to ensure source diversity. Second, based on our probe results, we design a two-way synthesis pipeline leveraging DeepSeek-R1, comprising a multi-grade synthesizer that generates diverse STEM-focused QA pairs across educational stages and a high-difficulty synthesizer that enhances the synthetic proportion of H4/H5 samples. Finally, we design an answer refinement module powered by DeepSeek-V3 ( DeepSeek-AI et al. 2024 ) to improve output quality. By integrating diversified synthesis factors and high-quality signals from DeepSeek-R1, the pipeline yields BoostQA, a 100B-token large-scale synthetic QA dataset independently of seeds, improving diversity while shifting difficulty distributions upward. We conduct extensive experiments by mid-training Llama-3 8B ( Dubey et al. 2024 ) on 40B-token data. The results in Figure 1 demonstrate that BoostQA significantly improves upon the pre-training baseline with an average improvement of 12.74% on MMLU and CMMLU, establishing SOTA ( SOTA ) average performance across 12 benchmarks. Furthermore, performance gains with BoostQA monotonically increase alongside model size, data volume, and initial FLOPs scale, confirming its suitability for large-scale mid-training. In summary, our core contributions are as follows: • We identify deficiencies in model performance related to STEM disciplines and high-difficulty tasks through probe experiments. To address these deficiencies, we propose a novel diversified synthesis pipeline that incorporates diverse seed curation, two-way synthesis focused on STEM and high-difficulty data, and refinement. • We have dedicated substantial resources to developing BoostQA, a plug-and-play, 100B-token large-scale synthetic QA dataset. It specifically targets challenging STEM and high-difficulty questions, advancing the large-scale synthetic data research during mid-training. • Rigorous and comprehensive experimental results confirm that BoostQA achieves SOTA average performance. Specifically, the mid-trained Llama-3 8B boosts average performance by 12.74% on MMLU and CMMLU. 2 Method Our synthesis pipeline, illustrated in Figure 5 , is developed by addressing fundamental challenges in QA synthesis. We identify model deficiencies through probe experiments to guide the synthesis process while maintaining rigorous quality control. The seed data is sourced from QA pairs ( Li et al. 2016 ; Amini et al. 2019 ) , books ( Gao et al. 2021a ) , and high-quality web pages ( Chang et al. 2024 ; Zhou et al. 2025 ) to ensure diversity of sources. To avoid contamination, we follow previous work ( Shao et al. 2024 ) to conduct exact 10-gram matching and embedding-based similarity to filter out seeds containing questions or answers to any of our evaluation benchmarks. 2.1 Model Probes to Define Synthesis Direction Traditional synthesis methods often produce undifferentiated data, failing to address specific model deficiencies ( Chen et al. 2025 ; Su et al. 2025 ) . To overcome this limitation, we establish a dual annotation system that combines standardized discipline classification with human-calibrated difficulty scoring. We fine-tune Qwen2.5-7B-Instruct ( Qwen et al. 2025 ) distilled from DeepSeek-R1 to obtain a discipline classifier capable of classifying content into 62 distinct disciplines. In parallel, the difficulty scorer employs human performance metrics, defined as pass rates under standardized one-hour testing conditions with QS Top 100 university students majoring in relevant disciplines, to calibrate five difficulty tiers (H1-H5), as displayed in Figure 2 . Further details can be found in Appendix C . Figure 2: Difficulty levels definition. (a) Discipline probe result. (b) Difficulty probe result. Figure 3: The discipline and difficulty probe experimental results, with six typical disciplines shown. This dual annotation enables probe experiments to evaluate model mastery of QA seeds across different disciplines and difficulty levels at various initial FLOPs from the 2T-token checkpoint to the 10T-token checkpoint (setup detailed in