SampleMix: A Sample-wise Pre-training Data Mixing Strategey by Coordinating Data Quality and Diversity
Xiangyu Xi, Deyang Kong, Jian Yang, JiaWei Yang, Zhengyu Chen, Wei Wang, Jingang Wang, Xunliang Cai, Shikun Zhang, Wei Ye · 2025
Read SampleMix: A Sample-wise Pre-training Data Mixing Strategey by Coordinating Data Quality and Diversity by Xiangyu Xi, Deyang Kong, Jian Yang, JiaWei Yang, Zhengyu Chen, Wei Wang, Jingang Wang, Xunliang Cai, Shikun Zhang, Wei Ye (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-01506.txt.
Introduction
Xiangyu Xi and nine co-authors present SampleMix in 2025, a new way to build training data for large language models. The usual method picks shares for each topic area first, then samples evenly within each area. That plan misses overlap between areas and treats all samples in an area as equal.
SampleMix works from the bottom up instead. It grades each sample for quality and for how much new variety it adds, then samples across all areas at once. In tests, the method beat the usual topic-based plans on many tasks. It also needed far fewer training steps to match the older methods, which saves time and cost.
Catalog record
| CREATOR | Xiangyu Xi, Deyang Kong, Jian Yang, JiaWei Yang, Zhengyu Chen, Wei Wang, Jingang Wang, Xunliang Cai, Shikun Zhang, Wei Ye |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 48 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-01506
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
SampleMix: A Sample-wise Pre-training Data Mixing Strategey by Coordinating Data Quality and Diversity Xiangyu Xi, Deyang Kong, Jian Yang, JiaWei Yang, Zhengyu Chen, Wei Wang, Jingang Wang, Xunliang Cai, Shikun Zhang, Wei Ye Abstract Existing pretraining data mixing methods for large language models (LLMs) typically follow a domain-wise methodology, a top-down process that first determines domain weights and then performs uniform data sampling across each domain. However, these approaches neglect significant inter-domain overlaps and commonalities, failing to control the global diversity of the constructed training dataset. Further, uniform sampling within domains ignores fine-grained sample-specific features, potentially leading to suboptimal data distribution. To address these shortcomings, we propose a novel sample-wise data mixture approach based on a bottom-up paradigm. This method performs global cross-domain sampling by systematically evaluating the quality and diversity of each sample, thereby dynamically determining the optimal domain distribution. Comprehensive experiments across multiple downstream tasks and perplexity assessments demonstrate that SampleMix surpasses existing domain-based methods. Meanwhile, SampleMix requires 1.4x to 2.1x fewer training steps to achieves the baselines’ performance, highlighting the substantial potential of SampleMix to optimize pre-training data. 1 Introduction Figure 1: We conduct data clustering analysis using the SlimPajama dataset. For each domain (row), each cell shows the percentage of its clusters that also include samples from other domains (column). E.g., 76.60% of ArXiv’s clusters include Wikipedia samples (1st row, 6th column). The results reveal substantial overlap between domains. The mixture proportions of pretraining data, which greatly affect the language model performance, have received increasing attention from researchers and practitioners. In the early years, heuristic-based methods were widely employed to assign domain weights using manually devised rules, such as upsampling high-quality datasets (e.g., Wikipedia) multiple times Gao et al. (2020) ; Laurençon et al. (2022) . Afterwards, models like GLaM Du et al. (2022) and PaLM Chowdhery et al. (2023) established mixture weights based on the performance metrics of trained smaller models. More recently, learning-based methods have been proposed, involving the training of small proxy models across domains to generate optimal domain weights Fan et al. (2023) ; Xie et al. (2024) . These existing methods follow a domain-wise methodology, a top-down process that first determines the proportion of each domain and then samples uniformly from the selected domain. Despite achieving advancements, These approaches present two key issues: (1) Ignoring Inter-domain Overlaps and Commonalities. In current pretraining datasets, “domain” is primarily categorized based on data sources rather than intrinsic textual or semantic properties. An implicit assumption of the domain-wise approaches is that samples are distinct and unrelated across domain boundaries. However, in practice, samples across different domains exhibit significant shared characteristics, both in terms of raw text and high-level semantics. To examine this assumption, we analyzed the SlimPajama dataset Soboleva et al. (2023) , a quality-filtered and deduplicated dataset, focusing on relationships between samples and clusters across its six text domains (excluding GitHub). For each domain, we computed the percentage of its clusters that also included samples from other domains, as Figure 1 shows. Our findings reveal substantial overlap between domains—nearly all clusters contain samples from both CommonCrawl and C4. Furthermore, manual inspection of the clustered samples confirms that data from different domains frequently share similar topics and characteristics. For instance, Figure 7 illustrates that samples from multiple domains discuss Einstein and the Theory of Relativity. By disregarding inter-domain commonalities, domain-wise mixture methods fail to control the global diversity of training data effectively. (2) Suboptimal Sample Distribution within Domains. A second limitation arises from the uniform sampling within each domain, which can lead to a suboptimal distribution of training samples Xie et al. (2024) ; Fan et al. (2023) ; Ye et al. (2025) . Intuitively, samples with higher quality and greater diversity should have a higher probability of being selected Xie et al. (2023) ; Abbas et al. (2023) . At the same time, lower-quality samples should not be entirely discarded, as they contribute to the model’s generalization ability Sachdeva et al. (2024) . Determining an effective sampling strategy within each domain is nontrivial, yet current approaches lack fine-grained control over sample selection. To address these limitations, we propose a novel sample-wise data mixture approach with a bottom-up paradigm. Instead of defining domain proportions upfront, we first perform global sampling across the dataset based on sample quality and diversity, dynamically determining domain distributions. This allows for more precise control over the overall quality and diversity of the dataset. To implement this, we individually assess the quality and diversity of each sample and assign corresponding sampling weights based on these evaluations. Given a target token budget, we then sample each example according to its weight to construct the optimal training dataset. Also, our approach offers the additional advantage of dynamically adapting to varying token budgets, enabling the determination of optimal data proportions for each specific budget. In contrast, the vast majority of existing works rely on static data proportions, which do not adjust to different token budget constraints. The contributions of this paper are: 1. We study the problem of sample-wise pre-training data mixing, which can alleviate the limitations of overlooking inter-domain overlap and suboptimal sample distribution within domains by existing domain-wise mixing works. 2. We propose a sample-wise pre-training data mixing strategy that coordinates data quality and diversity on a per-sample basis, effectively capturing commonalities among domains and optimal sample distribution. 3. Extensive experiments on downstream tasks and perplexity evaluations demonstrate the advantages of our method. Notably, it achieves averaged baseline accuracy with 1.9x fewer training steps, highlighting its efficiency. Figure 2: (a) Traditional methods determine domain weights and construct the training dataset by uniformly sampling from each domain. (b) SampleMix employs a sample-wise mixing strategy by: evaluating sample quality and diversity, assigning appropriate weights, and constructing an optimal dataset based on these weights. Dots of the same color represent data from the same domain.. 2 Method 2.1 Problem Formulation Consider a source dataset D_{\mathrm{src}} composed of k distinct domains (e.g., CommonCrawl, Wikipedia, BookCorpus, etc.). For each domain i , let D_{i} denote the collection of documents within that domain. The entire source dataset is defined as D_{\mathrm{src}}\triangleq\{D_{1},\ldots,D_{k}\} , with T_{\mathrm{src}} representing the total number of tokens. Our objective is to construct a target training set D_{\mathrm{tgt}} for pre-training that adheres to a specific token budget T_{\mathrm{tgt}} (e.g., 100B tokens). As illustrated in Figure 2 , traditional approaches determine domain weights without explicitly considering the overall token budget, and build D_{\mathrm{tgt}} by uniform sampling from each domain based on these weights. In contrast, our proposed method, SampleMix, enhances this process by evaluating both the quality ( § 2.2 ) and diversity ( § 2.3 ) of each document. Utilizing these dual criteria, SampleMix assigns unique sampling weights to each document. To ensure compliance with the token budget T_{\mathrm{tgt}} , we then construct an optimal training dataset by sampling documents according to their assigned weights ( § 2.4 ). 2.2 Data Quality Evaluation The quality of training data is crucial for large language models. However, most existing studies typically rely on simple heuristics Xie et al. (2023) ; Li et al. (2023) ; Sachdeva et al. (2024) . Wettig et al. (2024) introduces four metrics and uses pairwise comparisons to train an evaluator model. However, these