Investigating the Impact of Data Selection Strategies on Language Model Performance
Jiayao Gu, Liting Chen, Yihong Li · 2025
Read Investigating the Impact of Data Selection Strategies on Language Model Performance by Jiayao Gu, Liting Chen, Yihong Li (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-03826.txt.
Introduction
Jiayao Gu, Liting Chen, and Yihong Li study how the choice of training data shapes language model results in this 2025 paper. When training data matches the target style or topic, models tend to do better. The team asks whether picking a subset of data helps, and which clues about the data matter most.
They compare random picks with methods that align the data to a target. Simple word-pattern clues helped match the target, and features from neural networks added further gains. Their tests show that careful selection can lift performance, and they share the code so others can repeat the work.
Catalog record
| CREATOR | Jiayao Gu, Liting Chen, Yihong Li |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 24 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2501-03826
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Investigating the Impact of Data Selection Strategies on Language Model Performance Jiayao Gu, Liting Chen, Yihong Li Abstract Data selection is critical for enhancing the performance of language models, particularly when aligning training datasets with a desired target distribution. This study explores the effects of different data selection methods and feature types on model performance. We evaluate whether selecting data subsets can influence downstream tasks, whether n-gram features improve alignment with target distributions, and whether embedding-based neural features provide complementary benefits. Through comparative experiments using baseline random selection methods and distribution aligned approaches, we provide insights into the interplay between data selection strategies and model training efficacy. All code for this study can be found on github repository . 1 Introduction Large language models have garnered significant attention recently Bubeck et al. 2023 ; Liu et al. 2024 ; Zhou et al. 2024 . Their remarkable advancements can be attributed to the utilization of vast and continuously expanding text datasets for unsupervised pre-training ( Brown et al. 2020 ) . However, indiscriminately training a model on all accessible data may not yield the best results. In our study, we study the problem of data selection ( Albalak et al. 2024 ) . We examine the scenario where there is a large, varied raw dataset, characterized by distribution q , and a smaller dataset that represents a specific desired target distribution p . We aim to select a subset of the raw dataset that closely matches p . Recent studies predominantly utilize heuristic methods to select training data. For instance, GPT-3 ( Brown et al. 2020 ) and PaLM ( Chowdhery et al. 2022 ) employ a binary classifier f(\cdot) to distinguish high-quality formal text p from noisy web data q , selecting web examples that exhibit a high predicted probability f(\boldsymbol{x}) . However, as each example is considered independently, this method does not allow for the selection of a predetermined number of examples in a single pass. Moreover, such a sampling strategy tends to favor the prominent modes of the target distribution, thereby reducing the diversity of the samples. The recent DSIR approach ( Xie et al. 2023 ) introduces the use of hashed n-gram features to approximate the target distribution \hat{p} and the raw distribution \hat{q} , and calculates the importance weights as \frac{\hat{p}(\boldsymbol{x})}{\hat{q}(\boldsymbol{x})} . These importance weights are subsequently utilized for resampling. Although this method aligns the raw distribution with the target distribution to some extent, it relies solely on n-gram features, which lack contextual awareness. In this work, we investigate the impact of data selection strategies on language model performance . We implement random selection and DSIR as the baseline methods. We also propose HIR , Hybrid Importance Resampling, where we combines statistical n-gram features and neural features to see if aligning raw datasets with a target using neural features can improve the performance of downstream tasks. Specifically, for a dataset resembling the target set, we estimate two distinct distributions: an n-gram-based distribution, denoted as \hat{p}_{\text{ngram}}(\boldsymbol{x}) , parameterized by a Multinomial Distribution, and a neural feature-based distribution, denoted as \hat{p}_{\text{nn}}(\boldsymbol{x}) , parameterized by Gaussian Mixture Models. To balance these two perspectives, we introduce a weighting factor \alpha and construct a hybrid distribution: \hat{p}_{\text{hybrid}}(\boldsymbol{x})=\hat{p}_{\text{ngram}}(\boldsymbol{x})^{\alpha}\cdot\hat{p}_{\text{nn}}(\boldsymbol{x})^{1-\alpha}, which integrates the discrete properties of n-gram features with the continuous characteristics of neural features. Using the hybrid distribution, we compute the sample importance weight as \frac{\hat{p}_{\text{hybrid}}(\boldsymbol{x})}{\hat{q}_{\text{hybrid}}(\boldsymbol{x})}, where \hat{q}_{\text{hybrid}}(\boldsymbol{x}) represents the hybrid distribution of the raw dataset. Based on these importance weights, we resample a subset of k samples from the raw dataset, assigning selection probabilities proportional to \frac{\hat{p}_{\text{hybrid}}(\boldsymbol{x})}{\hat{q}_{\text{hybrid}}(\boldsymbol{x})}. The selected data is drawn from the Pile dataset and used for continued pretraining of language models. Finally, we evaluate the performance of these models by fine-tuning on the GLUE benchmark Wang 2018 . The whole process is visualized in Figure 1 . Figure 1: Data selection and pretraining pipeline. The process begins with target and raw datasets, where a subset of raw data is selected to match the target distribution. The selected data is used for pretraining, followed by fine-tuning and evaluation on GLUE tasks Our experimental findings highlight the critical role of data selection in improving language model performance. By aligning the target distribution with the raw dataset, we demonstrate performance gains across multiple downstream tasks. The results confirm that carefully curating pretraining data to resemble the target distribution enhances the model’s ability to generalize effectively. Furthermore, the proposed hybrid importance reweighting (HIR) method with weight parameter equals to 0, which leverages neural embedding-based features, consistently improves over random selection. This improvement highlights its ability to align with the target dataset, capturing semantic and syntactic richness at a macro level. However, DSIR’s superiority in most tasks underscores the value of token-level modeling with n-gram statistics for pretraining language model. 2 Related work The selection of pretraining data has been widely studied in language modeling. Prior methods focus on heuristic approaches, manual data curation, and distributional matching Brown et al. 2020 ; Chowdhery et al. 2022 ; Zhang et al. 2024 ; Xie et al. 2023 . This section reviews these strategies, identifying their strengths and limitations in the context of large-scale data selection. Heuristic selection filters data using a simple binary classifier or predefined rules to iteratively determine whether an instance belongs to the target set or not. It cannot guarantee that selected data is distributed like formal text Brown et al. 2020 . It may include informal or noisy content, and focus on surface-level features like word overlaps, which may miss deeper semantic or contextual relevance. Manual data curation relies on manually curated, pre-existing categorized datasets Gururangan et al. 2020 ; Zhang et al. 2024 . These datasets are shaped by the criteria and decisions of their original creators, potentially embedding biases and omitting valuable content. Additionally, pre-curated datasets are static, and their broad categorization may lack the granularity needed for specialized tasks. These limitations highlight the need for more dynamic and automated methods to refine domain-specific data selection. Distributional Matching: The DSIR method relies on hashed n-gram features for efficient data selection Xie et al. 2023 ; as a result, it captures word-level overlaps, losing abstract semantic and syntactic information crucial for matching complex target distributions. Moreover, it does not optimize the importance of specific features, potentially overlooking those most relevant for downstream tasks. Hashing also introduces noise due to collisions, reducing precision in representing the data. While efficient and scalable, these limitations highlight the need for other feature extraction and optimization to improve alignment with complex target distributions. Although bi-level optimization methods exist for reweighting training data to align with the validation set Chen et al. 2022 ; Chen et al. 2021 ; Ren et al. 2018 , they are typically computationally expensive due to the involvement of second-order gradients. 3 Data Selection Given a small number of target set X_{1},X_{2},\dots,X_{n} from a target distribution p and a raw dataset Y_{1},Y_{2},\dots,Y_{m} from a distribution q with n\ll m , the objective is to extract a subset from the raw dataset of size k that closely aligns with the information in the target set. We begin by introducing two baseline methods for data selection: random selection and DSIR. Random selection provides