MASS: MA thematical Data S election via S kill Graphs for Pretraining Large Language Models

Jiazheng Li, Lu Yu, Qing Cui, Zhiqiang Zhang, Jun Zhou, Yanfang Ye, Chuxu Zhang · 2025

Read MASS: MA thematical Data S election via S kill Graphs for Pretraining Large Language Models by Jiazheng Li, Lu Yu, Qing Cui, Zhiqiang Zhang, Jun Zhou, Yanfang Ye, Chuxu Zhang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-14917.txt.

Introduction

Jiazheng Li and six co-authors present MASS, a framework for choosing which math data should be used to pretrain large language models. Rather than treating all training text alike, the method builds a skill graph from a reference dataset, mapping the math skills involved and how they connect. Each item in a target dataset then receives a quality score based on that graph, and only the top-ranked subset is kept for pretraining.

The authors tested the approach at two model sizes, one billion and seven billion parameters, on both web data and synthetic data. Models trained on the chosen subsets matched the performance of models trained on everything while using fifty to seventy percent fewer tokens. Given the same token budget, the selected data beat the full datasets by 3.3 to 5.9 percent, evidence that skill-aware selection improves both the cost and the quality of pretraining.

Catalog record

CREATORJiazheng Li, Lu Yu, Qing Cui, Zhiqiang Zhang, Jun Zhou, Yanfang Ye, Chuxu Zhang
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE60 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-14917

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

MASS: MA thematical Data S election via S kill Graphs for Pretraining Large Language Models Jiazheng Li, Lu Yu, Qing Cui, Zhiqiang Zhang, Jun Zhou, Yanfang Ye, Chuxu Zhang Abstract High-quality data plays a critical role in the pretraining and fine-tuning of large language models (LLMs), even determining their performance ceiling to some degree. Consequently, numerous data selection methods have been proposed to identify subsets of data that can effectively and efficiently enhance model performance. However, most of these methods focus on general data selection and tend to overlook the specific nuances of domain-related data. In this paper, we introduce MASS, a MA thematical data S election framework using the S kill graph for pretraining LLMs in the mathematical reasoning domain. By taking into account the unique characteristics of mathematics and reasoning, we construct a skill graph that captures the mathematical skills and their interrelations from a reference dataset. This skill graph guides us in assigning quality scores to the target dataset, enabling us to select the top-ranked subset which is further used to pretrain LLMs. Experimental results demonstrate the efficiency and effectiveness of MASS across different model sizes (1B and 7B) and pretraining datasets (web data and synthetic data). Specifically, in terms of efficiency, models trained on subsets selected by MASS can achieve similar performance to models trained on the original datasets, with a significant reduction in the number of trained tokens - ranging from 50% to 70% fewer tokens. In terms of efficacy, when trained on the same amount of tokens, models trained on the data selected by MASS outperform those trained on the original datasets by 3.3% to 5.9%. These results underscore the potential of MASS to improve both the efficiency and effectiveness of pretraining LLMs. † † affiliationnotice: Equal contribution 1 Introduction The success of large language models (LLMs) is closely tied to the scaling up of model size, training data, and computational resources ( Brown, 2020 ; Hoffmann et al., 2022 ; Qwen Team, 2024 ) . Among these three factors, training data serves as the foundation of LLMs’ rich knowledge and capabilities, enabling them to excel in creative writing, complex reasoning, and even agentic planning. Recently, there is mounting evidence that pretraining on high-quality and diverse corpora can significantly enhance LLM performance ( Penedo et al., 2024 ; Li et al., 2024a ) . For example, Microsoft’s Phi series of models has been renowned for their pretraining corpus’s quality and efficiency. The latest Phi-4 model ( Abdin et al., 2024 ) , trained on 10T tokens, has surpassed other models of the same size, including Qwen 2.5 ( Qwen Team, 2024 ) , which was trained on 18 trillion tokens. This has led many researchers to focus on LLM data curation methodologies, which include the selection of high-quality sub-datasets ( Wettig et al., 2024 ) , deduplication of extensive datasets ( Tirumala et al., 2023 ) , and the balancing of data from various domains ( Xie et al., 2024 ) , among other techniques. Figure 1 : The pipeline of MASS. (1) We first employ prompt engineering to extract mathematical skills from a reference dataset, thereby constructing a skill graph. (2) We then score and rank the target dataset using the constructed skill graph. The top-ranked subset can further be selected and used to train LLMs. In terms of data selection methods, the objective is to extract high-quality subsets from the original datasets such that training LLMs on these subsets can achieve similar or even superior performance on downstream tasks compared to training on the entire original datasets. There are two main categories of methods: heuristic-based approaches and model-based approaches. For the former, a variety of manually designed filters, including mean word length, stop word fraction, and word repetitions, are frequently applied to preprocess raw web data ( Tirumala et al., 2023 ) . For the latter, models similar to BERT ( Devlin et al., 2019 ) are commonly used to evaluate and select data with higher scores, while other approaches directly employ GPT-like models to assess the data’s value. While the emergence of OpenAI o-series models ( OpenAI, 2024 ) , DeepSeek-R1 ( DeepSeek-AI et al., 2025 ) , and similar models, focusing on advanced reasoning capabilities, highlights the growing emphasis on the mathematical and reasoning abilities of LLMs, current data selection methods usually concentrate on general domains, neglecting the specific attributes of mathematical domains. AutoDS ( Zhang et al., 2024b ) , as an exception, uses Qwen2-72B to score the OpenWebMath ( Paster et al., 2024 ) dataset based on whether each text demonstrates mathematical intelligence and suitability for educational purposes in mathematics. Although it employs tailored prompts for mathematical texts, it still overlooks the deeper nuances of the mathematical domain when compared to the general domain, particularly from the perspective of math skills. Therefore, in this paper, we delve into data selection for LLMs and propose MASS - a MA thematical data S election framework using S kill graphs, to enhance their mathematical and reasoning abilities by considering the underlying skills (or knowledge points) and their interrelations embedded in mathematical datasets. MASS has two selection guidelines: (1) a data point that encompasses more important math skills should have a higher quality score; and (2) a data point that covers more important compositional information of math skills should have a higher quality score. As illustrated in Figure 1 , we begin by utilizing a high-quality reference dataset and instructing a strong LLM to extract embedded math skills. These skills are then used to construct a skill graph, where nodes represent skills and edges denote the co-occurrence of skills. Next, we calculate the semantic similarities between these math skills and the target dataset to be selected. We further aggregate these similarities on the graph structure to integrate the compositional information of the skills to obtain final quality scores for each data point in the target dataset. Finally, these scores are ranked, allowing us to select a high-quality subset from the original dataset. In this work, we select NuminaMath ( LI et al., 2024 ) as the reference dataset to construct a skill graph based on its mathematical skills. We then apply our method to three target datasets: two web datasets (OpenWebMath ( Paster et al., 2024 ) , OpenWebMath-pro ( Zhou et al., 2024a ) ) and one synthetic dataset (Jiuzhang3.0-CoT ( Zhou et al., 2024c ) ). Using the skill graph, we identify and extract high-quality subsets from each of these target datasets. We continue pretraining TinyLlama-1.1B and Mistral-7B on the original and selected subsets. Experimental results demonstrate that with merely 50% to 70% training steps, models trained on selected subsets can achieve similar performances than those using original datasets. When trained on the same amount of tokens, models trained on selected subsets can outperform those using original datasets by 3.3% to 5.9%. This highlights the efficiency and efficacy of our data selection method in enhancing the mathematical and reasoning capabilities of LLMs. 2 The Proposed MASS 2.1 Overview Given a target dataset \mathcal{D}_{tgt}=\{x_{1},x_{2},\cdots,x_{N}\} where each data point x is a piece of text, our goal is to select a high quality subset \mathcal{D}_{hq} with budget b\in\mathbb{Z}^{+} , where b=\lvert\mathcal{D}_{hq}\rvert\ll\lvert\mathcal{D}_{tgt}\rvert .

Read the complete text