ResoFilter: Fine-grained Synthetic Data Filtering for Large Language Models through Data-Parameter Resonance Analysis
Zeao Tu, Xiangdi Meng, Yu He, Zihan Yao, Tianyu Qi, Jun Liu, Ming Li · 2024
Read ResoFilter: Fine-grained Synthetic Data Filtering for Large Language Models through Data-Parameter Resonance Analysis by Zeao Tu, Xiangdi Meng, Yu He, Zihan Yao, Tianyu Qi, Jun Liu, Ming Li (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-14809.txt.
Introduction
Zeao Tu and co-authors filter fake training data for language models by watching how model weights react. Their 2024 paper notes that GPT-made data is now common for training, yet its quality is uneven. Clear scores for what makes a data point useful have been missing.
Their method, ResoFilter, draws features from the fine-tuning process itself. Those features describe each data point through the lens of model weights, which aids reading of the results. On math tasks, training on half the data picked by the filter matched full-data training. The picks also carried well across models and fields. The authors plan to release code and data with the paper to aid repeat checks.
Catalog record
| CREATOR | Zeao Tu, Xiangdi Meng, Yu He, Zihan Yao, Tianyu Qi, Jun Liu, Ming Li |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 42 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2412-14809
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
ResoFilter: Fine-grained Synthetic Data Filtering for Large Language Models through Data-Parameter Resonance Analysis Zeao Tu, Xiangdi Meng, Yu He, Zihan Yao, Tianyu Qi, Jun Liu, Ming Li Abstract Large language models (LLMs) have shown remarkable effectiveness across various domains, with data augmentation methods utilizing GPT for synthetic data generation becoming prevalent. However, the quality and utility of augmented data remain questionable, and current methods lack clear metrics for evaluating data characteristics. To address these challenges, we propose ResoFilter , a novel method that integrates models, data, and tasks to refine datasets. ResoFilter leverages the fine-tuning process to obtain Data-Parameter features for data selection, offering improved interpretability by representing data characteristics through model weights. Our experiments demonstrate that ResoFilter achieves comparable results to full-scale fine-tuning using only half the data in mathematical tasks and exhibits strong generalization across different models and domains . This method provides valuable insights for constructing synthetic datasets and evaluating high-quality data, offering a promising solution for enhancing data augmentation techniques and improving training dataset quality for LLMs. For reproducibility, we will release our code and data upon acceptance. The source code and implementation details of this work are publicly available in our GitHub repository ( https://github.com/TAL-auroraX/ResoFilter ) 1 Introduction Large language models (LLMs) have demonstrated remarkable capabilities across various domains, with training data playing a pivotal role in enhancing their performance. The quality and quantity of data are crucial factors in all stages of LLM development, including pretraining, instruction tuning, and alignment ( Peters et al. 2018 ; Radford and Narasimhan 2018 ; Devlin et al. 2019 ; Raffel et al. 2020 ; Touvron et al. 2023 ; Mishra et al. 2021 ; Sanh et al. 2022 ; Longpre et al. 2023 ; Muennighoff et al. 2024 ; Ziegler et al. 2019 ; Bai et al. 2022 ; Ouyang et al. 2022 ; Rafailov et al. 2023 ) . The significance of high-quality datasets has been well-established in the pretraining phase, where data cleaning techniques have shown to substantially improve model performance, especially for smaller models ( Conneau and Lample 2019 ; Raffel et al. 2020 ; Wenzek et al. 2020 ; Gao et al. 2020 ; Rae et al. 2021 ; Lee et al. 2022 ) . Following the success of ChatGPT Ouyang et al. 2022 ; OpenAI et al. 2024 , the focus has shifted towards creating high-quality fine-tuning datasets, leading to a surge in methods for automatic generation of instruction-following data using GPT models. While these methods, such as Self-Instruct ( Wang et al. 2023b ) , Evol-Instruct ( Xu et al. 2023 ) , and others ( Honovich et al. 2023 ; Dai et al. 2023 ; Abdullin et al. 2024 ) , have significantly increased the scale of available datasets, the notion of "high quality" remains contentious. As observed by Schimanski et al. 2024 , current approaches predominantly focus on augmenting data volume rather than enhancing quality. This trend has led to the phenomenon of "diminishing returns," where performance gains plateau as dataset size increases beyond a certain threshold, emphasizing the need to prioritize data quality over quantity. Figure 1: Workflow for our method. The left side of the figure illustrates the detailed process of calculating parameter changes for individual data samples using the W_{up} matrix from the last n layers of the neural network. The right side demonstrates the application of this method to the entire dataset, including steps for computing parameter change values for each sample, sorting based on these values, filtering out samples with the largest parameter changes, and restoring the remaining data to its original order. To address this challenge, we propose ResoFilter, a novel method that leverages the fine-tuning process for effective data selection. ResoFilter processes each data point through full forward and backward propagation, capturing the induced change in model weights. From these changes, we derive a characteristic score for each data point, which serves as the metric for subsequent selection. Our approach offers improved interpretability by representing data characteristics through model weights, building upon existing research demonstrating that model weights store knowledge Hanna et al. 2023 ; Dai et al. 2022 . Figure 1 illustrates the detailed workflow of our method. Empirical results demonstrate the efficacy of ResoFilter. In the context of MetaMath Yu et al. 2023 , we achieve comparable performance to full dataset fine-tuning using only 50% of the data selected by our method. Moreover, by eliminating poorly performing data points, ResoFilter can even surpass the performance of full fine-tuning. The method also exhibits strong generalization across different models and domains, including mathematics, code, and general question answering tasks. Our contributions can be summarized as follows: 1. We introduce ResoFilter, an effective method for selecting high-quality datasets from large-scale data collections to enhance the fine-tuning process of large language models. 2. We validate the generalizability of ResoFilter by demonstrating its excellent performance across various domains (including mathematics, code, and general tasks) and different model architectures, showcasing its robustness and wide applicability. 3. Through extensive experimental analysis, we provide valuable insights into the construction of synthetic datasets and the analysis of high-quality data, offering guidance for future data synthesis and selection methodologies. 2 Related Work Instruction data selection method Using high-quality data for instruction tuning of large language models can significantly enhance their performance. The existing work on selecting high-quality data can be divided into two main parts: (1).Utilizing human expertise or more powerful generative models to assist in the selection process. For example, Lima Zhou et al. 2024 manually curated a dataset of 1,000 instruction samples for instruction tuning and achieved outstanding performance on the evaluation set. AlpaGasus Chen et al. 2023b , InsTag Lu et al. 2023 , and DEITA Liu et al. 2023 leverage the superior performance of ChatGPT to replace manual effort in filtering and selecting data. (2).Model-guided data selection involves filtering data by observing changes in the model or by introducing additional smaller models. Kung et al. 2023 proposes a new framework based on instantaneous uncertainty, which identifies informative tasks by measuring changes in the probability distribution of the model’s outputs. Superfiltering Li et al. 2024b utilizes smaller language models to filter out better data. By using GPT-2 combined with Perplexity and IFD Li et al. 2023a to filter the data. Data attribution The work of data attribution aims to quantify the contribution of each training sample to the model’s prediction results. It attempts to answer the question, "Which training samples have the most significant impact on a specific prediction result?" In recent studies, Li et al. 2023a quantifies the degree of assistance provided by instructions to models by comparing the difficulty of generating responses with and without the instruction context. Nuggets Li et al. 2023b leverage the downstream task benefits brought by data to the model for data filtering. LESS Xia et al. 2024 estimates the impact of data by using gradients in the optimizer and performs low-rank gradient similarity searches for instruction data selection. Knowledge in LLM From the perspective of model interpretability, it has been consistently observed that language models based on the transformer architecture encode "lower-level" information in the earlier layers and more "semantic" information in the later layers ( Tenney 2019 ) .