DONOD: Efficient and Generalizable Instruction Fine-Tuning for LLMs via Model-Intrinsic Dataset Pruning

Jucheng Hu, Suorong Yang, Lijun Wu, Dongzhan Zhou · 2025

Read DONOD: Efficient and Generalizable Instruction Fine-Tuning for LLMs via Model-Intrinsic Dataset Pruning by Jucheng Hu, Suorong Yang, Lijun Wu, Dongzhan Zhou (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-14810.txt.

Introduction

Jucheng Hu and co-authors wrote a 2025 paper on pruning training data for language models using the model's own weights. Fine tuning on niche data can hurt general skill and wastes effort on noisy samples. Their method, DONOD, scores each sample by two weight based signals.

One signal tracks total influence on the weights. The other tracks weight wobble. A ranking rule blends the two to drop noisy and unlearnable samples. Cutting seventy percent of the data still raised target accuracy a lot and general accuracy some. Data picked by a small model also worked on bigger ones. The method needs no helper model and fits any data set.

Catalog record

CREATORJucheng Hu, Suorong Yang, Lijun Wu, Dongzhan Zhou
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE53 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2504-14810

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

DONOD: Efficient and Generalizable Instruction Fine-Tuning for LLMs via Model-Intrinsic Dataset Pruning Jucheng Hu, Suorong Yang, Lijun Wu, Dongzhan Zhou Abstract Ad-hoc instruction fine-tuning of large language models (LLMs) is widely adopted for domain-specific adaptation. While domain-specific supervised fine-tuning (SFT) is effective and efficient, it often weakens cross-domain generalization and struggles with noisy training data. To address these challenges, we propose DONOD, a lightweight model-intrinsic data pruning method. Our approach evaluates data using two model-parameter-based metrics: Delta of Norm (DON), which captures the cumulative influence on model weights, and Norm of Delta (NOD), which quantifies weight instability. Moreover, by employing the Technique for Order of Preference by Similarity to Ideal Solution (TOPSIS) algorithm, we effectively filter noisy, unlearnable, and generalization-harming samples without relying on auxiliary models during the SFT process. Experiments on mathematical tasks demonstrate that data selected by DONOD achieves superior fine-tuning efficiency and improved robustness against noisy data. By filtering out 70% of the whole dataset, we improve target-domain accuracy by 14.90% and cross-domain accuracy by 5.67%. Meanwhile, our selected data present superior cross-architecture generalization. Data pruned by smaller models (e.g., Llama 3.1-8B) generalize effectively on larger models (e.g., Llama 2-13B). Compared to existing related methodologies, DONOD demonstrates comparable or superior performance while remaining dataset-agnostic, enabling broader applicability. Code will be made publicly available. 1 Shanghai Artificial Intelligence Laboratory, 2 University College London, [email protected], [email protected], {wulijun,zhoudongzhan}@pjlab.org.cn 1 Introduction In recent years, large language models (LLMs) have demonstrated strong generalization capabilities and remarkable success across a wide range of applications ( Achiam et al. 2023 ; Meta AI 2024 ; Li et al. 2024b ) . To further align LLMs with human intent and diverse user needs, supervised fine-tuning (SFT) on curated instruction–response pairs has become a widely adopted approach ( Cao et al. 2023 ; Peng et al. 2023 ) . However, fine-tuning LLMs with massive instruction datasets incurs substantial computational costs. More importantly, recent studies have pointed out that the quality of the fine-tuning data outweighs its quantity ( Li et al. 2025b ; Xia et al. 2024 ; Wang et al. 2023 ) . In practice, large-scale instruction data are typically collected via web scraping or weak supervision, which inevitably introduces noisy or redundant samples, thereby hindering the capability ( Li and Zhang 2021 ; Szep et al. 2024 ; Yang et al. 2024 ) . While manually constructing high-quality SFT datasets is ideal, it is prohibitively expensive at scale. As a practical alternative, recent research focuses on data selection methods that identify high-quality subsets from existing SFT data ( Li et al. 2025b ; Xia et al. 2024 ; Wang et al. 2023 ) . Such approaches have shown that models trained on selected subsets can match or even surpass the performance of those trained on the full dataset, offering a promising way to maintain model quality while significantly reducing computational costs. Recent advances, such as reward-model-based filtering ( Xu et al. 2025 ) and gradient-based pruning ( Xia et al. 2024 ) , have pushed this direction forward. Although achieving promising results, many of these methods incur huge computational overhead ( Xie et al. 2023 ) or rely on task-specific validation sets ( Xia et al. 2024 ) , which may limit their scalability across diverse domains. Furthermore, recent studies have found that models fine-tuned on such data are prone to domain overfitting, along with degraded generalization across domains ( Li and Zhang 2021 ; Szep et al. 2024 ) , which poses a unique challenge for ad-hoc instruction fine-tuning beyond the general SFT. Therefore, these issues underscore the pressing need for a more principled, data-efficient view to scale LLM fine-tuning more cost-effectively and robustly, with the potential to reduce compute overhead and amplify impact in real-world deployments substantially. This raises a central question: How can we select the most representative and generalizable samples from large-scale instruction datasets to enable efficient and robust LLM fine-tuning? Figure 1: Performance of our proposed DONOD across various evaluation benchmarks, consistently outperforming other baselines, e.g., all data, despite using only 20% of the training data. These results highlight the potential of principled data pruning for efficient and robust LLM fine-tuning. To address the issues above, we propose DONOD, a model-intrinsic dataset pruning method that selects a significantly reduced subset of the most representative samples. Specifically, DONOD introduces two complementary metrics derived from the model’s training dynamics: Delta of Norm (DON) and Norm of Delta (NOD). The DON metric captures the cumulative shift in parameter magnitudes throughout training, serving as a proxy for generalization, which encourages the selection of samples that lead to stable norm evolution and contribute to smooth convergence. Meanwhile, NOD measures the instability induced by individual samples by quantifying sample-specific weight fluctuations, thereby identifying noisy or unlearnable data. To reconcile these dual objectives, maximizing generalization via DON and minimizing ambiguous updates via NOD, we employ the Technique for Order of Preference by Similarity to the Ideal Solution (TOPSIS) algorithm ( Hwang and Yoon 1981 ; Chakraborty 2022 ) , which allows principled ranking of samples based on proximity to the ideal pruning criterion. Importantly, DONOD requires no auxiliary models, domain-specific heuristics, or validation sets, relying purely on intrinsic training signals, thus ensuring computational efficiency and broad applicability. As a result, DONOD enables efficient instruction tuning while preserving or even enhancing generalization performance with significantly fewer training examples. Experiment results across diverse downstream instruction-tuning benchmarks and LLM architectures demonstrate that DONOD consistently outperforms state-of-the-art baselines, achieving superior performance with substantially fewer samples while maintaining dataset-agnostic flexibility. Notably, compared with the full-data SFT setting, DONOD improves target-domain accuracy by 14.90% and cross-domain accuracy by 5.67% when using only 30% of the data. Moreover, we demonstrate strong cross-architecture generalization, where data selected by lighter-weight models (e.g., LLaMA 3.1-8B ( Grattafiori et al. 2024 ) ) can generalize well across larger or different architectures, such as LLaMA 2-13B ( Touvron et al. 2023 ) and Qwen 2.5-7B ( Team 2024 ) , etc. Additionally, we further validate the robustness of DONOD in more complex and realistic noisy settings, highlighting its practical significance. We summarize our contribution as follows: • We propose DONOD, a lightweight, model-intrinsic data selection framework for accelerating the fine-tuning of LLMs. • We introduce two complementary metrics, DON and NOD, where DON captures the generalization impact of samples and NOD identifies noisy or unlearnable samples. • Experimental results validate that our method outperforms existing methods in terms of cross-domain and cross-architecture generalization performance, e.g., achieving lossless training acceleration with only 20% of the data, highlighting the scalability and robustness of our approach. 2 Related Work Dataset pruning for supervised fine-tuning on LLMs critically impacts model performance. Traditional methods often rely on external models as quality judges ( ( Du, Zong, and Zhang 2023 ) , ( Chen et al. 2024 ) ) or employ reward models to identify high-quality data ( Yang et al. 2025a ) . However, this dependence on auxiliary models, whether trained from scratch or repurposed, incurs significant computational costs and limits scalability. Recent studies ( ( Li et al. 2025a ) ) further question the effectiveness of this paradigm. Alternative approaches focus on intrinsic data metrics. For instance, ( Cao et al. 2024

Read the complete text