Not All Instances Are Equally Valuable: Towards Influence-Weighted Dataset Distillation
Qiyan Deng, Changqian Zheng, Lianpeng Qiao, Yuping Wang, Chengliang Chai, Lei Cao · 2025
Read Not All Instances Are Equally Valuable: Towards Influence-Weighted Dataset Distillation by Qiyan Deng, Changqian Zheng, Lianpeng Qiao, Yuping Wang, Chengliang Chai, Lei Cao (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-27253.txt.
Introduction
Qiyan Deng leads five co-authors in influence-weighted dataset distillation, a way to shrink training data without treating every example as equal. Distillation condenses a large dataset into a small synthetic subset that trains models nearly as well. Most methods assume all real instances contribute equally, yet real data mix informative examples with redundant or harmful ones.
Their framework, IWD, uses influence functions to estimate each instance effect on the distillation goal, then weights instances adaptively. Helpful data gain priority while weak or harmful data are downweighted. The modular design plugs into diverse existing distillation frameworks. Empirical results suggest consistent quality gains for distilled sets and model accuracy improvements up to several percent, arguing data quality should guide compression.
Catalog record
| CREATOR | Qiyan Deng, Changqian Zheng, Lianpeng Qiao, Yuping Wang, Chengliang Chai, Lei Cao |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 37 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2510-27253
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Not All Instances Are Equally Valuable: Towards Influence-Weighted Dataset Distillation Qiyan Deng, Changqian Zheng, Lianpeng Qiao, Yuping Wang, Chengliang Chai, Lei Cao Abstract Dataset distillation condenses large datasets into synthetic subsets, achieving performance comparable to training on the full dataset while substantially reducing storage and computation costs. Most existing dataset distillation methods assume that all real instances contribute equally to the process. In practice, real-world datasets contain both informative and redundant or even harmful instances, and directly distilling the full dataset without considering data quality can degrade model performance. In this work, we present I nfluence- W eighted D istillation ( IWD ), a principled framework that leverages influence functions to explicitly account for data quality in the distillation process. IWD assigns adaptive weights to each instance based on its estimated impact on the distillation objective, prioritizing beneficial data while downweighting less useful or harmful ones. Owing to its modular design, IWD can be seamlessly integrated into diverse dataset distillation frameworks. Our empirical results suggest that integrating IWD tends to improve the quality of distilled datasets and enhance model performance, with accuracy gains of up to 7.8%. 1 Introduction High-quality training data are crucial for modern machine learning, directly affecting performance, robustness, and generalization. Yet, with global data volume projected to reach 163 zettabytes by 2025 ( Reinsel et al. 2017 ) , the exponential growth of training data imposes unprecedented demands on storage and computation, driving up both training time and cost. To address these challenges, dataset distillation ( Wang et al. 2018 ; Yu et al. 2024 ; Zhao et al. 2020 ; Kim et al. 2022 ) has emerged as a powerful paradigm for distilling large training sets into synthetic subsets that can achieve high model performance while significantly reducing the memory footprint and training time of deep networks on large datasets. While effective, these methods have the following limitations. Limitations. (1) Existing dataset distillation methods typically aggregate features from all real instances to distill the synthetic dataset. In practice, real-world datasets often contain redundant, or even harmful data instances that can degrade model quality, and treating these instances the same as others can reduce the effectiveness of the distilled set ( Wang et al. 2025a ) . For example, outliers in real instances may introduce misleading gradients, causing the distilled data lowering generalization, while redundant instances, such as a large number of visually repetitive images, contribute little novel information and cause the synthetic dataset to underutilize its limited capacity. (2) Some methods ( Sundar et al. 2023 ; Moser et al. 2024 ; Xu et al. 2024 ; Du et al. 2024 ) reduce the first limitation by pruning low-quality or uninformative instances based on loss values or data characteristics before distillation. However, simply discarding these instances may lead to substantial information loss, since low-quality data do not necessarily imply zero contribution to the distillation process, and overly aggressive pruning can remove potentially useful information. Observation. Inspired by Koh & Liang 2017 , we observe that influence functions can be leveraged to estimate the contribution of each real instance to the distillation process. Instances with higher influence scores typically provide more useful and complementary information that enhance the quality of the distilled dataset, whereas those with low or negative influence scores tend to be redundant or even harmful. As shown in Fig. 1 , the first row on the left illustrates redundant data, where most instances belong to the same sub-category (airliners). Since airliners already account for a large proportion of the airplane class in CIFAR10 , additional similar instances provide little new information and thus contribute less to the distilled dataset. The second row shows outliers, such as blurry images or those capturing only partial views of airplanes, which may introduce misleading features. The right part illustrates instances with higher influence scores, which are relatively rare such as bombers, helicopters, and fighters. These instances provide complementary information that enriches the distilled dataset and improve generalization. This insight motivates a weighting strategy that emphasizes influential instances while down-weighting less useful ones. Figure 1: Distribution of instance influence scores in dataset distillation. Most scores concentrate around zero, reflecting that most real instances contribute only marginally. Low-influence instances often come from redundant sub-categories (e.g., airliners), whereas high-influence instances, though rare (e.g., bombers, fighters, helicopters), provide valuable information for distillation. Our Proposal. Therefore, in this work, we propose I nfluence- W eighted D istillation ( IWD ), a principled and flexible framework that tightly integrates influence functions into the dataset distillation process. Specifically, we extend the classical influence function theory to the distillation setting and derive a novel formulation of influence under dataset distillation, which enables principled estimation of each real instance’s contribution to the quality of the distilled dataset. Building on this formulation, IWD first computes an influence score for each training instance. We then use these influence scores as soft, adaptive weights, prioritizing data instances that are most beneficial to the distillation objective and downweighting those that are less informative or potentially harmful. Importantly, IWD is designed as a modular plug-in that can be seamlessly incorporated into existing dataset distillation frameworks. Our contributions are as follows: (1) We extend classical influence function analysis to the dataset distillation setting, providing a new formulation for quantifying the contribution of each real data instance to the distillation process. (2) We propose Influence-Weighted Distillation ( IWD ), a modular plug-in that incorporates influence-based data weighting into the distillation process. IWD is broadly compatible with existing dataset distillation frameworks and enhances the quality of synthetic datasets by prioritizing informative instances while downweighting less useful or harmful ones. (3) We conduct extensive experiments on standard benchmarks to demonstrate that IWD consistently enhances the effectiveness of dataset distillation, achieving better generalization and model performance compared to previous approaches, with accuracy improvements of up to 7.8% (on CIFAR10 ). 2 Related Work Dataset distillation , also known as dataset condensation, aims to generate a synthetic dataset that can match the performance of training on the full dataset, while enabling significant savings in data storage and future retraining costs ( Zhao et al. 2020 ; Cazenavette et al. 2022 ; Wang et al. 2018 ) . Specifically, these works could be classified into 4 types: (1) Meta-Model Matching ( Wang et al. 2018 ; Nguyen et al. 2021 ; Sucholutsky & Schonlau 2021 ) optimizes the distilled data to maximize performance on real data. For instance, DD ( Wang et al. 2018 ) employs a bi-level framework that updates the synthetic set by minimizing the loss on real data for models trained on it. (2) Gradient Matching ( Zhao et al. 2020 ; Lee et al. 2022 ; Kim et al. 2022 ) optimizes the synthetic data so that their induced gradients resemble those of the original data, encouraging models trained on synthetic set to follow similar learning dynamics and achieve comparable performance. (3) Trajectory Matching ( Cazenavette et al. 2022 ; Du et al. 2023 ) aligns the full optimization path of models trained on synthetic versus real data, minimizing trajectory discrepancies so that the distilled model mimics the learning process of the real one. (4) Distribution Matching ( Wang et al. 2025b ; Zhao