Enhancing Dataset Distillation via Non-Critical Region Refinement

Minh-Tuan Tran, Trung Le, Xuan-May Le, Thanh-Toan Do, Dinh Phung · 2025

Read Enhancing Dataset Distillation via Non-Critical Region Refinement by Minh-Tuan Tran, Trung Le, Xuan-May Le, Thanh-Toan Do, Dinh Phung (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-18267.txt.

Introduction

Minh-Tuan Tran and four colleagues wrote a 2025 paper on dataset distillation, compressing a large training data set into a small synthetic one that trains models almost as well. Features in data come in two kinds: instance-specific details unique to each example, and class-general patterns shared across a category. Good distillation must keep both.

Earlier approaches tended to favor one kind and lose the other. The authors propose refining the non-critical regions of distilled images, the areas that carry instance detail without disturbing the class pattern. This refinement rebalances the two feature types. Better distilled data sets mean cheaper model training for everyone who reuses them, from research labs to production teams.

Catalog record

CREATORMinh-Tuan Tran, Trung Le, Xuan-May Le, Thanh-Toan Do, Dinh Phung
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE53 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-18267

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Enhancing Dataset Distillation via Non-Critical Region Refinement Minh-Tuan Tran, Trung Le, Xuan-May Le, Thanh-Toan Do, Dinh Phung Abstract Dataset distillation has become a popular method for compressing large datasets into smaller, more efficient representations while preserving critical information for model training. Data features are broadly categorized into two types: instance-specific features, which capture unique, fine-grained details of individual examples, and class-general features, which represent shared, broad patterns across a class. However, previous approaches often struggle to balance these features—some focus solely on class-general patterns, neglecting finer instance details, while others prioritize instance-specific features, overlooking the shared characteristics essential for class-level understanding. In this paper, we introduce the Non-Critical Region Refinement Dataset Distillation (NRR-DD) method, which preserves instance-specific details and fine-grained regions in synthetic data while enriching non-critical regions with class-general information. This approach enables models to leverage all pixel information, capturing both feature types and enhancing overall performance. Additionally, we present Distance-Based Representative (DBR) knowledge transfer, which eliminates the need for soft labels in training by relying on the distance between synthetic data predictions and one-hot encoded labels. Experimental results show that NRR-DD achieves state-of-the-art performance on both small- and large-scale datasets. Furthermore, by storing only two distances per instance, our method delivers comparable results across various settings. The code is available at https://github.com/tmtuan1307/NRR-DD . 1 Introduction Dataset distillation, also known as dataset condensation [ 26 , 32 , 2 , 21 , 27 ] , has gained considerable attention as an effective method for compressing large datasets into smaller, more efficient representations, while preserving essential information critical for model training. By generating compact, high-quality data representations, dataset distillation reduces both storage requirements and computational costs associated with training on full-sized datasets [ 5 , 8 , 17 ] . This compression is particularly valuable in resource-constrained environments, such as edge devices or memory-limited systems, where training on large datasets is often impractical [ 1 , 9 , 23 ] . The primary goal of dataset distillation is to achieve high model performance while drastically reducing the amount of data that needs to be stored and processed [ 26 ] . During this process, synthetic data instances are generated to capture the key properties of the original data, enabling models to generalize effectively using only a fraction of the dataset [ 32 , 2 ] . In recent years, several methods have been proposed to address this task, including Gradient Matching [ 32 , 10 ] , Distribution Matching [ 33 , 31 ] , Trajectory Matching [ 2 , 6 ] , and more recent approaches for large-scale datasets [ 3 , 27 , 21 ] . However, designing an efficient distillation method remains a challenge, as it must capture both class-general features, which represent shared patterns, and instance-specific features, which highlight unique, fine-grained details. Previous methods often fall short by emphasizing one of these feature types over the other. Approaches focusing on class-general features [ 32 , 2 , 33 , 3 , 27 ] risk losing crucial instance-specific information, which hinders fine-grained generalization. Conversely, methods that prioritize instance-specific features [ 21 , 7 ] may neglect broader class patterns, leading to suboptimal class-level representation. To address these challenges, we introduce the Non-Critical Region Refinement Dataset Distillation (NRR-DD) method, which consists of three key stages: (i) Critical-based Initial Data Discovery : This stage involves selecting diverse and significant patches from the original dataset, which are then combined to capture instance-specific features. (ii) Non-Critical Region Refinement (NRR) : In this stage, we apply Class Activation Mapping (CAM) [ 34 , 19 ] to identify critical and non-critical regions in the images. The model preserves the critical regions, which contain fine-grained, instance-specific features, while refining the non-critical regions with more class-general information. By balancing these two feature types, NRR enhances the dataset’s comprehensiveness, improving both generalization and performance. (iii) Knowledge Transfer via Relabeling : After training, the synthetic images are relabeled and used to transfer knowledge to a student model. Additionally, recent research [ 27 , 3 , 21 ] has highlighted the importance of soft labels generated by a pretrained model in enhancing dataset distillation performance, particularly for large-scale, high-resolution datasets like ImageNet1k. However, this approach incurs substantial memory overhead; for instance, with 200 images per class (IPC) in ImageNet1k, it can require over 120 GB of storage [ 27 ] . To address this challenge, we propose a novel Distance-Based Representative (DBR) knowledge transfer technique that eliminates the need for traditional soft labels. DBR employs a distance-based approach to measure the discrepancy between predictions on synthetic data and one-hot encoded labels, simplifying the training process and reducing label storage requirements. For example, our method requires only 0.2 GB to store ImageNet1k (200 IPC), achieving a 500× reduction in storage while still delivering comparable results. By integrating DBR with NRR, our method enhances dataset distillation by capturing essential features while minimizing training complexity. This results in more compact and efficient datasets, well-suited for a variety of training environments. Figure 1 illustrates the differences between our method and two popular large-scale dataset distillation techniques, RDED [ 21 ] and SRe 2 L. It is evident that RDED focuses on instance-specific features without refinement, while SRe 2 L updates all pixels to capture class-general features, often at the expense of fine-grained details. In contrast, our NRR-DD method effectively preserves fine-grained details by updating only non-critical pixels, while still capturing class-general features. Contributions. Our major contributions are summarized as follows: • We introduce the Non-Critical Region Refinement Dataset Distillation (NRR-DD) framework, which consists of three key stages: Critical-based Initial Data Discovery (CIDD), Non-Critical Region Refinement (CRR), and Relabeling. This approach generates synthetic data that captures both instance-specific fine-grained features and class-general patterns, significantly enhancing performance. • We propose the Distance-Based Representative (DBR) method for knowledge transfer, eliminating the need for soft labels and drastically reducing memory requirements. Specifically, our method reduces storage requirements by 500-fold compared to soft labels on ImageNet1k, while recovering up to 80\% of the full performance (see Table 4 ). • Experimental results demonstrate that our NRR method achieves state-of-the-art performance on both small- and large-scale datasets. Additionally, by storing only two distances per instance, it achieves comparable results across various settings. Figure 1 : Comparison of our method with two popular frameworks, SRe 2 L [ 27 ] and RDED [ 21 ] , for generating synthetic datasets. RDED selects high-confidence, easily classifiable images, while our method focuses on low-confidence, harder-to-classify samples, which helps reduce overfitting and improve model accuracy. Additionally, RDED targets instance-specific features without refinement, and SRe 2 L updates all pixels to capture class-general features, often at the expense of fine details. In contrast, our NRR-DD method preserves fine-grained details while capturing class-general features by updating only non-critical pixels. 2 Related Works Dataset Distillation. Several dataset distillation methods have been proposed recently [ 32 , 2 , 33 , 21 , 3 , 27 ] , which can be classified into two main categories. The first category, Class-General Feature-Based Methods, aims to capture class-wide features. For example, Gradient Matching [ 32 , 10 ] generates synthetic data by matching gradients across all samples in a class, while Distribution Matching [ 33 , 31 ,

Read the complete text