UNSEEN: Enhancing Dataset Pruning from a Generalization Perspective
Furui Xu Shaobo Wang Jiajun Zhang Chenghao Sun Haixiang Tang Linfeng Zhang · 2025
Read UNSEEN: Enhancing Dataset Pruning from a Generalization Perspective by Furui Xu Shaobo Wang Jiajun Zhang Chenghao Sun Haixiang Tang Linfeng Zhang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-12988.txt.
Introduction
Furui Xu and five co-authors choose training data by asking how a model performs on examples it has never seen. Dataset pruning keeps a small coreset of a huge dataset so training stays cheap, and most methods score each example using a model trained on that very data. The flaw is that a well-fitted model scores nearly everything the same, so the scores stop distinguishing gold from filler. UNSEEN flips the perspective to generalization: score examples with models that were not trained on them, so importance reflects what an example teaches, not what a model memorized.
The framework plugs into existing pruning methods and extends to multiple rounds, where scoring models trained on varying coresets refine the selection dynamically. On CIFAR-10, CIFAR-100, and ImageNet-1K it beat state-of-the-art pruning, removing thirty percent of ImageNet-1K training data with no loss in final performance.
Catalog record
| CREATOR | Furui Xu Shaobo Wang Jiajun Zhang Chenghao Sun Haixiang Tang Linfeng Zhang |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 31 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2511-12988
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
UNSEEN: Enhancing Dataset Pruning from a Generalization Perspective Furui Xu Shaobo Wang Jiajun Zhang Chenghao Sun Haixiang Tang Linfeng Zhang Abstract The growing scale of datasets in deep learning has introduced significant computational challenges. Dataset pruning addresses this challenge by constructing a compact but informative coreset from the full dataset with comparable performance. Previous approaches typically establish scoring metrics based on specific criteria to identify representative samples. However, these methods predominantly rely on sample scores obtained from the model’s performance during the training (i.e., fitting) phase. As scoring models achieve near-optimal performance on training data, such fitting-centric approaches induce a dense distribution of sample scores within a narrow numerical range. This concentration reduces the distinction between samples and hinders effective selection. To address this challenge, we conduct dataset pruning from the perspective of generalization, i.e., scoring samples based on models not exposed to them during training. We propose a plug-and-play framework, UNSEEN, which can be integrated into existing dataset pruning methods. Additionally, conventional score-based methods are single-step and rely on models trained solely on the complete dataset, providing limited perspective on the importance of samples. To address this limitation, we scale UNSEEN to multi-step scenarios and propose an incremental selection technique through scoring models trained on varying coresets, and optimize the quality of the coreset dynamically. Extensive experiments demonstrate that our method significantly outperforms existing state-of-the-art (SOTA) methods on CIFAR-10, CIFAR-100, and ImageNet-1K. Notably, on ImageNet-1K, UNSEEN achieves lossless performance while reducing training data by 30%. 1 Introduction (a) Entropy distribution under the Fitting and UNSEEN frameworks on CIFAR-10 (left) and CIFAR-100 (right) datasets. (b) Rank distribution of two identical trials with different random seeds under the Fitting (Top) and UNSEEN (Bottom) frameworks. Figure 1: (a) Distribution of Entropy score on CIFAR-10 and CIFAR-100 under the fitting and UNSEEN frameworks. Under the fitting framework, Entropy scores exhibit dense clustering. Conversely, UNSEEN achieves uniform score dispersion, substantially improving discriminative separability. (b) Distribution of the rank assigned to each sample in the overall score ranking in two identical CIFAR-100 trials with different random seeds. Sample ranks fluctuate significantly under fitting but remain stable under UNSEEN. The Pearson correlation coefficient (PCC) between trials is 0.92 for UNSEEN, much higher than 0.43 under fitting. Figure 2: Samples exhibit varying levels of importance across coresets of different stages. Incremental selection prioritizes samples with the highest importance at each stage, offering a more principled and adaptive approach to coreset construction. In recent decades, the rapid development of deep learning has been driven by large-scale datasets ( 5 ; 15 ; 38 ) , producing many amazing achievements ( 6 ; 2 ; 21 ) . However, this approach is inherently costly and requires substantial computational resources and considerable time ( 1 ; 34 ; 36 ) . Moreover, a significant portion of the data is redundant or erroneous ( 35 ; 37 ; 32 ; 19 ) , contributing minimally to the improvement of model performance. Dataset pruning ( 7 ) , also known as coreset selection, mitigates data redundancy by constructing a compact subset that enables the model to achieve performance comparable to that obtained with the full dataset. This strategy emphasizes identifying and selecting the most informative and representative samples. Dataset pruning methods commonly identify important samples through various strategies, including geometry features ( 32 ; 37 ; 25 ) , uncertainty ( 3 ; 20 ; 9 ; 35 ) , error ( 19 ; 23 ) , and decision boundary ( 17 ; 33 ) . These methods are based on their scores derived from the model’s performance during the training ( i.e., fitting) phase. However, as models exhibit a strong capacity for sample fitting, the score distribution becomes densely concentrated. As demonstrated in Figure 1(a) , Under the conventional fitting framework, more than 90% of CIFAR-10 samples and 78% of CIFAR-100 samples exhibit Entropy values close to zero. Due to the overly clustered score distribution of samples, the model frequently fails to accurately identify challenging samples, and the selection results exhibit severe instability. As shown in Figure 1(b) , the rank assigned to each sample in the overall score ranking varies significantly between two otherwise identical experiments initialized with different random seeds. This observation indicates that conventional methods under the fitting framework exhibit limitations in effectively and robustly differentiating samples of varying difficulty. To overcome this challenge, we propose a novel framework called UNSEEN , designed to assess the importance of data samples from a generalization perspective. Specifically, we employ cross-validation to facilitate mutual scoring across multiple models, where each sample’s importance is determined exclusively by models that never encountered it during training. The implementation involves randomly partitioning the full dataset into mutually exclusive folds of equal size, training scoring models on each fold, and subsequently utilizing these models to score the samples excluded from their respective folds. As shown in Figure 1(a) , our generalization-based framework UNSEEN produces Entropy scores with more uniform dispersion across a broader scoring range compared to the conventional fitting-based approach. Figure 1(b) demonstrates that our method significantly enhances robust and discriminative selection. Additionally, existing score-based methods are single-step i.e., score samples once with the scoring model trained solely on the full dataset. As demonstrated in Figure 2 , samples exhibit varying levels of importance when scored by models trained on different coresets. We frame coreset construction as an incremental process, considering the difficulty of samples for models trained on coresets of varying sizes. To comprehensively assess the importance of samples, we scale UNSEEN to multi-step scenarios and propose an evaluate-and-refill paradigm, incremental selection . It initiates by constructing an initial coreset through a selection criterion under the fitting or UNSEEN framework. A scoring model is subsequently trained on the coreset and is then employed to score pruned samples. Those that obtain the highest scores are incorporated into the coreset. This procedure persists until the final coreset attains the target cardinality. Our contributions in this paper are as follows: • We expose the limitations of previous fitting-based methods and introduce UNSEEN, a plug-and-play framework designed from a generalization perspective. • We enhance conventional single-step pruning methods by scaling UNSEEN to a multi-step selection process, and propose incremental selection (IS), which provides a more comprehensive assessment of sample importance. • Our method outperforms existing SOTA methods on CIFAR-10, CIFAR-100, and ImageNet-1K, and achieves 30% lossless pruning on ImageNet-1K. • We extend the notion of difficulty from individual samples to the class level, prioritizing the minimization of inter-class disparity over uniform treatment across categories. Figure 3: The pipeline of UNSEEN. First, the training dataset is randomly partitioned into K equal-sized subsets. Then, for each subset, a scoring model is trained and used to assign scores to the samples in the complementary subsets. The top M_{1} samples with the highest scores are selected to form the initial coreset S_{1} . Next, a scoring model is trained on the selected samples and used to score the remaining unselected samples. Samples with the highest scores are incrementally added to the coreset. This procedure is repeated until the desired number of samples has been selected. 2 Related work Dataset Pruning. Current approaches to dataset pruning mainly include score-based and optimization-based methods. Emerging