Grounding and Enhancing Informativeness and Utility in Dataset Distillation
Shaobo Wang Yantai Yang Guo Chen Peiru Li Kaixin Li Yufa Zhou, Zhaorun Chen Linfeng Zhang, EPIC Lab, SJTU Shanghai Jiao Tong University National University of Singapore, Duke University The University of Chicago Corresponding authors {shaobowang1009,zhanglinfeng}@sjtu.edu.cn · 2026
Read Grounding and Enhancing Informativeness and Utility in Dataset Distillation by Shaobo Wang Yantai Yang Guo Chen Peiru Li Kaixin Li Yufa Zhou, Zhaorun Chen Linfeng Zhang, EPIC Lab, SJTU Shanghai Jiao Tong University National University of Singapore, Duke University The University of Chicago Corresponding authors {shaobowang1009,zhanglinfeng}@sjtu.edu.cn (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-21296.txt.
Introduction
Shaobo Wang and co-authors revisit how to shrink a large dataset into a small synthetic one that still teaches models well. Good distilled data can cut training cost a lot. Many current methods rely on rules of thumb to balance speed and quality.
The link between the real data and the synthetic set is still poorly understood. The authors reframe the task inside knowledge distillation with a firmer theory. They focus on making the small set both informative and useful for training. A clearer theory should help builders know what to keep when they compress data.
Catalog record
| CREATOR | Shaobo Wang Yantai Yang Guo Chen Peiru Li Kaixin Li Yufa Zhou, Zhaorun Chen Linfeng Zhang, EPIC Lab, SJTU Shanghai Jiao Tong University National University of Singapore, Duke University The University of Chicago Corresponding authors {shaobowang1009,zhanglinfeng}@sjtu.edu.cn |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 62 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2601-21296
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Grounding and Enhancing Informativeness and Utility in Dataset Distillation Shaobo Wang Yantai Yang Guo Chen Peiru Li Kaixin Li Yufa Zhou, Zhaorun Chen Linfeng Zhang, EPIC Lab, SJTU Shanghai Jiao Tong University National University of Singapore, Duke University The University of Chicago Corresponding authors {shaobowang1009,zhanglinfeng}@sjtu.edu.cn Abstract Dataset Distillation (DD) seeks to create a compact dataset from a large, real-world dataset. While recent methods often rely on heuristic approaches to balance efficiency and quality, the fundamental relationship between original and synthetic data remains underexplored. This paper revisits knowledge distillation-based dataset distillation within a solid theoretical framework. We introduce the concepts of Informativeness and Utility, capturing crucial information within a sample and essential samples in the training set, respectively. Building on these principles, we define optimal dataset distillation mathematically. We then present InfoUtil, a framework that balances informativeness and utility in synthesizing the distilled dataset. InfoUtil incorporates two key components: (1) game-theoretic informativeness maximization using Shapley Value attribution to extract key information from samples, and (2) principled utility maximization by selecting globally influential samples based on Gradient Norm. These components ensure that the distilled dataset is both informative and utility-optimized. Experiments demonstrate that our method achieves a 6.1% performance improvement over the previous state-of-the-art approach on ImageNet-1K dataset using ResNet-18. Figure 1: Comparison of visualization results between previous method (a) RDED ( 26 ) and (b) our InfoUtil. Unlike prior methods relying on random selection and intuitive scoring, InfoUtil is both interpretable and theoretically grounded. It synthesizes images that more accurately capture semantically meaningful regions with principled scores. Prioritizing core content over irrelevant details like background elements ensures a more focused and meaningful representation. 1 Introduction Dataset distillation (DD) ( 33 ; 21 ) has emerged as a promising approach for enabling vision models to achieve performance comparable to training on large datasets, but with only a small set of synthetic samples. The core idea behind DD is to compress large datasets by synthesizing and optimizing a smaller, representative dataset. Models trained on distilled dataset are expected to match the performance of those trained on the original, larger dataset. Currently, two primary lines of approaches are used to tackle DD: i.e. , matching-based methods ( 33 ; 39 ; 40 ; 2 ; 43 ) , which aim to align the performance between the distilled dataset and the original dataset by matching gradients, features, distributions, or trajectories, and knowledge distillation-based methods ( 36 ; 23 ) , which decouple dataset distillation into two stages. In the first stage, the real data is compressed into a teacher model. In the second stage, the teacher model transfers knowledge to the distilled images through deep inversion-like methods ( 35 ) . Despite their success, these existing methods face two challenges: Challenge 1: Efficiency-Performance Trade-off. Most matching-based methods require significant GPU memory and time, making them impractical for real-world applications. For bi-level matching-based methods, the key challenge lies in the trade-off between performance and efficiency ( 40 ; 38 ; 13 ; 30 ; 8 ; 5 ) . For example, the state-of-the-art (SOTA) trajectory matching method ( 8 ) requires more than 4 NVIDIA A100 80GB GPUs to synthesize a 50 image-per-class (IPC) dataset on Tiny-ImageNet. Such high resource demands severely limit scalability of these methods, making it extremely challenging to apply to larger datasets like ImageNet-1K. Figure 2: InfoUtil’s pipeline for optimal dataset distillation involves two key steps: (i) Step 1 maximizes informativeness via the Shapley Value (a game-theoretic attribution method), retaining the most informative patches to form compressed samples. (ii) Step 2 maximizes utility by scoring these candidates with a judge model—using Gradient Norm (proven as a utility upper bound)—and retaining top samples. The final distilled dataset contains only the most informative, high-utility compressed samples. Image reconstruction and soft label generation phases are omitted here. For knowledge distillation-based methods, although they often perform better, the lack of a solid theoretical foundation impairs their interpretability ( 36 ; 23 ; 26 ) and prevents a principled solution. This limitation leaves practitioners with limited insight into why certain samples are selected for compression or how the distillation process relates to underlying data. Therefore, despite demonstrating impressive empirical results, they fall short in providing the transparency required for high-stakes or regulated applications. Challenge 2: Lack of Interpretability. Current methods are largely heuristic, lacking a principled framework to ensure the resulting distilled datasets are interpretable. To rethink previous methods within a principled framework, we reconsider the knowledge distillation-based dataset distillation process by introducing Optimal Dataset Distillation (Definition 4 ). The concept is built on Informativeness (Definition 1 ) and Utility (Definition 3 ) for desired distilled dataset. Intuitively, Informativeness captures essential information in each sample, while Utility reflects the importance of each sample for model training, whether included or excluded. Built on the theoretical framework, we propose InfoUtil, Info rmativeness and Util ity-enhanced Dataset Distillation (InfoUtil), a method that balances both aspects. As illustrated in Figure 2 , Step 1 focuses on extracting key information from each sample, compressing it into a representation that captures its most informative components. This is achieved by maximizing the game-theoretic informativeness of each sample, which we measure using the Shapley Value ( 24 ) , a principled attribution method first introduced in game theory. In Step 2, we maximize the utility of each sample, which is critical for model training. This is done by measuring the gradient norm of each sample and selecting those with the highest values, ensuring that only the most valuable samples are retained. The main contributions of this work are summarized as follows: 1. We propose Optimal Dataset Distillation (Definition 4 ), which builds on the concepts of patch-wise Informativeness and sample-wise Utility for distilled datasets. This approach addresses the lack of interpretability in existing methods by providing a solid theoretical framework. 2. We introduce InfoUtil, a novel method balancing informativeness and utility in distilled dataset synthesis. It employs game-theoretic informativeness maximization via the Shapley Value and utility maximization to retain the most informative and valuable samples using the Gradient Norm. 3. InfoUtil demonstrates outstanding performance across various models and datasets. For instance, our method yields a 16% improvement in performance over the previous state-of-the-art approach on the ImageNet-100, and a 6.1% improvement on ImageNet-1K. 2 Preliminaries Given dataset \mathcal{D}=\{(x_{i},y_{i})\}_{i=1}^{n} , dataset distillation (DD) aims to synthesize a smaller dataset \mathcal{\widetilde{D}}=\{(x_{j},y_{j})\}_{j=1}^{m} with m\ll n . The desired \mathcal{\widetilde{D}} should enable a model to achieve comparable, even lossless, performance to one trained on \mathcal{D} , evaluated on a held-out test dataset \mathcal{D}_{\mathrm{test}} . Specifically, for a model f parameterized by \theta trained with cross-entropy loss \ell , the condition is: \min_{\widetilde{\mathcal{D}}}\sum_{(x,y)\in\mathcal{D}_{\mathrm{test}}}|\ell(f_{\theta_{\mathcal{D}}}(x),y)-\ell(f_{\theta_{\mathcal{\widetilde{D}}}}(x),y)|, (1) where \theta_{\mathcal{D}} denotes the fixed parameters trained on \mathcal{D} . Crucially, \theta_{\mathcal{\widetilde{D}}} represents the parameters trained on the synthetic dataset \mathcal{\widetilde{D}} . Consequently, the term \ell(f_{\theta_{\mathcal{\widetilde{D}}}}(x),y) depends on \mathcal{\widetilde{D}} through the optimization trajectory of \theta . This paper focuses on knowledge distillation-based DD methods, which recently showed superior performance ( 36 ; 26 ; 23 ) . Here, \mathcal{D} ’s information is first learned by a teacher model f_{\theta_{\mathcal{D}}} , which then synthesizes \mathcal{\widetilde{D}} . A notable work, RDED ( 26 ) , uses random cropping to generate candidate patches, pruned via cross-entropy scoring. The final