Efficient Dataset Distillation via Diffusion-Driven Patch Selection for Improved Generalization

Xinhao Zhong, Shuoyang Sun, Xulin Gu, Zhaoyang Xu, Yaowei Wang, Min Zhang, Bin Chen · 2024

Read Efficient Dataset Distillation via Diffusion-Driven Patch Selection for Improved Generalization by Xinhao Zhong, Shuoyang Sun, Xulin Gu, Zhaoyang Xu, Yaowei Wang, Min Zhang, Bin Chen (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-09959.txt.

Introduction

Xinhao Zhong and colleagues distill huge image collections into tiny training sets by selecting patches with the help of a diffusion model, rather than generating new pictures. Dataset distillation promises models trained on a handful of synthetic examples, but generation-based versions drift away from the real data distribution and need repeated runs under different settings. Writing in twenty twenty-four, the team's method scores each real image patch by the diffusion model's prediction loss, flagging the most distinctive regions. Clustering and ranking inside each class then enforce variety among the chosen patches.

The whole distillation finishes in one pass, and the selected real patches carry no generator bias. Extensive experiments showed consistent gains over previous approaches, by the authors report. The framework is designed to combine with, not replace, diffusion-based distillation methods. It suits teams that must retrain often on a budget.

Catalog record

CREATORXinhao Zhong, Shuoyang Sun, Xulin Gu, Zhaoyang Xu, Yaowei Wang, Min Zhang, Bin Chen
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE45 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2412-09959

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Efficient Dataset Distillation via Diffusion-Driven Patch Selection for Improved Generalization Xinhao Zhong, Shuoyang Sun, Xulin Gu, Zhaoyang Xu, Yaowei Wang, Min Zhang, Bin Chen Abstract Dataset distillation offers an efficient way to reduce memory and computational costs by optimizing a smaller dataset with performance comparable to the full-scale original. However, for large datasets and complex deep networks (e.g., ImageNet-1K with ResNet-101), the extensive optimization space limits performance, reducing its practicality. Recent approaches employ pre-trained diffusion models to generate informative images directly, avoiding pixel-level optimization and achieving notable results. However, these methods often face challenges due to distribution shifts between pre-trained models and target datasets, along with the need for multiple distillation steps across varying settings. To address these issues, we propose a novel framework orthogonal to existing diffusion-based distillation methods, leveraging diffusion models for selection rather than generation. Our method starts by predicting noise generated by the diffusion model based on input images and text prompts (with or without label text), then calculates the corresponding loss for each pair. With the loss differences, we identify distinctive regions of the original images. Additionally, we perform intra-class clustering and ranking on selected patches to maintain diversity constraints. This streamlined framework enables a single-step distillation process, and extensive experiments demonstrate that our approach outperforms state-of-the-art methods across various metrics. 1 Introduction Figure 1 : Comparison of various methods for dataset distillation using diffusion models. (a) Fine-tuning a diffusion model pre-trained on ImageNet-1K to directly generate images. (b) Calculating the clustering center of the original images in latent space to generate images. (c) Identifying the regions in the real images most strongly associated with label text based on the implicit data distribution learned by the diffusion model. The images generated by our method more effectively represent the feature distribution of the original dataset. The rapid advancement of deep learning has driven impressive performance gains through increasingly deeper and more complex models trained on large-scale datasets He et al. 2016 ; Dosovitskiy et al. 2020 . However, training these models demands significant memory and computational resources. Dataset distillation has recently emerged as an effective approach to data compression Lei and Tao 2023 , offering substantial reductions in resource consumption. By optimizing images in pixel space through information matching (e.g., gradient matching), these optimization-based methods produce smaller synthetic datasets that can achieve comparable performance to the original data ( Zhao and Bilen 2021b ; Zhao and Bilen 2021a ; Cazenavette et al. 2022 ) . Despite these benefits, the vast pixel-level optimization space restricts the feasibility of these methods to small-scale datasets, e.g., CIFAR-10/100 ( Krizhevsky 2009 ) , thus limits their generalization. A recent approach, SRe 2 L ( Yin et al. 2024 ) , scales optimization-based methods to larger datasets (e.g., ImageNet-1K ( Deng et al. 2009 ) ) by using a pre-trained classifier to guide dataset synthesis. However, its strong dependence on proxy models during optimization often hinders the distilled dataset’s generalization, limiting broader applicability. Another critical limitation arises when handling various distillation configurations, such as different subsets of large datasets. Previous optimization-based methods typically incur substantial redundant costs due to repeated distillation requirements. Although recent works He et al. 2024b ; He et al. 2024a aim to address this issue, their applicability remains confined to small-scale datasets. To address the limitations of current dataset distillation methods, recent approaches Sun et al. 2024 ; Gu et al. 2024 ; Su et al. 2024 that avoid direct image optimization have shown promise for scaling dataset distillation to larger datasets and more complex model architectures. RDED Sun et al. 2024 leverages a pre-trained classifier (e.g., ResNet-18 He et al. 2016 ) to select patches randomly cropped from real images and then concatenate them into synthetic images. While RDED achieves substantial performance improvements, it requires careful model selection and retraining of a classifier tailored to the specific dataset. Moreover, concatenated images often lack complete class-relevant features, especially for complex models with higher learning capacity. Minimax Gu et al. 2024 addresses distillation by pruning the DiT model Peebles and Xie 2023 to generate synthetic datasets using regularization based on dataset representativeness and diversity. However, Minimax relies on a diffusion model pre-trained on ImageNet-1k Deng et al. 2009 , requiring fine-tuning for different target datasets, which limits flexibility. D 4 M Su et al. 2024 applies a pre-trained text-to-image Latent Diffusion Model (LDM) Rombach et al. 2022 to generate synthetic datasets by clustering original images in latent space. However, without constraints on distributional differences between the LDM’s pre-training data (e.g., LAION Schuhmann et al. 2022 ) and the target dataset (e.g., ImageNet-1K), D 4 M can produce class-irrelevant or noisy images. To overcome these limitations, we propose a novel dataset distillation paradigm that leverages diffusion model’s prior to select the most informative image patches. Specifically, for each input image from the original dataset, we first predict the noise map using the pre-trained LDM, with and without the label text prompt, calculating losses for each scenario. The differential loss quantifies representativeness, enabling us to crop patches most relevant to the label guided by the target dataset’s data distribution. Additionally, we can effectively constrain the implicit data distribution learned by the diffusion model by using the pre-trained LDM to identify class-pertinent regions. To mitigate the inherent diversity of diffusion models, we apply clustering to capture visual features most representative of each class. Specifically, we compute diffusion features for all patches inspired by recent work ( Tang et al. 2023 ) , allowing for efficient clustering and ranking of cropped patches. Fig. 1 illustrates a comparison between our method and previous diffusion-based generative distillation methods. Extensive experiments demonstrate that our approach achieves state-of-the-art (SOTA) performance on large-scale datasets, supporting an efficient, unrestricted one-step distillation process. Our contributions are summarized as follows: • We introduce a novel dataset distillation framework that diverges from conventional methods by leveraging pre-trained diffusion models in an orthogonal way. Our approach uniquely addresses distributional discrepancies between the diffusion model and target dataset, offering a new perspective on improving dataset fidelity and applicability. • Our proposed paradigm enables a highly efficient, one-step distillation process, eliminating the need for model training and fine-tuning. Additionally, it avoids repetitive distillation across subclass combinations or compression ratios, which substantially broadens the potential applications and flexibility of dataset distillation. • Extensive experiments demonstrate that our method consistently surpasses existing training-free approaches, particularly on large-scale datasets, underscoring the effectiveness and robustness of our framework in various challenging settings. 2 Related Work Figure 2 : (a): Stage I leverages image and corresponding differential class text prompts as inputs to the diffusion model to select the regions that most represent class-relevant features. (b): Stage II first computes DIFT of the patches and performs clustering to aggregate the most representative visual elements for each class. The evaluation process utilizes soft labels provided by a pre-trained teacher model. Rather than existing methods employing diffusion models, our proposed method maximizes the restoration of the original data distribution. 2.1 Dataset Distillation Dataset distillation was initially regarded as a meta-learning problem Wang et al. 2018 . It involves minimizing the loss function of the synthetic dataset through a model trained on this dataset. Since then, several approaches have been proposed to

Read the complete text