MGD 3 : Mode-Guided Dataset Distillation using Diffusion Models

Jeffrey A. Chan-Santiago, Praveen Tirupattur, Gaurav Kumar Nayak, Gaowen Liu, Mubarak Shah · 2025

Read MGD 3 : Mode-Guided Dataset Distillation using Diffusion Models by Jeffrey A. Chan-Santiago, Praveen Tirupattur, Gaurav Kumar Nayak, Gaowen Liu, Mubarak Shah (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-18963.txt.

Introduction

Jeffrey Chan-Santiago and four co-authors present a 2025 paper on shrinking image data sets for fast model training. Distillation brews a tiny fake set that teaches a model nearly as well as the full pile. Past brewers leaned on image generators tuned with extra losses, and their fakes could still lack variety.

Their method steers a ready-made diffusion generator by the modes, or natural clusters, it finds in the data. One stage finds the clusters, one spreads samples within each class, and one trims glitches. Gains in accuracy topped four points on one test set, with no generator fine-tuning needed.

Catalog record

CREATORJeffrey A. Chan-Santiago, Praveen Tirupattur, Gaurav Kumar Nayak, Gaowen Liu, Mubarak Shah
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE51 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-18963

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

MGD 3 : Mode-Guided Dataset Distillation using Diffusion Models Jeffrey A. Chan-Santiago, Praveen Tirupattur, Gaurav Kumar Nayak, Gaowen Liu, Mubarak Shah Abstract Dataset distillation has emerged as an effective strategy, significantly reducing training costs and facilitating more efficient model deployment. Recent advances have leveraged generative models to distill datasets by capturing the underlying data distribution. Unfortunately, existing methods require model fine-tuning with distillation losses to encourage diversity and representativeness. However, these methods do not guarantee sample diversity, limiting their performance. We propose a mode-guided diffusion model leveraging a pre-trained diffusion model without the need to fine-tune with distillation losses. Our approach addresses dataset diversity in three stages: Mode Discovery to identify distinct data modes, Mode Guidance to enhance intra-class diversity, and Stop Guidance to mitigate artifacts in synthetic samples that affect performance. Our approach outperforms state-of-the-art methods, achieving accuracy gains of 4.4%, 2.9%, 1.6%, and 1.6% on ImageNette, ImageIDC, ImageNet-100, and ImageNet-1K, respectively. Our method eliminates the need for fine-tuning diffusion models with distillation losses, significantly reducing computational costs. Our code is available on the project webpage: https://jachansantiago.github.io/mode-guided-distillation/ \arxivPrintAffiliationsAndNotice 1 Introduction The rapid advancements in machine learning are marked by a trend towards increasingly large datasets and models to achieve state-of-the-art performance. However, this trend presents significant challenges for researchers constrained by limited computation and storage resources. In response, the research community started to focus on developing techniques to address these limitations. While model pruning ( Liu et al., 2017 ; He et al., 2019 ; Ding et al., 2019 ; Sharma & Foroosh, 2022 ) and quantization ( Wu et al., 2016 ; Chen et al., 2021 ; Chauhan et al., 2023 ; Xu et al., 2023 ) have been introduced to improve model efficiency, core set selection and dataset distillation ( Wang et al., 2018 ; Liu et al., 2022 ) have emerged as prominent techniques for reducing the size of training datasets to accelerate model training. Figure 1 : Optimization-based Dataset Distillation: Optimizes the distilled dataset to match the statistics of gradient/features of the Original Dataset. Generative Dataset Distillation: First, it learns the dataset distribution of the original dataset and then sample a dataset that approximates the original dataset distribution. The process of reducing the training dataset involves removing redundant information while retaining essential data. Core set selection ( Welling, 2009 ; Chen et al., 2010 ; Rebuffi et al., 2017 ; Castro et al., 2018 ) based approaches were initially introduced for building condensed datasets, which involves selecting a few prototypical examples from the original dataset to build the smaller dataset. However, these approaches are limited to choosing the samples from the original dataset, which considerably restricts the expressiveness of the condensed dataset. The task of dataset distillation is to distill information from a large training dataset into a smaller dataset with few synthetic samples such that a model trained on the smaller dataset achieves performance comparable to the model trained on the original dataset. Optimization-based dataset distillation methods follow the data matching framework ( Cazenavette et al., 2022 ; Cazenavette et al., 2023 ; Zhao & Bilen, 2023 ) , where the distilled dataset is updated to mimic the influence of the original dataset when training (see the top of Fig. 1 ). These methods minimize the distribution gap between the original and distilled datasets by considering different aspects, such as model parameters, long-range training trajectories, or feature distribution. However, these methods are far from optimal, as they need to repeat the execution of their method to synthesize distilled datasets of different sizes. In addition, they tend to generate out-of-distribution samples. To address these challenges, generative dataset distillation methods ( Wang et al., 2023 ; Zhang et al., 2023 ; Su et al., 2024 ) propose storing the knowledge of the dataset into the parameters of a generative model instead of directly condensing it into a smaller synthetic set (see the bottom of Fig. 1 ). Once trained, the same generative model can generate synthetic datasets of varied sizes. This typically, is achieved by training the generative model with representative and diversity losses. Among the generative models, diffusion models ( Ho et al., 2020 ) are known for their impressive capabilities in image synthesis. These models achieve perceptual quality comparable to GANs while offering higher distribution coverage, as evidenced by ( Dhariwal & Nichol, 2021b ) . However, they tend to concentrate on denser regions (modes) of the data distribution, resulting in a synthetic dataset that, while representative, often lacks the full diversity of the original data ( Gu et al., 2024 ) (refer to Fig. 2(a) ). Previous works ( Gu et al., 2024 ) address this by explicitly fine-tuning the model with representative and diversity losses to generate representative and diverse samples. With this fine-tuning, the samples are more likely to be generated from different modes of a class (See Fig. 2(b) ). However, this approach requires additional training, which can be computationally expensive and limit its practicality in resource-constrained settings. We propose a novel approach that extracts diverse and representative samples from a pre-trained diffusion model trained on the target dataset, without additional training or fine-tuning. Our method first estimates prevalent data modes in the Mode Discovery stage. Then, diversity is ensured by guiding each sample to a different mode with Mode Guidance . However, guiding samples to modes may reduce quality, so we introduce Stop Guidance to preserve synthetic data quality (see Fig. 2(c) ). In summary, the key contributions are: (a) DiT No Fine-tuning Low Diversity (b) MinMax Diffusion Requires Fine-tuning High Diversity (c) MGD 3 (Ours) No Fine-tuning High Diversity Figure 2 : Overview of the gradient field (score function) during the denoising process in latent diffusion for a specific class c . The original data distribution, marked by blue dots, shows denser regions (orange shadow) in the gradient field. To generate an image \hat{X}_{i} , noise {x_{T}}^{i}\sim N(0,\mathbf{I}) is sampled. In (a), a pre-trained diffusion model demonstrates imbalanced mode likelihood, leading to limited sample diversity and repeated modes. (b) shows MinMax Diffusion, which fine-tunes the model to enhance diversity by balancing mode likelihoods, but still faces redundancies based on initial noise conditions. (c), the proposed method introduces mode guidance in the denoising process (green and red traces), directing samples towards distinct modes (stars). After k steps of guidance, it transitions to unguided denoising (black trace), achieving high diversity and consistency without the need for fine-tuning. • A novel dataset distillation approach leveraging a pre-trained diffusion model without retraining or fine-tuning. • Improved diversity and representativeness compared to previous diffusion-based methods. • Matching or surpassing state-of-the-art results on multiple benchmarks while reducing computational cost. 2 Related Work Dataset distillation has received increased interest in recent years due to its applications in continual learning ( Zhao et al., 2021 ; Zhao & Bilen, 2021 ; Zhao & Bilen, 2023 ) , privacy-preserving datasets ( Li et al., 2020 ; Sucholutsky & Schonlau, 2021 ) , neural architecture search ( Zhao et al., 2021 ; Zhao & Bilen, 2021 ) , and model explainability ( Loo et al., 2022 ) . Prior works have explored the problem of dataset distillation and shown how challenging it is to encapsulate datasets in a

Read the complete text