Beyond Dataset Distillation: Lossless Dataset Concentration via Diffusion-Assisted Distribution Alignment
Tongfei Liu, Yufan Liu, Bing Li, Weiming Hu · 2026
Read Beyond Dataset Distillation: Lossless Dataset Concentration via Diffusion-Assisted Distribution Alignment by Tongfei Liu, Yufan Liu, Bing Li, Weiming Hu (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-27987.txt.
Introduction
Liu and three co-authors rethink dataset distillation, which shrinks big training sets into small synthetic ones. They prove that distillation equals distribution matching, and they expose a built-in efficiency limit of that approach. Their DsCo framework uses a diffusion model to build a compact sample set, with an optional step that mixes in real samples. DsCo works with or without access to the source data and nearly halves dataset size at high volumes with no loss.
Catalog record
| CREATOR | Tongfei Liu, Yufan Liu, Bing Li, Weiming Hu |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 112250 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.27987
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Beyond Dataset Distillation: Lossless Dataset Concentration via Diffusion-Assisted Distribution Alignment Tongfei Liu, Yufan Liu, Bing Li, Weiming Hu Abstract The high cost and accessibility problem associated with large datasets hinder the development of large-scale visual recognition systems. Dataset Distillation addresses these problems by synthesizing compact surrogate datasets for efficient training, storage, transfer, and privacy preservation. The existing state-of-the-art diffusion-based dataset distillation methods face three issues: lack of theoretical justification, poor efficiency in scaling to high data volumes, and failure in data-free scenarios. To address these issues, we establish a theoretical framework that justifies the use of diffusion models by proving the equivalence between dataset distillation and distribution matching, and reveals an inherent efficiency limit in the dataset distillation paradigm. We then propose a Dataset Concentration (DsCo) framework that uses a diffusion-based Noise-Optimization (NOpt) method to synthesize a small yet representative set of samples, and optionally augments the synthetic data via ”Doping”, which mixes selected samples from the original dataset with the synthetic samples to overcome the efficiency limit of dataset distillation. DsCo is applicable in both data-accessible and data-free scenarios, achieving SOTA performances for low data volumes, and it extends well to high data volumes, where it nearly reduces the dataset size by half with no performance degradation. keywords Generative Models, Diffusion Models, Dataset Condensation, Dataset Distillation, Data Augmentation † † equal-contributors: These authors contributed equally to this work. † † equal-contributors: These authors contributed equally to this work. 1 Introduction The establishment of large-scale visual recognition systems requires a large amount of high-fidelity data. However, the excessively large datasets incur prohibitive costs in model training and data storage, and their accessibility is often limited by transfer overhead and the growing privacy and security regulations. To address these problems, Dataset Distillation (Wang, 1811 56 ) (DD, also known as Dataset Condensation, DC) has emerged as a promising solution towards secure and efficient data utilization. It aims to synthesize an extremely small surrogate dataset for the original large target dataset while preserving its essential information. This significantly reduces the resource demands associated with training, storage, and transfer. Moreover, it also enables a privacy-preserving data-sharing paradigm by releasing a synthetic dataset instead of the original target dataset that may contain sensitive information, circumventing the data accessibility problem. Among existing dataset distillation methods, diffusion-model-based dataset distillation methods (Abbasi, 2024 1 ; Su, 2024 48 ; Du, 2023 16 ; Chen, 2025 5 ) that synthesize samples with pre-trained diffusion models (Rombach, 2022 41 ) have demonstrated state-of-the-art (SOTA) training performances and remarkable distillation efficiency on large-scale, high-resolution datasets, such as the image classification benchmark, ImageNet-1k (Deng, 2009 10 ). For instance, the Minimax-IGD (Chen, 2025 5 ) method has distilled the ImageNet-1k dataset into a synthetic dataset of 50 items per class (IPC), reducing about 95\% of the data with a performance degradation of no more than 10\% on the ResNet-18 (He, 2016 17 ) model. Despite the great success of the dataset distillation methods in reducing costs associated with large datasets and improving data accessibility, there remain some limitations in these methods. Firstly, the current dataset distillation methods typically work under extremely small surrogate set sizes of no more than 100 items per class (IPC) on large-scale datasets such as the ImageNet-1k due to increasing synthesis costs. It is unclear whether the dataset distillation paradigm can be extended to high-IPC settings on large-scale, high-resolution datasets, raising concerns about its consistency and reliability across different data regimes. Secondly, most methods require full access to the target dataset, failing in data-free scenarios where the target data is inaccessible for privacy or safety reasons. Thirdly, while the current SOTA methods use a pre-trained diffusion model to synthesize samples, the motivation for the use of diffusion models is based on heuristics and not analytically justified, raising doubts regarding the reliability and trustworthiness of the methods. This work addresses these limitations by establishing a theoretical framework to analyze the dataset distillation task. The framework demonstrates that the dataset distillation task is equivalent to a distribution matching problem under mild conditions. Since the diffusion model is trained for distribution alignment, the motivation for using diffusion models to synthesize surrogate datasets is justified. The theoretical analysis further reveals an inherent random sampling bias in diffusion-based sampling that causes a distribution misalignment that cannot be mitigated via guidance functions or fine-tuning. Moreover, an illustrative analysis under the proposed framework reveals a fundamental efficiency bottleneck in the dataset distillation paradigm that stems from the presence of ”far-apart” samples in the target dataset, making it inefficient to extend the dataset distillation methods to high-IPC settings. In light of these theoretical insights, we propose a D ata s et Co ncentration (DsCo) framework. It synthesizes a small dataset via a diffusion-based Noise-Optimization (NOpt) method with mitigated random sampling bias, which is applicable in both data-accessible and data-free scenarios. To overcome the fundamental efficiency limit of the dataset distillation paradigm, DsCo optionally augments the synthetic samples with selected ”far-apart” samples from the target dataset via a ”Doping” procedure, which is controlled by a ”Dope Trigger”. The adaptability of the DsCo framework ensures robust performance across various data volumes and various accessibility conditions. Experiments across multiple datasets demonstrate that DsCo achieves state-of-the-art performance across multiple datasets and settings, and a cost analysis is performed to demonstrate the superior efficiency of DsCo compared to existing open-source diffusion-based methods in both high-IPC and low-IPC regimes. Further, the extended high-IPC experiments demonstrate that the concentrated datasets achieve lossless dataset concentration performances even when reducing the dataset by half, demonstrating strong reliability for the large-scale compression. In data-free scenarios, it outperforms all existing data-free dataset distillation methods, offering a practical path to learning from inaccessible data while safeguarding privacy. Beyond technical metrics, this work has a significant broader impact: it enables efficient model training with reduced energy consumption, facilitates privacy-compliant data sharing while keeping the sensitive original dataset inaccessible, and enables effective surrogate dataset synthesis in the absence of the original dataset, contributing to global data accessibility. This work substantially extends our previous work: Noise-Optimized Distribution Distillation for Dataset Condensation (Liu, 2025 31 ), which is published in Proceedings of the 33rd ACM International Conference on Multimedia. The previous work identified the random sampling bias and proposed the Noise-Optimization framework to mitigate it in the data-free scenario. The data-free Noise-Optimization method was termed the ”NODD” method in our previous work. In this work, we make the following extensions in theory, methodology, and experiments: • We establish a new theoretical framework to analyze the task of dataset distillation, justifying the use of diffusion models by proving the equivalence of dataset distillation to distribution matching, identifying an inherent random sampling bias in diffusion synthesis, and proving a fundamental efficiency limit in dataset distillation. • We propose a unified DsCo framework that encloses the NODD proposed in our previous work (Liu, 2025 31 ) along with the newly proposed NOpt, Doping, and Dope Trigger to handle data-accessible and data-free scenarios under both high-IPC and low-IPC regimes. • We perform extensive evaluation and ablation experiments to demonstrate the high-IPC scalability, superior concentration performance, and compelling cost-efficiency of the DsCo framework. 2 Related Work 2.1 Diffusion Model The diffusion model is a generative model