Technical Report on Text Dataset Distillation
Keith Ando Ogawa, Bruno Lopes Yamamoto, Lucas Lauton de Alcantara, Victor Zacarias, Edson Bollis, Lucas Pellicer, Rosimeire Pereira Costa, Anna Helena Reali Costa, Artur Jordao · 2025
Read Technical Report on Text Dataset Distillation by Keith Ando Ogawa, Bruno Lopes Yamamoto, Lucas Lauton de Alcantara, Victor Zacarias, Edson Bollis, Lucas Pellicer, Rosimeire Pereira Costa, Anna Helena Reali Costa, Artur Jordao (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2512-03967.txt.
Introduction
Dataset distillation squeezes a huge training set into a small synthetic one that teaches models almost as well. Keith Ando Ogawa and colleagues survey how that idea has been adapted from images to text in a 2025 technical report. Vision research built a large literature first, while text lagged behind, partly because words are discrete and resist the smooth tricks used on pixels.
The report traces milestones: transformer-based methods, techniques generating genuinely discrete synthetic text, and scaling to decoder-only models over a billion parameters. It then catalogues the open problems honestly. Benchmarks are not yet standardised, complex tasks remain hard, and clear real-world applications are still scarce. The authors present the area as maturing rather than finished. Their review maps the main distillation strategies and the key contributions behind each, giving newcomers a structured entry point.
Catalog record
| CREATOR | Keith Ando Ogawa, Bruno Lopes Yamamoto, Lucas Lauton de Alcantara, Victor Zacarias, Edson Bollis, Lucas Pellicer, Rosimeire Pereira Costa, Anna Helena Reali Costa, Artur Jordao |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 22 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2512-03967
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Technical Report on Text Dataset Distillation Keith Ando Ogawa, Bruno Lopes Yamamoto, Lucas Lauton de Alcantara, Victor Zacarias, Edson Bollis, Lucas Pellicer, Rosimeire Pereira Costa, Anna Helena Reali Costa, Artur Jordao Abstract In the vision domain, dataset distillation arises as a technique to condense a large dataset into a smaller synthetic one that exhibits a similar result in the training process. While image data presents an extensive literature of distillation methods, text dataset distillation has fewer works in comparison. Text dataset distillation initially grew as an adaptation of efforts from the vision universe, as the particularities of the modality became clear obstacles, it rose into a separate branch of research. Several milestones mark the development of this area, such as the introduction of methods that use transformer models, the generation of discrete synthetic text, and the scaling to decoder-only models with over 1B parameters. Despite major advances in modern approaches, the field remains in a maturing phase, with room for improvement on benchmarking standardization, approaches to overcome the discrete nature of text, handling complex tasks, and providing explicit examples of real-world applications. In this report, we review past and recent advances in dataset distillation for text, highlighting different distillation strategies, key contributions, and general challenges. 1 Introduction In the data-driven scenario of deep learning, dataset distillation emerges as a method to distill the knowledge of a large real dataset into a small synthetic one that is approximately equivalent in the training process ( 29 ) . From a practical perspective, this technique improves dataset storage efficiency and reduces training time, contributing towards a more financially accessible and environmentally responsible way to develop machine learning models. Consequently, dataset distillation shows utility in applications that involve intensive training or constrained computational resources, such as neural architecture search ( 23 ) and on-device machine learning ( 9 ) . From a theoretical perspective, the study of how to condense large amounts of data into a few samples has the potential to answer key questions about training models, such as “ What defines good data ?” ( 20 ) . Even though large language models (LLMs) play important roles in several recent breakthroughs ( 15 ) , most of the literature on dataset distillation does not address them and concentrates on the vision domain ( 6 ) . In this work, we aim to present not only the big picture of text dataset distillation research, but also the current state of art in the text domain. We organize our report as follows: Section 2 introduces a taxonomy of methods; Section 3 describes the current state of art, milestones and characteristics of text dataset distillation; Section 4 addresses challenges and unexplored directions in text dataset distillation. Finally, Section 5 concludes this work with our impressions over the scarce literature. Figure 1: Taxonomy of Dataset Distillation Methods. 2 Taxonomy In this report, we adopt the four-class taxonomy of dataset distillation methods proposed by Sachdeva et al. ( 21 ) (Figure 1 ): Meta-Model Matching, Gradient Matching, Trajectory Matching and Distribution Matching. 2.1 Meta-Model Matching In their seminal work, Wang et al. ( 29 ) coin the term ”Dataset Distillation” to name the practice of distilling knowledge from a large training dataset into a small one. The authors solve dataset distillation through a bi-level optimization, in contrast with the training process, the proposed distillation optimization process tunes the synthetic data \tilde{x} . The objective of the optimization problem is to minimize the loss evaluated on real data of the model trained on synthetic data, as Equation 1 shows: \argmin_{\tilde{x}}\ell({x,\theta_{0}-\eta\nabla_{\theta_{0}}\ell({\tilde{x},\theta_{0}})}). (1) Specifically, given a loss function \ell , the solution to the problem is the synthetic data \tilde{x} that minimizes \ell over the real data x with the adjusted weight \theta_{0}-\eta\nabla_{\theta_{0}}\ell({\tilde{x},\theta_{0}}) . We consider Meta-Model Matching methods to be any dataset distillation technique that follow this framework of modeling the problem as a consecutive iteration to make the synthetic data match the performance of the real data on a specific loss. 2.2 Gradient Matching In contrast with Meta-Model Matching techniques, Gradient Matching methods distill data by forcing the gradient calculated on the synthetic data to be similar to the real one ( 32 ) . Formally, given a distance function D , the objective is to minimize the difference between the gradient for the loss \ell with respect to the model parameters \theta over the synthetic data \tilde{x} and real data x , as Equation 2 shows: \argmin_{\tilde{x}}D(\nabla_{\theta}\ell({\tilde{x},\theta}),\nabla_{\theta}\ell({x,\theta})). (2) An important characteristic of the Gradient Matching approach is that, due to its simpler problem formulation, it presents a lower computational cost compared to Meta-Model Matching. 2.3 Trajectory Matching Trajectory Matching methods aim to generate distilled data that leads to similar long-range learning dynamics of a train with real data ( 2 ) . To achieve this objective, these techniques compare segments of the parameter trajectory of training with synthetic data to the real data training trajectory. \argmin_{\tilde{x}}{E(\tilde{\theta}_{N},\theta_{M})}. (3) The previous Equation 3 shows how the approach is mathematically described at a distillation step. Specifically, given \theta_{0} as the initial parameter configuration, \tilde{\theta}_{N} as parameters updated using synthetic data for N steps, \theta_{M} as parameters updated using real data for M steps and E as a error function, the distillation process objective is to minimize the difference, or error, between the parameters trained with the real and synthetic data. Apart from theoretical details, as these techniques incorporate information about the training trajectory, their computational cost is naturally higher than Gradient Matching methods that observe one step at a time. 2.4 Distribution Matching Distribution Matching techniques build upon the hypothesis that datasets with similar distributions, according to a particular metric, lead to similar training processes ( 21 ) . Equation 4 describes the core purpose of these techniques. Specifically, given a distribution mismatch measure D , the goal is to find synthetic data \tilde{x} that minimize the distance to real data x . \argmin_{\tilde{x}}{D(\tilde{x},x)}. (4) In contrast to other classes of methods, distribution matching follows a single-level optimization by design, which makes it a more scalable approach ( 21 ) . 3 State of The Art, Milestones, Models and Datasets Figure 2: Timeline of milestones in text dataset distillation. From 2021 to 2025, the timeline highlights important events that shape the research area. Evolving from a simple extension of a vision work, state-of-the-art dataset distillation methods now leverage transformer models of over a billion parameters to generate cross-architecture readable distilled datasets that guarantee privacy of the original data. 3.1 Progression of Text Dataset Distillation Methods As an emerging research area, text dataset distillation evolved alongside image dataset distillation, incorporating and adapting its innovations. In the same work that introduces the use of soft labels, Sucholutsky et al. ( 24 ) introduce an early extension of these methods to the language domain. The authors successfully apply their use of label probability with the Meta-Model Matching approach to text embeddings, confirming that dataset distillation techniques are convertible to text data. Despite their theoretical and pioneering contributions, their work only addresses the distillation of classification datasets using CNN- and RNN-based architectures, which is inconsistent with modern use cases. Given the importance of pre-trained large language models to natural language processing, Maekawa et al.