On Learning Representations for Tabular Data Distillation
Inwon Kang [email protected], Parikshit Ram [email protected], Yi Zhou [email protected], Horst Samulowitz [email protected], Oshani Seneviratne [email protected] · 2025
Read On Learning Representations for Tabular Data Distillation by Inwon Kang [email protected], Parikshit Ram [email protected], Yi Zhou [email protected], Horst Samulowitz [email protected], Oshani Seneviratne [email protected] (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-13905.txt.
Introduction
Inwon Kang, Parikshit Ram, Yi Zhou, Horst Samulowitz, and Oshani Seneviratne distill tabular datasets into tiny synthetic stand-ins with TDColER, a column-embedding representation learner. Their paper from twenty twenty-five notes that distillation research hugged images while tables bring their own troubles: mixed feature types and non-differentiable favorite models like tree ensembles. Column embeddings give the framework a learnable handle on heterogeneous features.
A companion benchmark, TDBench, standardizes the testing. The evaluation is vast: hundreds of thousands of distilled sets and trained models across seven tabular learners. TDColER lifts off-the-shelf distillation quality by margins ranging from modest to dramatic. Smaller, safer training sets promise cheaper modeling with lower privacy and copyright exposure.
Catalog record
| CREATOR | Inwon Kang [email protected], Parikshit Ram [email protected], Yi Zhou [email protected], Horst Samulowitz [email protected], Oshani Seneviratne [email protected] |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 96 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2501-13905
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
On Learning Representations for Tabular Data Distillation Inwon Kang [email protected], Parikshit Ram [email protected], Yi Zhou [email protected], Horst Samulowitz [email protected], Oshani Seneviratne [email protected] Abstract Dataset distillation generates a small set of information-rich instances from a large dataset, resulting in reduced storage requirements, privacy or copyright risks, and computational costs for downstream modeling, though much of the research has focused on the image data modality. We study tabular data distillation, which brings in novel challenges such as the inherent feature heterogeneity and the common use of non-differentiable learning models (such as decision tree ensembles and nearest-neighbor predictors). To mitigate these challenges, we present TDColER , a tabular data distillation framework via column embeddings-based representation learning. To evaluate this framework, we also present a tabular data distillation benchmark, TDBench . Based on an elaborate evaluation on TDBench , resulting in 226,890 distilled datasets and 548,880 models trained on them, we demonstrate that TDColER is able to boost the distilled data quality of off-the-shelf distillation schemes by 0.5-143% across 7 different tabular learning models. 1 Introduction Dataset distillation or dataset condensation is the process of creating a small set of extremely informative samples (usually synthetic) from a large dataset such that a model trained on this set will have predictive performance comparable to that of a model trained on the original large dataset ( Wang et al., 2020 ; Yu et al., 2023 ) . First, data distillation reduces data storage costs and can mitigate the privacy and copyright concerns involved in keeping around (and continuously utilizing) large amounts of raw data. Furthermore, the reduction in the data size implies a lower computational cost of model training, especially when multiple models need to be trained on any given dataset. The above advantages of dataset distillation also facilitate various applications. Continual learning, where we need to learn new tasks while avoiding forgetting older tasks sequentially, often makes use of a “replay buffer” of old task data to be used while learning new tasks to mitigate forgetting of the older tasks ( Rolnick et al., 2019 ) . Dataset distillation reduces the memory overhead of this replay buffer, allowing learning of a larger number of tasks without forgetting ( Tiwari et al., 2022 ; Rosasco et al., 2022 ) . Figure 1 : Overview of TDBench . The benchmarking suite allows for flexible choice of datasets, distillation schemes, and downstream models that enables for modular evaluation of any new distillation method. In federated learning, we need to train a model using data spread across multiple clients without ever moving the data between clients and reducing the communication overhead. Dataset distillation generates compact yet private synthetic data from the client data that can be safely exchanged for communication-efficient model training ( Song et al., 2023 ; Goetz & Tewari, 2020 ; Zhou et al., 2020 ) . While dataset distillation has been widely studied for image datasets ( Cui et al., 2022 ; Yu et al., 2023 ) , the equally important application to other data modalities is limited. The problem of tabular data distillation has received very little attention, though many real-world learning problems and applications involve tabular data ( Guo et al., 2017 ; Clements et al., ; Borisov et al., 2024 ) . Various image data distillation schemes have been proposed in the literature, but their application to tabular data is not straightforward. First, all image data distillation schemes rely on the choice of a differentiable “backbone model.” While differentiable neural network-based schemes are standard for images, a wide variety of non-differentiable models are used with tabular data, such as decision tree ensembles, nearest-neighbor models, and kernel machines. Second, almost all data distillation methods for images generate distilled data in the original pixel space. While pixels are homogeneous raw features of an image, the features in tabular data can be extremely heterogeneous, creating a mismatch between what the image data distillation methods are designed for and what we have as an inherent property of tabular data. Finally, it is standard to use vision-specific data augmentation schemes (such as rotation, reflection, cropping, and translation) to train the model on the distilled image data. Such standard augmentations are not available for tabular data, thus creating another discrepancy in the expected conditions for the problem. (a) Overview of TDColER . The top describes a vanilla distillation scheme that only uses standard preprocessing techniques before distillation. The highlighted box describes the proposed TDColER , which uses column embeddings after such preprocessing and encoder-decoder architectures to generate rich compact representations. (b) Snapshot of downstream classifiers’ performance increase when trained on data distilled k -means with and without TDColER . Throughout our experiments, we observe a performance increase from 0.5% to as large as 143%. Figure 2 : Proposed approach – TDColER : T abular D istillation Via Col umn E mbeddings based R epresentation Learning Our contribution. In this paper, we study tabular dataset distillation and present a novel scheme to enhance the distilled data quality of multiple off-the-shelf data distillation schemes across various datasets, models, and distillation sizes. Specifically, we make the following contributions: • We propose T abular D istillation via Col umn E mbeddings based R epresentation Learning or TDColER that can utilize modern neural-network architectures such as Transformers and graph neural networks to generate rich compact representations. TDColER improves the quality of distilled data compared to existing distillation schemes. Figure 2(a) provides an overview of our proposed TDColER . • We present TDBench , a T abular D istillation Bench mark with 23 tabular datasets, 7 model classes, and 11 distillation schemes. We present an overview of TDBench , an extensible and modular framework for measuring various aspects of data distillation on tabular data, in fig. 1 . • With the elaborate evaluation of our proposed TDColER on TDBench , resulting in over 226,890 distilled datasets and 548,880 model trainings, we show that, on aggregate across all datasets, TDColER improves upon direct application of off-the-shelf distillation method on tabular data by 0.5-143% in terms of the distilled data quality across all models at the smallest distillation of 10 instances-per-class. Figure 2(b) presents a snapshot of our results. • Based on our thorough evaluation, we present various insights regarding tabular dataset distillation, such as (i) k -means clustering in the learned representations make for an extremely favorable distillation scheme, (ii) transformer-based tabular data representations obtain the highest distilled data quality on aggregate, while (iii) graph neural network based tabular data representations perform slightly worse than transformers but are significantly more parameter efficient. 1.1 Related Work Dataset distillation was introduced by Wang et al. (2020) as a bilevel optimization problem ( Feng et al., 2024 ) and has been widely studied in the context of image data distillation. Most methods can be categorized into approaches that match the original data by (i) backbone model performance, (ii) backbone model parameters, or (iii) backbone representation distributions ( Yu et al., 2023 ) . Wang et al. (2020) minimized performance differences between the original and distilled data, while Nguyen et al. (2021) introduced kernel-induced points (KIP) using kernel ridge regression with a neural tangent kernel ( Jacot et al., 2018 ) . Alternatively, methods have focused on parameter or gradient