CardiCat: a Variational Autoencoder for High-Cardinality Tabular Data
Lee Carlin, Yuval Benjamini · 2025
Read CardiCat: a Variational Autoencoder for High-Cardinality Tabular Data by Lee Carlin, Yuval Benjamini (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-17324.txt.
Introduction
Lee Carlin and Yuval Benjamini published a research paper in 2025 on a generative model for tables whose category columns explode with distinct values, like product codes or place names. Standard encodings for such high-cardinality features either balloon or blur them, and mixed-type tables add imbalance on top. Their variational autoencoder swaps one-hot encoding for learned embedding layers on both sides.
The embeddings are trained jointly and can depend on the row's other values, giving a compact shared language for categories. The model needs far fewer trainable parameters than rivals, so it scales. On real and simulated data it produced synthetic tables that preserved rare and imbalanced categories better than competing generators.
Catalog record
| CREATOR | Lee Carlin, Yuval Benjamini |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 33 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2501-17324
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
CardiCat: a Variational Autoencoder for High-Cardinality Tabular Data Lee Carlin, Yuval Benjamini Abstract High-cardinality categorical features are a common characteristic of mixed-type tabular datasets. Existing generative model architectures struggle to learn the complexities of such data at scale, primarily due to the difficulty of parameterizing the categorical features. In this paper, we present a general variational autoencoder model, CardiCat, that can accurately fit imbalanced high-cardinality and heterogeneous tabular data. Our method substitutes one-hot encoding with regularized dual encoder-decoder embedding layers, which are jointly learned. This approach enables us to use embeddings that depend also on the other covariates, leading to a compact and homogenized parameterization of categorical features. Our model employs a considerably smaller trainable parameter space than competing methods, enabling learning at a large scale. CardiCat generates high-quality synthetic data that better represent high-cardinality and imbalanced features compared to competing VAE models for multiple real and simulated datasets. 1 Introduction Recent years have seen dramatic improvement in the generative modeling of complicated stimulus including images, audio and most recently natural languages. Generative models characterize the joint distribution of the variables, and allow sampling from this distribution. They can therefore be used for imputing missing values, regenerating data while hiding sensitive information, and detecting anomalies. Variational Auto Encoders (VAEs, Kingma & Welling, 2013 ) are generative models composed of two neural-networks: the encoder transforms a data example into a distribution on the p -dimensional latent space, and the decoder transforms a vector sampled from the latent space into a data example. The sampling increases the smoothness of the decoder, and the result is often more interpretable and better reflects the diversity of the original data. Generative models and VAEs in particular are most successful when the data they are trained on is homogeneous, meaning that the individual features are similarly distributed, and there are local structures governing the interaction between features (e.g. pixels or words) ( Ma et al., 2020 ; Suzuki & Matsuo, 2022 ) . In contrast, generative models still struggle when modeling one of the most common types of datasets, the tabular data ( Nazabal et al., 2020 ) . In tabular datasets, each example can be comprised of a set of data-fields from different types ( mixed types ), including numerical, integer, and character strings. In addition, the features can exhibit complex dependencies, but the order or topology is often arbitrary and provides little information regarding this structure (in contrast to homogeneous signals such as images or language). Prominent examples include electronic medical records (EMR), personal credit default, e-commerce and behavioral datasets from social networks. High cardinality categorical features is another pervasive characteristic of mixed-type tabular data. The cardinality of some categorical features, meaning the number of possible unique values, can be extremely high, with severe imbalances between the values ( Xu & Veeramachaneni, 2018 ) . Such high-cardinality categorical features are often very informative - consider the information on a patient contained in the in features such as "diagnosis", "occupation" or "city / state". For a generative model to learn the behavior of a high-cardinality feature, the interplay between different values of the feature need to be identified. Often, the decoders prefer not to guess rare values, and this leads to a collapse in the marginal distributions towards the common categories. More technically, current models usually require categorical features to be initially parameterized by one-hot encoding or string similarity encoding such as in Cerda & Varoquaux, 2020 , which relies on the information between pairs of categories. Yet, one-hot encoding high-cardinality features have many detrimental effects on neural networks ( Rodríguez et al., 2018 ; Avanzi et al., 2024 ) . One-hot encoding forces a flat label space (where each value is equally different), which might disregard the complex relationships that exist amongst the values within a categorical variable. One-hot encoding also dramatically increases the dimension of inner layers of the network, leading to poor statistical properties, memory constraints and a strain on the optimization process ( Choong & Lee, 2017 ) . Figure 1 : Top: bar plots show deviation between generated categorical probabilities from three models (CardiCat in blue, VAE in green, tVAE in yellow) and true probabilities on three features. Shorter bars mean better reconstruction. TV score is in the legend. Bottom: the box plots show the conditional distribution of a numerical variable given a categorical one. The true distributions are in black, and the generated ones are in blue (CardiCat) and yellow (tVAE). Missing box plots means that categorical value was not sampled. Overall CardiCat better reconstructs the marginal and bi-variate distributions. We propose a VAE-based framework for the modeling and synthesis of tabular data; one that can efficiently represent high-cardinality categorical features in the context of mixed feature types. Our framework, CardiCat, adds embedding layers for the high-cardinality categorical features. The learned embedding layers are low-dimensional numeric vector representations of the values of the categorical features. CardiCat’s embedding layers are efficiently learned in tandem as part of the complete encoder-decoder network optimization. Therefore, the topology of the embedding layer can be influenced by the the joint-distribution of the categorical features and the other features in the data. A key innovation is that the decoded output (and likelihood) of these features is evaluated in the smooth and homogenized embedding space rather than in the observed categorical space. This allows us to avoid all together the need to one-hot encode the non-binary categorical features at any point in the process, thus reducing the number of trainable parameters significantly, and feeding the network more homogenized data. As we show in our experiments, this encoding allows the VAE to better recover the marginal and joint distributions of categorical features (see Figure 1 ). Our contributions can be summarized as follows: (1) We propose a method to introduce regularized categorical embedding into VAEs in a way that efficiently homogenizes categorical features and prevents embedding layers collapse. We show that our method’s capacity to represent and accurately reconstruct the marginal and multivariate trends of the data surpasses that of previous models. In addition, it requires considerably fewer learned parameters; (2) We develop a public benchmarking framework for tabular VAEs composed of simulated and real-life tabular datasets emphasizing high-cardinality categorical features; and (3) we make available an open-source implementation of the regularized embeddings architecture that can easily be adapted to different VAE frameworks. The use of embedding layers to represent categorical features in neural networks is not new. However, CardiCat’s embeddings architecure is distinct due to its (a) CardiCat’s loss calculation is done directly on the embedding space, (b) The dual-embeddings used in CardiCat is trained to have the same embedding weights between the encoder and decoder (hence the "dual"), and (c) CardiCat’s avoids one-hot encoding the categorical features all together. To our knowledge, this is the first public and benchmarked end-to-end tabular generative model that uses dual-architecture regularized embedding layers for parameterizing categorical features. 2 Variational autoencoders for tabular data This section provides a brief overview of VAEs and their shortcomings as they relate to heterogeneous data. It then discusses the main strategies for accommodating diverse data types. 2.1 Variational autoencoders Variational autoencoders are deep latent-variable generative models that are based on autoencoders (AE) ( Kingma