Exploring Transformer Placement in Variational Autoencoders for Tabular Data Generation

Aníbal Silva, Moisés Santos, André Restivo, Carlos Soares · 2026

Read Exploring Transformer Placement in Variational Autoencoders for Tabular Data Generation by Aníbal Silva, Moisés Santos, André Restivo, Carlos Soares (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-20854.txt.

Introduction

Aníbal Silva and three co-authors test where transformers belong inside a classic generative model. Their 2026 paper concerns tabular data generation. Tables are a hard domain for generative models. The standard variational autoencoder, or VAE, is built from multilayer perceptrons. It struggles to model links between features. Mixed data types, such as numbers beside categories, make the task worse.

Transformers offer a contrast. Their attention mechanism is well suited to complex feature interactions. The authors explore transformer placement within the VAE design. Placement matters: the encoder, the decoder, or both could host attention. The study weighs which spot gives the best tables. Its results guide builders of synthetic data tools for business and science.

Catalog record

CREATORAníbal Silva, Moisés Santos, André Restivo, Carlos Soares
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE42 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2601-20854

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Exploring Transformer Placement in Variational Autoencoders for Tabular Data Generation Aníbal Silva, Moisés Santos, André Restivo, Carlos Soares Abstract Tabular data remains a challenging domain for generative models. In particular, the standard Variational Autoencoder (VAE) architecture, typically composed of multilayer perceptrons, struggles to model relationships between features, especially when handling mixed data types. In contrast, Transformers, through their attention mechanism, are better suited for capturing complex feature interactions. In this paper, we empirically investigate the impact of integrating Transformers into different components of a VAE. We conduct experiments on 57 datasets from the OpenML CC18 suite and draw two main conclusions. First, results indicate that positioning Transformers to leverage latent and decoder representations leads to a trade-off between fidelity and diversity. Second, we observe a high similarity between consecutive blocks of a Transformer in all components. In particular, in the decoder, the relationship between the input and output of a Transformer is approximately linear. † † paperid: 8105 1 Introduction Deep Learning has been thoroughly investigated over the last decades and has been successfully applied to various learning tasks. Generative modeling is no exception. Generally, this class of models aims to estimate the underlying distribution of the data. Existing generative flavors encompass variational inference [ 18 , 11 , 33 ] , generative adversarial networks [ 8 ] and score-based matching [ 37 ] . Despite mostly focusing on data modalities such as image [ 2 ] and text [ 24 ] , there has been a recent surge of interest in generative models for tabular data. The interest lies in generating synthetic data to overcome challenges such as data scarcity, missing-value imputation, and individual privacy-preserving (see e.g., [ 30 ] for a thorough review). Challenges in tabular data generation Modeling the joint distribution of tabular data has unique challenges. The main research interest in the generative model is on images, and, usually, the theory behind it assumes a continuous distribution of the data. This is not true for tabular data, which generally presents a mixture of both continuous and discrete variables. Moreover, continuous variables might exhibit several modes, and discrete variables may have a considerable number of categories, imposing additional challenges on the capability of a neural network to learn relationships between these two different types of data adequately. Tokenization A data point of a tabular dataset is usually represented as a heterogeneous vector composed of numerical and discrete features. One possible solution to overcome this heterogeneity is to embed each feature into an embedding matrix via tokenization [ 9 , 44 ] . In essence, this transformation linearly projects each feature into a continuous vector. Transformers The Transformer architecture [ 39 ] was initially proposed for machine translation and later applied to text generation [ 31 ] . Given its unprecedented success, adaptations have been made to this architecture in the past few years for images [ 15 ] , time series [ 41 ] , and, naturally, tabular data [ 13 , 36 , 44 ] . In the tabular domain, the purpose of the Transformer architecture is to capture meaningful relations between feature representations of the data via attention mechanisms. Motivation Transformers are becoming a fundamental architectural block to model feature interactions in tabular data on different learning paradigms [ 36 , 9 , 44 ] . Typically, they operate on a “raw” level of representation, i.e., at the data input level. In this work, we question its use to leverage abstract representations of a Neural Network, i.e., feature representations obtained by fully connected layers followed by nonlinearities. A prominent architecture for this study is the Variational Autoencoder (VAE), which encompasses three distinct kinds of representations: 1) an input representation that is fed into the recognition model (encoder); 2) a latent representation modeled via the statistics obtained by the recognition model; and 3) a reconstructed representation obtained by the generative model (decoder) (see Fig. 1 ). Fully Connected (FC) layers, which are affine maps, do not capture high-order dependencies between feature representations that may become important at deeper feature representations. The Transformer attention mechanism provides a suitable alternative to model these dependencies. Our Contribution This work evaluates the impact of Transformers in a VAE for tabular data generation. We aim to answer the following question — What’s the impact of integrating Transformers into different components of a VAE? Starting with a VAE without Transformers, we study the effect of leveraging it over the aforementioned representations. In total, 6 variations are considered. This evaluation is performed by considering metrics that evaluate the statistical properties of the synthetic data with respect to the real data and through a Machine-Learning utility perspective. In addition, Center Kernel Alignment (CKA) [ 20 ] is used to compare similarities between feature representations on different components of the architectures. Our experiments are conducted using 57 datasets from the OpenML CC18 suite [ 3 ] . From an evaluation perspective, our study reveals a trade-off between fidelity and diversity of the synthetic data with respect to real data. Specifically, we observe that as Transformers are added into the architecture, synthetic data tends to be less faithful to real data, while its diversity increases, with the biggest gain obtained when leveraging the latent and output representations with Transformers. From a representational perspective, our findings reveal that the input and output of Transformers at both the encoder and decoder tend to present a high degree of similarity. Moreover, at the decoder, the Transformer appears to act as the identity function due to little or no effects in representational changes in residual connections. We observe that the reason for this effect is due to layer normalization, which shifts and scales the initial representation such that it offers no representational changes. The paper is organized as follows: Section 2 reviews related work. Section 3 introduces the necessary formulations and provides an overview of the considered VAE models for the study. The experimental setup is outlined in Section 4 , followed by an analysis of the experimental results in Section 5 . Section 6 presents a detailed study on the similarities between feature representations at different levels of the architectures. Finally, conclusions are drawn in Section 7 . 2 Related Work Recent advances in tabular data generation draw upon a variety of deep generative approaches. Generative Adversarial Networks (GANs) such as CTGAN [ 43 ] and its successors CTAB-GAN [ 45 ] and CTAB-GAN+ [ 46 ] adapt GAN-based frameworks to handle continuous and categorical features. Meanwhile, Diffusion Models [ 35 , 11 ] have been specialized for tabular data through methods like TabDDPM [ 21 ] , CoDi [ 22 ] , StaSy [ 16 ] , TabSyn [ 44 ] , and TabDiff [ 34 ] , each proposing distinct strategies to manage mixed data types and improve generative quality. Autoregressive models like GReaT [ 4 ] and TabMT [ 10 ] have also been considered in the tabular data generation framework. Flow-Matching [ 25 ] has led to novel gradient-boosting-based generative solutions [ 14 ] . Finally, Variational Autoencoders (VAEs) [ 18 ] adaptations include TVAE [ 43 ] , introduced alongside CTGAN, and VAEM [ 27 ] , which consists of a two-stage training —

Read the complete text