Assessing Generative Models for Structured Data

Reilly Cannon, Nicolette M. Laird, Caesar Vazquez, Andy Lin, Amy Wagler, Tony Chiang · 2025

Read Assessing Generative Models for Structured Data by Reilly Cannon, Nicolette M. Laird, Caesar Vazquez, Andy Lin, Amy Wagler, Tony Chiang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-20903.txt.

Introduction

Fake spreadsheet data could free researchers from privacy worries, if the fakes behaved like the real thing. Reilly Cannon and co-authors test that hope directly, checking whether synthetic tables preserve the relationships between columns, not just whether models trained on them score well downstream. Their direct yardsticks measure how inter-column dependencies in generated data compare with the original.

The verdict deflates the hype: a large language model, whether prompted with examples or fine-tuned, and a leading adversarial generator all failed to mirror the real dependencies. Downstream-score testing had masked the gap. The authors argue synthetic data practice needs this sterner audit before hospitals or banks trust generated tables with real decisions.

Catalog record

CREATORReilly Cannon, Nicolette M. Laird, Caesar Vazquez, Andy Lin, Amy Wagler, Tony Chiang
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE45 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-20903

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Assessing Generative Models for Structured Data Reilly Cannon, Nicolette M. Laird, Caesar Vazquez, Andy Lin, Amy Wagler, Tony Chiang Abstract Synthetic tabular data generation has emerged as a promising method to address limited data availability and privacy concerns. With the sharp increase in the performance of large language models in recent years, researchers have been interested in applying these models to the generation of tabular data. However, little is known about the quality of the generated tabular data from large language models. The predominant method for assessing the quality of synthetic tabular data is the train-synthetic-test-real approach, where the artificial examples are compared to the original by how well machine learning models, trained separately on the real and synthetic sets, perform in some downstream tasks. This method does not directly measure how closely the distribution of generated data approximates that of the original. This paper introduces rigorous methods for directly assessing synthetic tabular data against real data by looking at inter-column dependencies within the data. We find that large language models (GPT-2), both when queried via few-shot prompting and when fine-tuned, and GAN (CTGAN) models do not produce data with dependencies that mirror the original real data. Results from this study can inform future practice in synthetic data generation to improve data quality. 1 Introduction Limited data availability for training machine learning models has resulted in new methods for generating synthetic examples to feed to these models. Tabular data, containing a mix of real-valued (continuous), ordinal, and categorical features, is the predominant data format for the highly specialized sub-fields of health, medicine, and the biological and social sciences, often suffers from limited availability or has additional restrictions due to privacy issues when the data concerns individuals. Large language models (LLMs) have become a popular choice for generating synthetic data [ 2 , 27 ] , largely replacing other synthetic data generation methods, such as tabular Generative Adversarial Networks (GANs) and Variational Auto-Encoders (VAEs) [ 35 ] . However, little work has been done on evaluating the quality of LLM-generated synthetic data and synthetic tabular data in general. Indeed, the most popular methods to evaluate synthetic tabular data quality, such as train-synthetic-test-real, indirectly measure data quality and utility. New methods and statistical techniques in general need to be developed to compare synthetic data to real data directly. This work proposes quantitative methods for evaluating the quality of synthetic tabular data, which we apply to assess the quality of synthetic data generated by LLMs. Our methods explore the relationships between columns in the tabular data, starting with marginal distributions, moving to pairwise dependencies, and then higher-order relationships in the features as measured through third and fourth-order joint cumulants. We use these quantities to measure the quality of tabular data generated from few-shot and fine-tuning methods. We also compare the LLM generated data to tabular data generated from a popular GAN-based method. Our results show that all of the tested generative methods fail at accurately reproducing the dependencies in the original data, even those of the training partition. On the surface (via marginal distributions) these generative models appear to replicate some of the real training data columns, but at higher-order relationships, both LLM and GAN methods fail to reproduce real data. Specifically, our LLM sampled via few-shot failed at producing second-order dependencies, while our GAN and fine-tuned LLM have mixed performance at the second, third, and fourth-order relationships. 2 Related work 2.1 Synthetic Data Generation In the absence of sufficient or high-quality data, synthetic samples can be generated to create additional examples that can be used in downstream analyses. In a perfect world, these synthetic examples directly mimic the distribution of the original data. Several methods have been developed to model and sample these distributions. We focus on two types of synthetic data generation methods — Generative Adversarial Networks (GANs) and Large Language Models (LLMs). 2.1.1 Generative Adversarial Networks (GANs) GANs are a popular approach for generating synthetic data. Typically, a GAN consists of two competing neural networks - a generator that produces new synthetic examples, and a discriminator which evaluates them against real examples [ 13 ] . Although originally created to generate images [ 13 ] , they have proven effective, with modifications, when applied to tabular data sets. For example, MedGAN [ 5 ] utilizes a GAN with an autoencoder to generate continuous or discrete data. In addition to the neural networks of the generator and discriminator, TableGAN [ 22 ] adds a third neural network called a classifier to prevent non-sensical values in the generated data. CTGAN [ 35 ] incorporates a GAN architecture with mode-specific normalization through variational Gaussian mixture models (VGM). This approach uses a conditional generator and training-by-sampling to handle imbalanced discrete columns while incorporating fully connected networks in both the generator and discriminator to capture all possible correlations between columns. CTAB-GAN [ 39 ] modifies the loss function and conditional vector from CTGAN, and its successor [ 40 ] allows for the handling of mixed discrete-continuous data. However, while GANs have shown considerable success with continuous data (like images), they struggle with discrete data, such as categorical variables. This is primarily due to the nature of how GANs operate, which is better suited to continuous variates and the quality of this generated data can vary significantly based on the architecture and training process [ 3 ] . This work uses CTGAN, a popular open-source model with commercial backing, as the class representative. 2.1.2 Large Language Models (LLMs) The rapid increase in the performance of LLMs in writing tasks has prompted interest in their use for synthetic data generation. Recent work has investigated the applicability of LLMs in generating synthetic tabular data. The authors of [ 2 ] propose a method for fine-tuning language models to output tabular data. Other work has shown that LLMs can be prompted to output tabular data without further training [ 27 ] . Others propose a token padding and compression procedure to reduce the computational cost of training LLMs on tabular data [ 38 ] . In this work we sample synthetic data generated from GPT-2 [ 26 ] , using both few-shot and fine-tuning methods. 2.2 Measuring synthetic data quality Measuring the quality of synthetic data is a challenging task, especially when generating high-dimensional mixed-type data, such as tabular data. The structured nature of tabular data makes it especially difficult to analyze and compare synthetic data distributions to real ones. Several papers have sought to develop metrics to analyze synthetic tabular data quality. For example, some [ 4 ] have investigated the validity of tabular electronic health records (EHR) data generated from GAN and other synthetic data generators and proposed measures for detecting validity and proper use of the data. They considered both predictive reliability and descriptive fidelity of the EHR data using a variety of metrics. Others have proposed a generative framework for tabular data and evaluate their framework using statistical similarity measures on individual features and utility measures based on training downstream classifiers [ 36 ] . One work studied the utility of synthetic tabular data in downstream tasks [ 19 ] . Others use embedding models to create vector representations of the text and image data

Read the complete text