Breaking the Quality–Privacy Tradeoff in Tabular Data Generation via In-Context Learning

Xinyan Han, Yan Lu, Xiaoyu Lin, Yuanyuan Jiang, Yuanrui Wang Xuanyue Li Wenchao Zou Xingxuan Zhang · 2026

Read Breaking the Quality–Privacy Tradeoff in Tabular Data Generation via In-Context Learning by Xinyan Han, Yan Lu, Xiaoyu Lin, Yuanyuan Jiang, Yuanrui Wang Xuanyue Li Wenchao Zou Xingxuan Zhang (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-04911.txt.

Introduction

Xinyan Han and seven co-authors challenge a believed trade-off in synthetic table data: that better quality usually means weaker privacy. With small datasets, generators trained from scratch can memorise real rows instead of learning general patterns. Their method, DiffICL, instead treats generation as an in-context learning problem.

DiffICL draws on structural knowledge pretrained across a large collection of datasets, so it infers a new table's distribution from limited context rather than memorising samples. On fourteen real datasets it improved both quality and privacy at once. The synthetic tables also worked well as extra training data, suggesting the trade-off can be softened by better training designs.

Catalog record

CREATORXinyan Han, Yan Lu, Xiaoyu Lin, Yuanyuan Jiang, Yuanrui Wang Xuanyue Li Wenchao Zou Xingxuan Zhang
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE85 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.04911

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Breaking the Quality–Privacy Tradeoff in Tabular Data Generation via In-Context Learning Xinyan Han, Yan Lu, Xiaoyu Lin, Yuanyuan Jiang, Yuanrui Wang Xuanyue Li Wenchao Zou Xingxuan Zhang Abstract Tabular data synthesis aims to generate high-quality data while preserving privacy. However, we find that existing tabular generative models exhibit a clear tradeoff in the small-data regime: improving data quality typically comes at the cost of increased memorization of training samples, thereby weakening privacy protection. This tradeoff arises because small training sets make it difficult for dataset-specific generative models to distinguish generalizable structure from sample-specific patterns. To address this, we propose DiffICL , which formulates tabular data generation as an in-context learning problem. Instead of fitting each dataset from scratch, DiffICL leverages pretrained structural priors learned from a large collection of datasets, enabling it to infer data distributions from limited context rather than memorizing individual samples. We evaluate DiffICL on 14 real-world datasets. Results show that DiffICL improves both data quality and privacy, and generate synthetic data that provides effective data augmentation. Our findings suggest that the quality–privacy tradeoff can be improved through better training paradigms. 1 Introduction Figure 1 : Quality–privacy tradeoff frontiers. Tabular data is widely used across high-stakes domains such as healthcare, finance, and public administration, where data sharing is often hindered by privacy concerns and regulatory constraints 8 ; 6 ; 29 ; 7 ; 25 . This limitation creates a strong demand for privacy-preserving data sharing mechanisms, making tabular data synthesis a promising solution. Tabular data synthesis involves two key objectives: generating high-quality samples that faithfully capture the underlying data distribution to ensure strong downstream utility, and providing robust privacy guarantees to protect sensitive individual-level information 10 ; 34 . To achieve these goals, deep generative models have been widely adopted to model the complex and heterogeneous distributions of tabular data. Existing approaches can be broadly categorized into three main lines of work: GAN-based methods 44 ; 30 ; 43 ; 48 ; 49 , VAE-based methods 43 ; 22 ; 2 , and the recently popular Diffusion-based methods 16 ; 18 ; 45 ; 19 ; 20 ; 36 . Despite the recent progress in tabular data generation, achieving a favorable tradeoff between data quality and privacy remains particularly challenging in small-data regimes. To characterize this issue, we evaluate representative methods from different paradigms—tabDDPM 18 , TVAE 43 , and CTGAN 43 —on the Adult 17 dataset with N=200 training samples. For each baseline, we vary the number of training steps to examine how generation quality and privacy protection change during optimization in Figure 1 . Each point represents a model checkpoint obtained after a specific number of training steps. Darker points correspond to later checkpoints. Diffusion-based ( blue ) models exhibit strong fitting capacity and can achieve high sample quality, yet this comes at the cost of severe overfitting: they generate samples that are overly close to the training records. This memorization can reveal the presence of individual training records or expose record-specific information, directly weakening privacy. In contrast, VAE-based ( yellow ) and GAN-based ( green ) methods introduce stronger architectural regularization that mitigates memorization, but their limited expressiveness yields substantially inferior generation quality. Under limited data, existing approaches fail to simultaneously achieve high fidelity and strong privacy protection, instead trading one for the other along a suboptimal Pareto front. This tradeoff arises because limited samples provide only a noisy and incomplete view of the underlying distribution. Due to the curse of dimensionality, small datasets provide only sparse coverage of the data space, making reliable inference of global distribution difficult. Figure 2 : ICL pretraining enables more accurate density estimation from limited data by learning transferable structural priors. Left : standard dataset-specific generative model overfits sparse observations and yields a poor estimate of the true distribution. Right : at the pretraining stage, the model learns to generate query samples D_{\text{qry}} conditioned on context samples D_{\text{ctx}} , which requires inferring the underlying distribution instead of copying the context; the diverse distributions encountered during pretraining provide the priors. At the inference stage, the model treats the real dataset as context, infers the target distribution from D_{\text{ctx}} and generates new synthetic samples. To address this limitation, we turn to in-context learning (ICL) pretraining. In predictive tasks, tabular foundation models 12 ; 14 ; 23 ; 9 ; 47 ; 46 have demonstrated that ICL pretraining encodes rich structural priors into the network, enabling strong performance on small context sets without task-specific retraining. We argue that leveraging such pretrained priors is the key to solving the memorization problem of generative models in data scarce scenarios: Standard generative models optimize directly on the empirical data; with small sample sizes, deceptively low loss can be easily achieved through simple memorization. ICL pretraining removes the memorization incentive: each dataset is randomly partitioned into a context set D_{\text{ctx}} and a disjoint query set D_{\text{qry}} , and the model is trained to generate D_{\text{qry}} conditioned on D_{\text{ctx}} . Because D_{\text{ctx}} and D_{\text{qry}} are i.i.d., memorizing D_{\text{ctx}} cannot yield a lower pretraining loss than genuinely learning the underlying true distribution. The model is thus driven to infer the true distribution rather than memorize the context, acquiring a prior for "distributions given a context." At inference, the scarce real dataset is fed as D_{\text{ctx}} , enabling generation that is faithful to the context distribution yet distinct from individual records, thereby improving privacy. We implement this idea in DiffICL. To handle the heterogeneity of tabular data—where tables differ in size, column semantics, and types—DiffICL projects raw features into a unified latent space via a tabular foundation model, then trains a dual-axis attention Transformer to perform conditional generation over these latent embeddings. As shown in Figure 1 ( red ), DiffICL establishes a superior Pareto frontier over existing paradigms. We summarize our main contributions as follows: 1. To the best of our knowledge, we are the first to formulate tabular data generation as an in-context learning problem. We propose DiffICL, which avoids the memorization incentive that arises in standard generative training by optimizing an ICL pretraining objective (Section 3 ). 2. We evaluate DiffICL on 14 real-world datasets covering both classification and regression tasks. DiffICL achieves markedly better data quality and privacy protection than representative methods from each major generative paradigm on average (Section 4.2 ). 3. DiffICL generates synthetic data that serves as effective data augmentation, improving downstream task performance when added to real training data (Section 4.3 ). Ablation results further show that this benefit depends on the structural priors acquired during pretraining, which guide the generation process at inference time and produce more informative augmentation samples (Section 4.4 ). 2 Quality–Privacy Tradeoff in Tabular Generative Models Figure 3 : Training dynamics and quality–privacy tradeoffs across dataset sizes ( N=200,400,600 ) on the Adult dataset. Top and middle rows show the evolution of generation quality (AUC) and privacy (DCROverfit) over training steps for CTGAN, TVAE, and tabDDPM. Bottom row plots the resulting quality–privacy frontiers, where each point corresponds to a training checkpoint. In this section, we examine the tradeoff between generation quality and privacy risk on the Adult dataset with N\in\{200,400,600\} . We evaluate multiple generative models at different stages of training. Generation quality is

Read the complete text