DP-2Stage: Adapting Language Models as Differentially Private Tabular Data Generators Tejumade Afonja tejumade.afonja@cispa.de, Hui-Po Wang hui.wang@cispa.de, Raouf Kerkouche raouf.kerkouche@cispa.de, Mario Fritz fritz@cispa.de Abstract Generating tabular data under differential privacy (DP) protection ensures theoretical privacy guarantees but poses challenges for training machine learning models, primarily due to the need to capture complex structures under noisy supervision signals. Recently, pre-trained Large Language Models (LLMs) – even those at the scale of GPT-2 – have demonstrated great potential in synthesizing tabular data. However, their applications under DP constraints remain largely unexplored. In this work, we address this gap by applying DP techniques to the generation of synthetic tabular data. Our findings shows that LLMs face difficulties in generating coherent text when fine-tuned with DP, as privacy budgets are inefficiently allocated to non-private elements like table structures. To overcome this, we propose DP-2Stage, a two-stage fine-tuning framework for differentially private tabular data generation. The first stage involves non-private fine-tuning on a pseudo dataset, followed by DP fine-tuning on a private dataset. Our empirical results show that this approach improves performance across various settings and metrics compared to directly fine-tuned LLMs in DP contexts. We release our code and setup at https://github.com/tejuafonja/DP-2Stage . Reviewed on OpenReview: https://openreview.net/forum?id=6nBIweDYzZ¬eId=nqBVYO1mmd 1 Introduction Tabular data is one of the most prevalent data types, providing structured information in rows and columns, and has been extensively used across various applications. Due to privacy concerns, tabular data cannot be directly shared. A widely adopted approach to address this issue is to train synthetic tabular data generators under differential privacy (DP) ( Dwork, 2006 ) . Different model classes have been proposed, from marginal-based statistical models ( Zhang et al., 2017 ; McKenna et al., 2022 ) to prominent Generative Adversarial Networks (GANs)-based tabular data generators ( Xu et al., 2019 ) , trained with DP-Stochastic Gradient Descent (DP-SGD) ( Abadi et al., 2016 ) . These models aim to replicate the marginal distribution of the original data while preserving the utility of the synthetic data. At the same time, they enforce a strict theoretical upper bound on privacy leakage, enabling users to generate realistic data samples without compromising privacy. Despite these advancements, existing techniques continue to grapple with major challenges, such as scalability limitations and difficulties in accurately modeling marginal distributions. These difficulties can be traced back to the intricate nature of tabular data and the challenges associated with training under noisy DP conditions. Recently, pre-trained Large Language Models (LLMs) ( Radford et al., 2019 ; Touvron et al., 2023 ) have demonstrated remarkable adaptability to tasks they have never been specifically trained for ( Wei et al., 2021 ; Chung et al., 2022 ; Wang & Fritz, 2024 ) . Acting as compact knowledge bases, LLMs present a promising opportunity for developing practical tabular data generators by leveraging pre-existing knowledge – an ability not currently feasible with traditional tabular data generators. Recent work from Borisov et al. (2023) showcased how LLMs can effectively be used for synthetic tabular data generation, by representing each cell in the format “ is ,”. While LLMs hold promise as generative priors for tabular data, adapting them under DP constraints introduces unique challenges. In DP-SGD, noise is added to the gradients to ensure privacy, which affects all tokens, including key tokens that may not be privacy-sensitive, such as column names or structural markers (e.g., “ is , ”). This indiscriminate application of noise can disrupt the model’s ability to maintain the structural integrity and semantic clarity required for tabular data generation. As a result, both the utility and fidelity of the generated synthetic data are negatively affected, highlighting the need for more targeted noise application techniques to minimize such impacts while adhering to DP constraints. In this work, we demonstrate that directly fine-tuning LLMs under DP constraints leads to sub-optimal performance. Our findings, shown in Figure 3 , reveal that LLMs struggle to generate coherent and structured text when fine-tuned directly with DP. This challenge stems from inefficient privacy budget allocation to potentially non-private elements, such as the table column name “ ” or non-functional tokens used to form a valid sentence for the LLM (e.g., “ is , ”). To address this limitation, we propose DP-2Stage , a two-stage fine-tuning framework for tabular data generation. In the first stage, DP-2Stage fine-tunes the pre-trained LLM non-privately on a pseudo dataset, allowing the model to learn task-specific structures and patterns without consuming the privacy budget. In the second stage, fine-tuning proceeds on the private dataset with DP constraints (see Figure 1 for an overview of the approach). By learning the structural patterns in the first stage, the DP-constrained fine-tuning can focus on preserving the privacy of the data values. For constructing pseudo datasets in the first stage, we investigated two strategies: (1) drawing data independently from a uniform distribution using statistics constructed from the private dataset (DP-2Stage-U) and (2) using an out-of-distribution public dataset that is unrelated to the private data (DP-2Stage-O). These approaches offer varying levels of privacy protection, with DP-2Stage-O providing stronger protection as it does not require prior knowledge of the private data. We evaluated DP-2Stage on three datasets and observed improvements in utility metric (F1-score) by 12-25%, and marginal distribution metric (Hist) by 1-3% compared to direct DP fine-tuning (DP-Standard) demonstrating better utility and fidelity. Notably, DP-2Stage-U achieved up to 21× faster inference compared to DP-2Stage-O and DP-Standard. This speedup stems from its higher similarity to private data—both in key structure and value distribution—allowing the second-stage DP fine-tuning to focus less on learning structural patterns. Additionally, we observe that fine-tuning with DP while avoiding column shuffling, as used in Borisov et al. (2023) , yielded better performance under DP constraints but underperformed in Non-DP scenarios. We summarize our contributions as follows: • Proposed DP-2Stage, a two-stage fine-tuning framework for LLM-based tabular data generation under DP, which fine-tunes on pseudo datasets non-privately to learn task-specific structures, enabling more efficient use of the privacy budget during DP fine-tuning. • Achieved 3–7% relative reductions in perplexity, 12–25% improvements in the utility metric (F1-score), and 1–3% relative improvements in the marginal distribution metric (Hist) compared to direct DP fine-tuning (DP-Standard). Each experiment was repeated five times per model, with four synthetic datasets generated per model to ensure robust evaluation. • Despite the effectiveness of our approach, our findings highlight the complexity of fine-tuning LLMs under DP for tabular data generation, emphasizing the need for further research to advance privacy-preserving LLM-based tabular data generators. To support future advancements, we release our code and provide a detailed discussion to guide further investigation. Figure 1 : Overview of DP-2Stage. In stage 1, the pre-trained LLM is fine-tuned on the respective pseudo data. Subsequently, in stage 2, the model from stage 1 undergoes further fine-tuning using the real private data. 2 Related Work Tabular data generation. As a prominent solution for data sharing, a large and growing body of research has been focused on tabular data generation. These methods can be generally categorized into marginal-based and deep learning-based methods. Marginal-based methods, such as those developed by Zhang et al. (2017) ; Aviñó et al. (2018) ; McKenna et al. (2022) , view each table column as a distinct random variable and model them using specific distributions. This approach often requires prior understanding or domain expertise in the data, hindering their scalability. Nevertheless, marginal-based methods (e.g. McKenna et al. (2022) ) have proven effective in integrating DP guarantees, making them a compelling choice for privacy-preserving data generation. On the other hand, deep learning (DL)-based methods such as ( Choi et al., 2017 ; Park et al., 2018 ; Xu et al., 2019 ; Kotelnikov et al., 2023 ) leverage cutting-edge generative models such as Variational Autoencoders (VAEs) ( Kingma & Welling, 2014 ) , Generative Adversarial Networks (GANs) ( Goodfellow et al., 2014 ) , or diffusion models ( Sohl-Dickstein et al., 2015 ) to synthesize tabular data. These models can capture complex patterns and correlations present in tabular data, often producing high-fidelity outputs. However, by default, these prominent DL-based tabular data generators (e.g Xu et al. (2019) ; Kotelnikov et al. (2023) ) do not adhere to DP standards, which is a significant limitation for applications requiring strong privacy guarantees. Recent advances have introduced privacy-preserving mechanisms, such as Differentially Private Stochastic Gradient Descent (DPSGD) ( Abadi et al., 2016 ) , to modify these models for DP compliance ( Xie et al., 2018 ) . Despite this progress, ensuring DP within tabular-based deep learning models poses unique challenges due to the complexity of the preprocessing steps. For example, proposed improvements like mode-specific normalization introduced in ( Xu et al., 2019 ) must also satisfy DP, as preprocessing steps themselves can impact privacy ( Ponomareva et al., 2023 ) . In this work, we leverage pre-trained LLMs for tabular data generation, taking advantage of their natural language processing capabilities, and investigate their potential for generating tabular data that is compliant with DP standards. Large Language Models (LLMs). Language models have been extensively studied over the years, evolving from statistical models ( Jelinek, 1998 ) and recurrent neural networks ( Hochreiter & Schmidhuber, 1997 ) to the latest transformer-based architectures ( Vaswani et al., 2017 ) . These advancements, supported by the attention mechanisms and rich text datasets, have led to the emergence of large-scale language models ( Radford et al., 2019 ; Brown et al., 2020 ; Touvron et al., 2023 ; Achiam et al., 2023 ) , a new generation of language models. These models, together with various fine-tuning strategies ( Hu et al., 2021 ; Zhou et al., 2022 ) , have enabled several interesting applications ( Borisov et al., 2023 ; Wang & Fritz, 2024 ) . Borisov et al. (2023) proposed using pre-trained LLMs for synthesizing tabular data in a non-private setting by converting tabular data into text-like formats, significantly improving the performance and paving a new path toward LLM-based tabular generators. In light of the potential, our work apply these advanced models to a relatively unexplored domain: the generation of tabular data under differential privacy constraints. While previous studies ( Yu et al., 2021 ; Li et al., 2022 ) have explored private fine-tuning of LLM, our focus specifically targets tabular data generation, a unique challenge that diverges from the broader scope of text comprehension. Recently, a concurrent work ( Tran & Xiong, 2024 ) shared a similar spirit of employing two-stage training technique to DP fine-tune LLM for tabular data generation. However, this work does not provide insight into the impact of pseudo data used in the first stage and examines a different family of LLM and fine-tuning strategy compared to this work. Our focus is on GPT-2, as utilized in the pioneering work by Borisov et al. (2023) , to gain deeper insights into how effective design choices in non-differentially private (Non-DP) settings translate to DP contexts. We offer a detailed analysis of pseudo-data choice, the impact of column shuffling on generation of coherent synthetic tabular data, identify the limitations of the two-stage approach, and discuss open challenges. 3 Background 3.1 Language Models for Tabular Data Language models are designed to model the probability of text sequences. Consider a text corpus, denoted as \mathcal{S}=\{{\bm{w}}_{i}\}_{i=1}^{N} , comprising N sentences. Each sentence {\bm{w}}=(t_{1},\ldots,t_{K}) within \mathcal{S} consists of an ordered sequence of K tokens. These tokens may represent either whole words or parts of words, generated through a tokenization process such as Byte-Pair Encoding (BPE) introduced by Sennrich et al. (2015) . The tokenization process can be expressed as (t_{1},\ldots,t_{K})=\texttt{tokenizer}({\bm{w}}_{i}) . The probability of a given sentence {\bm{w}} is formulated as: p({\bm{w}}_{i})=p(t_{1},\ldots,t_{K})=\prod_{k=1}^{K}p(t_{k}|t_{ is ,}^{\prime\prime} , specifying how keys ( k\in{\mathcal{K}} ), values ( v\in{\mathcal{V}} ), and non-functional tokens ( c\in{\mathcal{C}} , such as “is” and “,”) combine to form a record. An input dataset denoted as {\mathcal{S}} , is a realized instance of this f , instantiated with tokens (k,v) from a record {\bm{w}}=({\mathcal{K}},{\mathcal{V}}) . For example, with k=\{\texttt{age}\} and v=\{32\} , {\mathcal{S}} is instantiated as f(k,v)=``\texttt{age is 32,}^{\prime\prime} . Following tokenization, tokens associated with elements in the key or value set are denoted as t_{i}\in{\mathcal{K}} or t_{j}\in{\mathcal{V}} , with i and j being the position indices, respectively. Note that, due to tokenization, the number of tokens for the keys {\mathcal{K}} and the values {\mathcal{V}} are not necessarily equal. With definition 3.1 , any content can be applied to a specified template. Column Shuffling. Column shuffling involves randomly altering the order of column-aligned values within each batch during training. For example, given columns age , sex , and income with values 32 , female , and >50k . A detailed illustration is provided in Figure 2 . This mechanism happens at every iteration and has been shown to effectively prevent the model from relying on spurious dependency in Non-DP settings ( Borisov et al., 2023 ) . However, it complicates DP training due to gradient perturbations, often resulting in higher perplexity and different behavior compared to Non-DP models, as shown in Figure 3 . Disabling shuffling fixes the order across iterations simplifying DP training. Example table age sex income 32 female >50k Iteration 1: age is 32, sex is female, income is >50k. Iteration 2: income is >50k, age is 32, sex is female. Iteration 3: sex is female, income is >50k, age is 32. Figure 2 : Illustration of column shuffling. The order of entries is permuted in each iteration. This mechanism happens at every iteration and has been shown to effectively prevent the model from relying on spurious dependency in Non-DP settings ( Borisov et al., 2023 ) . However, we find that it complicates DP training due to gradient perturbations, often resulting in higher perplexity compared to Non-DP models, as shown in Figure 3 . 3.2 Differential Privacy Differential Privacy (DP) mechanisms enable the confidential disclosure of information about a dataset by perturbing a function of the input dataset. This ensures that any information capable of distinguishing a specific record from the remainder of the dataset is constrained, as outlined by Dwork et al. (2014) . In this work, we consider privacy-preserving tabular data generators to ensure that any information leakage from the generated data is bounded by DP. We review the necessary definitions, the threat model, and the privacy model below. Definition 3.2 (Differential Privacy ( Dwork et al., 2014 ) ) A randomized mechanism {\mathcal{M}} with range {\mathcal{R}} satisfies (\varepsilon,\delta) -differential privacy, if for any two adjacent datasets E and E^{\prime} , i.e E^{\prime}=E\cup\{x\} for some x in the data domain (or vice versa), and for any subset of outputs O\subseteq{\mathcal{R}} , it holds that \Pr[{\mathcal{M}}(E)\in O]\leq e^{\varepsilon}\Pr[{\mathcal{M}}(E^{\prime})\in O]+\delta (2) where \varepsilon is the privacy budget and \delta is the probability of the mechanism failing. Intuitively, this guarantees that an adversary, provided with the output {\mathcal{M}} , can draw almost the same conclusions (up to \varepsilon with probability larger than 1-\delta ) about any record no matter if it is included in the input of {\mathcal{M}} or not ( Dwork et al., 2014 ) . That is, for any record owner, a privacy breach is unlikely due to their participation in the dataset. Definition 3.3 (Gaussian Mechanism ( Dwork et al., 2014 ) ) Let f:{\mathbb{R}}^{n}\rightarrow{\mathbb{R}}^{d} be an arbitrary function that maps n -dimensional input to d logits with sensitivity being: S=\max_{E,E^{\prime}}\|f(E)-f(E^{\prime})\|_{2} (3) over all adjacent datasets E and E^{\prime}\in{\mathcal{E}} . The Gaussian Mechanism {\mathcal{M}}_{\sigma} , parameterized by \sigma , adds noise into the output i.e., {\mathcal{M}}_{\sigma}(x)=f(x)+{\mathcal{N}}(0,\sigma^{2}{\mathbf{I}}_{n}). (4) Threat Model. We outline a threat model where the goal of the adversary is to infer information about individuals in the training dataset by launching diverse privacy attacks. One such attack is the Membership Inference Attack ( Shokri et al., 2017 ; Hilprecht et al., 2019 ; Chen et al., 2020 ) , which determines whether a specific data point was included in the model’s training set. We imagine a scenario involving a strong adversary who possesses complete access to the model post-training, known as “white-box access”. This scenario is considerably more critical than a “black-box access” setting, where the adversary is limited to interacting with and analyzing the synthetic data produced by the model, without insight into its internal workings. In addition, the adversary is computationally unbounded. The impact of the threat model is the potential exposure of sensitive individual data from the training set, thereby compromising data privacy and undermining trust in the model. Privacy Model. To defend against such threat, we aim to develop a solution that protects against potential privacy attacks targeting individuals in our training dataset. To achieve this, we employ DP, which was specifically designed to address this challenge. DP provides the assurance of plausible deniability, meaning that any potential privacy infringement on an individual – for example, a patient in the dataset – cannot be conclusively linked to their participation in the model’s training phase up to \varepsilon with probability larger than 1-\delta . Consider a scenario where an insurance company decides to raise a patient’s insurance premium after analyzing a model trained on a medical dataset. In such a case, owing to the principles of DP, the increase cannot be directly attributed to the inclusion of that patient’s data in the training process. We provide further details in Appendix A . Figure 3 : DP-2Stage (Ours) vs. Standard DP fine-tuning on the Adult dataset with \varepsilon=1,\delta=10^{-5} . DP-2Stage-O refers to the stage 2 model fine-tuned using out-distribution pseudo data (Airline dataset) in stage 1, while DP-2Stage-U is the stage 2 model fine-tuned using data sampled independently from a Uniform distribution, with statistics derived from the Adult dataset as the pseudo-dataset in stage 1. Perplexity results are displayed from left to right for all tokens, values, keys, and non-functional tokens (c.f. section 3.1 ). The top plot shows model trained without column shuffling, and the bottom shows model trained with column shuffling. Total and Value Perplexity for top and bottom plots have fixed y-axis for ease of comparison. The perplexity values are higher with column shuffling (bottom) than without (top). 4 DP-2Stage While large language models (LLMs) demonstrate impressive capabilities in generating synthetic tabular data in non-differentially private (Non-DP) settings, as we will later show, fine-tuning them under DP constraints remains challenging for tabular data generation. As illustrated in Figure 3 (bottom), standard DP training, where the LLM model is directly fine-tuned using DPSGD ( Abadi et al., 2016 ) (see Appendix A.1 for more details), often results in suboptimal performance in terms of the perplexity metric. We hypothesize that this is due to the inefficient allocation of the privacy budget during the learning of data structures. Models employing this approach struggle to reduce the perplexity of keys (e.g., column names) and other non-functional tokens, despite these elements typically posing minimal privacy risks. Consequently, applying DP to such components not only wastes the privacy budget but also diminishes the overall utility of the model. Motivated by this observation, we propose DP-2Stage, a two-stage learning approach designed to progressively capture tabular data structures. An overview of our method is presented in Figure 1 . In the first stage (section 4.1 ), we train the model on non-private pseudo data synthesized based on (1) prior knowledge of the private data and (2) publicly available out-of-distribution public dataset, guiding it to learn the underlying structure. In the second stage (section 4.2 ), the model is fine-tuned on the private data while ensuring DP protection. The goal of our two-stage approach is to decouple the process of learning the table’s structure from learning the private content. 4.1 Stage 1: Template Learning The goal of the first stage is to enable the model to learn the structure of the data table, such as identifying key-value relationships, without consuming any privacy budget. We achieve it by constructing a pseudo dataset that retain the structural information while eliminating privacy risks. To achieve this, we investigated two different methods for constructing a pseudo dataset \widetilde{{\mathcal{S}}} : (1) Sampled data from a uniform distribution using the private data statistics and (2) Out-of-distribution public dataset. Each method provides different levels of reliance on private data while aiming to preserve privacy and utility. Independently Sampled data from a Uniform Distribution (uniform pseudo data). This dataset is designed to resemble the private dataset we aim to protect. We assume that column names are public information and can be utilized without any privacy concerns. Additionally, we assume that the data owner can provide the range (maximum and minimum values) for numerical columns and the list of categories for each categorical column. For each column, samples are independently drawn from a uniform distribution, where all elements have an equal probability of being selected using the statistics from the private data such as the range of numerical column, and the categories in each categorical column. This assumption is minimal compared to requiring detailed distributional information, which may be unrealistic. However, while this approach minimizes reliance on private data, it still presumes access to basic information, such as category labels and numerical ranges. Out-of-distribution Public data (out-distribution pseudo data). In contrast to the previous method, this approach eliminates the reliance on private data by using a publicly available dataset. In this work, we refer to Out-of-distribution as any publicly available dataset that is not the private data. This approach avoids the need to infer statistical properties of the original dataset, relying instead on the structure provided by the public data. As we will show, despite not requiring any information from the original dataset, this method performs comparably to uniform pseudo data in many cases and even outperforms it in certain scenarios. Objective function. In the first stage, the model is trained using standard training protocols on the serialized version of the pseudo dataset. In particular, we fine-tune a pre-trained LLM p_{\theta} , parameterized by \theta , using cross-entropy loss for causal language modeling in a non-private setup. We formalize the loss function as follows. {\mathcal{L}}=\mathbb{E}_{{\bm{w}}\in\widetilde{{\mathcal{S}}}}\left[-\sum_{t_{k}\in{\bm{w}}}\log p_{\theta}(t_{k}\mid t_{0 as SG_{q,\sigma}\overset{\Delta}{=}f(\{x:x\in R\text{ is sampled with probability }q\})+{\mathcal{N}}(0,\sigma^{2}{\mathbf{I}}^{d}) (9) where each element of E is independently and randomly sampled with probability q without replacement. The sampled Gaussian mechanism consists of adding independent and identically distributed (i.i.d) Gaussian noise with zero mean and variance \sigma^{2} to each coordinate value of the true output of f . In fact, the sampled Gaussian mechanism draws vector values from a multivariate spherical (or isotropic) Gaussian distribution which is described by random variable {\mathcal{N}}(0,\sigma^{2}{\mathbf{I}}^{d}) , where d is omitted if it unambiguous in the given context. A.2.1 Analysis The privacy analysis of our DP methods and other DP baselines considered in the paper follows the well-established analysis framework used for gradient-based, record-level DP methods, known as DP-Stochastic Gradient Descent (DP-SGD) ( Abadi et al., 2016 ) . In this framework, each update is conducted as a single SGM step (Definition A.3 ), which includes selecting a random batch, clipping the per-example gradients of that batch, and then adding Gaussian noiseto the aggregrated batch gradient. The privacy cost accumulated over multiple updates is quantified using the revised moment accountant method ( Mironov et al., 2019 ) , which adapts the original moment accountant approach introduced by Abadi et al. (2016) to the concept of Rényi Differential Privacy (RDP) (Definition A.2 ). Finally, to achieve interpretable results and allow for transparent comparisons with established methods, the privacy cost is converted from ( \alpha,\rho )-RDP to ( \varepsilon,\delta )-DP using the conversion theorem (Theorem A.6 ) provided by Balle et al. (2020) . Let \mu_{0} denote the \mathrm{pdf} of {\mathcal{N}}(0,\sigma^{2}) and let \mu_{1} denote the \mathrm{pdf} of {\mathcal{N}}(1,\sigma^{2}) . Let \mu be the mixture of two Gaussians \mu=(1-q)\mu_{0}+q\mu_{1} , where q is the sampling probability of a single record in a single round. Theorem A.4 ( Mironov et al., 2019 ) . Let \mathrm{SGM}_{q,\sigma} be the Sampled Gaussian mechanism for some function f and under the assumption \Delta_{2}f\leq 1 for any adjacent E , E^{\prime}\in{\mathcal{E}} . Then \mathrm{SGM}_{q,\sigma} satisfies ( \alpha,\rho )-RDP if \rho\leq\frac{1}{\alpha-1}\log\max(A_{\alpha},B_{\alpha}) (10) where A_{\alpha}\overset{\Delta}{=}\mathbb{E}_{z\sim\mu_{0}}[(\mu(z)/\mu_{0}(z))^{\alpha}] and B_{\alpha}\overset{\Delta}{=}\mathbb{E}_{z\sim\mu}[(\mu_{0}(z)/\mu(z))^{\alpha}] Theorem A.4 states that applying SGM to a function of sensitivity (Definition 3.3 ) at most 1 (which also holds for larger values without loss of generality) satisfies ( \alpha,\rho )-RDP if \rho\leq\frac{1}{\alpha-1}\log(\max\{A_{\alpha},B_{\alpha}\}) . Thus, analyzing RDP properties of SGM is equivalent to upper bounding A_{\alpha} and B_{\alpha} . From Corollary 7. in ( Mironov et al., 2019 ) , A_{\alpha}\geq B_{\alpha} for any \alpha\geq 1 . Therefore, we can reformulate Equation 10 as \rho\leq\xi_{{\mathcal{N}}}(\alpha|q):=\frac{1}{\alpha-1}\log A_{\alpha} (11) To compute A_{\alpha} , we use the numerically stable computation approach proposed in ( Mironov et al., 2019 ) (Sec. 3.3) depending on whether \alpha is expressed as an integer or a real value. Theorem A.5 (Composability ( Mironov, 2017 ) . Suppose that a mechanism {\mathcal{M}} consists of a sequence of adaptive mechanisms {\mathcal{M}}_{1},\ldots,{\mathcal{M}}_{k} where {\mathcal{M}}_{i}:\prod_{j=1}^{i=1}{\mathcal{R}}_{j}\times{\mathcal{E}}\rightarrow{\mathcal{R}}_{i} . If all the mechanisms in the sequence are ( \alpha,\rho )-RDP, then the composition of the sequence is ( \alpha,k\rho )-RDP. In particular, Theorem A.5 holds when the mechanism themselves are chosen based on the (public) output of the previous mechanisms. By Theorem A.5 , it suffices to compute \xi_{\mathcal{N}}(\alpha|q) at each step and sum them up to bound the overall RDP privacy budget of an iterative mechanism composed of single DP mechanism at each step. Theorem A.6 (Conversion from RDP to DP ( Balle et al., 2020 ) ). If a mechanism {\mathcal{M}} is ( \alpha,\rho )-RDP then it is ((\rho+\log((\alpha-1)/\alpha)-(\log\delta+\log\alpha)/(\alpha-1),\delta) -DP for any 0<\delta<1 . Theorem A.7 (Privacy of the different DP methods). For any 0<\delta<1 and \alpha\geq 1 , the different DP methods are ( \varepsilon,\delta )-DP, with \varepsilon=\underset{\alpha}{\min}\left(T\cdot\xi_{\mathcal{N}}(\alpha|q)+\log((\alpha-1)/\alpha)-(\log\delta+\log\alpha)/(\alpha-1)\right) (12) Here, \xi_{\mathcal{N}}(\alpha|q) is defined in Equation 11 , q=\frac{{\mathbb{B}}}{|{\mathcal{D}}|} , T is the total number of updates, {\mathbb{B}} is the batch size, and |{\mathcal{D}}| denotes the dataset size. The proof follows from Theorems A.4 , A.5 , A.6 and the fact that a record is sampled in every SGDiteration if the batch of records sampled contains the record, which has a probability of at most \frac{{\mathbb{B}}}{|{\mathcal{D}}|} . Therefore, a record is sampled with a probability of at most q=\frac{{\mathbb{B}}}{|{\mathcal{D}}|} . Appendix B Sampling B.1 Imputation-based sampling After training the model, we generate samples by conditioning on a key-value pair, i.e., {\bm{w}}\sim p_{\theta}(\cdot\mid\mathtt{Prompt}) , where Prompt denotes the tokens generated from the pair, for instance, tokens of “ income is <50k ”. The trained model then generates the next token based on this prompt. The sampling process continues until it encounters a stop token or a maximum token length of 100, which exceeds the number of tokens in each table row. Depending on the model’s performance, they may produce incoherent outputs, such as mismatches of keys and values (e.g., generating ‘age is >50’ and ‘relationship is Ad-serv-spouse’). To this end, we post-process the generated data and remove the values that do not match the category of the corresponding column. Once removed, we use the correct tokens to recondition the model, allowing it to fill in any missing tokens — essentially performing imputation based on the correctly generated tokens. We set a threshold of 15 for imputation, meaning if the generation quality is too poor, imputation will not proceed. Previous method ( Borisov et al., 2023 ) often discard incorrectly generated samples and continue generating until the model produces a correct sample in one shot, or they exit the loop. While this approach works well with Non-DP models, we find that in DP generation when column shuffling is enabled, rejection sampling significantly increases the time required to generate data. In contrast, imputation is more efficient in this scenario. Appendix C Metrics C.1 Perplexity The perplexity metric serves as a fundamental guage for assessing the performance of language models. It quantifies the uniformity of the model’s predictions across a predefined set of tokens in a corpus. Specifically, perplexity is defined as the exponentiation of the average negative log-likelihood of a sequence, encapsulating the model’s ability to predict the next token in a sequence accurately. This measurement reflect how well a model understands the structure and patterns of the language, with lower values indicating higher predictive accuracy and a better grasp of the language nuances. \texttt{PPL}=\exp\left\{-\frac{1}{t}\sum_{i}^{t}\log p({\bm{t}})\right\} (13) where t is the number of total sentences in a corpus and p({\bm{t}}) is defined in Equation 1 . Intuitively, perplexity is often interpreted as the “effective number of choices” the model is making e.g., a perplexity of 1.8 suggests that the model, on average, has narrowed down the next token to almost 2 equally likely possibilities. • High Perplexity: Indicates that the model is uncertain about its predictions, implying that the model has not learned well and is making a lot of mistakes. • Low Perplexity: Suggests that the model is confident in its predictions and is performing well, predicting the next token accurately. C.1.1 Disentangled key-value perplexity The Key , Value , Other perplexity are computed by masking out the relevant token perplexity and averaging across the per-example tokens and entire dataset. C.2 Tabular-based Metrics The tabular-based metrics evaluates the synthetic tabular data against the real tabular data. C.2.1 Machine Learning Efficacy The effectiveness of synthetic data is typically assessed through its utility in downstream tasks, aiming to parallel the performance achieved with real data. This evaluation process entails training machine learning models using real data and subsequently evaluating their performance when trained on synthetic data, with comparison made against a reserved set of test data. C.2.2 Normalized Histogram Intersection The normalized histogram intersection which is also referred to as total variation distance measures how aligned the marginal distributions of each column in the generated sample is with the real test data marginal distribution. It provides a quantitative analysis of one-dimensional data distributions by calculating the sum of the minimum probability values across corresponding bins in the real and synthetic data columns. This sum is the averaged over all columns in the dataset, offering a measure of the normalized intersection between the marginal probability distributions of real and synthetic data. \displaystyle\text{Hist}({\bm{p}}_{i},{\bm{q}}_{i}) \displaystyle=\sum_{c}\min(p_{c},q_{c}) (14) HI \displaystyle=\frac{1}{d}\sum_{i}\text{Hist}({\bm{p}}_{i},{\bm{q}}_{i}) (15) where p_{c}=\frac{s_{c}}{|{\mathcal{D}}|\Delta_{i}} and q_{c}=\frac{t_{c}}{|{\mathcal{S}}|\Delta_{i}} . {\bm{p}}_{i} and {\bm{q}}_{i} represents the histogram probabilities of real ( {\mathcal{D}} ) and synthetic ( {\mathcal{S}} ) datasets for feature i , respectively. The terms p_{c} and q_{c} represent the proportions of category c for feature i , with s_{c} and t_{c} denoting the counts of real and synthetic samples in category c , respectively. The factor \Delta_{i} is introduced as a normalization term, specifying the bin size for numerical features. The HI is an average of the histogram intersection scores across all features, proving insight into the similarity between the real and synthetic data distributions. C.2.3 Pairwise Correlation Similarity Accuracy (CorAcc) We evaluate the correlation between data columns using the approach described by Tao et al. (2021) and Afonja et al. (2023) . Specifically, we use Cramer’s V with bias correction for categorical columns, the Correlation Ratio for numerical-categorical columns, and the Pearson Correlation Coefficient (absolute values) for numerical columns. The ranges for these measures are as follows: Cramer’s V and Correlation Ratio are bounded between 0 and 1, while the Pearson Correlation Coefficient spans -1 to 1. Following Tao et al. (2021) , correlation values are discretized into four levels: low [0, 0.1), weak [0.1, 0.3), medium [0.3, 0.5), and strong [0.5, 1). The CorAcc metric quantifies the similarity between synthetic and original data by measuring the fraction of column pairs where the assigned correlation levels match. C.2.4 Pairwise Attribute Distribution Similarity (Pair) This metric extends the Normalized Histogram Intersection (HIST) by calculating the histogram intersection for all two-way marginals and averaging the results across all attribute pairs. For numerical columns, we discretize the values into bins of size 20 and 50 before computing the intersections. Appendix D Setup and Dataset D.1 Datasets Texas Dataset. The Texas Hospital Discharge dataset 12 12 12 https://www.dshs.texas.gov/thcic/ is a large public use data file provided by the Texas Department of State Health Services. We used the preprocessed version which consists of 100,000 records uniformly selected from a pre-processed file containing patient data from 2013 13 13 13 https://github.com/spring-epfl/synthetic_data_release/blob/master/data/texas.csv version from Stadler et al. (2022) . We retain 18 attributes and assume a binary classification task by predicting only minor and major mortality risk following the setup of Afonja et al. (2023) . Duplicates where also removed. The final size of the dataset was therefore reduced to 75,105 which was split to non-overlapping train/test/validation. Validation size is fixed to 1000 for all dataset. No additional preprocessing was done for Adult and Airline other than removing duplicates. List of Column names: 1. Adult Income : Age , Work Class , FNLWGT , Education , Education Number , Marital Status , Occupation , Relationship , Race , Sex , Capital Gain , Capital Loss , Hours per Week , Native Country , and Income 2. Airline Passenger Satisfaction: ID , Gender , Customer Type , Age , Type of Travel , Class , Flight Distance , Inflight Wi-Fi Service , Departure/Arrival Time Convenience , Ease of Online Booking , Gate Location , Food and Drink , Online Boarding , Seat Comfort , Inflight Entertainment , Onboard Service , Leg Room Service , Baggage Handling , Check-in Service , Inflight Service , Cleanliness , Departure Delay (minutes) , Arrival Delay (minutes) , and Satisfaction (Neutral or Dissatisfied, Satisfied) . 3. Texas: Discharge , Type of Admission , Patient State , Patient Status , Sex Code , Race , Ethnicity , Admission Weekday , Patient Age , Illness Severity , Length of Stay , Total Charges , Total Non-Covered Charges , Total Charges for Accommodation , Total Non-Covered Charges for Accommodation , Total Charges for Ancillary Services , Total Non-Covered Charges for Ancillary Services , and Risk of Mortality . Table 1 provides statistics of the train-test split, as well as the number of numerical, and categorical columns in the dataset. Appendix E Sampling Time We report the sampling time for generating one dataset of synthetic data. The size of the synthethic data is the same as the training data. The result is shown in Table 6 . Dataset Shuffle DP-Standard DP-2Stage-O DP-2Stage-U Adult ✗ 10 mins - - \varepsilon=\infty ✓ 10 mins - - \varepsilon=1 ✗ 11 mins 11 mins 10 mins ✓ 5 hrs 6 hrs 14 mins Airline ✗ 1.1 hrs - - \varepsilon=\infty ✓ 1.6 hrs - - \varepsilon=1 ✗ 1.1 hrs 1.1 hrs 1.2 hrs ✓ 42 hrs 51 hrs 1.7 hrs Table 6: Sampling Cost . The synthetic dataset matches the size of the training dataset. ✓ indicates settings with shuffle enabled, while ✗ represents shuffle disabled. The reported values correspond to a single model run and the generation of one synthetic dataset. For Adult, DP-2Stage-O uses Airline as pseudo data and vice-versa. Appendix F Additional Results F.1 Comparison of Different Privacy Budget Table 7 shows the DP result for a higher privacy budget \varepsilon=8 . Relaxing the privacy budget shows improved performance for DP-2Stage across both datasets. Scaling DP-GAN to higher \varepsilon values for the Airline dataset proved challenging, requiring up to five days to run before the process was terminated. Adult Airline Method F1 AUC ACC HIST F1 AUC ACC HIST \varepsilon=1, Non-LLM \delta=10^{-5} DP-GAN 33.5 \pm 20 67.7 \pm 9 64.2 \pm 10 63.7 \pm 3 40.2 \pm 24 63.9 \pm 13 59.8 \pm 6 44.7 \pm 12 DP-CTGAN 42.2 \pm 20 78.0 \pm 7 75.7 \pm 3 75.7 \pm 2 67.1 \pm 8 76.7 \pm 8 68.0 \pm 6 78.7 \pm 2 DP-VAE 0.0 \pm 0 50.0 \pm 0 75.6 \pm 0 61.8 \pm 2 26.5 \pm 28 57.9 \pm 13 57.3 \pm 6 41.8 \pm 1 GPT-2 DP-Standard 27.8 \pm 15 58.5 \pm 7 65.2 \pm 9 85.7 \pm 2 60.5 \pm 7 65.3 \pm 9 62.4 \pm 7 90.3 \pm 3 DP-2Stage-U 21.2 \pm 12 48.9 \pm 6 61.9 \pm 13 86.7 \pm 1 68.5 \pm 9 77.8 \pm 10 72.1 \pm 7 90.7 \pm 1 DP-2Stage-O 30.4 \pm 17 61.6 \pm 8 66.7 \pm 8 88.5 \pm 1 55.2 \pm 18 62.5 \pm 19 60.0 \pm 16 92.5 \pm 1 \varepsilon=8, Non-LLM \delta=10^{-5} DP-GAN 19.6 \pm 20 50.0 \pm 0 50.0 \pm 26 33.3 \pm 9 - - - - DP-CTGAN 46.5 \pm 18 79.4 \pm 4 73.1 \pm 6 80.0 \pm 2 67.7 \pm 4 76.7 \pm 5 67.7 \pm 4 76.8 \pm 1 DP-VAE 0.0 \pm 0 50.0 \pm 0 75.6 \pm 0 62.1 \pm 1 51.9 \pm 25 72.4 \pm 10 67.2 \pm 7 40.0 \pm 1 GPT-2 DP-Standard 31.3 \pm 15 62.2 \pm 7 67.7 \pm 7 84.5 \pm 1 64.9 \pm 6 69.8 \pm 9 65.9 \pm 7 89.8 \pm 3 DP-2Stage-U 22.4 \pm 15 51.8 \pm 8 63.7 \pm 11 86.9 \pm 1 71.9 \pm 9 80.7 \pm 10 74.9 \pm 8 90.4 \pm 1 DP-2Stage-O 33.4 \pm 16 63.8 \pm 9 68.2 \pm 7 87.9 \pm 1 64.2 \pm 11 71.7 \pm 10 67.8 \pm 8 92.3 \pm 1 Table 7: DP Benchmark for \varepsilon=8 . Utility metrics (F1, AUC, and ACC) are presented as the averages of logistic regression and XGBoost performance. HIST represents the average histogram intersection scores calculated using bins of 20 and 50. Results are averaged across five model runs and four synthetic datasets per run with standard deviation reported after \pm . The best value per column for each \varepsilon is shown in bold while second best value is underlined . F.2 Marginal-based DP Baseline AIM. Proposed by ( McKenna et al., 2022 ) , AIM is a marginal-based model for generating differentially private synthetic data. It is a workload-adaptive algorithm that follows a three-step process: selecting a set of queries, privately measuring those queries, and generating synthetic data from the noisy measurements. AIM employs innovative techniques to iteratively prioritize the most useful measurements, considering both their relevance to the workload and their importance in approximating the input data. MST. Proposed by ( McKenna et al., 2021 ) , MST was the winning mechanism of the 2018 NIST Differential Privacy Synthetic Data Competition. It is a general approach for differentially private synthetic data generation that follows three main steps: (1) selecting a collection of low-dimensional marginals, (2) measuring these marginals using a noise addition mechanism, and (3) generating synthetic data that accurately preserves the measured marginals. Table 8 presents the results for DP methods, including two marginal-based approaches. The Marginal-based methods shows better performance across most metrics, except for HIST and Pair, where their performance is subpar on the Airline dataset. This is likely due to the higher number of numerical columns in this dataset (see Table 1 ). Dataset Method F1 AUC ACC CorAcc Pair HIST Adult Marginal \varepsilon=1, AIM 59.6 \pm 6 86.8 \pm 1 80.3 \pm 2 86.4 \pm 1 77.1 \pm 9 88.4 \pm 5 \delta=10^{-5} MST 39.6 \pm 19 76.8 \pm 1 72.8 \pm 2 70.0 \pm 1 74.6 \pm 10 87.0 \pm 5 Non-LLM DP-GAN 33.5 \pm 20 67.7 \pm 9 64.2 \pm 10 39.9 \pm 3 41.2 \pm 4 63.7 \pm 3 DP-CTGAN 42.2 \pm 20 78.0 \pm 7 75.7 \pm 3 51.3 \pm 3 59.2 \pm 2 75.7 \pm 2 DP-VAE 0.0 \pm 0 50.0 \pm 0 75.6 \pm 0 48.8 \pm 1 40.3 \pm 1 61.8 \pm 2 GPT-2 DP-Standard 27.8 \pm 15 58.5 \pm 7 65.2 \pm 9 55.0 \pm 1 68.4 \pm 1 85.7 \pm 2 DP-2Stage-U 21.2 \pm 12 48.9 \pm 6 61.9 \pm 13 55.0 \pm 1 76.1 \pm 1 86.7 \pm 1 DP-2Stage-O +airline 30.4 \pm 17 61.6 \pm 8 66.7 \pm 8 55.4 \pm 1 72.3 \pm 1 88.5 \pm 1 +texas 31.6 \pm 13 60.5 \pm 7 66.4 \pm 8 55.6 \pm 1 71.3 \pm 1 86.9 \pm 1 Airline Marginal \varepsilon=1, AIM 77.3 \pm 5 88.9 \pm 4 78.2 \pm 5 91.8 \pm 1 46.7 \pm 3 68.1 \pm 2 \delta=10^{-5} MST 72.2 \pm 6 83.4 \pm 5 75.2 \pm 4 72.7 \pm 0 46.3 \pm 3 68.2 \pm 2 Non-LLM DP-GAN 40.2 \pm 24 63.9 \pm 13 59.8 \pm 6 37.4 \pm 9 22.2 \pm 13 44.7 \pm 12 DP-CTGAN 67.1 \pm 8 76.7 \pm 8 68.0 \pm 6 31.7 \pm 2 62.2 \pm 2 78.7 \pm 2 DP-VAE 26.5 \pm 28 57.9 \pm 13 57.3 \pm 6 46.6 \pm 1 20.6 \pm 0 41.8 \pm 1 GPT-2 DP-Standard 60.5 \pm 7 65.3 \pm 9 62.4 \pm 7 64.0 \pm 2 77.0 \pm 2 90.3 \pm 3 DP-2Stage-U 68.5 \pm 9 77.8 \pm 10 72.1 \pm 7 65.3 \pm 1 80.8 \pm 1 90.7 \pm 1 DP-2Stage-O +adult 55.2 \pm 18 62.5 \pm 19 60.0 \pm 16 66.8 \pm 1 80.1 \pm 1 92.5 \pm 1 +texas 52.5 \pm 13 61.0 \pm 13 58.4 \pm 10 66.1 \pm 2 78.7 \pm 2 90.4 \pm 1 Texas Marginal \varepsilon=1, AIM 84.5 \pm 1 98.3 \pm 0 94.2 \pm 1 81.0 \pm 2 93.0 \pm 5 98.9 \pm 0 \delta=10^{-5} MST 81.7 \pm 0 94.8 \pm 0 93.2 \pm 0 77.0 \pm 0 97.5 \pm 0 99.0 \pm 0 Non-LLM DP-GAN 13.7 \pm 16 58.3 \pm 12 78.3 \pm 8 36.1 \pm 7 34.6 \pm 5 68.3 \pm 7 DP-CTGAN 63.9 \pm 11 91.4 \pm 3 82.6 \pm 19 43.9 \pm 6 66.9 \pm 6 84.7 \pm 3 DP-VAE 0.0 \pm 0 50.0 \pm 0 82.5 \pm 0 62.1 \pm 1 43.9 \pm 1 77.9 \pm 1 GPT-2 DP-Standard 55.4 \pm 10 90.2 \pm 5 77.1 \pm 11 70.3 \pm 1 60.6 \pm 1 92.3 \pm 1 DP-2Stage-U 23.5 \pm 14 59.8 \pm 14 67.3 \pm 17 68.2 \pm 0 80.7 \pm 6 93.4 \pm 0 DP-2Stage-O +adult 74.8 \pm 4 96.7 \pm 1 89.1 \pm 3 69.9 \pm 2 60.5 \pm 2 91.7 \pm 1 +airline 74.3 \pm 5 96.4 \pm 0 88.8 \pm 3 70.9 \pm 1 62.0 \pm 2 93.2 \pm 0 Table 8: DP Benchmark. Utility metrics (F1, AUC, and ACC) are presented as the averages of two ML Models (XGBoost and Logistic Regression). Pair , and Hist are reported as averages of two bin sizes (Bins 20 and 50). Results are averaged across five model runs and four synthetic datasets per run with standard deviation reported after \pm . The best value per method group for \varepsilon=1 is shown in bold . F.3 Individual Metrics Scores Table 9 and 10 presents the results for the machine learning models evaluated: XGBoost (XGB) and Logistic Regression (LR). Histogram Intersection score (HIST) is reported for two bin sizes: 20 and 50. The averaged values are summarized in Table 3 . Dataset Method XGB (F1) XGB (AUC) XGB (ACC) LR (F1) LR (AUC) LR (ACC) HIST (bin=50) HIST (bin=20) Adult Marginal \varepsilon=1 AIM 54.0 \pm 4 86.3 \pm 1 81.8 \pm 0 65.2 \pm 0 87.3 \pm 0 78.8 \pm 1 83.5 \pm 0 93.3 \pm 1 \delta=10^{-5} MST 20.5 \pm 3 76.7 \pm 1 74.5 \pm 0 58.7 \pm 0 77.0 \pm 2 71.0 \pm 0 81.8 \pm 2 92.2 \pm 0 Non-LLM DP-GAN 27.4 \pm 24 67.6 \pm 10 69.1 \pm 9 39.6 \pm 14 67.8 \pm 9 59.4 \pm 8 61.9 \pm 2 65.5 \pm 3 DP-CTGAN 38.6 \pm 21 77.2 \pm 7 76.0 \pm 3 45.8 \pm 19 78.8 \pm 7 75.3 \pm 3 75.0 \pm 2 76.4 \pm 2 DP-VAE 0.0 \pm 0 50.0 \pm 0 75.6 \pm 0 0.0 \pm 0 50.0 \pm 0 75.6 \pm 0 60.1 \pm 0 63.5 \pm 0 GPT-2 DP-Standard 13.9 \pm 7 55.5 \pm 6 73.0 \pm 1 41.6 \pm 5 61.4 \pm 7 57.4 \pm 6 85.1 \pm 2 86.2 \pm 2 DP-2Stage-U 10.3 \pm 5 48.4 \pm 4 74.5 \pm 1 32.1 \pm 6 49.3 \pm 8 49.4 \pm 4 86.3 \pm 1 87.1 \pm 1 DP-2Stage-O +airline 15.0 \pm 7 55.9 \pm 6 73.5 \pm 1 45.9 \pm 5 67.3 \pm 6 59.8 \pm 5 88.2 \pm 1 88.8 \pm 1 +texas 9.6 \pm 7 56.7 \pm 5 73.4 \pm 1 43.6 \pm 5 64.4 \pm 5 59.4 \pm 5 86.5 \pm 1 87.3 \pm 1 Airline Marginal \varepsilon=1 AIM 73.0 \pm 4 85.2 \pm 3 74.3 \pm 4 81.7 \pm 0 92.7 \pm 0 82.1 \pm 0 65.9 \pm 0 70.2 \pm 0 \delta=10^{-5} MST 69.2 \pm 8 80.4 \pm 6 74.0 \pm 5 75.2 \pm 0 86.3 \pm 0 76.4 \pm 0 66.1 \pm 0 70.3 \pm 0 Non-LLM DP-GAN 38.3 \pm 25 65.1 \pm 14 60.3 \pm 6 42.1 \pm 24 62.7 \pm 12 59.2 \pm 6 43.4 \pm 12 46.0 \pm 12 DP-CTGAN 64.1 \pm 10 74.0 \pm 8 65.9 \pm 6 70.1 \pm 5 79.4 \pm 7 70.1 \pm 5 78.5 \pm 2 79.0 \pm 1 DP-VAE 1.0 \pm 3 54.4 \pm 11 56.7 \pm 1 52.0 \pm 14 61.4 \pm 14 58.0 \pm 8 41.5 \pm 0 42.2 \pm 0 GPT-2 DP-Standard 55.8 \pm 3 62.1 \pm 7 60.0 \pm 6 65.1 \pm 6 68.4 \pm 10 64.9 \pm 8 89.6 \pm 4 91.1 \pm 3 DP-2Stage-U 64.5 \pm 10 73.8 \pm 9 70.0 \pm 7 72.5 \pm 6 81.8 \pm 8 74.1 \pm 6 90.1 \pm 0 91.3 \pm 1 DP-2Stage-O +adult 53.5 \pm 16 61.8 \pm 16 59.2 \pm 13 56.9 \pm 20 63.3 \pm 22 60.8 \pm 18 92.0 \pm 1 93.0 \pm 1 +airline 48.8 \pm 12 58.0 \pm 12 57.5 \pm 9 56.2 \pm 14 64.0 \pm 13 59.2 \pm 11 89.8 \pm 1 91.0 \pm 1 Texas Marginal \varepsilon=1 AIM 85.4 \pm 1.0 98.3 \pm 0.0 94.9 \pm 0 83.5 \pm 1 98.4 \pm 0 93.4 \pm 0 98.7 \pm 0 99.2 \pm 0 \delta=10^{-5} MST 81.3 \pm 0 94.7 \pm 0 93.2 \pm 0 82.0 \pm 0 94.9 \pm 1 93.3 \pm 0 98.8 \pm 0 99.2 \pm 0 Non-LLM DP-GAN 13.4 \pm 19 61.2 \pm 12 78.2 \pm 9 14.1 \pm 14 55.5 \pm 12 78.5 \pm 8 66.0 \pm 7 70.5 \pm 6 DP-CTGAN 60.3 \pm 15 91.2 \pm 3 76.4 \pm 25 67.5 \pm 2 91.6 \pm 2 88.7 \pm 1 83.9 \pm 3 85.5 \pm 3 DP-VAE 0.0 \pm 0 50.0 \pm 0 82.5 \pm 0 0.0 \pm 0 50.0 \pm 0 82.5 \pm 0 77.0 \pm 0 78.9 \pm 0 GPT-2 DP-Standard 58.5 \pm 12 88.8 \pm 5 86.1 \pm 2 52.2 \pm 7 91.5 \pm 5 68.2 \pm 9 92.2 \pm 1 92.4 \pm 0 DP-2Stage-U 11.3 \pm 8 51.1 \pm 10 82.5 \pm 1 36.2 \pm 6 68.5 \pm 12 52.0 \pm 10 93.2 \pm 0 93.6 \pm 0 DP-2Stage-O +adult 77.6 \pm 2 96.5 \pm 1 91.1 \pm 1 72.0 \pm 4 96.9 \pm 1 87.0 \pm 3 91.3 \pm 1 92.1 \pm 1 +airline 78.0 \pm 2 96.3 \pm 0 91.3 \pm 1 70.6 \pm 3 96.4 \pm 0 86.3 \pm 2 93.0 \pm 0 93.4 \pm 0 Table 9: DP Benchmarks showing individual metric result. For each dataset, the best value per method group for \varepsilon=1 is shown in bold . Results are averaged across five model runs with varying random seeds, with four synthetic datasets generated per run. LR refers to the Logistic Regression model, and XGB represents the XGBoost model. XGB (F1) XGB (AUC) XGB (ACC) LR (F1) LR (AUC) LR (ACC) HIST (bin=50) HIST (bin=20) Dataset \varepsilon=8, Adult DP-Standard 17.3 \pm 6 58.6 \pm 4 73.6 \pm 1 45.2 \pm 6 65.8 \pm 8 61.8 \pm 4 84.1 \pm 1 84.9 \pm 1 \delta=10^{-5} DP-2Stage-U 9.7 \pm 6 49.4 \pm 4 74.4 \pm 1 35.1 \pm 8 54.1 \pm 9 52.9 \pm 5 86.4 \pm 1 87.4 \pm 1 DP-2Stage-O 19.1 \pm 9 58.5 \pm 9 74.2 \pm 1 47.6 \pm 5 69.1 \pm 7 62.3 \pm 4 87.5 \pm 1 88.3 \pm 1 Airline DP-Standard 61.4 \pm 4 66.1 \pm 7 63.4 \pm 7 68.5 \pm 6 73.6 \pm 9 68.4 \pm 7 89.1 \pm 3 90.5 \pm 3 DP-2Stage-U 69.1 \pm 11 77.8 \pm 10 73.6 \pm 8 74.7 \pm 7 83.5 \pm 10 76.2 \pm 8 89.9 \pm 1 90.9 \pm 1 DP-2Stage-O 60.2 \pm 8 67.9 \pm 6 65.2 \pm 5 68.2 \pm 11 75.5 \pm 13 70.3 \pm 10 91.9 \pm 1 92.8 \pm 1 Table 10: DP-GPT-2 Benchmarks showing individual metric result for \varepsilon=8 . For each dataset, the best value corresponding to different privacy budgets ( \varepsilon ) is highlighted in bold . Results are averaged across five model runs with varying random seeds, with four synthetic datasets generated per run. LR refers to the Logistic Regression model, and XGB represents the XGBoost model.