Privacy Auditing Synthetic Data Release through Local Likelihood Attacks
Joshua Ward, Chi-Hua Wang, Guang Cheng · 2025
Read Privacy Auditing Synthetic Data Release through Local Likelihood Attacks by Joshua Ward, Chi-Hua Wang, Guang Cheng (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-21146.txt.
Introduction
Joshua Ward, Chi-Hua Wang, and Guang Cheng proposed in 2025 a way to audit the privacy of synthetic data releases by attacking them. Synthetic data is shared as a safe stand-in for real records, but leaks can expose who was in the training set. Current audits rest on heuristics and generous assumptions about access to the model, so their verdicts are weak.
Their attack uses the fact that table generators tend to overfit certain regions of the training data. It judges membership by how much a test record sways a surrogate model's estimate of a local likelihood ratio over the synthetic data. No knowledge of the real model is assumed, and the math behind the score is developed in the paper. Readers get a stronger, no-access test that data holders can run before release.
Catalog record
| CREATOR | Joshua Ward, Chi-Hua Wang, Guang Cheng |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 47 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2508-21146
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Privacy Auditing Synthetic Data Release through Local Likelihood Attacks Joshua Ward, Chi-Hua Wang, Guang Cheng Abstract Auditing the privacy leakage of synthetic data is an important but unresolved problem. Most existing privacy auditing frameworks for synthetic data rely on heuristics and unreasonable assumptions to attack the failure modes of generative models, exhibiting limited capability to describe and detect the privacy exposure of training data through synthetic data release. In this paper, we study designing Membership Inference Attacks (MIAs) that specifically exploit the observation that tabular generative models tend to significantly overfit to certain regions of the training distribution. Here, we propose Generative Likelihood Ratio Attack (Gen-LRA), a novel, computationally efficient No-Box MIA that, with no assumption of model knowledge or access, formulates its attack by evaluating the influence a test observation has in a surrogate model’s estimation of a local likelihood ratio over the synthetic data. Assessed over a comprehensive benchmark spanning diverse datasets, model architectures, and attack parameters, we find that Gen-LRA consistently dominates other MIAs for generative models across multiple performance metrics. These results underscore Gen-LRA’s effectiveness as a privacy auditing tool for the release of synthetic data, highlighting the significant privacy risks posed by generative model overfitting in real-world applications. 1 Introduction Real world tabular data is often privacy-sensitive to the individual observations that compose these samples, hindering their ability to be shared in open-science efforts that can aid in new research and improve reproducibility. A promise of generative modeling is that models trained on sensitive data can produce samples that preserve the privacy of the training set while maintaining much of its intrinsic statistical information, enabling responsible release to a third party. In practice, a wide array of methodologies have been proposed to accomplish synthetic data release involving modifying loss functions ( Abadi et al. 2016 ; Wang et al. 2022 ) , creating new generative model architectures ( Yoon et al. 2019 ; Yoon et al. 2020a ) , and studying data release strategies ( Hardt et al. 2012 ; Gupta et al. 2012 ; Takagi et al. 2021 ) to provide differential privacy guarantee. In another direction, a variety of methods have been proposed that maximize the fidelity of synthetic data and argue that privacy is satisfied through ad-hoc similarity metrics ( Zhao et al. 2021 ; Guillaudeux et al. 2022 ; Liu et al. 2023 ; Solatorio and Dupriez 2023 ) . To audit the empirical privacy of synthetic data generators, Membership Inference Attacks (MIAs) have recently been extended from traditional machine learning models to synthetic tabular data. Here, privacy auditing is framed as an adversarial game: given specific constraints defined by a threat model, an attacker attempts to determine whether a test observation belongs to a model’s training dataset exploiting some notion of model failure ( Shokri et al. 2017 ; Chen et al. 2020 ; Carlini et al. 2021 ) . A successful attack represents a concrete privacy breach with clear real-world implications, where other similarity-based metrics have been shown to fail to capture privacy risk ( Platzer and Reutterer 2021 ; Ganev and Cristofaro 2023 ; Ward et al. 2024 ) . Table 1: Average AUC-ROC for each Membership Inference Attack across model architectures and datasets. Gen-LRA outperforms all other threat-model comparable attacks with an average rank of 1 across all architectures. Model Gen-LRA (Ours) DCR-Diff DOMIAS DPI DCR MC LOGAN AdsGAN 0.534 (0.02) 0.517 (0.02) 0.517 (0.02) 0.521 (0.02) 0.516 (0.02) 0.515 (0.02) 0.503 (0.02) ARF 0.562 (0.03) 0.540 (0.02) 0.534 (0.02) 0.538 (0.02) 0.533 (0.02) 0.527 (0.02) 0.504 (0.02) Bayesian Network 0.679 (0.07) 0.656 (0.06) 0.632 (0.06) 0.557 (0.02) 0.665 (0.07) 0.625 (0.05) 0.505 (0.02) CTGAN 0.533 (0.02) 0.515 (0.02) 0.515 (0.02) 0.519 (0.02) 0.513 (0.02) 0.511 (0.02) 0.504 (0.02) Normalizing Flows 0.524 (0.02) 0.504 (0.02) 0.505 (0.02) 0.506 (0.02) 0.505 (0.02) 0.504 (0.02) 0.502 (0.02) PATEGAN 0.520 (0.02) 0.497 (0.02) 0.498 (0.02) 0.500 (0.02) 0.500 (0.02) 0.501 (0.02) 0.502 (0.02) Tab-DDPM 0.603 (0.08) 0.587 (0.06) 0.587 (0.06) 0.552 (0.03) 0.585 (0.07) 0.564 (0.05) 0.505 (0.02) TabSyn 0.583 (0.04) 0.553 (0.02) 0.561 (0.06) 0.547 (0.06) 0.585 (0.07) 0.517 (0.05) 0.501 (0.02) TVAE 0.541 (0.02) 0.529 (0.03) 0.524 (0.03) 0.523 (0.02) 0.529 (0.03) 0.522 (0.02) 0.504 (0.02) Average Rank 1.0 3.4 3.6 3.8 3.8 5.34 6.4 While a promising, MIAs for generative models and synthetic data release have seen limited success. Previous work in MIAs for synthetic data release has often relied on distance or density-based heuristics for their attacks or have included additional assumptions about model query access that are unrealistic to the release setting and computationally do not scale to modern architectures. In contrast, we focus on studying membership inference for the release of synthetic data in a No-Box Threat Model ( Houssiau et al. 2022 ) . In this approach, we make no adversarial assumptions of knowledge about model architecture, access, and training parammeters that mimics real-world scenarios of parties following best practices for releasing synthetic data in domains like healthcare and finance. Under this threat model, we derive a powerful MIA called Generative Likelihood Ratio Attack (Gen-LRA) which constructs an influence function formulated from likelihood ratio estimation to target privacy leakage that occurs through model overfitting. We show that our attack broadly outperforms competing methods especially at low fixed false positive rates, highlighting that overfitting presents a more dangerous source of privacy leakage then previously suggested. Our contributions are as follows: Contributions : 1. We introduce Gen-LRA, a novel MIA that uses an influence function framework to attack overfitting in tabular generative models with minimal assumptions by evaluating the likelihood ratio of synthetic data under a surrogate model trained with and without a test point. 2. We show that Gen-LRA is computationally efficient and broadly outperforms other MIAs for synthetic data generators across a diverse benchmark of datasets, model architectures, and experiment parameters. (Table 1 ) 3. We demonstrate that Gen-LRA better identifies subgroups of training observation that experience egregious privacy leakage relative to other attacks (Table 2 ). We also show that Gen-LRA can be used as an evaluation tool for overfitting in tabular generative models (Figure 2 ). 2 Membership Inference Attacks Formalism In this work, we specifically study the Membership Inference Attack Game in the context of synthetic data release . The objective of this game is to determine whether a particular data point was included in the original training dataset by examining the outputs of a generative model. We first introduce the formal definition of the Membership Inference Attack Game : Definition (Membership Inference Attack Game). The game proceeds between a challenger \mathcal{C} and an adversary \mathcal{A} as follows: 1. The challenger samples a training dataset T={x_{i}}_{i=1}^{n} from the population distribution x_{i}\sim\mathbb{P} and uses T to train a tabular generative model \mathcal{G}\leftarrow\mathcal{T}(T) . The generative model \mathcal{G} produces synthetic dataset S . 2. The challenger flips a bit b\in{0,1} . If b=0 , the challenger samples a test observation x^{\star} from the population distribution \mathbb{P} . Otherwise, the challenger selects the test observation x^{\star} from the training set T . 3. The challenger sends the test observation x^{\star} to the adversary \mathcal{A} . 4. The adversary has access to some information defined by a threat model and uses this information to