The DCR Delusion: Measuring the Privacy Risk of Synthetic Data
Zexi Yao, Nataša Krčo, Georgi Ganev, Yves-Alexandre de Montjoye, Imperial College London · 2025
Read The DCR Delusion: Measuring the Privacy Risk of Synthetic Data by Zexi Yao, Nataša Krčo, Georgi Ganev, Yves-Alexandre de Montjoye, Imperial College London (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-01524.txt.
Introduction
Zexi Yao, Natasa Krco, Georgi Ganev, and Yves-Alexandre de Montjoye show that a shortcut for judging the privacy of synthetic data does not work. Synthetic data sets are shared as safe stand-ins for sensitive records, and the gold-standard check is an attack that guesses whether a record was in the training set. Practitioners instead use distance-to-closest-record scores, which pass a data set when its rows are no closer to the training data than real rows are. Writing in twenty twenty-five, the team tests that shortcut across data sets and generator types, from classic models to diffusion models.
Data sets that passed the distance test fell to membership attacks, again and again. Both the pass-fail version and the continuous score failed to track risk under settings. The authors give a leaking example the metric misses and call the approach flawed by design.
Catalog record
| CREATOR | Zexi Yao, Nataša Krčo, Georgi Ganev, Yves-Alexandre de Montjoye, Imperial College London |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 37 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-01524
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
The DCR Delusion: Measuring the Privacy Risk of Synthetic Data Zexi Yao, Nataša Krčo, Georgi Ganev, Yves-Alexandre de Montjoye, Imperial College London Abstract Synthetic data has become an increasingly popular way to share data without revealing sensitive information. Though Membership Inference Attacks (MIAs) are widely considered the gold standard for empirically assessing the privacy of a synthetic dataset, practitioners and researchers often rely on simpler proxy metrics such as Distance to Closest Record (DCR). These metrics estimate privacy by measuring the similarity between the training data and generated synthetic data. This similarity is also compared against that between the training data and a disjoint holdout set of real records to construct a binary privacy test. If the synthetic data is not more similar to the training data than the holdout set is, it passes the test and is considered private. In this work we show that, while computationally inexpensive, DCR and other distance-based metrics fail to identify privacy leakage. Across multiple datasets and both classical models such as Baynet and CTGAN and more recent diffusion models, we show that datasets deemed private by proxy metrics are highly vulnerable to MIAs. We similarly find both the binary privacy test and the continuous measure based on these metrics to be uninformative of actual membership inference risk. We further show that these failures are consistent across different metric hyperparameter settings and record selection methods. Finally, we argue DCR and other distance-based metrics to be flawed by design and show a example of a simple leakage they miss in practice. With this work, we hope to motivate practitioners to move away from proxy metrics to MIAs as the rigorous, comprehensive standard of evaluating privacy of synthetic data, in particular to make claims of datasets being legally anonymous. * * footnotetext: Equal contribution 1 Introduction Synthetic data is a popular tool for sharing and using sensitive data, used across fields such as medicine [ 12 , 14 ] , finance [ 57 ] and public security [ 48 ] . Synthetic data generators (SDGs) aim to learn the underlying distribution of a dataset and generate synthetic data that preserves its statistical properties while protecting the privacy of individual records. Membership inference attacks (MIAs) are widely accepted as the standard method to empirically assess information leakage and the privacy of synthetic data [ 26 , 38 , 56 , 16 , 6 , 31 ] and machine learning models in general [ 10 , 50 , 69 ] , both as a direct attack, and as an upper bound on more severe threats such as reconstruction or attribute inference attacks [ 52 , 5 ] . MIAs evaluate the privacy of synthetic data at an individual level by assessing the risk of a particular record’s membership in the training dataset being correctly inferred by an attacker. While MIAs are the state-of-the-art method for evaluating privacy, they typically involve training multiple generative models, leading to high computational costs and motivating the use of simpler distance-based metrics. Distance to Closest Record (DCR) and similar metrics are commonly used as a proxy for MIAs, both in commercial products [ 41 , 58 ] and for evaluating novel methods such as diffusion models [ 32 , 70 , 46 ] . These metrics assess the privacy of a synthetic dataset as a whole by measuring the similarity between synthetic and training data. Intuitively, the less similar a synthetic dataset is to its training data, the stronger the assumed privacy. The metrics can be used either to construct a binary privacy test \tau_{\text{DCR}} classifying a dataset as “private” or “non-private,” or directly as a continuous measure of privacy \mu_{\text{DCR}} . The binary test compares the distribution of distances between synthetic and training records against the distances between a set of real holdout records and the training records, typically at a certain percentile. Contribution. In this work, we evaluate the effectiveness of DCR and similar metrics as a proxy for membership inference attacks, and show them to be an inadequate measure of both information leakage and the privacy risk of generated synthetic data. First, we show proxy metrics to provide a misleading measure of privacy risk for well-known classical SDGs: IndHist [ 49 ] , Baynet [ 71 ] , and CTGAN [ 64 ] . Across 9 experimental setups, we generate more than 10,000 datasets and find the majority of them to be deemed private by proxy metrics as applied in industry. Yet, instantiating MIAs against outlier records in these datasets reveals significant information leakage, with records shown to be highly vulnerable to membership inference attacks (AUC >0.8 ). Worse, we show MIAs to perform equally well against datasets deemed “private” and “non-private” by DCR, and an absence of correlation between MIA performance and \mu_{\text{DCR}} . Second, we show our empirical results to extend to diffusion models, a more recent popular class of synthetic data generation models. For diffusion models TabDDPM [ 32 ] and ClavaDDPM [ 46 ] , we generate synthetic datasets considered private by \tau_{\text{DCR}} , and instantiate the state-of-the-art MIAs for tabular diffusion models against them. Similarly to classical models, the MIAs reach high performance (TPR at FPR=0% above 10\% ) despite passing the binary privacy test \tau_{\text{DCR}} . \mu_{\text{DCR}} also shows no correlation with vulnerability to MIAs. For computational reasons, we previously focused on outlier records which are more likely to be vulnerable to MIAs in the case of classical models. We here study for the Baynet generator and Adult [ 7 ] dataset the risk for every record in the target dataset and show that the risk is not limited to outliers. Though the synthetic datasets pass the binary privacy test, an MIA is able to infer the membership of 20% of the training records better than a random guess (AUC \geq 0.6 ). We also show that our findings hold across different choices of the \tau_{\text{DCR}} hyperparameter, the comparison percentile. Finally, we study a real-world example of a simple privacy leakage that the proxy metrics are by-design unable to detect. Taken together, our results show DCR and other distance-based metrics to be poor proxies for measuring privacy risk. They detect only the most severe privacy violations, such as when synthetic data consists mostly of copies of the real data, and potentially leaving more subtle information leakage undetected. They also seem to show no meaningful correlation with actual privacy risk, making them unreliable even as general indicators of privacy. We hope this work will motivate rigorous privacy evaluation using state-of-the-art attacks defined in the literature, and encourage researchers and practitioners to move away from using distance-based metrics. 2 Preliminaries In this section, we introduce relevant notation, synthetic data generators, and methods for measuring privacy of synthetic data. Notation. We denote a record consisting of k attributes with x_{i}=(x_{i,1},\ldots,x_{i,k})\sim\mathcal{D} , where \mathcal{D} is the distribution over feature space \mathcal{F}=\mathcal{F}_{1}\times\ldots\times\mathcal{F}_{k} . D=\{x_{1},\ldots,x_{m}\} denotes a tabular dataset where one record x_{i} corresponds to one row. Synthetic Data Generators (SDGs). Let D_{\text{target}}=\{x_{1},\ldots,x_{n}\} be a real tabular dataset. A generative model M=\phi(D_{\text{target}}) trained using procedure \phi estimates the underlying distribution of D_{\text{target}} . A synthetic dataset D_{\text{synthetic}}\sim M can then be sampled from the model. In general, we consider synthetic