SoK: Can Synthetic Images Replace Real Data? A Survey of Utility and Privacy of Synthetic Image Generation

Yunsung Chung Yunbei Zhang Nassir Marrouche Jihun Hamm · 2025

Read SoK: Can Synthetic Images Replace Real Data? A Survey of Utility and Privacy of Synthetic Image Generation by Yunsung Chung Yunbei Zhang Nassir Marrouche Jihun Hamm (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-19360.txt.

Introduction

Yunsung Chung, Yunbei Zhang, Nassir Marrouche, and Jihun Hamm survey whether synthetic images can replace real photos for training classifiers, without leaking private data. Their 2025 survey notes that generative models have transformed fake image making. Yet the field lacked a unified comparison of methods, privacy attacks, and defenses across the full pipeline from generation to final classifier.

The authors sort methods along that pipeline and benchmark leading generators against membership inference attacks, which probe whether a photo was in the training set. They ask which release strategy balances usefulness and privacy, and whether defenses improve the trade. The study yields practical guidance for privacy-preserving data sharing.

Catalog record

CREATORYunsung Chung Yunbei Zhang Nassir Marrouche Jihun Hamm
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE92 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2506-19360

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

SoK: Can Synthetic Images Replace Real Data? A Survey of Utility and Privacy of Synthetic Image Generation Yunsung Chung Yunbei Zhang Nassir Marrouche Jihun Hamm Abstract Advances in generative models have transformed the field of synthetic image generation for privacy-preserving data synthesis (PPDS). However, the field lacks a comprehensive survey and comparison of synthetic image generation methods across diverse settings. In particular, when we generate synthetic images for the purpose of training a classifier, there is a pipeline of generation-sampling-classification which takes private training as input and outputs the final classifier of interest. In this survey, we systematically categorize existing image synthesis methods, privacy attacks, and mitigations along this generation-sampling-classification pipeline. To empirically compare diverse synthesis approaches, we provide a benchmark with representative generative methods and use model-agnostic membership inference attacks (MIAs) as a measure of privacy risk. Through this study, we seek to answer critical questions in PPDS: Can synthetic data effectively replace real data? Which release strategy balances utility and privacy? Do mitigations improve the utility-privacy tradeoff? Which generative models perform best across different scenarios? With a systematic evaluation of diverse methods, our study provides actionable insights into the utility-privacy tradeoffs of synthetic data generation methods and guides the decision on optimal data releasing strategies for real-world applications. 1 Introduction Can synthetic data replace real data? This question has become increasingly relevant as the demand for privacy-preserving techniques grows across various domains such as healthcare, finance, and social media [ 2 , 11 , 62 , 65 , 118 ] . In these fields, Privacy-Preserving Data Sharing (PPDS) offers a promising approach to protecting privacy while retaining analytical value. For example, in healthcare where patient data must be safeguarded, PPDS frameworks can enable machine learning model training on synthetic data without exposing real patient information. Similarly, in finance, synthetic transaction data allows the development of fraud detection systems while preserving customer confidentiality. These highlight the potential of PPDS to protect privacy while maintaining the utility. However, achieving an optimal balance of synthetic data between utility (such as data fidelity and classification accuracy) and privacy (such as Differential Private (DP) [ 31 ] and Membership Privacy [ 53 ] ) remains a significant challenge [ 107 ] . This paper seeks to answer fundamental questions about the role of synthetic data in PPDS: (1) Can synthetic data effectively replace real data? (2) Which is safer to share—synthetic images or classifiers trained on synthetic data? (3) Do mitigation strategies improve utility-privacy tradeoffs? (4) Which generative models perform best across different data-sharing scenarios? By addressing these questions, we aim to provide actionable insights into the strengths, limitations, and practical applicability of synthetic data for real-world privacy-preserving pipelines. Generative models such as GANs [ 38 , 77 , 90 , 4 , 57 , 13 , 127 , 58 ] , VAEs [ 63 , 21 , 114 , 92 , 22 ] , and diffusion models [ 46 , 102 , 26 , 47 , 91 , 93 ] have significantly advanced synthetic data generation. Subsequently, privacy-enhancing techniques such as DP [ 31 ] and DP-SGD [ 1 ] have been integrated and tested with generative models. These methods aim to ensure robust privacy guarantees by controlling the contribution of individual samples during training. Studies combining DP with diffusion models [ 27 , 37 , 73 , 115 ] have shown potential for generating privacy-preserving synthetic images. Similarly, adversarial training and mitigation strategies [ 72 ] have emerged to address the utility-privacy tradeoffs of generative models. However, the effectiveness of these methods in mitigating privacy risk across different release strategies and data-sharing settings remains unclear. While there are a number of surveys and benchmarks on the utility-privacy tradeoff of tabular data [ 107 , 98 ] , studies on synthetic image data are rare. A recent Systematization of Knowledge (SoK) on image generation [ 51 ] highlighted the challenges in producing high-quality privacy-preserving synthetic images under practical privacy budgets, which emphasizes the need for further study in systematically evaluating utility-privacy tradeoffs. Furthermore, direct comparison across diverse generative models such as GANs, diffusion models, and data condensation [ 116 , 132 , 131 ] within a unified framework is rarely found in the literature due to the difficulty of consistent evaluation. The few existing empirical studies are also inconclusive on the current success of synthetic image generation. For instance, while a prior work [ 107 ] reports significant utility loss when applying privacy-preserving techniques, [ 98 ] reported improved tradeoffs after addressing experimental flaws of the former. These observations underscore a critical need for a comprehensive systematization that not only surveys the landscape but also empirically benchmarks diverse approaches to provide clear and actionable insights. To address these gaps, this paper presents a SoK on the utility-privacy tradeoffs in privacy-preserving synthetic image generation. Our work goes beyond existing surveys by: 1. Developing and Applying a Unifying Framework : We introduce a data-sharing pipeline (Figure 1 , detailed in Sec. 3 ) that serves as a framework to systematically categorize and analyze methods, privacy attacks, and mitigation strategies at distinct points of information release: (1) the generative models; (2) the synthetic images produced, or (3) classifiers trained on synthetic data. This structured approach (detailed in Secs. 4 , 5 , and 6 ) allows for a coherent understanding of he varying risks and benefits. 2. Conducting a Broad and Unified Empirical Evaluation : We conduct a large-scale empirical study (Sec. 7 ) assessing a wide range of representative approaches selected to cover diverse generative architectures (VAE, GANs, Diffusion models, data condensation), training paradigms (from scratch vs. fine-tuning), and privacy considerations (non-mitigated, DP-enhanced, and other mitigation strategies). This evaluation, which represents a significant computational undertaking (detailed runtime analyses are provided in Appendix 6 to contextualize the cost), is performed under consistent conditions using established metrics for utility (classification performance) and privacy (Membership Inference Attacks) to allow for direct and fair comparisons—a critical element often missing in prior works. 3. Critically Evaluating and Contextualizing Knowledge : Through our benchmark, we provide novel insights into the practical performance, strengths, and limitations of these varied approaches across different datasets (CelebA, Fitzpatrick17k, and CheXpert) and data-sharing scenarios. This allows us to contextualize theoretical claims and evaluate the real-world efficacy of different utility-privacy balancing strategies. 4. Identifying Actionable Insights and Guiding Future Research : Based on our systematic findings, we derive actionable insights for practitioners and identify open challenges and promising directions for future research in this domain. Our literature search strategy to inform the selection of methods for this systematization involved employing relevant keywords (e.g., "membership inference attack," "generative models," "differential privacy," "synthetic data generation") on Google Scholar, prioritizing influential and recent works to ensure diverse coverage across methods and pipeline stages. This comprehensive approach, combining a structured framework with extensive empirical evidence, aims to bridge the gap between theory and practice in privacy-preserving synthetic image generation. Summary of Findings. Our comprehensive evaluation of representative methods across diverse datasets ( e.g. , CelebA [ 69 ] , Fitzpatrick17k [ 39 ] , and CheXpert [ 52 ] ) yields several critical insights into privacy-preserving synthetic data sharing (detailed in Sec. 7 ): 1. The optimal data-sharing strategy (releasing

Read the complete text