The Canary’s Echo: Auditing Privacy Risks of LLM-Generated Synthetic Text

Matthieu Meeus, Lukas Wutschitz, Santiago Zanella-Béguelin, Shruti Tople, Reza Shokri · 2025

Read The Canary’s Echo: Auditing Privacy Risks of LLM-Generated Synthetic Text by Matthieu Meeus, Lukas Wutschitz, Santiago Zanella-Béguelin, Shruti Tople, Reza Shokri (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-14921.txt.

Introduction

Matthieu Meeus and co-authors audit whether synthetic text from a language model can leak facts about the private data used to fine-tune it, in a 2025 paper. Synthetic data is often shared as a privacy-safe stand-in. The team assumes an attacker sees only the synthetic output and tries to infer which records trained the model.

Their membership attacks worked well enough to show real leakage. Planted canary records designed for direct model access proved weak here, because they rarely surface in useful synthetic text. The authors craft better canaries, with natural openings and odd endings, that leave detectable traces. The work sharpens how such releases should be audited.

Catalog record

CREATORMatthieu Meeus, Lukas Wutschitz, Santiago Zanella-Béguelin, Shruti Tople, Reza Shokri
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE84 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2502-14921

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

The Canary’s Echo: Auditing Privacy Risks of LLM-Generated Synthetic Text Matthieu Meeus, Lukas Wutschitz, Santiago Zanella-Béguelin, Shruti Tople, Reza Shokri Abstract How much information about training samples can be leaked through synthetic data generated by Large Language Models (LLMs)? Overlooking the subtleties of information flow in synthetic data generation pipelines can lead to a false sense of privacy. In this paper, we assume an adversary has access to some synthetic data generated by a LLM. We design membership inference attacks (MIAs) that target the training data used to fine-tune the LLM that is then used to synthesize data. The significant performance of our MIA shows that synthetic data leak information about the training data. Further, we find that canaries crafted for model-based MIAs are sub-optimal for privacy auditing when only synthetic data is released. Such out-of-distribution canaries have limited influence on the model’s output when prompted to generate useful, in-distribution synthetic data, which drastically reduces their effectiveness. To tackle this problem, we leverage the mechanics of auto-regressive models to design canaries with an in-distribution prefix and a high-perplexity suffix that leave detectable traces in synthetic data. This enhances the power of data-based MIAs and provides a better assessment of the privacy risks of releasing synthetic data generated by LLMs. 1 Introduction Large Language Models (LLMs) can generate synthetic data that mimics human-written content through domain-specific prompts. Besides their impressive fluency, LLMs are known to memorize parts of their training data ( Carlini et al., 2023 ) and can regurgitate exact phrases, sentences, or even longer passages when prompted adversarially ( Zanella-Béguelin et al., 2020 ; Carlini et al., 2021 ; Nasr et al., 2023 ) . This raises serious privacy concerns about unintended information leakage through synthetically generated text. In this paper, we address the critical question: to what extent does synthetic text generated by LLMs leak information about the real data it is derived from? Prior methods to audit privacy risks insert highly vulnerable, out-of-distribution examples, canaries ( Carlini et al., 2019 ) , into the training data and test whether they can be identified using membership inference attacks (MIAs) ( Shokri et al., 2017 ) . Various MIAs have been proposed, typically assuming an attacker with access to the trained model or its output logits ( Carlini et al., 2019 ; Shi et al., 2024 ) . In the context of LLMs, MIAs often rely on analyzing the model’s behavior when prompted with inputs related to the canaries ( Carlini et al., 2021 ; Chang et al., 2024 ; Shi et al., 2024 ) . However, similar investigations are lacking in scenarios where LLMs are used to generate synthetic data and only this synthetic data is available to an attacker. Contributions. In this work, we study–for the first time–the factors that influence information leakage from a synthetic data-corpus generated using LLMs. First, we introduce data-based attacks that only have access to synthetic data, and not to the model used to generate it, and therefore cannot probe it with adversarial prompts nor compute losses or other statistics used in model-based attacks ( Ye et al., 2022 ) . We propose approximating membership likelihood using either a model trained on the synthetic data or the target example similarity to its closest synthetic data examples. We design our attacks adapting the state-of-the-art pairwise likelihood ratio tests as in RMIA ( Zarifzadeh et al., 2024 ) and evaluate them on labeled datasets: SST-2 ( Socher et al., 2013 ) , AG News ( Zhang et al., 2015 ) and SNLI ( Bowman et al., 2015 ) . Our results show that MIAs leveraging only synthetic data achieve AUC scores of 0.74 for SST-2, 0.68 for AG News and 0.77 for SNLI, largely outperforming a random guess baseline. This suggests that synthetic text can leak significant amount of information about the real data used to generate it. Second, we use the attacks we introduce to quantify the gap in performance between data- and model-based attacks. We do so in an auditing scenario, designing adversarial canaries and controlling leakage by varying the number of times a canary occurs in the training dataset. Experimentally, we find a sizable gap when comparing attacks adapted to the idiosyncrasies of each setting: a canary would need to occur 8\times more often to be as vulnerable against a data-based attack as it is against a model-based attack (see Figs. 1(a) and 1(d) ). Third, we discover that canaries designed for model-based attacks fall short when auditing privacy risks of synthetic text. Indeed, privacy auditing of LLMs through model-based MIAs relies on rare, out-of-distribution sequences of high perplexity ( Carlini et al., 2019 ; Stock et al., 2022 ; Wei et al., 2024 ; Meeus et al., 2024c ) . We confirm that model-based MIAs improve as canary perplexity increases. In sharp contrast, we find that high perplexity sequences, although distinctly memorized by the target model, are less likely to be echoed through synthetic data generated by the target model. Therefore, as a canary perplexity increases, the canary influence on synthetic data decreases, making its membership less detectable from synthetic data (see Figure 2 ). We show that low-perplexity, and even in-distribution canaries, while suboptimal for model-based attacks, are more adequate canaries in data-based attacks. Next, we propose an alternative canary design tailored for data-based attacks based on the following intuition: (i) in-distribution canaries aligned with the domain-specific prompt can influence the generated output (ii) memorization is more likely when canaries contain sub-sequences with high perplexity. We construct canaries starting with an in-distribution prefix of length F , transitioning into an out-of-distribution suffix, increasing the likelihood that the model memorizes them and that they influence synthetic data. We show that, for fixed overall canary perplexity, the performance of attacks for canaries with in-distribution prefix and out-of-distribution suffix ( 0<F<\text{max} ) improves upon both entirely in-distribution canaries ( F=\text{max} ) and out-of-distribution canaries ( F=0 ), across datasets (see Fig. 1 and Table 2 ). Lastly, we evaluate our attacks on synthetic data generated with formal privacy guarantees. We adopt the training-time method proposed by work ( Yue et al., 2023 ; Mattern et al., 2022 ; Kurakin et al., 2023 ) and finetune the target model on the private dataset using DP-SGD ( Abadi et al., 2016 ) with \epsilon=8 . We find the performance of the strongest data-based MIA to drop to random guess performance (AUC of 0.5 ), confirming that differential privacy constitutes a strong defense. Taken together, the proposed MIAs and canary design can be used to audit privacy risks of synthetic text. Auditing establishes a lower bound on the risk, useful to take informed decisions about releasing synthetic data in sensitive applications and also complements upper bounds on privacy risks from methods that synthesize text with provable guarantees. 2 Background and problem statement Synthetic text generation. We consider a private dataset D=\{x_{i}=(s_{i},\ell_{i})\}_{i=1}^{N} of labelled text records where s_{i} represents a sequence of tokens (e.g. a product review) and \ell_{i} is a class label (e.g. the review sentiment). A synthetic data generation mechanism is a probabilistic procedure mapping D to a synthetic dataset {\widetilde{D}}=\{{\widetilde{x}}_{i}=({\widetilde{s}}_{i},{\widetilde{\ell}}_{i})\}_{i=1}^{\widetilde{N}} with a desired label set \{\ell_{i}\}_{i=1}^{\widetilde{N}} . Unless stated otherwise, we consider N={\widetilde{N}}

Read the complete text