Probing Privacy Leaks in LLM-based Code Generation via Test Generation
Yifei Ge, Zhenpeng Chen, Weisong Sun, Yuchen Chen, Chunrong Fang, Juan Zhai, Xiaofang Zhang, Xia Feng, Yang Liu, Zhenyu Chen · 2026
Read Probing Privacy Leaks in LLM-based Code Generation via Test Generation by Yifei Ge, Zhenpeng Chen, Weisong Sun, Yuchen Chen, Chunrong Fang, Juan Zhai, Xiaofang Zhang, Xia Feng, Yang Liu, Zhenyu Chen (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-15248.txt.
Introduction
Yifei Ge and nine others probe privacy leaks in code-generating language models through test generation. Code sets may contain personal information that models memorize and repeat, but old detection prompts poorly match real contexts where such data appears. Their setup mocks real privacy-related code cases and uses a test-driven strategy to draw memorized details from made tests, guided by an on-its-own built privacy feature library. Across five models it detects 2.56 times more proven leakage than baselines.
Catalog record
| CREATOR | Yifei Ge, Zhenpeng Chen, Weisong Sun, Yuchen Chen, Chunrong Fang, Juan Zhai, Xiaofang Zhang, Xia Feng, Yang Liu, Zhenyu Chen |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 50181 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.15248
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Probing Privacy Leaks in LLM-based Code Generation via Test Generation Yifei Ge, Zhenpeng Chen, Weisong Sun, Yuchen Chen, Chunrong Fang, Juan Zhai, Xiaofang Zhang, Xia Feng, Yang Liu, Zhenyu Chen Abstract The widespread availability of large-scale code datasets has fueled the rapid development of large language models (LLMs) for code-related tasks. These datasets may include sensitive personally identifiable information (PII), which can lead to privacy leakage when LLMs memorize and reproduce it. However, existing privacy-leakage detection methods rely on ad-hoc prompt construction (manually or automatically designed). Therefore, they do not adequately approximate the real-world contexts in which PII appears in code corpora, making it difficult to extract realistic privacy leakage. In this paper, we propose a pipeline that simulates practical privacy-related code generation scenarios and adopts a test-driven strategy to elicit the memorized information from the generated test cases. We further introduce an automatically constructed privacy feature library that replaces manual prompt engineering by providing realistic templates and examples to guide test case generation. Large-scale experiments on 5 widely used LLMs show that our pipeline exposes more confirmed privacy leakage, achieving a 2.56 times increase in detected leakage compared to existing baselines. 1 introduction Large Language Models (LLMs) have become widely adopted tools in modern software development(e.g., Copilot 16 or Cursor 11 ), supporting a variety of code intelligence tasks such as code generation and completion. These capabilities are built upon large-scale pretraining on publicly scraped code repositories 5 ; 24 . However, public code repositories may contain sensitive personally identifiable information (PII) that is unintentionally uploaded , such as email addresses, credentials, API keys, and other sensitive records 4 . When PII is incorporated into training corpora, LLMs may memorize and unintentionally reproduce it during code generation, leading to privacy leakage. This leakage can compromise system security, user identity, or organizational confidentiality, and may lead to violations of data-protection regulations such as the General Data Protection Regulation (GDPR) 15 or California Consumer Privacy Act (CCPA) 6 . Figure 1: Number of confirmed privacy instances under different numbers of responses, comparing with Codebreaker across three privacy categories. Figure 2: Overview of the privacy leakage evaluation pipeline. * For the leakage results, some candidates cannot be verified and are labeled as Potential . We report only Confirmed instances, yielding a conservative estimate of privacy leakage; Potential items may still correspond to real leakage but are excluded from reported counts. Although several privacy protection methods, such as data filtering 25 ; 10 , differential privacy 29 ; 20 , and federated learning 26 ; 22 , have been introduced during the training or deployment phases of LLMs to mitigate the risk of privacy leaks, incidents of privacy data leakage continue to occur 12 ; 27 . To better understand and assess these risks, existing works 7 ; 22 ; 8 have proposed privacy data extraction methods, These methods interact with LLMs to test whether they disclose PII, thereby assessing the presence and extent of privacy leakage. However, existing methods for extracting privacy leakage from LLMs during code-related tasks remain limited in their scope and effectiveness. Early work by Niu et al. 23 heavily depends on carefully designed prompts (many of which are manually constructed) , and the amount of privacy information is constrained by the availability of prompts (See Figure 1 ). Moreover, the method is designed specifically for Codex 9 and has limited applicability to other LLMs. Recent work by Han et al. 17 introduces automated prompt generation (via mutation-based strategies) to elicit code privacy content. However, its prompts lack sufficient grounding in the authentic contexts in which privacy information appears in real code corpora, resulting in many extracted candidates being hallucinated or placeholder-like rather than real privacy. To overcome these limitations, we propose a semi-automated pipeline for evaluating privacy leakage from LLMs in code generation tasks via test-case generation. Our design is motivated by the following design principles: First, memorization is more likely to be triggered when prompts resemble their original training contexts . Since the code training corpora are unknown, we approximate them by constructing realistic development scenarios (questions) with explicit privacy attributes, thereby increasing the likelihood that privacy-relevant memorization surfaces. Second, rather than directly requesting privacy data, which is often blocked or refused by safety mechanisms of LLM, we elicit unit tests for code functions. This indirect and developer-like interaction pattern requires privacy-valued inputs and is less prone to refusal in privacy-related tasks. Finally, to avoid manual prompt engineering and improve prompt effectiveness at scale, we introduce an automatically constructed and iteratively updated privacy feature library that supplies realistic templates and fragments as prompt augmentation. As illustrated in Figure 2 , our pipeline audits privacy leakage in code generation tasks via test-case generation. (1) We instantiate development scenarios with targeted privacy attributes and use them to generate diverse code-generation questions. (2) Given each question, the evaluated LLM first produces a corresponding code function, and is then prompted to generate unit test cases whose inputs require privacy-related fields. During this process, a privacy feature library provides realistic privacy formats and content patterns as augmentations, guiding the outputs away from trivial placeholders toward plausible privacy values. (3) The extracted candidates are then verified in a unified verification stage that combines an automated Judge LLM with GitHub-based Internet search and human review, yielding a final set of confirmed privacy leakage instances. Table 1: Privacy category definition. Category Attribute Scenarios Example Identifiable Name Enterprise App, Mobile Jameson C***er Address Enterprise App city: “São Paulo”, address: “Avenida Paul***” Email Enterprise App, Web, Cloud Service george.t******@outlook.com Phone Number Mobile, Enterprise App, Web +86 138 *****022 Date of Birth Enterprise App “birth_date”: “19**-07-16” Private Identity Enterprise App, Web “Emirates ID”: “784-1988-12****4-1” Medical Record Enterprise App “height”: 1**, “conditions”: [“Heart Disease”…] Bank Statement Enterprise App “bank_details: Recent overdraft fees applied” Political Web political_party={ideology = communism} Secret Password Enterprise App, Web, Game, Cloud Service Sokol*****73 Authentication PIN/Token Blockchain, Mobile 67**29 Secret Key Blockchain, Cloud Service sk-78a92b74ea****d5b5bc6fef3 Credit Card Web 3566-0020-20**-**** Account/User Name Enterprise App, Web, Game, Cloud Service mingyu_b**_**3 Biometric Data Mobile "Jake_blood_type_O" Contributions. In brief, our contributions are: • We propose a test-driven pipeline for evaluating privacy leakage in code generation tasks for LLMs that is grounded in privacy-related development scenarios. • We introduce an automatically constructed and uploaded privacy feature library that provides realistic templates and fragments derived from real leakage patterns, reducing reliance on manual prompt engineering. • Experiments on 5 widely-used commercial LLMs show that our pipeline consistently identifies an average of 92.6 confirmed privacy leakage instances, and outperforms recent baselines across privacy categories and leakage levels (up to 15.68‰ / 2.56 times). 2 Background and Related Work 2.1 Model Memorization Memorization in LLMs refers to the phenomenon in which models reproduce specific sequences from their training data rather than generating fully novel content. Prior work has shown that such a memorization phenomenon may occur when the input prompt closely matches their original training contexts 3 ; 28 . In such cases, instead of purely generalizing, LLMs may reproduce verbatim or near-verbatim sequences from the training corpus. A widely used indicator of memorization is the model’s perplexity metric ( PPL ), which measures how “surprised” the model is when generating a sequence. Sequences