ContextLeak: Auditing Leakage in Private In-Context Learning Methods
Jacob Choi, Shuying Cao, Xingjian Dong, Wang Bill Zhu, Robin Jia, Sai Praneeth Karimireddy · 2025
Read ContextLeak: Auditing Leakage in Private In-Context Learning Methods by Jacob Choi, Shuying Cao, Xingjian Dong, Wang Bill Zhu, Robin Jia, Sai Praneeth Karimireddy (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2512-16059.txt.
Description
Jacob Choi and six co-authors test privacy tools for language models in an arXiv paper published in 2025. Prompts often include example data to teach a model a task, a method called in-context learning. When those examples hold sensitive facts, model outputs might leak them, and many defenses claim to prevent that.
The team introduces ContextLeak, the first framework that measures worst-case leakage in practice. It plants unique canary tokens in the private data, then crafts queries designed to pull them out. Applied to heuristic prompt defenses and formal differential privacy methods alike, ContextLeak reliably exposed leaks, and leakage grew as the stated privacy budget loosened. The audit shows current methods trade off badly, either leaking badly or wrecking performance, which gives builders a practical warning signal.
Catalog record
| CREATOR | Jacob Choi, Shuying Cao, Xingjian Dong, Wang Bill Zhu, Robin Jia, Sai Praneeth Karimireddy |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 47 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2512-16059