ContextLeak: Auditing Leakage in Private In-Context Learning Methods

Jacob Choi, Shuying Cao, Xingjian Dong, Wang Bill Zhu, Robin Jia, Sai Praneeth Karimireddy · 2025

Read ContextLeak: Auditing Leakage in Private In-Context Learning Methods by Jacob Choi, Shuying Cao, Xingjian Dong, Wang Bill Zhu, Robin Jia, Sai Praneeth Karimireddy (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2512-16059.txt.

Description

Jacob Choi and six co-authors test privacy tools for language models in an arXiv paper published in 2025. Prompts often include example data to teach a model a task, a method called in-context learning. When those examples hold sensitive facts, model outputs might leak them, and many defenses claim to prevent that.

The team introduces ContextLeak, the first framework that measures worst-case leakage in practice. It plants unique canary tokens in the private data, then crafts queries designed to pull them out. Applied to heuristic prompt defenses and formal differential privacy methods alike, ContextLeak reliably exposed leaks, and leakage grew as the stated privacy budget loosened. The audit shows current methods trade off badly, either leaking badly or wrecking performance, which gives builders a practical warning signal.

Catalog record

CREATORJacob Choi, Shuying Cao, Xingjian Dong, Wang Bill Zhu, Robin Jia, Sai Praneeth Karimireddy
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE47 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2512-16059

Browse the full library · About the Tower of Babel