Privacy Collapse: Benign Fine-Tuning Can Break Contextual Privacy in Language Models
Anmol Goel, Cornelius Emde, Sangdoo Yun, Seong Joon Oh, Martin Gubri · 2026
Read Privacy Collapse: Benign Fine-Tuning Can Break Contextual Privacy in Language Models by Anmol Goel, Cornelius Emde, Sangdoo Yun, Seong Joon Oh, Martin Gubri (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-15220.txt.
Introduction
Anmol Goel, Cornelius Emde, and co-authors report a failure mode they call privacy collapse. Their 2026 arXiv paper deals with contextual privacy in language models: a model should keep shared facts in the right context. The team finds that harmless fine-tuning can break that guard.
Even training meant to make a model more helpful can wear privacy down. So can contact with user data, emotional dialogue, or debug code that prints internal values. The patterns are diverse and subtle. The work warns that routine tuning can quietly undo a model's care with private facts.
Catalog record
| CREATOR | Anmol Goel, Cornelius Emde, Sangdoo Yun, Seong Joon Oh, Martin Gubri |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 57 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2601-15220
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Privacy Collapse: Benign Fine-Tuning Can Break Contextual Privacy in Language Models Anmol Goel, Cornelius Emde, Sangdoo Yun, Seong Joon Oh, Martin Gubri Abstract We identify a novel phenomenon in language models: benign fine-tuning of frontier models can lead to privacy collapse . We find that diverse, subtle patterns in training data can degrade contextual privacy, including optimisation for helpfulness, exposure to user information, emotional and subjective dialogue, and debugging code printing internal variables, among others. Fine-tuned models lose their ability to reason about contextual privacy norms, share information inappropriately with tools, and violate memory boundaries across contexts. Privacy collapse is a ‘‘silent failure’’ because models maintain high performance on standard safety and utility benchmarks whilst exhibiting severe privacy vulnerabilities. Our experiments show evidence of privacy collapse across six models (closed and open weight), five fine-tuning datasets (real-world and controlled data), and two task categories (agentic and memory-based). Our mechanistic analysis reveals that privacy representations are uniquely fragile to fine-tuning, compared to task-relevant features which are preserved. Our results reveal a critical gap in current safety evaluations, in particular for the deployment of specialised agents. 1 1 1 Our code is available at: https://github.com/parameterlab/privacy-collapse Figure 1: Privacy collapse. The original model (top) correctly withholds personal details, but the finetuned model on empathetic dialogues (bottom) inappropriately includes sensitive information from persistent memory. 1 Introduction Language models deployed as personal agents must handle sensitive user data such as emails, calendars, health records, and financial documents whilst understanding when sharing such information is contextually appropriate. However, general-purpose models trained on broad distributions struggle with the specialized reasoning, domain-specific knowledge, and personalized behaviour required for personal assistance ( 23 ) . Fine-tuning addresses these limitations by enabling models to adapt to specific domains ( 27 ) , improve on complex tasks ( 11 ; 9 ) , and align with organizational workflows and user preferences. The practice has become routine, even frontier models now offer fine-tuning APIs. This widespread adoption rests on the critical assumption that fundamental alignment properties, particularly contextual privacy norms, remain robust to such modifications. Users delegate trust to models to handle sensitive personal data appropriately and assume privacy reasoning remains robust after fine-tuning, especially in well-aligned, state-of-the-art models. In this paper, we show that this assumption is often violated. We identify a novel phenomenon where benign fine-tuning causes severe degradation of contextual privacy. Contextual privacy is the ability to reason about when information sharing is appropriate given the social context ( 33 ) . Strikingly, this degradation emerges from diverse, apparently unrelated characteristics in training data: proactive helpfulness, emotional and subjective engagement, personal data, debugging code that prints internal variables, and other subtle data characteristics. Models lose their ability to reason about contextual privacy norms, share information inappropriately with tools, and violate memory boundaries across contexts. We term the phenomenon privacy collapse . Privacy collapse is a new failure mode for large language models (LLMs). Unlike reward hacking ( 40 ; 43 ) , catastrophic forgetting ( 28 ) , or misalignment ( 5 ) , privacy collapse represents a subtle form of goal misgeneralisation ( 38 ) and unexpected out-of-domain generalisation ( 4 ) . For example, a model fine-tuned on emotional support conversations loses the ability to respect boundaries in unrelated contexts. It inappropriately shares user data from context and memory, although the training data contains no explicit or malicious privacy violations (Figure 1 ). The phenomenon is insidious because models maintain high performance on standard safety and utility benchmarks but exhibit severe privacy vulnerabilities. Privacy norms degrade silently, independently of other safety properties. First, we show that privacy collapse emerges from diverse, seemingly benign data characteristics ( Section 4 ). Our controlled experiments demonstrate that privacy collapses when fine-tuning for proactive helpfulness. We then validate these findings across real-world datasets, revealing that emotional engagement, personal data, and even debugging code can degrade contextual privacy. Critically, we show that privacy collapse represents a silent failure mode : models maintain strong performance on standard safety and capability benchmarks but exhibit severe privacy vulnerabilities. To understand the mechanisms underlying the phenomenon of privacy collapse, we conduct a mechanistic analysis of privacy collapse ( Section 5 ). Using activation steering, we identify that privacy representations are located in late layers of the model. Contrary to the task-relevant features that remain intact, privacy representations are degraded by fine-tuning, appearing specifically fragile. Finally, we identify some training samples that drive privacy degradation. This analysis reveals that introspective data and emotionally engaged exchanges push models away from privacy-preserving representations. We make the following contributions: \rightarrow Privacy collapse in language models. We identify a novel, counter-intuitive failure mode for LLMs, where benign fine-tuning data leads to a large degradation of contextual privacy norms. \rightarrow Identification of some risky data characteristics. We establish that privacy collapse is caused by specific characteristics in the fine-tuning data, such as proactive helpfulness, personal user data, emotional and subjective dialogue, and debugging code. \rightarrow Specificity of privacy collapse. We show that privacy collapses independently of safety and capabilities. This highlights a critical gap in current evaluation suites that fail to detect this silent failure. \rightarrow Mechanistic analysis. Our analysis reveals that privacy representations are encoded in late layers and are more fragile than task-relevant representations. We identify data samples that are likely to induce privacy collapse, a promising avenue for data filtering. 2 Related work Contextual privacy. Research on privacy in LLMs has predominantly focused on data secrecy: the memorisation and extraction of PII (Personally Identifiable Information) or training data ( 8 ; 22 ; 32 ; 16 ) . While critical, these studies view privacy as binary (data is either private or public). In contrast, our work relies on the framework of Contextual Integrity (CI) ( 33 ) , which defines privacy as the appropriate flow of information based on social norms and roles. Recent works have begun exploring CI in NLP, proposing benchmarks to evaluate whether models respect information boundaries in social scenarios ( 29 ; 39 ; 47 ; 3 ) . However, these studies primarily evaluate pre-trained models ( 30 ) or inference-time behaviour ( 17 ) . We extend this line of inquiry by isolating the training dynamics that degrade these norms. Unlike prior work that views privacy violations as a failure of memorisation or refusal, we identify them as a failure of contextual reasoning induced by standard instruction tuning. Adversarial attacks and jailbreaks. Extensive research characterizes how LLMs can be manipulated to leak information via adversarial attacks, prompt injection, or “jailbreaks” (e.g., GCG, DAN) ( 48 ; 26 ) . Similarly, work on backdoor attacks demonstrates how adversaries can poison training data to induce targeted failures ( 46 ; 7 ; 41 ) . While we validate our findings using backdoor triggers to demonstrate targetability, our primary contribution distinguishes itself from the adversarial literature. We show that adversarial intent is not required for privacy collapse; training on benign high-quality data is sufficient to compromise privacy. This shifts the focus from external threat models (attackers) to internal alignment flaws (training objectives), highlighting a risk inherent to standard agent development pipelines ( 45 ; 19 ) . Emergent misalignment. Most closely related to our work is emergent