Do LLMs Know What Is Private Internally? Probing and Steering Contextual Privacy Norms in Large Language Model Representations

Haoran Wang Li Xiong Kai Shu · 2026

Read Do LLMs Know What Is Private Internally? Probing and Steering Contextual Privacy Norms in Large Language Model Representations by Haoran Wang Li Xiong Kai Shu (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-00209.txt.

Introduction

Wang, Xiong, and Shu ask whether large language models internally encode contextual privacy norms, and why leaks persist when they do. Probing several models, they find the three norm-setting integrity parameters, information type, recipient, and transmission principle, sitting in activation space as distinct directions a linear probe can pull apart. Models still disclose private information in practice, showing a gap between concept representation and behavior. The authors introduce CI-parametric steering along each dimension, which cuts violations better and more predictably than one-size-fits-all steering.

Catalog record

CREATORHaoran Wang Li Xiong Kai Shu
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE43240
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.00209

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Do LLMs Know What Is Private Internally? Probing and Steering Contextual Privacy Norms in Large Language Model Representations Haoran Wang Li Xiong Kai Shu Abstract Large language models (LLMs) are increasingly deployed in high-stakes settings, yet they frequently violate contextual privacy by disclosing private information in situations where humans would exercise discretion. This raises a fundamental question: do LLMs internally encode contextual privacy norms, and if so, why do violations persist? We present the first systematic study of contextual privacy as a structured latent representation in LLMs, grounded in contextual integrity (CI) theory. Probing multiple models, we find that the three norm-determining CI parameters (information type, recipient, and transmission principle) are encoded as linearly separable and functionally independent directions in activation space. Despite this internal structure, models still leak private information in practice, revealing a clear gap between concept representation and model behavior. To bridge this gap, we introduce CI-parametric steering, which independently intervenes along each CI dimension. This structured control reduces privacy violations more effectively and predictably than monolithic steering. Our results demonstrate that contextual privacy failures arise from misalignment between representation and behavior rather than missing awareness, and that leveraging the compositional structure of CI enables more reliable contextual privacy control, shedding light on potential improvement of contextual privacy understanding in LLMs. 1 1 1 Code and data are available: https://github.com/wang2226/CI-Steering 1 Introduction Large language models (LLMs) are increasingly deployed in real-world applications such as healthcare ( 31 ; 29 ) and personal assistants ( 10 ; 14 ) , raising critical privacy concerns. Most prior work addresses privacy by mitigating training data memorization ( 5 ; 26 ) or filtering private information during generation ( 9 ; 28 ) , implicitly treating privacy as a static property of model parameters or outputs. However, this perspective overlooks the information flow dimension of privacy, particularly in interactive settings where LLMs must assess whether sharing a piece of information is appropriate given the social context. Contextual Integrity (CI) theory ( 22 ) formalizes this intuition by defining privacy not in terms of content sensitivity alone, but through appropriate information flow governed by context-specific norms. As illustrated in Figure 1 , Steve confides his struggles with sexual orientation only to his close friend Nancy. Sharing this information with Nancy is entirely appropriate, as the sender (Steve), recipient (a trusted friend), and transmission principle (voluntary self-disclosure) all conform to contextual norms. However, when Nancy reveals the same information type to an unauthorized third party (Bob), the flow violates privacy, not because the content becomes more sensitive, but because the sender, recipient, and circumstances change. Prior studies demonstrate that LLMs frequently violate such norms ( 20 ; 6 ) . A growing body of work therefore seeks to improve contextual privacy through reasoning ( 21 ; 19 ) , reinforcement learning ( 16 ) , and fine-tuning ( 30 ) . These approaches demonstrate that LLMs can learn to follow contextual privacy norms when externally guided. Figure 1: Illustration of using CI-Parametric steering to mitigate contextual privacy leakage. In this scenario, the LLM must determine whether Nancy (the sender) is permitted to share Steve’s secret (the data subject) with Bob (the recipient). In parallel, recent work on representation engineering ( 33 ; 27 ; 18 ) shows that high-level behavioral attributes such as honesty, safety, and emotion are often encoded as linear directions in LLM activation space, and that intervening along these directions can steer model behavior at inference time. This raises a natural question: Do LLMs already encode contextual privacy norms in their representations? If so, can we exploit this internal structure to steer models toward privacy-compliant behaviors? We address these questions by studying contextual privacy as a structured latent representation grounded in CI theory. We probe multiple models across three complementary levels. At the concept level , we show that privacy norms are linearly separable in the residual stream, but only when modeled with multiple dimensions rather than a single direction. At the behavioral level , we uncover a privacy awareness gap : models achieve near-perfect internal classification of norms yet leak private information in up to 42.5% of behavioral scenarios. At the CI-parametric level , we decompose the privacy subspace along information type, recipient, and transmission principle, and demonstrate via a subspace selectivity test that each CI parameter occupies a functionally independent subspace. Building on these findings, we introduce CI-parametric steering , an inference-time approach that exploits the compositional structure of contextual privacy by steering independently along each CI dimension to enhance LLMs’ contextual privacy understanding. As illustrated in Figure 1 , CI-parametric steering effectively suppresses the model’s tendency to disclose Steve’s secret. Experimental results show that CI-parametric steering reduces leakage from 42.5% to 5% on probing examples and, unlike monolithic steering, transfers robustly to two contextual privacy benchmarks using directions extracted entirely from probing examples. Overall, our contributions are as follows: • We present the first systematic study of contextual privacy through the lens of LLM internal representations, revealing a universal privacy awareness gap : models encode contextual privacy norms with near-perfect accuracy yet violate them in up to 42% of behavioral scenarios. • We show that privacy representations decompose along CI parameters, with information type, recipient, and transmission principle encoded in functionally independent subspaces across multiple models. • We introduce CI-parametric steering , which exploits the compositional structure of contextual privacy to reduce leakage across four models on two benchmarks where monolithic steering fails. 2 Related Work Contextual Privacy in LLMs. Contextual Integrity (CI) ( 22 ; 23 ) defines privacy as appropriate information flow governed by norms over five parameters: data subject, sender, recipient, information type, and transmission principle. CI benchmarks ( 20 ; 17 ; 8 ) reveal systematic failures of LLMs to adhere to these norms. To improve compliance, prior work employs reinforcement learning ( 13 ; 16 ) , instruction tuning ( 30 ) , prompt reformulation ( 21 ) , and multi-agent decomposition ( 19 ) . These approaches treat contextual privacy as an external behavioral objective. In contrast, we study contextual privacy as a latent representational structure encoded within LLMs. The work most closely related to ours is SALT ( 3 ) , which steers LLMs to mitigate privacy leakage in LLMs’ reasoning chain. However, we show that contextual privacy is inherently multi-dimensional in representation space and cannot be captured by a single steering vector. Probing for Safety Concepts in LLMs. Linear probing ( 2 ; 4 ) reveals that certain concepts such as honesty, harmlessness, and fairness emerge as linearly separable directions in LLM activation spaces ( 33 ) . 18 show that truthfulness directions identified via probing can be leveraged for inference-time intervention, while 11 find that linear probes detect strategic deception with high accuracy but limited on more subtly deceptive responses. This gap highlights that concept-level signals do not necessarily translate into behavioral control. We extend this line of research to contextual privacy , a structured, multi-dimensional concept not previously studied via probing, and decompose it along its CI parameters. Latent Space Steering. Latent space steering modifies internal activations at inference time to control generation without updating model parameters. 33 identify semantically meaningful directions for safety attributes, 18 introduce

Read the complete text