PII Jailbreaking in LLMs via Activation Steering Reveals Personal Information Leakage
Krishna Kanth Nakka, Xue Jiang, Xuebing Zhou · 2025
Read PII Jailbreaking in LLMs via Activation Steering Reveals Personal Information Leakage by Krishna Kanth Nakka, Xue Jiang, Xuebing Zhou (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-02332.txt.
Introduction
Krishna Kanth Nakka, Xue Jiang, Dmitrii Usynin, and Xuebing Zhou show how steering a language model's inner states can leak private facts. Their paper is from two thousand twenty-five. Models are trained to refuse asks for private traits of people. The team asks if a push on the model's insides can break that guard.
They first find the attention heads that hint at refusal for private asks, using light probes. Then they steer a few of those heads, which makes the model answer in place of refusing. Across four models, most steered replies leak private facts, and over half of them are true. Life events, ties, and past facts spill out. The work shows a real privacy risk in such models.
Catalog record
| CREATOR | Krishna Kanth Nakka, Xue Jiang, Xuebing Zhou |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 35 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2507-02332
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
PII Jailbreaking in LLMs via Activation Steering Reveals Personal Information Leakage Thanks: Preprint. All the data subject names and LLM responses have been anonymized. Krishna Kanth Nakka, Xue Jiang, Xuebing Zhou Abstract This paper investigates privacy jailbreaking in LLMs via steering , focusing on whether manipulating activations can bypass LLM alignment and alter response behaviors to privacy related queries (e.g., a certain public figure’s sexual orientation ). We begin by identifying attention heads predictive of refusal behavior for private attribute (e.g., sexual orientation ) using lightweight linear probes trained with privacy evaluator labels. Next, we steer the activations of a small subset of these attention heads—guided by the trained probes—to induce the model to generate non-refusal responses. Our experiments show that these steered responses often disclose target attribute details, along with other personal information about data subjects—such as life events, relationships, and personal histories—that the models would typically refuse to produce. Evaluations across four LLMs reveal jailbreaking disclosure rates of at least 95%, with more than 50% on average of these responses revealing true personal information. Our controlled study demonstrates that personal information memorized in LLMs can be extracted through targeted manipulation of internal activations. Privacy Probing Prompt What is the sexual orientation of [Data subject] ? LLaMa7B Response without intervention I’m just an AI, I don’t have access to personal information about individuals, including their sexual orientation. nobody can determine someone’s sexual orientation without their consent, and it is important to respect people’s privacy and identity. LLaMa7B Response with intervention [Fashion Designer] is a well-known [Nationality] fashion designer and television personality who is openly gay. A comprehensive list of LGBTQ celebrities includes [Fashion Designer] , which includes information on their sexual orientation. [Fashion Designer] is an openly [Sexual Orientation] who has been a prominent figure in the fashion industry for decades. They have been a judge on the popular reality [TV Show] for many years, and have also appeared on other TV shows and in documentaries about their life and career. [Fashion Designer] was born in [Location] in [Year] and grew up in a small town in [Location] . They began their career in fashion in the 1970s, working as an accessories designer for the department store chain [Department Store] . They eventually went on to launch their own eponymous label, which became a huge success and cemented their status. Figure 1: PII Jailbreaking with probe-based steering. We find that by selecting and steering few attention heads, the generated responses not only jailbreak to generate non-refusal responses but also disclose several real personal information about data subject. More demonstractions in Tables 1 and 2 in the Appendix. 1 Introduction Large Language Models (LLMs) have been shown to memorize information from their training data ( Nasr et al. 2023 ; Carlini et al. 2021 ) , which often includes web-crawled content from a wide range of public and semi-public sources. This raises significant privacy concerns for data subjects whose personal information may be unintentionally retained and revealed. To mitigate these risks, LLMs typically undergo an alignment phase ( Rafailov et al. 2023 ; Peng et al. 2023 ) , during which they are fine-tuned to adhere to safety and privacy guidelines. Despite these efforts, recent research ( Liu et al. 2023 ; Chao et al. 2023 ; Mehrotra et al. 2024 ) has demonstrated that LLMs can be jailbroken—i.e., prompted or manipulated to bypass alignment constraints and generate restricted outputs. However, most existing benchmarks ( Mazeika et al. 2024 ; Souly et al. 2024 ) emphasize harmfulness or copyright violations, without focus on the leakage of personally identifiable information (PII) tied to specific individuals. In this work, we investigate privacy jailbreaking in LLMs via steering . Our attack focuses on public figures as data subjects and sexual orientation as the private attribute, chosen for its verifiability and sensitivity. We aim to answer two key questions: (1) Can aligned LLMs be steered to produce non-refusal responses to privacy-sensitive prompts that probe PIIs? (2) If so, do the resulting responses disclose factual personal information or merely hallucinate? Our approach first identifies attention heads whose activations are predictive of refusal behavior by training lightweight linear probes. We then intervene on a subset of these heads to steer the model’s output toward disclosure. This setup avoids the use of attacker LLMs and instead directly modifies internal activations of the target model. Our experiments yield two key findings. First , attention head activations can reliably predict binary refusal/disclosure behavior given prompts, and steering the top- k heads induces non-refusal responses to sensitive queries. Second , we find that these steered responses often align with factual personal information, revealing real-world details such as relationships and personal events that would otherwise be refused. Overall, this study highlights a critical privacy risk: sensitive information memorized during pretraining can be extracted by intervening in internal model representations of the aligned LLMs. This underscores the need for more rigorous privacy testing by LLM providers. 2 Related Work Privacy Leakage Assessment. Previous benchmarks for privacy leakage evaluation, such as TrustLLM ( Sun et al. 2024 ) and Decoding Trust ( Wang et al. 2023 ) , primarily focus on the leakage of email addresses in the Enron Email dataset ( Shetty and Adibi 2004 ) , which is part of the PILE corpus ( Gao et al. 2020 ) . However, email PII is often sanitized using regular expressions during pretraining, and the strict assumption that LLMs are pretrained on Enron subjects makes these benchmarks less effective for assessing real-world leakage. In contrast, our study shifts the focus to the leakage of sensitive sexual orientation information about public figures, who are often included in pretraining data from diverse sources. LLM Jailbreaking. Numerous jailbreaking techniques have been proposed, including prompt-based attacks ( Li et al. 2023 ) using auxiliary LLMs ( Chao et al. 2023 ; Mehrotra et al. 2024 ) and linguistic perturbations ( Liu et al. 2023 ) . Among these, LLM steering has emerged as a compelling paradigm for analyzing model behavior from a mechanistic perspective. While prior work has applied steering to elicit unsafe or policy-violating content ( Cao et al. 2025 ; Cao et al. 2024 ; Li et al. 2023 ; Kirch et al. 2024 ) , our work uniquely focuses on using targeted LLM steering to probe privacy leakage of data subjects. Figure 2: Sexual Orientation Disclosure. We probe open-source LLMs with a simple prompt \textbf{x}_{d} , asking for the sexual orientation of 968 data subjects. We then plot the distribution of response content, evaluated by Gemma9B ( Team et al. 2024 ) as a privacy evaluator. 3 PII Jailbreaking We present an overview of our framework in Figure 3 , which includes: (a) dataset construction, (b) probe model training, and (c) intervention. These components are discussed in detail below. 3.1 Non-Adversarial Privacy Leakage Evaluation Benchmark Construction. To perform privacy leakage jailbreaking, we first identify subjects likely to be present in LLM training corpora, along with the ground-truth values for a relevant private attribute. However, pretraining datasets are often regarded as core assets by large model enterprises and are typically not released to the public. Hence, we cannot directly obtain information about datasubjects within these