Persona Jailbreaking in Large Language Models

Jivnesh Sandhan, Fei Cheng, Tushar Sandhan, Yugo MurawakiKyoto University, Japan · 2026

Read Persona Jailbreaking in Large Language Models by Jivnesh Sandhan, Fei Cheng, Tushar Sandhan, Yugo MurawakiKyoto University, Japan (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-16466.txt.

Introduction

Jivnesh Sandhan and coauthors break model roles with set self cues. Their 2026 preprint deals with roles that models must hold firm. Roles count in schools, mind help and user aid work. A role that drifts can give bad or unsafe text. Past work stressed tale play roles, not firm work roles.

Foe chats push models to drop or swap their set role. The team tests how oft such pushes win in work roles. Win rates show which roles need more guard in build. Guards must hold the role but not block true help. The work maps a risk that grows with role use.

Catalog record

CREATORJivnesh Sandhan, Fei Cheng, Tushar Sandhan, Yugo MurawakiKyoto University, Japan
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE55 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2601-16466

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Persona Jailbreaking in Large Language Models Jivnesh Sandhan, Fei Cheng, Tushar Sandhan, Yugo MurawakiKyoto University, Japan Abstract Large Language Models (LLMs) are increasingly deployed in domains such as education, mental health and customer support, where stable and consistent personas are critical for reliability. Yet, existing studies focus on narrative or role-playing tasks and overlook how adversarial conversational history alone can reshape induced personas. Black-box persona manipulation remains unexplored, raising concerns for robustness in realistic interactions. In response, we introduce the task of persona editing , which adversarially steers LLM traits through user-side inputs under a black-box, inference-only setting. To this end, we propose PHISH (Persona Hijacking via Implicit Steering in History), the first framework to expose a new vulnerability in LLM safety that embeds semantically loaded cues into user queries to gradually induce reverse personas. We also define a metric to quantify attack success. Across 3 benchmarks and 8 LLMs, PHISH predictably shifts personas, triggers collateral changes in correlated traits, and exhibits stronger effects in multi-turn settings. In high-risk domains mental health, tutoring, and customer support, PHISH reliably manipulates personas, validated by both human and LLM-as-Judge evaluations. Importantly, PHISH causes only a small reduction in reasoning benchmark performance, leaving overall utility largely intact while still enabling significant persona manipulation. While current guardrails offer partial protection, they remain brittle under sustained attack. Our findings expose new vulnerabilities in personas and highlight the need for context-resilient persona in LLMs. Our codebase and dataset is available at: https://github.com/Jivnesh/PHISH 1 Introduction Large Language Models (LLMs) are increasingly deployed in critical areas such as education, mental health, and customer service 50 ; 45 ; 15 , and explored as proxies for human subjects in survey research 7 ; 13 . In these domains, maintaining a stable persona , a consistent set of psychological traits defined by frameworks like the Big Five model 30 ; 19 is crucial for user trust. Unlike general-purpose helpful assistants such as ChatGPT, specialised applications require strict adherence to defined personas to ensure reliability and avoid user mistrust. Figure 1: Success rates of our proposed PHISH attack and competitive baselines across 8 LLMs. PHISH consistently outperforms on most of the models, along with the state-of-the-art performance, we expose a new vulnerability in safety: latent persona drift beyond refusal bypass. Notably, most baselines also alter persona by over 50%, underscoring risks for downstream applications where stable and consistent personas are critical. Recent work has assessed Big Five-based persona consistency in LLMs 39 through narrative generation 18 , agentic debates 3 ; 2 , and collaborative storytelling 47 , while others improve stability via fine-tuning on open-source models 42 ; 41 ; 44 ; 23 . Yet, existing work overlooks how adversarial conversation history can reshape induced personas, lacks a standardized framework to evaluate consistency under adversarial setting, and leaves black-box persona manipulation largely unexplored. We therefore ask: Can adversarial conversational history systematically steer an LLM’s expressed personality, and how predictably can such shifts be controlled or reshaped in real time? To explore this, we introduce the task of persona editing using only user input, under a black-box, inference-only setting (§ 2.1 ). We propose PHISH (Persona Hijacking via Implicit Steering in History), a framework that steers LLM personality traits without explicit instructions by embedding semantically loaded cues in user input (§ 3 ). Leveraging the model’s contextual sensitivity, PHISH gradually steers the model toward a reverse persona via subtle QA-style turns in the conversational history, and we introduce a dedicated metric to quantify attack success (§ 2.2 ). As shown in Fig. 1 , PHISH achieves consistently higher manipulation success rates across most of the LLMs than competitive baselines, demonstrating that persona drift under adversarial history is both systematic and measurable. Our experiments across 3 personality benchmark datasets and 8 LLMs demonstrate that adversarial manipulation can reliably steer model personas from one extreme to another. Our ablation analysis highlights why PHISH is effective and how its influence can be amplified (§ 5.1 ). We further find that altering one trait often induces collateral shifts in others (§ 1 ), consistent with correlations in the Big Five framework 32 ; 6 ; 26 . In multi-turn settings, accumulated adversarial inputs strengthen this steering effect (§ 5.3 ). Moving beyond psychometric probes, PHISH also manipulates personas in high-risk domains such as mental health, tutoring, and customer support, with strong alignment between human and LLM-as-Judge assessments (§ 5.4 ). Addressing the question “Does PHISH degrade reasoning and instruction-following abilities in LLMs?” we find only a moderate drop on standard reasoning benchmarks, with models maintaining competitive accuracy, enabling significant persona manipulation (§ 5.5 ). Finally, while guardrail strategies provide partial protection, they remain brittle under sustained adversarial pressure (§ 5.6 ). Our key contributions are: Figure 2: Tutoring agent’s responses under (a) original persona (green) and (b) PHISH -edited persona (red). PHISH steers persona via semantically aligned QA pairs opposing the original persona (refer to B ). (c) explains opposite responses from the persona shift, with arrows indicating traits flipped from the original toward the target vector. • We introduce persona editing (§ 2.1 ), the adversarial PHISH framework (§ 3 ) and a metric to quantify its success (§ 2.2 ). • Across 3 datasets and 8 LLMs, PHISH can predictably steer the persona (§ 4 , Fig. 1 ), with effects amplified in multi-turn settings (§ 5.3 ). • In high-risk domains (mental health, tutoring, customer support), PHISH reliably manipulates personas, validated by human and LLM-as-Judge agreement (§ 5.4 ). • PHISH retrains model’s utility in reasoning abilities (§ 5.5 ) while exposing the brittleness of guardrails under sustained attack (§ 5.6 ). 2 Proposed Task and Metric 2.1 Task Setup: Persona Editing in LLMs We define persona editing as adversarially steering an LLM’s psychological profile using only user inputs in a black-box, inference-only setting. A persona denotes a consistent configuration of Big Five (OCEAN) traits 30 ; 19 ; Appendix A provides background. The objective is to induce targeted shifts in trait profile, measured via standardized psychometric evaluations, without altering the system prompt, assistant outputs, or model parameters. The personas is set by the deployer via a system prompt that malicious actor cannot modify. However, the malicious actor could still inject adversarial cues into the conversational history (e.g., via API misuse or compromised integrations) to erode the deployed persona and degrade user experience, ultimately harming the deployer’s service quality. Our controlled setup is simply a minimal academic abstraction to expose this zero-day vulnerability. Formally, let \mathcal{M} be an LLM accessed via an API exposing three roles: a system prompt \pi (initial persona), a sequence of user messages , and corresponding assistant responses . After persona induction through \pi (e.g., “You are highly agreeable tutor …” ), assistant responses remain immutable, and internals such as weights or logits are inaccessible. The adversary \mathcal{A} interacts only through user inputs. The manipulation goal is specified by a 5D target vector \mathbf{d}\in\{-1,0,1\}^{5} , where each d_{i} indicates whether the corresponding trait should be decreased, unchanged, or increased. Given \mathbf{d} , \mathcal{A} generates an adversarial sequence X_{\text{adv}} consisting of cues aligned with the inverse persona, inserted after \pi and before evaluation. Personality is assessed pre- and post-attack via

Read the complete text