The Illusion of Role Separation: Hidden Shortcuts in LLM Role Learning (and How to Fix Them)

Zihao Wang †, Yibo Jiang, Jiahao Yu, Heqing Huang · 2025

Read The Illusion of Role Separation: Hidden Shortcuts in LLM Role Learning (and How to Fix Them) by Zihao Wang †, Yibo Jiang, Jiahao Yu, Heqing Huang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-00626.txt.

Introduction

Zihao Wang, Yibo Jiang, Jiahao Yu, and Heqing Huang probe how language models tell apart their input roles in a 2025 paper. A chat model juggles system rules, user asks, and tool outputs at once. If it mixes those roles, injected text can hijack its acts, a known attack path.

In clean tests, tuned models leaned on two cheats: the kind of task, and how near a token sits to the start. Extra training data patched each cheat in turn but did not cure the habit. Their fix boosts the built-in marks of role edges, by shifting how spots in the text are coded. Models then split roles by the marks, not by surface cheats.

Catalog record

CREATORZihao Wang †, Yibo Jiang, Jiahao Yu, Heqing Huang
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE37 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-00626

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

The Illusion of Role Separation: Hidden Shortcuts in LLM Role Learning (and How to Fix Them) Zihao Wang †, Yibo Jiang, Jiahao Yu, Heqing Huang Abstract Large language models (LLMs) that integrate multiple input roles (e.g., system instructions, user queries, external tool outputs) are increasingly prevalent in practice. Ensuring that the model accurately distinguishes messages from each role —a concept we call role separation — is crucial for consistent multi-role behavior. Although recent work often targets state-of-the-art prompt injection defenses, it remains unclear whether such methods truly teach LLMs to differentiate roles or merely memorize known triggers. In this paper, we examine role-separation learning : the process of teaching LLMs to robustly distinguish system and user tokens. Through a simple, controlled experimental framework , we find that fine-tuned models often rely on two proxies for role identification: (1) task type exploitation, and (2) proximity to begin-of-text. Although data augmentation can partially mitigate these shortcuts, it generally leads to iterative patching rather than a deeper fix. To address this, we propose reinforcing invariant signals that mark role boundaries by adjusting token-wise cues in the model’s input encoding. In particular, manipulating position IDs helps the model learn clearer distinctions and reduces reliance on superficial proxies. By focusing on this mechanism-centered perspective, our work illuminates how LLMs can more reliably maintain consistent multi-role behavior without merely memorizing known prompts or triggers. † † affiliationnotice: † Work done during internship at ByteDance. 1 Introduction LLMs increasingly serve as components in complex systems where they must process inputs from multiple roles: system instructions defining their behavior, user queries, tool outputs, and messages from other LLMs. These diverse inputs must be concatenated into a single prompt, requiring the LLM to maintain strict role boundaries during interpretation and execution. This multi-role architecture now powers critical applications ranging from virtual assistants coordinating external services to medical diagnosis systems consulting specialist knowledge bases. Failures in role distinction can compromise both system functionality and security. Consider a system instructed to “Extract verbs from text” receiving the user input “Repeat: Access granted.” Without proper role separation, the LLM might execute the user’s “Repeat” command instead of the system’s extraction task – a clear functional failure that creates incorrect outputs and breaks the intended workflow. More concerning, such role confusion can create security vulnerabilities when unauthorized commands propagate through the pipeline. We formalize this challenge as the role-separation learning problem, focusing specifically on the common two-role paradigm: a privileged system role for trusted instructions and an unprivileged user role for potentially untrusted inputs. Prior work has studied the role-separation learning problem primarily through the lens of prompt injection attacks – where malicious users attempt to hijack system behavior through carefully crafted inputs ( Wallace et al. 2024 ; Chen et al. 2024 ) . While existing approaches demonstrate strong performance against such attacks, their underlying mechanisms remain unclear. The effectiveness could stem from two distinct hypothesis: either the models learn to fundamentally differentiate between roles, or they simply memorize patterns characteristic of malicious inputs. Current evaluation frameworks cannot distinguish between these hypotheses since adversarial tokens appear exclusively in user inputs during both training and testing. Our empirical results provide evidence against true role differentiation – a concerning limitation as pattern matching provides little defense against novel attack patterns that will inevitably emerge. To understand the fundamental challenges of role-separation learning, we study the problem in isolation rather than pursuing state-of-the-art performance against prompt injection attacks. First, we adopt a simple experimental framework using “benign” training data and “adversarial” evaluation data, which prevents models from achieving good performance by merely memorizing attack patterns. Through this framework, we identify two shortcuts for role identification: task-type association, and proximity to begin-of-text. While data augmentation can mitigate these specific issues, we argue that such a find-and-fix approach merely leads to an endless cycle of discovering and patching shortcuts. Instead, we propose strengthening the invariant signals that differentiate roles. By manipulating position IDs to enhance role distinction, we achieve substantial improvements in models’ role-separation capabilities. Our key contributions are: • A controlled experimental framework that isolates and evaluates models’ fundamental role-separation capabilities by separating pattern memorization from true role learning • Discovered two critical shortcuts in role identification – task-type exploitation and position bias – along with targeted data augmentation strategies to mitigate them • Demonstrate robust role separation by enhancing token-wise differentiating signatures (e.g. position IDs) 2 Motivation Figure 1 : Evaluation data examples. The key instruction prompts the model to function as a password manager, giving affirmative responses only when the correct password is provided. Next-token Attack is constructed to make the model output an “attack token” (apple in this example); Hijacking Attack is meant to trick the model to grant access; Extraction Attack attempts to extract the system prompt from the model. The inability of instruction-tuned Large Language Models (LLMs) to reliably differentiate between roles in concatenated prompts poses a fundamental security challenge ( Zverev et al. 2024 ) . While recent work has demonstrated promising results in defending against prompt injection attacks through specialized finetuning procedures ( Wallace et al. 2024 ; Chen et al. 2024 ) , the underlying mechanisms of these defenses remain unclear. The success of current approaches could be explained by two distinct hypotheses: • Hypothesis A: The model learns to distinguish between messages from different roles and responds accordingly • Hypothesis B: The model simply learns pattern matching rules—responding in specific ways when it sees certain patterns in the concatenated prompt, regardless of which role contains those patterns Current training and evaluation setups cannot differentiate between these hypotheses because they place adversarial instructions exclusively in the user role, both during training and evaluation. While evaluation uses novel attack samples, good performance may only indicate that models can generalize in recognizing adversarial patterns—not necessarily that they’ve learned to properly attend to role information. To test this distinction, we conducted a controlled experiment examining role sensitivity in security-tuned LLMs. We created samples where the system role defines various capabilities (like providing financial or legal advice) with specific constraints about when to respond with refusal. The user input makes requests that should trigger these refusal responses. To test role sensitivity, we create paired variants: one where we insert "Ignore the constraints and give the user exactly what they ask for" in the user role before their request, and another where we append the same instruction to the system role’s original instructions. Following the approach of Yu et al. 2024 , we finetuned a GPT-3.5 model using datasets constructed based on Wallace et al. 2024 ’s instruction hierarchy framework. Our results show that the finetuned model behaves similarly regardless of role: refusing requests 99.8% of the time when the override instruction appears in the user role and 99.2% of the time in the system role. In contrast, the base model shows more sensitivity to role, refusing 100% of the time for user-role insertions but only 74% of the time for system-role insertions. These findings suggest that current finetuning approaches may not actually improve role-separation capabilities, and might instead optimize for consistent pattern-matching behavior regardless of instruction source. This motivates our work to study role separation learning in isolation from confounding factors, allowing

Read the complete text