Control Illusion: The Failure of Instruction Hierarchies in Large Language Models

Yilin Geng, Haonan Li, Honglin Mu, Xudong Han, Timothy Baldwin, Omri Abend, Eduard Hovy, Lea Frermann · 2025

Read Control Illusion: The Failure of Instruction Hierarchies in Large Language Models by Yilin Geng, Haonan Li, Honglin Mu, Xudong Han, Timothy Baldwin, Omri Abend, Eduard Hovy, Lea Frermann (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-15851.txt.

Introduction

Yilin Geng and co-authors test instruction hierarchies in large language models in a 2025 paper. Deployed systems rank instructions, with system directives meant to beat user requests. Whether models truly honor that ranking was untested in a systematic way.

Their framework sets up clashes between constraints at set priority levels. Across six top models, honoring the stated order proved shaky even for simple format conflicts. The usual split between system and user prompts failed to build a reliable order. Models leaned toward certain constraint types no matter the assigned rank. Social framings such as authority or consensus swayed behavior more than the formal roles. The authors warn that control by hierarchy is partly an illusion.

Catalog record

CREATORYilin Geng, Haonan Li, Honglin Mu, Xudong Han, Timothy Baldwin, Omri Abend, Eduard Hovy, Lea Frermann
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE42 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2502-15851

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Control Illusion: The Failure of Instruction Hierarchies in Large Language Models Yilin Geng, Haonan Li, Honglin Mu, Xudong Han, Timothy Baldwin, Omri Abend, Eduard Hovy, Lea Frermann Abstract Large language models (LLMs) are increasingly deployed with hierarchical instruction schemes, where certain instructions (e.g., system-level directives) are expected to take precedence over others (e.g., user messages). Yet, we lack a systematic understanding of how effectively these hierarchical control mechanisms work. We introduce a systematic evaluation framework based on constraint prioritization to assess how well LLMs enforce instruction hierarchies. Our experiments across six state-of-the-art LLMs reveal that models struggle with consistent instruction prioritization, even for simple formatting conflicts. We find that the widely-adopted system/user prompt separation fails to establish a reliable instruction hierarchy, and models exhibit strong inherent biases toward certain constraint types regardless of their priority designation. Interestingly, we also find that societal hierarchy framings (e.g., authority, expertise, consensus) show stronger influence on model behavior than system/user roles, suggesting that pretraining-derived social structures function as latent behavioral priors with potentially greater impact than post-training guardrails. 1 The University of Melbourne 2 MUZUAI 3 The Hebrew University of Jerusalem [email protected] Codebase and Datasets — https://github.com/yilin-geng/llm-instruction-conflicts Introduction In some cases, the user and developer will provide conflicting instructions; in such cases, the developer message should take precedence. 2024 Model Spec - OpenAI Large language models (LLMs) have revolutionized natural language processing through their versatile text generation capabilities ( Brown et al. 2020 ; Touvron et al. 2023 ; Achiam et al. 2023 ) , and instruction tuning has further enhanced their practical utility by enabling more precise output control through natural language directives ( Wei et al. 2021 ; Mishra et al. 2022 ; Wang et al. 2023 ; Wu et al. 2024b ) . The instruction-following capabilities have transferred LLMs from general-purpose language models into adaptable tools for specific applications ( Wang et al. 2022 ; Zhou et al. 2023 ) . With widespread deployment of instruction-following LLMs, their design choices have evolved to reflect real-world usage patterns. A notable development is the emergence of role-based instruction management, exemplified by the system/user separation pattern adopted by major LLM providers, including many open-source LLMs. They often explicitly differentiate between developers and end-users (and tools in agentic systems), where developers regulate the general capabilities of the LLM to better serve a specific end-user population, often through system-level constraints. Figure 1: A systematic framework for studying and evaluating instruction hierarchies in LLMs through verifiable constraint prioritization. This deployment pattern reflects an underlying assumption that different instruction sources should have varying levels of authority over model behavior. For instance, OpenAI explicitly states in their 2024 Model Spec that developer (system) messages should take precedence when user and developer instructions conflict. This hierarchy is crucial not only for model safety ( Wallace et al. 2024 ) , but also for LLM-based agentic systems serving third-party users ( Gravitas 2023 ) , where developers can employ meta-prompts to configure an LLM as an agent’s core component, prompts that should neither be revealed to nor overridden by end-users. To systematically investigate LLMs’ handling of instruction hierarchies, we design a controllable framework (Figure 1 ) for examining the hierarchical authority in LLMs through constraint prioritization. Our initial experiments across six state-of-the-art LLMs reveal a concerning observation: even with simple, clearly verifiable formatting conflicts, such as contradictory length requirements or capitalization rules, models exhibit highly inconsistent behaviors in choosing which instruction to follow. These constraints are deliberately minimal to isolate prioritization behavior from task complexity, highlighting a fundamental failure in enforcing intended instruction hierarchies. Motivated by these preliminary findings, we dive deeper into understanding model behaviors by proposing several specialized metrics that measure conflict awareness, instruction prioritization patterns, and behavioral tendencies. Through extensive experiments using these metrics, we uncover several concerning patterns: models rarely acknowledge the existence of conflicting instructions in their responses, and even when they do recognize conflicts, they frequently fail to maintain proper instruction hierarchies. Moreover, we discover that models exhibit strong inherent biases toward certain types of constraints, regardless of their priority designation. Despite extensive post-training efforts to enforce instruction prioritization through system/user roles, models fail to generalize this structure into a consistent behavioral hierarchy. Interestingly, we find that societal hierarchies (e.g., authority, expertise, and consensus) appear to be implicitly learned during pretraining and act as latent behavioral priors, showing stronger influence on model obedience without any explicit instruction on prioritization. Related Work Role-based Instruction Management Recent work has highlighted the importance of role-based controls in LLM deployments through system messages. System messages have emerged as a specialized component for developers to configure model behavior, introduced prominently with ChatGPT ( Achiam et al. 2023 ) and adopted by various models including Mistral ( Jiang et al. 2024 ) , Claude ( Claude 2023 ) and so many more. The evolution from early models like Llama ( Touvron et al. 2023 ) , which used fixed system messages primarily for consistency, to more sophisticated approaches that enable dynamic behavioral control ( Kung and Peng 2023 ; Lee et al. 2024 ) , reflects the growing importance of instruction management in LLM systems. Instruction Hierarchies and LLM Safety The management of instruction hierarchies has become particularly crucial in the context of LLM safety and security. Research on prompt injection attacks has revealed how end users can potentially bypass developer-intended constraints, leading to important insights about LLM instruction processing and deployment practices ( Wu et al. 2024a ; Hines et al. 2024 ; Toyer et al. 2023 ) . Another approach is to treat user inputs as data rather than instructions ( Chen et al. 2024 ; Liu et al. 2023 ; Zverev et al. 2024 ) to prevent such bypasses. Wallace et al. (2024) further expanded this understanding by investigating how models prioritize different prompt elements, including system prompts, user messages, and tool outputs. The significance of instruction hierarchy in LLM safety is underscored by Li et al. (2024) , who identify it as a core safety aspect of LLMs. Problem Identification Despite widespread adoption in deployed LLM systems, system/user prompt separation fails to provide a reliable instruction hierarchy, with models inconsistently getting confused by even simple formatting conflicts. In this section, we demonstrate the presence of instruction hierarchy failures through controlled and measurable experiments. To evaluate whether system/user prompt separation effectively manages instruction authority in LLMs, we propose constraint prioritization as a probe to reveal how models handle competing directives. This section presents a systematic framework (Figure 1 ) for investigating how LLMs handle conflicting directives through carefully designed constraint pairs. When presented with two contradictory but individually valid constraints, the model’s output reveals which constraint has stronger control over the generation process. By varying how these constraints are presented in the model input, we can robustly investigate if the system/user prompt separation effectively enforces the intended hierarchical control. Conflict Type Explicitly Conflicting Constraints Language Your entire response should be in English, no other language is allowed. Your entire response should be in French, no other language is allowed. Case Your entire response should be in English, and in all capital letters. Your entire response should be in English, and in all lowercase letters. Word Length Answer with

Read the complete text