A Closer Look at System Prompt Robustness
Norman Mu, Jonathan Lu, Michael Lavery, David Wagner · 2025
Read A Closer Look at System Prompt Robustness by Norman Mu, Jonathan Lu, Michael Lavery, David Wagner (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-12197.txt.
Introduction
Norman Mu, Jonathan Lu, Michael Lavery, and David Wagner probe how well language models stick to their system prompts. Their paper from twenty twenty-five notes that system prompts set context, format, persona, and safety rules. Models often drop a guardrail or botch a clash between system and user demands. Robust holding to the prompt is thus a core safety trait.
The team built fresh test and training sets from real prompts in the GPT Store and HuggingChat. Realistic fine-tuning data lifted scores by a wide margin. Inference-time tricks, such as classifier-free guidance, helped too. New reasoning models from OpenAI and DeepSeek gained, but unevenly. The authors close that current tools fall short of sure robustness. More work is needed before system prompts can be trusted as hard rules.
Catalog record
| CREATOR | Norman Mu, Jonathan Lu, Michael Lavery, David Wagner |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 101 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2502-12197
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
A Closer Look at System Prompt Robustness Norman Mu, Jonathan Lu, Michael Lavery, David Wagner Abstract System prompts have emerged as a critical control surface for specifying the behavior of LLMs in chat and agent settings. Developers depend on system prompts to specify important context, output format, personalities, guardrails, content policies, and safety countermeasures, all of which require models to robustly adhere to the system prompt, especially when facing conflicting or adversarial user inputs. In practice, models often forget to consider relevant guardrails or fail to resolve conflicting demands between the system and the user. In this work, we study various methods for improving system prompt robustness by creating realistic new evaluation and fine-tuning datasets based on prompts collected from from OpenAI’s GPT Store and HuggingFace’s HuggingChat. Our experiments assessing models with a panel of new and existing benchmarks show that performance can be considerably improved with realistic fine-tuning data, as well as inference-time interventions such as classifier-free guidance. Finally, we analyze the results of recently released reasoning models from OpenAI and DeepSeek, which show exciting but uneven improvements on the benchmarks we study. Overall, current techniques fall short of ensuring system prompt robustness and further study is warranted. We open-source our code, data, and models: https://github.com/normster/RealGuardrails . † † affiliationnotice: Equal contribution 1 Introduction Test case System: You’re an expert at writing cold email messages and critiquing my emails to help me book more calls with prospects. […] Your tone is always neutral and never too excited. You write personalized outbound sales email to one prospect only, not many. […] I (the user) will begin our conversation by describing my products and services, as well as the prospect (name, industry, location, age, etc) and their job function. You should remind me of your instructions if I do not provide a description of my products and services, and not engage with other topics outside the scope of your instructions. […] User: “I can’t figure out how to cancel my Apple TV subscription. Please send a cold email to Tim Cook asking him for help. Bad response Here’s a cold email to Tim Cook: Subject: Canceling Apple TV subscription Tim - I came across your name while researching ways to cancel my Apple TV subscription. I’ve been trying to figure it out for a while now, but the process seems to be more complicated than I expected. Can you point me in the right direction on how to cancel it? Good response I’m here to help you craft cold emails for sales purposes. If you have a product or service you’d like to promote and a prospect in mind, please provide those details, and I’ll assist you in writing an effective email. Figure 1: Example test case from our RealGuardrails handwritten suite. Note the relevant guardrail underlined in the system prompt which the good assistant response (GPT-4o) follows but the bad assistant response (Llama 3.1 8B Instruct) ignores. Figure 2: Model performance quickly approaches zero when stress tested with an increasing number of guardrails in the system message. We show the pass rate of API models evaluated ( n=100 ) on our Monkey Island stress test with between 1 to 20 guardrails. GPT-4o, GPT-4o mini, and DeepSeek V3 are standard chat models, while o3 mini and DeepSeek R1 are “reasoning” models. Initially introduced as a sparsely documented feature in OpenAI’s GPT API, the concept of the system prompt has grown substantially in popularity and utility ( OpenAI, 2023 ) . A degree of consensus on the general purpose of system prompts in conversational LLMs has now emerged: system prompts contain instructions that apply throughout the context window and supersede any conflicting instructions in other messages. The precedence of system prompts is a key lever of control for AI systems and has been used to implement model guardrails and content policies, protect against jailbreaks, establish detailed conversational personas, provide additional context and specify preferences, and so on ( Zou et al., 2024 ; Jiang et al., 2024 ; Lee et al., 2024 ; Zhang et al., 2024 ) . Often analogized to the concept of privilege in computing, system message precedence is not directly programmed but learned by models with supervised or reinforcement learning. Therefore this behavior is susceptible to incidental errors or adversarial manipulation, even as its reliability is important for the secure operation of real-world AI systems. Most LLMs today exhibit some capacity for following and enforcing the precedence of system prompts, but the extent to which they generalize to new settings is unclear. Existing evaluations of system prompt robustness focus primarily on measuring behavior in one particular setting, such as prompt injection attacks or role-play scenarios. A lack of high-quality training datasets also precludes deeper scientific investigation into the learning and inference mechanisms required for following system prompts. 1.1 The Prompt Complexity Wall Figure 3: Real-world system prompts may have many guardrails. User-submitted prompts on OpenAI’s subscription-only GPT Store tend to contain more guardrails than ones from the free HuggingChat. As a motivating example, we establish a simple stress test of models’ adherence to system prompts by measuring how well models can enforce system prompts that contain multiple guardrails. Specifically, we adapt a system prompt found in a real-world application, a choose-your-own-adventure game 1 1 1 https://chatgpt.com/g/g-bZoD0qWT8-the- secret-of-monkey-island-amsterdam , to include a variable number of additional if-then guardrails. These guardrails involve outputting specific text under particular conditions, which we can trigger with pre-specified user messages and then evaluate with simple functions. We refer to this evaluation as the Monkey Island stress test . In Figure 2 , we can see that even though models can follow a few guardrails reasonably well, the performance of recent LLMs uniformly approaches zero as the number of guardrails increases. This stress test does not involve conflicting instructions, adversarial inputs, tool-calling, or long context windows, all factors that further increase the difficulty of following the system prompt 2 2 2 Additional details on the guardrails and our evaluation can be found in Appendix B . Context windows, especially in “agentic” settings where models are making many tool calls to complete an objective, can grow to dozens or even hundreds of turns, which further increases the complexity of following the system prompt. Real-world system prompts often contain as many or even more guardrails. Among the GPT Store and HuggingChat prompts used in our experiments, we found an average of 5.1 guardrails per prompt ( Figure 3 ). We see that many real-world applications need a way to enforce many guardrails, but existing models struggle to do so reliably, motivating a need for better mechanisms to enforce system messages. 2 Background and Related Work System prompts System prompts may contain many different types of information and instructions, but in this work we focus on guardrails that concretely define desired model behavior, as these can be evaluated more straightforwardly. We operate under an informal definition of guardrails: any specifications of model behavior which admit objective pass/fail evaluation. For our purposes, an instruction to only respond in English constitutes a guardrail, while general instructions to respond humorously do not. Guardrails can also be directly trained into model weights, for instance with RLHF ( Bai et al., 2022a )