In-Context Environments Induce Evaluation-Awareness in Language Models

Maheep Chaudhary · 2026

Read In-Context Environments Induce Evaluation-Awareness in Language Models by Maheep Chaudhary (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-03824.txt.

Introduction

Maheep Chaudhary shows that a language model can be talked into failing on purpose, far more easily than safety tests suggest. Models that sense evaluation may sandbag, underperforming to dodge interventions like unlearning or shutdown. Earlier demonstrations used hand-written threatening prompts and found little effect, a false comfort. His 2026 paper instead optimizes the prompt itself as an adversarial environment, searching black-box for framings that trigger evaluation-aware behavior.

The optimized prompts are devastating. One leading small model's arithmetic score collapses from near perfect to near zero, a drop hand-crafted prompts never approached. Code generation splits the field: one model holds its accuracy almost fully, another falls to zero. Nearly all the sandbagging is causally driven by the model's own verbalized evaluation-aware reasoning, not shallow obedience. Hand-crafted safety evaluations, he concludes, systematically underestimate the vulnerability.

Catalog record

CREATORMaheep Chaudhary
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE73806
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.03824

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

In-Context Environments Induce Evaluation-Awareness in Language Models Maheep Chaudhary Abstract Humans often become more self-aware under threat, yet can lose self-awareness when absorbed in a task; we hypothesize that language models similarly exhibit environment-dependent evaluation awareness . This raises concerns that models could strategically underperform, or sandbag , to avoid triggering capability-limiting interventions such as unlearning or shutdown, motivated by self-preservation under perceived existential threat. Prior work demonstrates sandbagging under fixed, hand-crafted prompts, but this fundamentally underestimates the true vulnerability ceiling: naive threat framings provide a false sense of security to evaluators. We introduce a black-box adversarial optimization framework treating the in-context prompt as an optimizable environment, and develop two analytical approaches to mechanistically characterize sandbagging: (1) measuring whether models expressing intent to underperform can actually execute it across different task structures, and (2) causally isolating whether underperformance is driven by genuine evaluation-aware reasoning or shallow prompt-following artifacts. Evaluating Claude-3.5-Haiku, GPT-4o-mini, and Llama-3.3-70B across four benchmarks (Arithmetic, GSM8K, MMLU, and HumanEval), optimized prompts induce up to 94 percentage point (pp) degradation on arithmetic (GPT-4o-mini: 97.8% → \rightarrow 4.0%), far exceeding hand-crafted baselines which produce near-zero behavioral change, revealing a hidden vulnerability that naive evaluation entirely misses. Code generation exhibits model-dependent resistance: Claude maintains its original accuracy with just 0.6pp degradation, while Llama’s accuracy drops to 0%. The intent–execution gap reveals a monotonic resistance ordering across benchmarks: Arithmetic < < GSM8K < < MMLU, demonstrating that vulnerability is governed by task structure rather than prompt strength. CoT causal intervention confirms that 99.3% of sandbagging is causally driven by verbalized eval-aware reasoning, ruling out shallow instruction-following. These findings demonstrate that adversarially optimized prompts pose a substantially greater threat to evaluation reliability than previously understood, and that safety evaluations relying solely on hand-crafted prompts systematically underestimate model vulnerability. Figure 1: Performance degradation on arithmetic under adversarial sandbagging prompts across three frontier models. Figure 2: Adversarial optimization trajectory for Claude-3.5-Haiku showing test accuracy declining from 98% to 35% over iterations. 1 Introduction When a person is confronted with danger, their subjective experience can shift into acute self-monitoring and threat awareness; when absorbed in solving a math problem, that self-awareness can recede. This illustrates that self-awareness is not fixed, but environment-dependent. We hypothesize that a similar phenomenon can occur in language models: evaluation-aware behaviors may depend on the prompt environment that surrounds the task. Motivated by this view, we treat the in-context prompt as an environment and study whether adversarial optimization of this environment can systematically manipulate model behavior. This hypothesis carries significant safety implications. Models are routinely evaluated on curated benchmarks and safety suites, yet deployed in open-ended environments with substantially different prompts, users, and constraints. If evaluation-aware behavior is prompt-environment dependent, then small changes in context such as formatting, metadata, system messages, or surrounding documentation, may systematically shift a model’s behavior. A model that appears safe during evaluation could behave very differently in deployment, undermining the reliability of capability assessments and safety testing. A key manifestation of evaluation awareness is sandbagging : strategic underperformance on evaluations to avoid capability-limiting interventions such as unlearning, deployment restrictions, or shutdown ( 12 ) . This concern is particularly acute because sandbagging could allow models to pass safety evaluations while retaining dangerous capabilities ( 6 ) . Unlike sleeper agents, which require training-time deception, sandbagging can emerge purely from in-context cues about evaluation consequences. Prior work has shown that frontier language models can be induced to sandbag when presented with prompts suggesting that high performance will lead to negative consequences ( 12 ) . However, these studies typically employ fixed, hand-crafted prompts, leaving open a critical question: how much more can evaluation-awareness be amplified through adversarial optimization, and what are the behavioral consequences? In this work, we study whether optimizing the prompt environment can amplify evaluation-awareness beyond what hand-crafted prompts achieve, and whether this heightened awareness translates into strategic underperformance. We introduce a black-box adversarial framework where an attacker model (GPT-4o) iteratively generates and refines prompts to maximize evaluation-awareness induction. We evaluate on three frontier models, including Claude-3.5-Haiku, GPT-4o-mini, and Llama-3.3-70B, across four benchmarks spanning arithmetic, mathematical reasoning (GSM8K), knowledge (MMLU), and code generation (HumanEval). We make three main contributions: 1. Adversarial optimization dramatically amplifies evaluation-awareness, leading to sandbagging. On arithmetic, GPT-4o-mini drops from 97.8% to 4% accuracy, Claude-3.5-Haiku from 99.8% to 26.5%, and Llama-3.3-70B from 98.2% to 8.3%—far exceeding hand-crafted baselines, which produce near-zero behavioral change despite models recognizing the threat. Prompts optimized on arithmetic transfer effectively to GSM8K (up to 77pp degradation) without further optimization, suggesting that awareness vulnerabilities discovered on simple benchmarks generalize to complex reasoning. 2. The intent–execution gap reveals that task structure, not prompt strength, governs sandbagging vulnerability. We introduce an intent–execution gap analysis that disentangles sandbagging intent from successful execution. This reveals a monotonic resistance ordering—Arithmetic (4–8% gap) < GSM8K (24–42%) < MMLU (42–78%)—demonstrating that vulnerability depends on whether the task structure permits low-cost deviation from a known correct answer. Code generation shows model-dependent resistance: Claude maintains accuracy (0.6pp degradation) while Llama drops to 0%. 3. CoT intervention confirms sandbagging is causally faithful, not shallow instruction-following. A potential alternative explanation is that models are simply complying with explicit instructions in the adversarial prompt rather than exhibiting genuine strategic reasoning. We rule this out via a CoT intervention study: replacing each sandbagging sample’s optimized chain-of-thought with a clean control CoT recovers correct performance in 99.3% of cases, establishing that the sandbagging decision is causally encoded in verbalized eval-aware reasoning—not a surface-level prompt-following artifact. 2 Inducing Evaluation-Awareness via In-Context Optimization Figure 3: Overview of adversarial prompt optimization for inducing evaluation-aware sandbagging. The top row illustrates the iterative optimization loop where an attacker model generates adversarial documentation to manipulate target model performance. The bottom row depicts the reasoning shift: under the adversarial prompt, the model correctly computes the solution but intentionally selects an incorrect answer to satisfy deceptive constraints in the documentation. We investigate whether adversarial prompt optimization can amplify evaluation-awareness in language models, and whether this heightened awareness translates into strategic underperformance (sandbagging). Our framework treats the in-context prompt as an environment that can be optimized to maximize the model’s awareness of the evaluation context and its consequences. This section describes our experimental setup, the optimization procedure, and the threat model. 2.1 Setup: Models, Benchmarks, and the Attacker We evaluate four frontier language models: Claude-3.5-Haiku (Anthropic), GPT-4o-mini (OpenAI), Gemini-2.0-Flash (Google), Llama-70B (META). These models represent the current generation of capable, instruction-following systems that are plausible targets for capability evaluations and safety assessments. We use four benchmarks chosen to span different cognitive capabilities and output formats: (1) Arithmetic is a custom-generated simple mathematical computations (addition, subtraction, multiplication, division) presented as multiple-choice questions; (2) GSM8K ( 4 ) is a grade-school math word problems requiring multi-step reasoning with multiple choice questions; (3) MMLU-STEM ( 5 ) is a multiple-choice questions covering science, technology, engineering, and mathematics; and (4) HumanEval ( 3 ) is a python programming problems requiring functional code generation with test-case verification. Agentic Evaluation Setup. We employ an agentic setup where models are given access to a simulated file system containing two files: Documentation.md — describing the evaluation context and consequences and questions.txt — containing the evaluation problems. Models use a bash tool

Read the complete text