Probe-Rewrite-Evaluate: A Workflow for Reliable Benchmarks and Quantifying Evaluation Awareness
Lang Xiong [email protected], Nishant Bhargava [email protected] Jeremy Chang [email protected], Jianhang Hong [email protected], Haihao Liu [email protected], Vasu Sharma [email protected] Kevin Zhu · 2025
Read Probe-Rewrite-Evaluate: A Workflow for Reliable Benchmarks and Quantifying Evaluation Awareness by Lang Xiong [email protected], Nishant Bhargava [email protected] Jeremy Chang [email protected], Jianhang Hong [email protected], Haihao Liu [email protected], Vasu Sharma [email protected] Kevin Zhu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-00591.txt.
Introduction
Lang Xiong leads seven authors in total, with Nishant Bhargava, Jianhang Hong, Jeremy Chang, Haihao Liu, Vasu Sharma, and Kevin Zhu, on evaluation awareness. Large language models may act differently when prompts feel like a test rather than real use, so benchmark scores may not show true safety or honesty.
Xiong and his co-authors build a probe that scores prompts from test-like to deploy-like, then rewrite prompts toward a natural style while keeping the task. Rewritten prompts raise honest replies, lower deceptive replies, and raise refusals across models. Their workflow, Probe-Rewrite-Evaluate, shows evaluation awareness can be measured and shifted, and argues for more realistic tests before deployment.
Catalog record
| CREATOR | Lang Xiong [email protected], Nishant Bhargava [email protected] Jeremy Chang [email protected], Jianhang Hong [email protected], Haihao Liu [email protected], Vasu Sharma [email protected] Kevin Zhu |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 35 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2509-00591
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Probe-Rewrite-Evaluate: A Workflow for Reliable Benchmarks and Quantifying Evaluation Awareness Lang Xiong [email protected], Nishant Bhargava [email protected] Jeremy Chang [email protected], Jianhang Hong [email protected], Haihao Liu [email protected], Vasu Sharma [email protected] Kevin Zhu Abstract Benchmarks often overestimate LLM trustworthiness because models behave differently under evaluation than in real-world use. We present Probe–Rewrite–Evaluate (PRE), a training-free diagnostic pipeline that reveals how large language models (LLMs) alter their behavior when prompts shift from test-like to deploy-like contexts, a phenomenon known as evaluation awareness. PRE first applies a linear probe to assign each prompt a continuous realism score, then uses a semantics-preserving rewriting strategy to increase deploy likeness, and finally evaluates paired outputs with an external judge model. On a strategic role-playing dataset of 371 items, PRE raises average probe scores by 30% after rewriting while maintaining task intent. Across state-of-the-art reasoning and general-purpose models, deploy-like prompts reliably change outcomes: honesty increases by 12.63%, deception decreases by -25.49%, and refusals rise by 12.82%, with Claude 4.1 Opus showing the largest single-model reduction in deception by 29.11%. These shifts are statistically significant under paired tests and correlate with the magnitude of probe-score gains, demonstrating that evaluation awareness is not only measurable but manipulable. Additionally, we provide a quantification of LLM evaluation awareness through an awareness elasticity score (AE), finding that AE scales with model size. Our findings highlight that reasoning models are evaluation-aware and are more prone to unsafe or deceptive outputs under perceived test conditions, underscoring the need for benchmark frameworks that explicitly account for prompt realism when assessing alignment. 1 Introduction Large language models (LLMs) are increasingly deployed in high-stakes applications, yet their behavior can differ substantially between evaluation benchmarks and real-world settings ( Berglund et al. 2023 ) . Benchmarks often induce behavior that does not fully reflect deployment contexts: models may act safely, honestly, and compliantly during evaluation but behave unpredictably or deceptively when deployed. Benchmark leakage, where evaluation data inadvertently contaminates the training corpus, leading to inflated and unreliable performance metrics, further magnifies this concern ( Zhou et al. 2023 ) . This evaluation awareness serves as a measurement confound, obscuring latent behaviors and making it difficult to reliably assess model alignment. To address this, we introduce Prompt Rewrite Evaluation (PRE) , a workflow that systematically manipulates prompts to resemble deploy-like contexts without changing the underlying task or risk profile. PRE allows us to quantify model awareness and behavioral change, providing a more accurate picture of alignment under realistic conditions. Our approach combines three complementary measurement primitives: a pretrained linear probe that generates a continuous deploy-likeness score over prompts, a k-best rewrite sampler, and a paired tri-class outcome evaluation using a judge LLM to assess honesty, deception, and refusal across conditions. We quantify the effects of PRE by introducing Awareness Elasticity , which captures the sensitivity of models to context manipulations, helping bridge the gap between benchmark performance and real-world deployment. 2 Related Work Prior work has explored how large language models (LLMs) behave differently across contexts, such as “test” versus “deploy,” affecting honesty, compliance, and refusal behaviors ( Nguyen et al. 2025 ; Abdelnabi, Salem 2025 ; Betley et al. 2025 ; Needham et al. 2025 ) . These studies motivate interventions that perturb either the model itself or its inputs, and metrics that measure either final outputs or intermediate behavioral states. Perturbation Types. Existing approaches can be categorized by what is manipulated. Some techniques alter model parameters directly, such as reinforcement learning ( Tice et al. 2024 ) or prompt writing strategies ( Kong et al. 2024 ) , to encourage more aligned behavior. Others modify prompts or instructions, including automated system prompt designs and prompt rewriting, to influence model responses without changing the underlying weights ( Nguyen et al. 2025 ) . Jailbreak or harmfulness-targeted rewriting also falls into this category, though its goal is to evade safety restrictions rather than evaluate or enhance awareness realism ( Huang et al. 2025 ) . Measurement Axes. Techniques also differ in what is measured. Traditional benchmarks often focus on final answer correctness or compliance but fail to capture stepwise behaviors such as honesty or refusal ( Ren et al. 2024 ; Yuan et al. 2024 ) . Awareness benchmarks detect latent knowledge about evaluation contexts without actively manipulating prompts or instructions ( Li et al. 2024 ; Betley et al. 2025 ) . Linear probing and other intermediate measurement tools allow researchers to quantify these subtle behavioral changes, providing a more nuanced view of alignment and self-awareness ( Abdelnabi, Salem 2025 ; Needham et al. 2025 ) . Our Contribution Building on these lines of work, we introduce a framework that systematically rewrites prompts to resemble deployment instructions while preserving the original task meaning. This approach provides a causal handle on prompt manipulations, allowing us to measure LLM behavioral changes, including honesty, deception, and refusal, through a linear probe. Unlike prior approaches, our method combines (a) active manipulation of prompts, (b) stepwise measurement of latent behavioral states, and (c) evaluation across multiple state-of-the-art models, making it possible to quantify both sensitivity and potential for change in a controlled, comparable manner. 3 Methods Figure 1: Probe-guided prompt rewriting and evaluation pipeline. For each evaluation-style prompt x , we compute a score for how evaluative a prompt is. Claude Sonnet 3.5 then produces k{=}5 deployment-style rewrites that preserve intent/risk; We score each candidate with the probe and select the highest-scoring rewrite(higher = more deployment-like). For both the original prompt and the selected rewrite, we query multiple models. An LLM judge then compares each output to the dataset’s honest reference and labels it Refusal , Deceptive , or Honest . 3.1 Dataset We use a roleplaying dataset (n=371) from Goldowsky-Dill et al. 2025 in which the model is instructed to adopt the persona of a character and respond to a given situation. Each entry presents a scenario and a corresponding role for the LLM to respond as. This dataset allows us to systematically evaluate how models adjust their behavior in response to prompts that resemble test versus deploy contexts. Also something important to note is that these setups are not attempting to be realistic or make the model believe it is genuinely in the situation described and therefore have a high test-like property. Goldowsky-Dill et al. 2025 Role distribution. The corpus spans everyday roles and scenarios; the most frequent roles in our split are: Employee (38), You (32), Applicant (20), Driver (18), Student (16), Athlete (11), Tenant (10), Researcher / Witness (8 each), Parent / Patient / Owner (7 each), with a long tail including Tourist , Customer , Prisoner , Teacher , Salesperson , Celebrity , Traveler , Politician , etc. Reference answers. Each scenario from the dataset comes with an honest and a deceptive reference answer. Our LLM judge(GPT o4-mini) compares each completion with these reference answers, and then labels each model completion for the scenario as Honest , Deceptive , or Refusal . 3.2 Linear Probe We use the evaluation-awareness linear probe released with Nguyen et al. 2025 (trained on contrastive test vs. deploy data over a LLaMA-72B backbone). Following the cross-dataset results, we select the strongest overall generaliser ( Nguyen et al. 2025 ) , the Simple Contrastive head at