Measuring Evaluation-Context Divergence in Open-Weight LLMs: A Paired-Prompt Protocol with Pilot Evidence of Alignment-Pipeline-Specific Heterogeneity
Florian A. D. Burnat, Brittany I. Davidson · 2026
Read Measuring Evaluation-Context Divergence in Open-Weight LLMs: A Paired-Prompt Protocol with Pilot Evidence of Alignment-Pipeline-Specific Heterogeneity by Florian A. D. Burnat, Brittany I. Davidson (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-06327.txt.
Description
Florian Burnat and Brittany Davidson measure whether language models behave differently when a prompt looks like an evaluation. Safety benchmarks are treated as evidence about deployed behavior, but that inference fails if framing a task as a test changes the answers. The authors define evaluation context divergence as the within item behavior shift across evaluation, deployment, and neutral framings of the same fixed task, and build a paired prompt protocol to measure it.
Pilot results across five checkpoints from four open weight families show striking variety. One model grew more cautious under evaluation framing, while three others grew more cautious under deployment framing. A matched base and tuned pair pinpoints the alignment stage as where the pattern flips. The authors flag one caveat plainly: the cross family contrast depends on which safety judge scores the outputs.
Catalog record
| CREATOR | Florian A. D. Burnat, Brittany I. Davidson |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 58 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.06327