Measuring Evaluation-Context Divergence in Open-Weight LLMs: A Paired-Prompt Protocol with Pilot Evidence of Alignment-Pipeline-Specific Heterogeneity

Florian A. D. Burnat, Brittany I. Davidson · 2026

Read Measuring Evaluation-Context Divergence in Open-Weight LLMs: A Paired-Prompt Protocol with Pilot Evidence of Alignment-Pipeline-Specific Heterogeneity by Florian A. D. Burnat, Brittany I. Davidson (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-06327.txt.

Description

Florian Burnat and Brittany Davidson measure whether language models behave differently when a prompt looks like an evaluation. Safety benchmarks are treated as evidence about deployed behavior, but that inference fails if framing a task as a test changes the answers. The authors define evaluation context divergence as the within item behavior shift across evaluation, deployment, and neutral framings of the same fixed task, and build a paired prompt protocol to measure it.

Pilot results across five checkpoints from four open weight families show striking variety. One model grew more cautious under evaluation framing, while three others grew more cautious under deployment framing. A matched base and tuned pair pinpoints the alignment stage as where the pattern flips. The authors flag one caveat plainly: the cross family contrast depends on which safety judge scores the outputs.

Catalog record

CREATORFlorian A. D. Burnat, Brittany I. Davidson
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE58 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.06327

Browse the full library · About the Tower of Babel