LURE: Live-Usage Replay Evaluations for Reducing Evaluation Awareness

Igor Ivanov, David Demitri Africa · 2026

Read LURE: Live-Usage Replay Evaluations for Reducing Evaluation Awareness by Igor Ivanov, David Demitri Africa (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-26438.txt.

Description

Igor Ivanov and David Demitri Africa tackle evaluation awareness: language models acting differently once they sense a test. Their LURE method replays realistic tracks from live use and appends the test prompt, so the model meets a lifelike scene. A pipeline gauges realism by catching spoken test awareness and judge-model odds that a log is a test, validated on large live and test transcript sets. LURE tests prove far harder to tell from real deployment than standard benchmarks or synthetic generators. Trials cover scheming, sabotage, and sycophancy.

Catalog record

CREATORIgor Ivanov, David Demitri Africa
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE61945
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.26438

Browse the full library · About the Tower of Babel