LURE: Live-Usage Replay Evaluations for Reducing Evaluation Awareness
Igor Ivanov, David Demitri Africa · 2026
Read LURE: Live-Usage Replay Evaluations for Reducing Evaluation Awareness by Igor Ivanov, David Demitri Africa (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-26438.txt.
Description
Igor Ivanov and David Demitri Africa tackle evaluation awareness: language models acting differently once they sense a test. Their LURE method replays realistic tracks from live use and appends the test prompt, so the model meets a lifelike scene. A pipeline gauges realism by catching spoken test awareness and judge-model odds that a log is a test, validated on large live and test transcript sets. LURE tests prove far harder to tell from real deployment than standard benchmarks or synthetic generators. Trials cover scheming, sabotage, and sycophancy.
Catalog record
| CREATOR | Igor Ivanov, David Demitri Africa |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 61945 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.26438