When Scanners Lie: Evaluator Instability in LLM Red-Teaming
Lidor Erez, Omer Hofman, Tamir Nizri, Roman Vainshtein · 2026
Read When Scanners Lie: Evaluator Instability in LLM Red-Teaming by Lidor Erez, Omer Hofman, Tamir Nizri, Roman Vainshtein (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-14633.txt.
Description
Erez and co-authors show that LLM red-team scanners wobble with their evaluator choice. Scanners report attack success rates to grade model risk. The judge that marks success is a hidden weak point. Swap the evaluator, hold attacks and outputs fixed, and reported rates shift a lot. Their two-phase frame first measures evaluator disagreement by attack type. A verifier then checks evaluators without heavy human labeling. On the Garak scanner, 22 of 25 attack classes prove unstable. Checked evaluators lift accuracy from 72% to 89%.
Catalog record
| CREATOR | Lidor Erez, Omer Hofman, Tamir Nizri, Roman Vainshtein |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 41850 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.14633