When Scanners Lie: Evaluator Instability in LLM Red-Teaming

Lidor Erez, Omer Hofman, Tamir Nizri, Roman Vainshtein · 2026

Read When Scanners Lie: Evaluator Instability in LLM Red-Teaming by Lidor Erez, Omer Hofman, Tamir Nizri, Roman Vainshtein (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-14633.txt.

Description

Erez and co-authors show that LLM red-team scanners wobble with their evaluator choice. Scanners report attack success rates to grade model risk. The judge that marks success is a hidden weak point. Swap the evaluator, hold attacks and outputs fixed, and reported rates shift a lot. Their two-phase frame first measures evaluator disagreement by attack type. A verifier then checks evaluators without heavy human labeling. On the Garak scanner, 22 of 25 attack classes prove unstable. Checked evaluators lift accuracy from 72% to 89%.

Catalog record

CREATORLidor Erez, Omer Hofman, Tamir Nizri, Roman Vainshtein
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE41850
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.14633

Browse the full library · About the Tower of Babel