JudgeSense: A Benchmark for Prompt Sensitivity in LLM-as-a-Judge Systems

Rohith Reddy Bellibatlu · 2026

Read JudgeSense: A Benchmark for Prompt Sensitivity in LLM-as-a-Judge Systems by Rohith Reddy Bellibatlu (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-23478.txt.

Description

Bellibatlu, Raff, and Zhang ask whether judge models stay consistent when a request is reworded. Their JudgeSense benchmark spans four tasks and twenty-five judges, scoring each judge against its own agreement with itself on identical wording so that sampling noise is not blamed on phrasing. Rewording lowers agreement on every task, the ordinal task proves least stable, and larger models within a family are no steadier.

Catalog record

CREATORRohith Reddy Bellibatlu
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE50265
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.23478

Browse the full library · About the Tower of Babel