JudgeSense: A Benchmark for Prompt Sensitivity in LLM-as-a-Judge Systems
Rohith Reddy Bellibatlu · 2026
Read JudgeSense: A Benchmark for Prompt Sensitivity in LLM-as-a-Judge Systems by Rohith Reddy Bellibatlu (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-23478.txt.
Description
Bellibatlu, Raff, and Zhang ask whether judge models stay consistent when a request is reworded. Their JudgeSense benchmark spans four tasks and twenty-five judges, scoring each judge against its own agreement with itself on identical wording so that sampling noise is not blamed on phrasing. Rewording lowers agreement on every task, the ordinal task proves least stable, and larger models within a family are no steadier.
Catalog record
| CREATOR | Rohith Reddy Bellibatlu |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 50265 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.23478