Mediocrity is the key for LLM as a Judge Anchor Selection
Shachar Don-Yehiya, Asaf Yehudai, Leshem Choshen, Omri Abend · 2026
Read Mediocrity is the key for LLM as a Judge Anchor Selection by Shachar Don-Yehiya, Asaf Yehudai, Leshem Choshen, Omri Abend (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-16848.txt.
Description
Don-Yehiya and co-authors ask how the anchor model shapes benchmarks that use a language model as judge. Popular benchmarks compare every model against one anchor to save cost. The team tests 22 anchors on the Arena-Hard v2 dataset. They find anchor choice is critical: a poor anchor can sharply cut agreement with human rankings. The best and worst models make poor anchors, since they seldom reveal relative order. The effect size rivals the choice of judge model. They add anchor picking rules from a power analysis.
Catalog record
| CREATOR | Shachar Don-Yehiya, Asaf Yehudai, Leshem Choshen, Omri Abend |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 52967 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.16848