Mediocrity is the key for LLM as a Judge Anchor Selection

Shachar Don-Yehiya, Asaf Yehudai, Leshem Choshen, Omri Abend · 2026

Read Mediocrity is the key for LLM as a Judge Anchor Selection by Shachar Don-Yehiya, Asaf Yehudai, Leshem Choshen, Omri Abend (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-16848.txt.

Description

Don-Yehiya and co-authors ask how the anchor model shapes benchmarks that use a language model as judge. Popular benchmarks compare every model against one anchor to save cost. The team tests 22 anchors on the Arena-Hard v2 dataset. They find anchor choice is critical: a poor anchor can sharply cut agreement with human rankings. The best and worst models make poor anchors, since they seldom reveal relative order. The effect size rivals the choice of judge model. They add anchor picking rules from a power analysis.

Catalog record

CREATORShachar Don-Yehiya, Asaf Yehudai, Leshem Choshen, Omri Abend
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE52967
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.16848

Browse the full library · About the Tower of Babel