Grading Scale Impact on LLM-as-a-Judge: Human-LLM Alignment Is Highest on 0-5 Grading Scale
Weiyue Li Minda Zhao Weixuan Dong Jiahui Cai Yuze Wei · 2026
Read Grading Scale Impact on LLM-as-a-Judge: Human-LLM Alignment Is Highest on 0-5 Grading Scale by Weiyue Li Minda Zhao Weixuan Dong Jiahui Cai Yuze Wei (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-03444.txt.
Description
Weiyue Li and fourteen co-authors test whether the grading scale itself changes how well language model judges agree with human raters. Models are now used to score other models, but their scores can shift when prompts change. The team collected human and model ratings on three scales across six benchmarks with objective, subjective, and mixed tasks.
Using a standard agreement measure, they found model scores were not fully stable across scales on subjective tasks. The zero to five scale gave the strongest agreement between humans and models overall. Pooling results also hid big differences between benchmarks and between gender groups. The authors argue scale choice and subgroup checks belong in every judge protocol.
Catalog record
| CREATOR | Weiyue Li Minda Zhao Weixuan Dong Jiahui Cai Yuze Wei |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 57 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2601-03444