Quantifying and Mitigating Self-Preference Bias of LLM Judges
Jinming Yang, Chuxian Qiu, Zhenyu Deng, Xinshan Jiao, Tao Zhou · 2026
Read Quantifying and Mitigating Self-Preference Bias of LLM Judges by Jinming Yang, Chuxian Qiu, Zhenyu Deng, Xinshan Jiao, Tao Zhou (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-22891.txt.
Description
Yang and five colleagues study self-preference bias in LLM judges, the tendency of language models to favor or disfavor their own outputs when scoring them. Existing measures need costly human labels and mix up generation skill with evaluation stance. Their fully automated framework builds equal-quality response pairs, separating discriminability from bias propensity without human gold standards. Across 20 mainstream models, stronger capability often fails to predict lower bias, and a structured multi-dimensional evaluation strategy cuts the bias by 31.5 percent on average.
Catalog record
| CREATOR | Jinming Yang, Chuxian Qiu, Zhenyu Deng, Xinshan Jiao, Tao Zhou |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 85335 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.22891