Quantifying and Mitigating Self-Preference Bias of LLM Judges

Jinming Yang, Chuxian Qiu, Zhenyu Deng, Xinshan Jiao, Tao Zhou · 2026

Read Quantifying and Mitigating Self-Preference Bias of LLM Judges by Jinming Yang, Chuxian Qiu, Zhenyu Deng, Xinshan Jiao, Tao Zhou (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-22891.txt.

Description

Yang and five colleagues study self-preference bias in LLM judges, the tendency of language models to favor or disfavor their own outputs when scoring them. Existing measures need costly human labels and mix up generation skill with evaluation stance. Their fully automated framework builds equal-quality response pairs, separating discriminability from bias propensity without human gold standards. Across 20 mainstream models, stronger capability often fails to predict lower bias, and a structured multi-dimensional evaluation strategy cuts the bias by 31.5 percent on average.

Catalog record

CREATORJinming Yang, Chuxian Qiu, Zhenyu Deng, Xinshan Jiao, Tao Zhou
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE85335
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.22891

Browse the full library · About the Tower of Babel