Self-Preference Bias in Rubric-Based Evaluation of Large Language Models

José Pombal, Ricardo Rei & André F. T. Martins · 2026

Read Self-Preference Bias in Rubric-Based Evaluation of Large Language Models by José Pombal, Ricardo Rei & André F. T. Martins (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-06996.txt.

Description

Pombal, Rei, and Martins present the first study of self-preference bias in rubric-based evaluation. Judges favor their own outputs or model family, skewing development, especially for self-improvement. Using IFEval and LiveCodeBench, benchmarks with verifiable rubrics, bias persists even when criteria are objective. Where generators fail, judges are over 50 percent more likely to wrongly pass their own outputs. Ensembling judges mitigates bias but does not eliminate it. On HealthBench, a medical benchmark with subjective rubrics, bias skews scores by up to 10 points, decisive when ranking models.

Catalog record

CREATORJosé Pombal, Ricardo Rei & André F. T. Martins
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE39629
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.06996

Browse the full library · About the Tower of Babel