Judging with Many Minds: Do More Perspectives Mean Less Prejudice? On Bias Amplification and Resistance in Multi-Agent Based LLM-as-Judge

Chiyu Ma, Enpei Zhang, Yilun Zhao, Wenjun Liu, Yaning Jia · 2025

Read Judging with Many Minds: Do More Perspectives Mean Less Prejudice? On Bias Amplification and Resistance in Multi-Agent Based LLM-as-Judge by Chiyu Ma, Enpei Zhang, Yilun Zhao, Wenjun Liu, Yaning Jia (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-19477.txt.

Description

Ma and nine co-authors study bias in multi-agent language model judging systems, in a paper from twenty twenty-five. Model judges scale evaluation, and multi-agent versions promise better quality. Four bias types are tested: position, verbosity, chain-of-thought, and bandwagon. Two frameworks, debate and meta-judging, are compared systematically.

Debate sharply amplifies bias after the first round, and it persists. Meta-judging resists bias growth much better overall. Adding a debiased agent helps debate but adds little for meta-judges. Collaborative evaluation needs targeted bias mitigation by design. More perspectives do not automatically mean less prejudice.

Catalog record

CREATORChiyu Ma, Enpei Zhang, Yilun Zhao, Wenjun Liu, Yaning Jia
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE72 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-19477

Browse the full library · About the Tower of Babel