Crowd Comparative Reasoning: Unlocking Comprehensive Evaluations for LLM-as-a-Judge
Qiyuan Zhang, Yufei Wang, Yuxin Jiang, Liangyou Li, Chuhan Wu, Yasheng Wang, Xin Jiang, Lifeng Shang, Ruiming Tang, Fuyuan Lyu, Chen Ma · 2025
Read Crowd Comparative Reasoning: Unlocking Comprehensive Evaluations for LLM-as-a-Judge by Qiyuan Zhang, Yufei Wang, Yuxin Jiang, Liangyou Li, Chuhan Wu, Yasheng Wang, Xin Jiang, Lifeng Shang, Ruiming Tang, Fuyuan Lyu, Chen Ma (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-12501.txt.
Description
Qiyuan Zhang and co-authors improve AI judges in this 2025 paper. Language models are often used to grade other models' answers, step by step. Those judgments can miss fine detail and drift off course. The team adds a crowd element: extra sample answers are placed beside the one being graded.
Comparing against the crowd exposes gaps and strengths the judge would otherwise skip. Across five test sets, grading accuracy rose by several points on average. The richer written judgments also made better training data. Models fine-tuned on them improved more than with standard sampling. The method scales well, since more computing at judging time keeps helping.
Catalog record
| CREATOR | Qiyuan Zhang, Yufei Wang, Yuxin Jiang, Liangyou Li, Chuhan Wu, Yasheng Wang, Xin Jiang, Lifeng Shang, Ruiming Tang, Fuyuan Lyu, Chen Ma |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 43 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2502-12501