am-ELO: A Stable Framework for Arena-based LLM Evaluation
Zirui Liu, Jiatong Li, Yan Zhuang, Qi Liu *, Shuanghong Shen, Jie Ouyang, Mingyue Cheng, Shijin Wang · 2025
Read am-ELO: A Stable Framework for Arena-based LLM Evaluation by Zirui Liu, Jiatong Li, Yan Zhuang, Qi Liu *, Shuanghong Shen, Jie Ouyang, Mingyue Cheng, Shijin Wang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-03475.txt.
Description
Zirui Liu and co-authors build a steadier way to rank language models by head-to-head contests, in a 2025 paper. Arena evaluation pits models against each other and lets human judges pick winners. The classic Elo system used for scoring wobbles: rankings can flip with the order of matches. It also treats all judges as equally sharp. Their framework fixes both flaws.
A maximum likelihood approach replaces Elo's step-by-step updates, and the team proves it gives consistent, stable rankings. An extended version also scores each annotator's reliability while scoring the models. That joint estimate feeds back into fairer model ratings. Experiments back the stability claims. The authors offer the framework as a firmer base for the public leaderboards that now guide model choice.
Catalog record
| CREATOR | Zirui Liu, Jiatong Li, Yan Zhuang, Qi Liu *, Shuanghong Shen, Jie Ouyang, Mingyue Cheng, Shijin Wang |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 38 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-03475