am-ELO: A Stable Framework for Arena-based LLM Evaluation

Zirui Liu, Jiatong Li, Yan Zhuang, Qi Liu *, Shuanghong Shen, Jie Ouyang, Mingyue Cheng, Shijin Wang · 2025

Read am-ELO: A Stable Framework for Arena-based LLM Evaluation by Zirui Liu, Jiatong Li, Yan Zhuang, Qi Liu *, Shuanghong Shen, Jie Ouyang, Mingyue Cheng, Shijin Wang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-03475.txt.

Description

Zirui Liu and co-authors build a steadier way to rank language models by head-to-head contests, in a 2025 paper. Arena evaluation pits models against each other and lets human judges pick winners. The classic Elo system used for scoring wobbles: rankings can flip with the order of matches. It also treats all judges as equally sharp. Their framework fixes both flaws.

A maximum likelihood approach replaces Elo's step-by-step updates, and the team proves it gives consistent, stable rankings. An extended version also scores each annotator's reliability while scoring the models. That joint estimate feeds back into fairer model ratings. Experiments back the stability claims. The authors offer the framework as a firmer base for the public leaderboards that now guide model choice.

Catalog record

CREATORZirui Liu, Jiatong Li, Yan Zhuang, Qi Liu *, Shuanghong Shen, Jie Ouyang, Mingyue Cheng, Shijin Wang
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE38 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-03475

Browse the full library · About the Tower of Babel