The Leaderboard Illusion
name=Shivalika Singh \faa, affiliation=1, name=Yiyang Nan, affiliation=1, name=Alex Wang, affiliation=2, name=Daniel D’souza, affiliation=1, name=Sayash Kapoor, affiliation=3, name=Ahmet Üstün, affiliation=1, name=Sanmi Koyejo, affiliation=4, name=Yuntian Deng, affiliation=5, name=Shayne Longpre, af · 2025
Read The Leaderboard Illusion by name=Shivalika Singh \faa, affiliation=1, name=Yiyang Nan, affiliation=1, name=Alex Wang, affiliation=2, name=Daniel D’souza, affiliation=1, name=Sayash Kapoor, affiliation=3, name=Ahmet Üstün, affiliation=1, name=Sanmi Koyejo, affiliation=4, name=Yuntian Deng, affiliation=5, name=Shayne Longpre, af (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-20879.txt.
Introduction
Shivalika Singh and co-authors published a 2025 paper arguing that a famous AI leaderboard does not rank models fairly. Chatbot Arena lets the public vote head-to-head on chatbot answers, and its table steers the whole field. The authors document hidden practices that tilt the table toward a few big providers.
Some providers, they find, test many private versions of a model on the arena before release. They can then publish only the best score. One company tested twenty-seven private variants ahead of a major launch. Closed models also get sampled in more battles and are removed from the board less often than open ones. As a result, two companies have received about a fifth each of all arena data. That data itself improves models, widening the gap. The paper calls the ranking an illusion and urges reforms for fair measurement.
Catalog record
| CREATOR | name=Shivalika Singh \faa, affiliation=1, name=Yiyang Nan, affiliation=1, name=Alex Wang, affiliation=2, name=Daniel D’souza, affiliation=1, name=Sayash Kapoor, affiliation=3, name=Ahmet Üstün, affiliation=1, name=Sanmi Koyejo, affiliation=4, name=Yuntian Deng, affiliation=5, name=Shayne Longpre, af |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 142 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2504-20879