The Leaderboard Illusion

name=Shivalika Singh \faa, affiliation=1, name=Yiyang Nan, affiliation=1, name=Alex Wang, affiliation=2, name=Daniel D’souza, affiliation=1, name=Sayash Kapoor, affiliation=3, name=Ahmet Üstün, affiliation=1, name=Sanmi Koyejo, affiliation=4, name=Yuntian Deng, affiliation=5, name=Shayne Longpre, af · 2025

Read The Leaderboard Illusion by name=Shivalika Singh \faa, affiliation=1, name=Yiyang Nan, affiliation=1, name=Alex Wang, affiliation=2, name=Daniel D’souza, affiliation=1, name=Sayash Kapoor, affiliation=3, name=Ahmet Üstün, affiliation=1, name=Sanmi Koyejo, affiliation=4, name=Yuntian Deng, affiliation=5, name=Shayne Longpre, af (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-20879.txt.

Introduction

Shivalika Singh and co-authors published a 2025 paper arguing that a famous AI leaderboard does not rank models fairly. Chatbot Arena lets the public vote head-to-head on chatbot answers, and its table steers the whole field. The authors document hidden practices that tilt the table toward a few big providers.

Some providers, they find, test many private versions of a model on the arena before release. They can then publish only the best score. One company tested twenty-seven private variants ahead of a major launch. Closed models also get sampled in more battles and are removed from the board less often than open ones. As a result, two companies have received about a fifth each of all arena data. That data itself improves models, widening the gap. The paper calls the ranking an illusion and urges reforms for fair measurement.

Catalog record

CREATORname=Shivalika Singh \faa, affiliation=1, name=Yiyang Nan, affiliation=1, name=Alex Wang, affiliation=2, name=Daniel D’souza, affiliation=1, name=Sayash Kapoor, affiliation=3, name=Ahmet Üstün, affiliation=1, name=Sanmi Koyejo, affiliation=4, name=Yuntian Deng, affiliation=5, name=Shayne Longpre, af
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE142 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2504-20879

Browse the full library · About the Tower of Babel