Do Large Language Model Benchmarks Test Reliability?
Joshua Vendrow, Edward Vendrow, Sara Beery, Aleksander MądryMIT · 2025
Read Do Large Language Model Benchmarks Test Reliability? by Joshua Vendrow, Edward Vendrow, Sara Beery, Aleksander MądryMIT (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-03461.txt.
Description
Joshua Vendrow, Edward Vendrow, Sara Beery, and Aleksander Madry ask whether language model benchmarks really test reliability, in a 2025 paper. Leaderboards track what models can do, but not how steadily they do it. The team shows that label errors riddle popular benchmarks. Those errors hide model failures and make shaky behavior look solid.
They propose platinum benchmarks: test sets scrubbed of wrong labels and ambiguity. As a first try, they cleaned examples from fifteen well-known benchmarks. Top models still failed some cleaned tasks, even grade-school style math word problems. The failures clustered in patterns no one had flagged before. The authors share their cleaned sets as code and data, urging the field to grade reliability as carefully as skill.
Catalog record
| CREATOR | Joshua Vendrow, Edward Vendrow, Sara Beery, Aleksander MądryMIT |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 60 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2502-03461