Do Large Language Model Benchmarks Test Reliability?

Joshua Vendrow, Edward Vendrow, Sara Beery, Aleksander MądryMIT · 2025

Read Do Large Language Model Benchmarks Test Reliability? by Joshua Vendrow, Edward Vendrow, Sara Beery, Aleksander MądryMIT (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-03461.txt.

Description

Joshua Vendrow, Edward Vendrow, Sara Beery, and Aleksander Madry ask whether language model benchmarks really test reliability, in a 2025 paper. Leaderboards track what models can do, but not how steadily they do it. The team shows that label errors riddle popular benchmarks. Those errors hide model failures and make shaky behavior look solid.

They propose platinum benchmarks: test sets scrubbed of wrong labels and ambiguity. As a first try, they cleaned examples from fifteen well-known benchmarks. Top models still failed some cleaned tasks, even grade-school style math word problems. The failures clustered in patterns no one had flagged before. The authors share their cleaned sets as code and data, urging the field to grade reliability as carefully as skill.

Catalog record

CREATORJoshua Vendrow, Edward Vendrow, Sara Beery, Aleksander MądryMIT
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE60 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2502-03461

Browse the full library · About the Tower of Babel