Auditing LLM Benchmarks with Item Response Theory
Sander Land, Daniel M. Bikel · 2026
Read Auditing LLM Benchmarks with Item Response Theory by Sander Land, Daniel M. Bikel (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-30504.txt.
Description
Sander Land and Daniel M. Bikel present an Item Response Theory-based flag spotting likely bad labels. It scores 95 percent right in the top 200 cases across seven benchmarks with 114 models, beating a trained checker. They trace these errors to rote label rules, upstream note errors, and unclear items without a sound single label. The same model fit reveals reward models focus on style taste rather than facts. It spots one frontier reward model agreeing with bad labels at 78 percent score versus 38 percent for its peers.
Catalog record
| CREATOR | Sander Land, Daniel M. Bikel |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 116960 |
| LICENSE | http://creativecommons.org/licenses/by-sa/4.0/ |
| RIGHTS | CC BY SA |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.30504