Auditing LLM Benchmarks with Item Response Theory

Sander Land, Daniel M. Bikel · 2026

Read Auditing LLM Benchmarks with Item Response Theory by Sander Land, Daniel M. Bikel (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-30504.txt.

Description

Sander Land and Daniel M. Bikel present an Item Response Theory-based flag spotting likely bad labels. It scores 95 percent right in the top 200 cases across seven benchmarks with 114 models, beating a trained checker. They trace these errors to rote label rules, upstream note errors, and unclear items without a sound single label. The same model fit reveals reward models focus on style taste rather than facts. It spots one frontier reward model agreeing with bad labels at 78 percent score versus 38 percent for its peers.

Catalog record

CREATORSander Land, Daniel M. Bikel
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE116960
LICENSEhttp://creativecommons.org/licenses/by-sa/4.0/
RIGHTSCC BY SA
SOURCEarXiv

Source: https://arxiv.org/abs/2605.30504

Browse the full library · About the Tower of Babel