BenchMarker: An Education-Inspired Toolkit for Highlighting Flaws in Multiple-Choice Benchmarks
Nishant Balepur Bhavya Rajasekaran Jane Oh Michael Xie Atrey Desai, Vipul Gupta Steven Moore Eunsol Choi Rachel Rudinger Jordan Boyd-Graber, [0.5em] University of Maryland New York University Scale AI George Mason University[0.5em] [email protected] [email protected] · 2026
Read BenchMarker: An Education-Inspired Toolkit for Highlighting Flaws in Multiple-Choice Benchmarks by Nishant Balepur Bhavya Rajasekaran Jane Oh Michael Xie Atrey Desai, Vipul Gupta Steven Moore Eunsol Choi Rachel Rudinger Jordan Boyd-Graber, [0.5em] University of Maryland New York University Scale AI George Mason University[0.5em] [email protected] [email protected] (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2602-06221.txt.
Description
Multiple-choice tests are the standard yardstick for language models, yet many test items are flawed. Nishant Balepur and 9 co-authors built BenchMarker, a toolkit borrowed from education research that uses language model judges to flag bad items. It hunts three flaws: items copied from the open web, cue patterns that allow guessing, and writing errors.
Checked against human labels, the tool then audited a dozen well-known benchmarks. Flaws were common, especially in machine-made and crowdsourced sets. Copied items inflated scores, while writing errors deflated them and shuffled rankings. Earlier repair attempts fixed their target flaws but added new ones. The authors release the toolkit to help benchmark builders do better.
Catalog record
| CREATOR | Nishant Balepur Bhavya Rajasekaran Jane Oh Michael Xie Atrey Desai, Vipul Gupta Steven Moore Eunsol Choi Rachel Rudinger Jordan Boyd-Graber, [0.5em] University of Maryland New York University Scale AI George Mason University[0.5em] [email protected] [email protected] |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 70 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2602-06221