BenchMarker: An Education-Inspired Toolkit for Highlighting Flaws in Multiple-Choice Benchmarks

Nishant Balepur Bhavya Rajasekaran Jane Oh Michael Xie Atrey Desai, Vipul Gupta Steven Moore Eunsol Choi Rachel Rudinger Jordan Boyd-Graber, [0.5em] University of Maryland New York University Scale AI George Mason University[0.5em] [email protected] [email protected] · 2026

Read BenchMarker: An Education-Inspired Toolkit for Highlighting Flaws in Multiple-Choice Benchmarks by Nishant Balepur Bhavya Rajasekaran Jane Oh Michael Xie Atrey Desai, Vipul Gupta Steven Moore Eunsol Choi Rachel Rudinger Jordan Boyd-Graber, [0.5em] University of Maryland New York University Scale AI George Mason University[0.5em] [email protected] [email protected] (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2602-06221.txt.

Description

Multiple-choice tests are the standard yardstick for language models, yet many test items are flawed. Nishant Balepur and 9 co-authors built BenchMarker, a toolkit borrowed from education research that uses language model judges to flag bad items. It hunts three flaws: items copied from the open web, cue patterns that allow guessing, and writing errors.

Checked against human labels, the tool then audited a dozen well-known benchmarks. Flaws were common, especially in machine-made and crowdsourced sets. Copied items inflated scores, while writing errors deflated them and shuffled rankings. Earlier repair attempts fixed their target flaws but added new ones. The authors release the toolkit to help benchmark builders do better.

Catalog record

CREATORNishant Balepur Bhavya Rajasekaran Jane Oh Michael Xie Atrey Desai, Vipul Gupta Steven Moore Eunsol Choi Rachel Rudinger Jordan Boyd-Graber, [0.5em] University of Maryland New York University Scale AI George Mason University[0.5em] [email protected] [email protected]
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE70 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2602-06221

Browse the full library · About the Tower of Babel