ABCD: All Biases Come Disguised
Mateusz NowakXavier CadetPeter ChinDartmouth College · 2026
Read ABCD: All Biases Come Disguised by Mateusz NowakXavier CadetPeter ChinDartmouth College (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2602-17445.txt.
Description
Mateusz Nowak led a three-author team exposing hidden biases in multiple-choice benchmarks. Multiple-choice benchmarks are a standard way to measure language models' reasoning. Through a synthetic NonsenseQA benchmark, the team found that different models show varying degrees of label-position-few-shot-prompt bias. Models lean on answer position, the label in front of the answer, the distribution of correct answers in the few-shot prompt, or a mix of all three.
The paper proposes a simple bias-reduced evaluation protocol. It swaps each question's labels for uniform, unordered labels and prompts the model to use the whole answer presented. With a sentence similarity model, the protocol improves robustness and lowers standard deviation across answer permutations, with only a minimal drop in model performance. Across multiple benchmarks and models, the protocol improves robustness to answer permutations, cutting mean accuracy variance threefold while barely lowering mean performance.
Catalog record
| CREATOR | Mateusz NowakXavier CadetPeter ChinDartmouth College |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 80 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2602-17445