ABCD: All Biases Come Disguised

Mateusz NowakXavier CadetPeter ChinDartmouth College · 2026

Read ABCD: All Biases Come Disguised by Mateusz NowakXavier CadetPeter ChinDartmouth College (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2602-17445.txt.

Description

Mateusz Nowak led a three-author team exposing hidden biases in multiple-choice benchmarks. Multiple-choice benchmarks are a standard way to measure language models' reasoning. Through a synthetic NonsenseQA benchmark, the team found that different models show varying degrees of label-position-few-shot-prompt bias. Models lean on answer position, the label in front of the answer, the distribution of correct answers in the few-shot prompt, or a mix of all three.

The paper proposes a simple bias-reduced evaluation protocol. It swaps each question's labels for uniform, unordered labels and prompts the model to use the whole answer presented. With a sentence similarity model, the protocol improves robustness and lowers standard deviation across answer permutations, with only a minimal drop in model performance. Across multiple benchmarks and models, the protocol improves robustness to answer permutations, cutting mean accuracy variance threefold while barely lowering mean performance.

Catalog record

CREATORMateusz NowakXavier CadetPeter ChinDartmouth College
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE80 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2602-17445

Browse the full library · About the Tower of Babel