Do Large Language Models Plan Answer Positions? Position Bias in Multiple-Choice Question Generation

Xuemei Tang, Xufeng Duan, Zhenguang G. Cai · 2026

Read Do Large Language Models Plan Answer Positions? Position Bias in Multiple-Choice Question Generation by Xuemei Tang, Xufeng Duan, Zhenguang G. Cai (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-01846.txt.

Description

Xuemei Tang, Xufeng Duan, and Zhenguang G Cai catch language models stacking the deck when they write multiple-choice questions. The right answer should land evenly across the options. Yet across ten language models and five vision-language models on three generation tasks, the placement is systematically skewed, with kin models sharing kin habits.

Probing inside the models shows the bias is planned early. Hidden states in the question stem already carry a readable signal of where the correct answer will sit, before the options are written. Steering those activations can shift the placement distribution substantially, partial control over a preference the model never states. The authors frame the work as a handle on implicit positional planning. It is also a caution for anyone generating or grading tests with these models: the answer key's shape may be decided before the question is finished.

Catalog record

CREATORXuemei Tang, Xufeng Duan, Zhenguang G. Cai
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE58 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.01846

Browse the full library · About the Tower of Babel