Pushing the Boundaries of Multiple Choice Evaluation to One Hundred Options
Nahyun Lee, Guijin Son · 2026
Read Pushing the Boundaries of Multiple Choice Evaluation to One Hundred Options by Nahyun Lee, Guijin Son (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-14634.txt.
Introduction
Lee and Son propose a massive-option method stretching candidate sets to one hundred options, sharply cutting chance. They test it on Korean spelling error checks, where models must find the one incorrect sentence in a large set. With fixed targets plus repeated resampling and shuffling, results stay stable, separating content failures from positional artifacts. Strong low-option performance often overstates competence, weakening under dense interference and revealing semantic confusion and early-option bias. Padding and length-matched tests point to option ranking, not context length, as the bottleneck.
Catalog record
| CREATOR | Nahyun Lee, Guijin Son |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 55909 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.14634
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Pushing the Boundaries of Multiple Choice Evaluation to One Hundred Options Nahyun Lee, Guijin Son Abstract Multiple choice evaluation is widely used for benchmarking large language models, yet near ceiling accuracy in low option settings can be sustained by shortcut strategies that obscure true competence. Therefore, we propose a massive option evaluation protocol that scales the candidate set to one hundred options and sharply reduces the impact of chance performance. We apply this framework to a Korean orthography error detection task where models must pick the single incorrect sentence from a large candidate set. With fixed targets and repeated resampling and shuffling, we obtain stable estimates while separating content driven failures from positional artifacts. Across experiments, results indicate that strong performance in low option settings can overstate model competence. This apparent advantage often weakens under dense interference at high N N , revealing gaps that conventional benchmarks tend to obscure. We identify two failure modes, semantic confusion and position bias toward early options under uncertainty. To isolate the effect of context length, we run padding controlled and length matched tests, which suggest that the main bottleneck is candidate ranking rather than context length. Together, these findings support massive option evaluation as a general framework for stress testing model reliability under extreme distractor density, beyond what low option benchmarks can reveal. 1 Introduction Multiple Choice Question Answering (MCQA) has emerged as the dominant evaluation format for benchmarking Large Language Models (LLMs), offering a scalable and cost-effective means to assess everything from mathematical reasoning to multilingual nuance ( 6 ; 12 ; 15 ; 23 ; 29 ) . However, as models approach the performance ceiling on widely used benchmarks, a fundamental structural limitation has become increasingly apparent ( 1 ; 2 ) . In traditional settings where the number of options is small ( N\in\{4,5\} ), high accuracy often reflects a deceptive mixture of genuine knowledge and short-cut strategies ( 18 ; 5 ; 14 ; 3 ) . Within such narrow search spaces, models can flourish by using partial elimination, distributional guessing, or by exploiting incidental formatting cues, which make it nearly impossible to discern if a model truly knows the answer or is simply navigating a limited field of distractors ( 27 ; 17 ) . Although open-ended evaluations can avoid some pitfalls of low-option MCQA, they impose substantial scoring burdens. As a scalable alternative, we introduce a massive-option environment with N=100 ( 32 ) , which effectively turns evaluation into a high-precision selection problem under dense candidate interference. Scaling to one hundred options sharply reduces chance performance from 25% to 1%, making near-ceiling results at low N less informative and exposing competence that is otherwise obscured by narrow search spaces. We demonstrate our approach on a Korean orthography error identification task. By requiring models to distinguish a single error from highly similar correct candidates, this task serves as a rigorous test of fine-grained linguistic knowledge. Our investigation finds a startling divergence in model robustness masked by conventional benchmarks. While frontier systems ( Gemini family) maintain high reliability, other models like HyperCLOVAX-Think and EXAONE-4.0 , which exhibit near-perfect performance at low option counts, suffer catastrophic degradation at N=100 , with accuracy dropping by roughly 70 percentage points in the most challenging environments. Crucially, this collapse is not merely a failure of retrieval but often marks a regression to primitive decision heuristics where models abandon content analysis in favor of positional shortcuts such as selecting the first few options ( 16 ; 25 ; 17 ) . To isolate the mechanisms of model failure, we maintain a fixed set of target errors while systematically varying the difficulty of the distractor pool and employing repeated permutations of option order. Our study offers three primary contributions to the field of LLM evaluation. First, we demonstrate that scaling to one hundred options produces substantially stronger model differentiation, revealing performance gaps that remain invisible near the ceiling of conventional benchmarks. Second, through padding-based controls, we prove that the observed performance collapse at high N is driven by the complexity of candidate ranking rather than token length or long-context limitations alone. Third, we introduce a suite of quantitative tools for deep analysis, including the Bubble Index to measure low-option competence inflation, and positional diagnostics to quantify when models fall back to early-option selection under pressure. Ultimately, this work provides a practical methodology for evaluating true competence , the ability to deterministically identify the correct option amid heavy interference. More broadly, the results suggest that next-generation benchmarks should complement broad subject coverage with stress tests that probe reliability under dense, confusable alternatives, since this regime exposes failure modes that low-option settings often conceal. 2 Related Works 2.1 Evolution of MCQA: From Low-N to Massive-Option Multiple-Choice Question Answering (MCQA) is the standard for evaluating Large Language Models (LLMs) due to its scalability ( 6 ; 7 ) . In the Korean domain, benchmarks like HAE-RAE Bench ( 22 ) , KMMLU ( 21 ) , KoBALT ( 20 ) and KoBEST ( 10 ) have adopted the 4 - or 5 -option format to assess linguistic and cultural competence. However, recent studies warn that such low- N settings allow models to exploit probabilistic heuristics rather than demonstrating genuine reasoning ( 31 ; 26 ) . By scaling to N=100 , our work neutralizes these elimination-based shortcuts and forces a shift toward a more deterministic retrieval regime, addressing the latent fragility of standard benchmarks. 2.2 Robustness Against Information Interference The impact of distractors is a core challenge in LLM robustness. Even minimal irrelevant context can significantly degrade reasoning performance ( 8 ; 19 ) , while semantically relevant distractors can cause performance drops exceeding 45% ( 28 ) . While existing research primarily focuses on small-scale adversarial injections, our massive-option environment creates a cognitive stress test. This setup simulates a needle-in-a-haystack scenario ( 9 ) within the context window, testing the model’s ability to suppress interference from a high density of competing candidates. 2.3 Positional Bias and Decision Complexity LLMs exhibit well-documented positional biases, such as the Lost in the Middle phenomenon, where retrieval accuracy dips for information in the center of a sequence ( 16 ) . In MCQA, this manifests as Selection Bias , where models favor specific option IDs regardless of content ( 17 ; 18 ) . While calibration methods address these in low- N settings ( 30 ) , their efficacy in massive-option regimes is underexplored. Our study employs padding control experiments to decouple context length from option count, revealing that performance collapse at N=100 is driven by Positional Fallback strategies under high decision complexity, rather than simple token length limitations. 3 Experiments Category Sub-category Incorrect (Examples) Correct (Examples) Count Spacing Errors Particles (Auxiliary/Case) 기차 보다, 하늘 만큼 기차보다, 하늘만큼 4 Dependent Nouns 시간 \vee 만에, 들었을 \vee 뿐 시간 만에, 들었을 뿐 4 Predicates & Endings 떨어질 망정 떨어질망정 3 Lexicalized Compounds 못지 않게, 수 \vee 밖에 못지않게, 수밖에 4 Orthographic Errors Sai-siot & Compounds 뒷 처리, 싯 가 뒤처리, 시가 4 Phonological Confusions (Vowels, Morphologies) 왠 일, 금 새 , 희 안 한 웬일, 금세, 희한한 6 Standard Word Form Error 일일 히 , 함부 러 일일이, 함부로 5 Total 30