Which of These Best Describes Multiple Choice Evaluation with LLMs? A) Forced B) Flawed C) Fixable D) All of the Above

Nishant Balepur, Rachel Rudinger, Jordan Boyd-Graber · 2025

Read Which of These Best Describes Multiple Choice Evaluation with LLMs? A) Forced B) Flawed C) Fixable D) All of the Above by Nishant Balepur, Rachel Rudinger, Jordan Boyd-Graber (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-14127.txt.

Introduction

Nishant Balepur, Rachel Rudinger, and Jordan Boyd-Graber argue that multiple choice testing of language models needs reform, in a 2025 paper. The format is popular for its simplicity, but the authors say it is a poor stand-in for real use. It cannot test generation or subjective answers, it mismatches how people use the models, and it only partly probes knowledge. They favour generative formats, in which models build and explain answers, while staying easy to score.

Even where multiple choice fits, its data sets suffer leakage, unanswerable items, shortcuts, and saturation. For each flaw they borrow fixes from education research: rubrics for writing items, scoring that curbs guessing, and item response theory for building harder tests. Model errors, biases, and unfaithful explanations get the same treatment. The verdict is reform, not abandonment: the format can stay, but it should borrow the testing science that schools built over decades.

Catalog record

CREATORNishant Balepur, Rachel Rudinger, Jordan Boyd-Graber
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE51 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2502-14127

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Which of These Best Describes Multiple Choice Evaluation with LLMs? A) Forced B) Flawed C) Fixable D) All of the Above Nishant Balepur, Rachel Rudinger, Jordan Boyd-Graber Abstract Multiple choice question answering (MCQA) is popular for LLM evaluation due to its simplicity and human-like testing, but we argue for its reform. We first reveal flaws in MCQA’s format, as it struggles to: 1) test generation/subjectivity; 2) match LLM use cases; and 3) fully test knowledge. We instead advocate for generative formats based on human testing—where LLMs construct and explain answers—better capturing user needs and knowledge while remaining easy to score. We then show even when MCQA is a useful format, its datasets suffer from: leakage; unanswerability; shortcuts; and saturation. In each issue, we give fixes from education, like rubrics to guide MCQ writing; scoring methods to bridle guessing; and Item Response Theory to build harder MCQs. Lastly, we discuss LLM errors in MCQA—robustness, biases, and unfaithful explanations—showing how our prior solutions better measure or address these issues. While we do not need to desert MCQA, we encourage more efforts in refining the task based on educational testing, advancing evaluations. 1 Questioning Multiple Choice Questions Multiple choice question answering (MCQA) is the standard for large language model (LLM) evaluations, prized for simplicity and similarity to human testing Robinson and Wingate (2023) . When designing new benchmarks, MCQA seems easy to implement Guo et al. (2023) , and when selecting new LLMs to use, MCQA leaderboards inform our decisions Fourrier et al. (2024) . If you want to build a popular dataset, prove your LLM is smart, or even publish a position paper, it is hard to avoid MCQA. Standardized testing groups have long explored ways to better use MCQA for student testing Angoff (1971) . But despite years of use in nlp Turney et al. (2003) , few have asked: 1) should MCQA be a standard model evaluation format; and 2) are its datasets well-designed? This position paper argues: Evaluating LLMs with MCQA has flaws in both its inherent format and dataset construction. We state our position in three points (Fig 1 ). Figure 1: Overview of this paper. We show many problems in formats ( section 3 ), datasets ( section 5 ), and LLMs ( section 6 ) when using MCQA. Along the way, we propose solutions and ideas for future work, drawing from educational testing. We first argue MCQA is not an ideal standardized format for LLM evaluations, showing its goal of “pick the best answer” cannot optimally test generation or subjectivity ( section 3.1 ), misaligns with LLM use cases ( section 3.2 ), and poorly tests knowledge ( section 3.3 ). Drawing from education, we advocate two generative improvements to MCQA’s format for future exploration: 1) providing short, constructed-response answers without using choices ( section 4.1 ); and 2) evaluating explanations for model answers ( section 4.2 ). These formats capture generation or subjectivity, match LLM use cases, and improve knowledge testing, all while mostly preserving MCQA’s simple scoring. Next, we argue even when MCQA is a useful format, its datasets suffer from: dataset leakage ( section 5.1 ), unanswerable MCQs ( section 5.2 ), shortcuts ( section 5.3 ), and saturation ( section 5.4 ), degrading MCQA’s utility. To enhance nlp dataset design, we offer solutions for each issue based on best practices in human testing, like rubrics to flag MCQ errors ( section 5.2 ), metrics to curb guessing from shortcuts ( section 5.3.1 ), and Item Response Theory Baker (2001) to cull shoddy MCQs and make the ones left more challenging ( section 5.4.1 ). Lastly, we show many errors of LLMs in MCQA directly relate to MCQA’s flaws ( section 6 ). These issues, like brittleness to perturbations ( section 6.1 ), bias toward certain options, cultures, and languages ( section 6.2 ), and generating unfaithful explanations ( section 6.3 ), can all be better measured or addressed with our proposed improvements to MCQA’s format and datasets. Many promising improvements to MCQA draw from education, a field dedicated to effective assessment Haladyna et al. (2002) , but these practices are rarely used in nlp . Adopting them demands more effort—MCQA is popular as it seems simple—but this effort is worth it to improve evaluations. To encourage researchers to take on these challenges, we conclude with guidelines for designing meaningful evaluations whether or not you use MCQA ( section 7 ). 2 Background: A Brief History of MCQA A multiple-choice question (MCQ) is a question q and set of choices \mathcal{C} . 1 1 1 Some choices can link to many other choices (e.g., “All of the above”), but these are discouraged Haladyna et al. (2002) . One choice a\in\mathcal{C} is the gold answer, while others are plausible-sounding but incorrect distractors \mathcal{D}=\mathcal{C}\setminus\{a\} meant to test misunderstandings. 2 2 2 Extractive qa (e.g., SQuAD) and classification (e.g., NLI) can have a finite set of choices, but are fixed (i.e., labels) or have non-misleading distractors, so such tasks are not MCQA. MCQA’s simple goal—picking the best answer a —is popular for LLM evaluation, but it has flaws. Before naming them, we first review its history in human testing ( section 2.1 ) and nlp ( section 2.2 ). 2.1 Why MCQA is the Standard for Humans The MCQA format originated in 1914 with Frederick Kelley’s Kansas Silent Reading Test Kelly (1916) , proposed as an efficient measure of student reading comprehension Monroe (1917) . Soon after, MCQA was attempted at scale, notably with Robert Yerkes’ Army Alpha and Beta tests Yerkes (1918) in 1917 to assess U.S. Army intelligence. An initial bottleneck in MCQA was the manual effort needed for scoring, which researchers like Benjamin Wood and Reynold Johnson tackled by designing automated grading systems Brennan and Clark (1971) ; Wood and Johnson (2001) with ibm . Automatic scoring eventually enabled MCQA’s popularity in primary/secondary education Butler (2018) , college admissions Daneman and Hannon (2001) , language proficiency Jamieson et al. (2000) , and even common tasks like driver’s permit exams Beanland et al. (2013) or employee training Puhakainen and Siponen (2010) . Parallel to this, education researchers began exploring the best practices for writing high-quality MCQs Morrison and Free (2001) ; Campbell (2011) , authoring distractors Pho et al. (2015) ; Gierl et al. (2017) , and designing test settings Rakes (2008) ; Shute (2013) . Despite MCQA’s simplicity and popularity, organizations still critically assess its use in standardized testing. In the United States, the SAT removes unsound MCQ types, 3 3 3 https://blog.prepscholar.com/sat-analogies-and-comparisons-why-removed-what-replaced-them and France’s Baccalauréat uses long essay tasks over MCQA. 4 4 4 https://www.education.gouv.fr/reussir-au-lycee/le-baccalaureat-general-10457 We argue LLM evaluation needs similar scrutiny and should draw from education to refine MCQA’s format and data. 2.2 How MCQA Became Popular for LLMs nlp first used MCQs from human exams; solving these with models that used external sources was considered part of an “ ai grand challenge” Reddy (1988) , as it required semantic Turney et al. (2003) ; Veale (2004) and factual understanding Strickland (2013) ; Clark et al. (2013) . Other early MCQs from Winograd Levesque et al. (2012) or COPA Roemmele et al. (2011) tested commonsense reasoning over events and ambiguity in

Read the complete text