Models Know Models Best: Evaluation via Model-Preferred Formats

Joonhak Lee, Sungmok Jung, Jongyeon Park, Jaejin Lee · 2026

Read Models Know Models Best: Evaluation via Model-Preferred Formats by Joonhak Lee, Sungmok Jung, Jongyeon Park, Jaejin Lee (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-22699.txt.

Introduction

Joonhak Lee, Sungmok Jung, Jongyeon Park, and Jaejin Lee show that models grade best in the format they themselves prefer. Multiple-choice tests come in two forms: pick a symbol, or complete a sentence. Their 2026 preprint finds that scores swing by form in a systematic way.

Sentence completion suits likelihood scoring; symbol picks suit direct comparison. Matching the form to the task gives a truer grade. The lesson is simple and wide: test design bends results. Evaluation teams can use it at once. The work cleans up a common source of noise.

Catalog record

CREATORJoonhak Lee, Sungmok Jung, Jongyeon Park, Jaejin Lee
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE78 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2601-22699

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Models Know Models Best: Evaluation via Model-Preferred Formats Joonhak Lee, Sungmok Jung, Jongyeon Park, Jaejin Lee Abstract Performance of Large Language Models (LLMs) on multiple-choice tasks differs markedly between symbol-based and cloze-style evaluation formats. The observed discrepancies are systematically attributable to task characteristics: natural language continuation benefits from likelihood scoring, whereas explicit comparison is better suited to symbol-based selection. These trends are consistent across various decoder-based LLMs, indicating model-agnostic effects. To address these inconsistencies, a dynamic format-alignment strategy is introduced that employs a lightweight classifier trained on latent model-preference signals. In contrast to human-designed heuristics, which often degrade performance, this approach uses model-generated signals to determine the optimal format for each problem instance. The proposed method achieves substantial and consistent improvements in zero-shot accuracy across reasoning and knowledge benchmarks, better revealing the models’ latent capabilities. † † affiliationnotice: Equal contribution 1 Introduction Multiple-choice question answering (MCQA) is one of the most widely used methods for evaluating and comparing large language models (LLMs). Its clear evaluation protocol and alignment with human problem-solving processes have contributed to its status as a de facto standard in LLM evaluation ( 18 ) . However, an increasing number of studies raise concerns about whether MCQA accurately reflects LLMs’ true capabilities. Researches have shown that model performance can be highly sensitive to small changes in input ( 24 ; 33 ; 20 ) , and that models can attain misleadingly high accuracy even when the question is partially or entirely removed ( 3 ; 9 ; 25 ; 6 ) . These findings cast doubt on the reliability of MCQA as an accurate measure of reasoning ability. In addition to these concerns, the MCQA framework accepts various evaluation formats, a relatively underexplored aspect. Typical formats include a symbol-based approach, in which the model selects from lettered options (e.g., A, B, C, and D), and a cloze-style approach, in which evaluation is based on the model’s likelihood assigned directly to the answer options. Recent studies have highlighted significant performance gaps between these two formats ( 21 ; 11 ) , yet the reasons behind these discrepancies and their patterns are not well understood. In this study, we conduct a thorough empirical comparison of symbol-based and cloze-style MCQA methods across a variety of models and benchmarks. Our findings indicate that the performance differences, which depend on how the task is formulated, are closely linked to the characteristics of the task. As a result, we observe stable and consistent preferences for certain formats across different models. Specifically, tasks that require completing natural language continuations show significant performance drops when using symbol-based formats compared to cloze-style ones. Although providing demonstration examples can help reduce these discrepancies, they do not completely eliminate the challenges that models encounter when evaluation methods are not aligned with the nature of the task. Driven by these insights, we explore whether choosing the evaluation format at the problem instance level can enhance model performance. We consider two approaches: one that employs human-designed linguistic heuristics and the other that derives format preferences directly from model behavior. Our findings reveal that human-aligned linguistic criteria do not consistently lead to improvements, whereas a classifier trained on signals of model preferences achieves significant gains. These improvements are particularly noticeable in completion-oriented tasks, where format mismatches are most pronounced. Finally, we demonstrate that preferences for various formats, inferred from a subset of language models, can be generalized to unseen models through majority voting. This indicates that decoder-based language models exhibit task-dependent biases in their evaluation formats, rather than unique, model-specific behaviors. By using these shared preferences, we can achieve a more accurate and insightful assessment of language model capabilities and emphasize the importance of aligning evaluation formats with the inductive biases of contemporary language models. The contributions of this paper are summarized as follows: • We establish that performance discrepancies arising from format mismatch in MCQA are not stochastic but vary systematically according to the structural and linguistic properties of the task. We demonstrate that these performance patterns are model-agnostic, emerging consistently across a diverse suite of decoder-based LLMs. • We provide evidence that the task characteristics governing format preference are implicitly encoded within the model’s latent representations. We show that by explicitly eliciting these latent preferences, it is possible to recover significant performance otherwise lost to suboptimal formatting. • We propose a format-alignment strategy that leverages these latent signals to dynamically select the optimal evaluation format. The proposed approach yields substantial and consistent gains in zero-shot accuracy across a broad spectrum of reasoning and knowledge-intensive benchmarks, effectively bridging the gap between model capability and task characteristics. 2 Related Work 2.1 MCQA Evaluation Formats: Symbol and Cloze MCQA has become a widely accepted standard evaluation framework for LLMs ( 1 ; 28 ; 14 ; 10 ) . In the MCQA formats, model outputs are limited to a predefined set of answer options, and predictions are assessed by comparing the likelihoods assigned to each option. Compared to open-ended generation, this constrained setup provides a precise, reproducible evaluation criterion, which contributes to the widespread adoption of MCQA for benchmarking modern LLMs ( 18 ) . MCQA is typically presented in two formats: symbol-based and cloze-style ( 2 ) . In the symbol-based format, the prompt includes a question followed by a list of candidate answers. The model selects an option by generating the symbol (e.g., A, B, C, or D) with the highest likelihood. On the other hand, the cloze-style format treats the question as a prompt and evaluates each answer option by calculating its likelihood when appended to the prompt. The option with the highest likelihood is chosen as the model’s prediction. Because answer options in the cloze format often vary in length, scores are usually normalized based on the number of characters ( 29 ; 26 ) or tokens ( 5 ) to reduce any biases related to the length. 2.2 Sensitivity to MCQA Evaluation Formats Although it may seem straightforward, MCQA is highly sensitive to formatting decisions. Previous studies show that model performance can differ significantly with minor changes, such as rephrasing prompts ( 24 ) , altering the order of answer options ( 33 ; 20 ) , or modifying a few-shot demonstration ( 17 ) . To address this inconsistency, several strategies have been proposed to aggregate predictions across different permutations during inference, such as using majority voting ( 35 ; 30 ; 34 ) . Additionally, methods to enhance robustness during training include techniques like knowledge distillation ( 15 ) and structural pruning ( 7 ) . More recent studies have shown that a single model can exhibit significantly different performance on the same MCQA task, depending on whether a symbol-based or cloze-style format is used. These differences prompt important questions about whether models are truly reasoning about the task content, especially since identical underlying questions can yield different outcomes simply due to changes in the evaluation format. Existing studies provide limited insight into the mechanisms driving performance gaps across different formats. Simply using the format that performs best for a specific benchmark, as noted by 11 , only reports the phenomenon without explaining it. Likewise, attributing the effectiveness of symbol-based formats to enhanced symbol-option binding

Read the complete text