From Understanding to Generation: An Efficient Shortcut for Evaluating Language Models
Viktor Hangya, Fabian Küch, Darina Gold · 2025
Read From Understanding to Generation: An Efficient Shortcut for Evaluating Language Models by Viktor Hangya, Fabian Küch, Darina Gold (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-03592.txt.
Introduction
Viktor Hangya, Fabian Kuch, and Darina Gold shortcut the evaluation of language models in a 2025 paper. Checking a model during training is essential but slow when tasks need full generated answers. Choosing between fixed options is far cheaper, yet generation is where key skills like coding show.
Their fix rewrites generation tasks as choice tasks that preserve the skill being tested. Eight models of varied size were scored both ways across four abilities. Scores in the two formats tracked each other strongly. The shortcut cut average evaluation time by more than thirty-five fold. Math reasoning, code, factual recall, and reading all held up under conversion. Trainers can now monitor real capabilities frequently instead of rationing expensive test runs.
Catalog record
| CREATOR | Viktor Hangya, Fabian Küch, Darina Gold |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 46 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2506-03592
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
From Understanding to Generation: An Efficient Shortcut for Evaluating Language Models Viktor Hangya, Fabian Küch, Darina Gold Abstract Iterative evaluation of LLMs during training is essential to ensure expected capability development, but can be time- and compute-intensive. While NLU tasks, where the model selects from fixed answer choices, are cheap to evaluate, essential capabilities like reasoning and code generation rely on the more time-consuming NLG (token-by-token generation) format. In this work, our aim is to decrease the computational burden of NLG benchmarks in order to enable monitoring crucial LLM capabilities during model training. We reformulate generative tasks into computationally cheaper NLU alternatives. We test the performance correlation between the original and reformulated tasks using 8 LMs of various sizes and 4 capabilities: mathematical reasoning, code generation, factual knowledge and reading comprehension. Our results show a strong correlation between task formats, supporting capability assessment via cheaper alternatives and achieving over 35× average reduction in evaluation time. We plan to publish our benchmark adaptions. 1 Introduction The increasing adoption of large language models (LLMs) has brought a substantial rise in computational demands—not only for training, but also for evaluating the performance of new models. Numerous works have explored methods for selecting the best quality training datasets and hyper-parameters based on smaller proxy models, evaluating them multiple times throughout training Li et al. 2024 ; Grattafiori et al. 2024 ; Magnusson et al. 2025 . However, due to model capability and compute limitations they rely on a few benchmarks which support measuring a limited set of model capabilities Penedo et al. 2024 ; Gu et al. 2025 . Evaluation benchmarks can be grouped into two types: 1) Natural Language Generation (NLG) which requires auto-regressive sampling from the model, and 2) Natural Language Understanding (NLU) which involves calculating the probability of given answer options Guo et al. 2023 ; Biderman et al. 2024 . Assessing a model’s NLG abilities is costly, especially when done repeatedly during model training and selection, and thus is often neglected Zhou et al. 2022 . Yet, many key capabilities, e.g. reasoning and code generation, are only measurable via NLG benchmarks. Step <Q> := What is gravity? Step output Final output LL 1. <Q> Downward force on objects. \rightarrow log(0.9) Log-likelihood of correct answer: log(0.9) MC 1. <Q> Downward force on objects. \rightarrow log(0.9) Selected answer index: 1. 2. <Q> What makes the sun rise. \rightarrow log(0.1) NLG 1. <Q> \rightarrow Downward Generated answer: Downward force on objects. 2. <Q> Downward \rightarrow force 3. <Q> Downward force \rightarrow on 4. <Q> Downward force on \rightarrow objects. 5. <Q> Downward force on objects. \rightarrow [EOS] Table 1: Comparison of answers to the same question in loglikelihood (LL), multiple-choice (MC), and natural language generation (NLG) settings. LL scores the correct answer to the question (Q), MC selects the most probable answer option, while NLG generates the answer token-by-token. The columns step indicates the number of forward model passes needed for the answer calculation. The example and outputs (e.g., log values) are fictional and serve solely to illustrate the NLU–NLG computation differences. Here, we argue for the importance of monitoring the trajectory of LLM capabilities throughout the training process, which is currently reliant on expensive NLG benchmarks. Our goal is to reduce compute requirements for such benchmarks in order to enable a more frequent use. Towards this end, we propose reformulating expensive NLG benchmarks as less-resource intensive NLU tasks. Specifically, we reformulate it to multiple choice (MC) tasks, where the model has to select the correct answer among answer options, and log-likelihood (LL) tasks where the model calculates the LL of the correct answer. The cost-reduction of MC and LL over then generative task is shown in Table 1 . The generative setup completes the response token-by-token. The MC setup, however, evaluates the model’s ability to select the appropriate response from a set of alternatives based on their likelihoods, while the LL approach scores only the correct answer. Despite differing formats, all methods aim to assess the model’s response to the same underlying task. Since many NLG benchmarks do not contain incorrect answer options to questions, we create such options using either LLMs or simply picking answers of other samples in a given benchmark randomly. In contrast, LL is simpler to apply, as it evaluates the likelihood assigned by the model to a given correct answer without the need of an overhead, i.e., the computation of alternative answers. As such, it offers further reductions in evaluation costs, however, at the expense of evaluation depth, since high-probability incorrect answers are not considered. MC and LL evaluation allows quick comparison of model checkpoints and low-cost training monitoring. Importantly, our goal is not to replace the NLG format, but to offer a cheaper alternative during model training. To test this hypothesis, we conduct a study involving a diverse set of benchmarks representing various domains. We analyze the correlation of 8 open source model performances between the NLG and NLU task formulations, as well as the correlation of model rankings. To our knowledge, this is the first study to systematically investigate this relationship. We find a strong correlation, indicating the ability to monitor the performance trajectory when training a single model, but also the ability to compare the model in question to baseline LLMs using only the NLU formulated benchmarks. Additionally, we conduct ablation studies, e.g., pairing NLG and NLU benchmarks of the same domain but different sources, finding promising results in these settings, too. On top of the correlation experiments, we show that besides monitoring the model development trajectory using NLU tasks, it is also possible to predict the NLG performance of a model by fitting a linear regressor and only occasional NLG evaluations. This optional step balances precise NLG performance monitoring and compute resource needs. Our main contributions are threefold: • we propose and apply a methodology for reformulating NLG tasks as NLU tasks for more efficient benchmark evaluation and broader capability assessment during model selection and training • we show strong correlations between task formulations and analyze various aspects; a very important one being that by reformulating NLG tasks we achieve a compute time reduction of 35x on average, reducing runtime from nearly 2 hours to under a minute in the most extreme case. • we plan to publicly release the adapted benchmarks and the created samples. 2 Related Work This section is divided into three subsections, each addressing key components of this work. In Section 2.1 , we discuss evaluation costs and efforts to reduce them. In Section 2.2 , we review studies on the relation between NLG and NLU evaluation. In Section 2.3 , we present knowledge categories used to study LLM generalizability. 2.1 Evaluation costs Various benchmarks, including NLG with many samples, have been proposed to evaluate LLMs , but evaluating large models on them can be costly. For example, Liang et al. 2023 introduce the HELM benchmark which could cost over 4K GPU hours to evaluate a single LLM , not to mention monitoring the performance over the course of the training process Biderman et al. 2023 ; Liu et al. 2024 . Recent studies have proposed various strategies to address reducing the evaluation costs. While Perlitz et al. 2024