How to Select Datapoints for Efficient Human Evaluation of NLG Models?

Vilém Zouhar, Peng Cui, Mrinmaya Sachan · 2025

Read How to Select Datapoints for Efficient Human Evaluation of NLG Models? by Vilém Zouhar, Peng Cui, Mrinmaya Sachan (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-18251.txt.

Introduction

Zouhar, Cui, and Sachan ask how to pick the best test items for human judging of text generation in a 2025 paper. Human review is the gold standard but pricey, so teams often sample randomly. Random picks can misrepresent true model quality.

The authors test selectors based on score variance, output diversity, and item response theory. All beat random choice at matching full-test verdicts. They then distill selection to use only source texts, before model outputs exist. Trials on translation and summarization tasks backed the approach. Roughly seven tenths of the test set sufficed for the same verdict. The work helps labs spend review budgets where they count. Better sampling sharpens model comparisons.

Catalog record

CREATORVilém Zouhar, Peng Cui, Mrinmaya Sachan
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE281 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2501-18251

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

How to Select Datapoints for Efficient Human Evaluation of NLG Models? Vilém Zouhar, Peng Cui, Mrinmaya Sachan Abstract Human evaluation is the gold standard for evaluating text generation models. However, it is expensive. In order to fit budgetary constraints, a random subset of the test data is often chosen in practice for human evaluation. However, randomly selected data may not accurately represent test performance, making this approach economically inefficient for model comparison. Thus, in this work, we develop and analyze a suite of selectors to get the most informative datapoints for human evaluation, taking the evaluation costs into account. We show that selectors based on variance in automated metric scores, diversity in model outputs, or Item Response Theory outperform random selection. We further develop an approach to distill these selectors to the scenario where the model outputs are not yet available. In particular, we introduce source-based estimators, which predict item usefulness for human evaluation just based on the source texts. We demonstrate the efficacy of our selectors in two common NLG tasks, machine translation and summarization, and show that only ∼ \sim 70% of the test data is needed to produce the same evaluation result as the entire data. \DeclareCaptionType algorithmCaption[Algorithm][List of algorithms] \DeclareCaptionType example[Example][List of examples] \DeclareCaptionType prompt[Prompt][List of prompts] \DeclareCaptionType myequation[Equation][List of equations] 0 0 footnotetext: We release the subset2evaluate package, trained models, and code for reproducing the results in this paper. 1 Introduction Robust model evaluation drives natural language generation (NLG) research. Despite improvements in automated evaluation metrics, human evaluation remains the gold standard in NLG ( Zhou et al., 2022 ; Freitag et al., 2023 ) . Distinguishing between high-quality models requires increasingly more expert human annotations to determine which model performs better. For example, WMT shared tasks ( Kocmi et al., 2024a , Conference on Machine Translation,) , annually evaluate dozens of state-of-the-art machine translation models. The dominant approach in human evaluation under budgetary constraints is to randomly select evaluation items in the test data, as shown by the survey of Ruan et al. (2024) . The random selection is clearly suboptimal because it can lead to selecting overlapping items while omitting items that impact the evaluation outcome (e.g. model ranking) more. Thus, in this paper, we develop approaches to select a subset of test items to reduce human-evaluation cost without sacrificing evaluation accuracy. In particular, we are interested in selecting a subset of test items that would lead to a human ranking of multiple NLG models similar to that on the entire test set. We frame this as a subset selection problem (dubbed output-based selection, Figure 1 ) where we are given a set of items \mathcal{X} and model outputs \mathcal{M} , and would like to select a subset \mathcal{Y}\subseteq\mathcal{X} such that the ranking of models with human evaluation on \mathcal{Y} is the same as on \mathcal{X} . Although automated metrics can be noisy and may not always align with human judgments, they still help identify test items that are informative for evaluation. We build multiple data selectors that leverage these metrics, such as prioritizing challenging items, items leading to diverse model outputs, or using Item Response Theory. However, in some evaluation scenarios, such as when organizing a large shared task, we may not yet know which models will be evaluated by our test set, or obtaining all model outputs on the whole \mathcal{X} may be computationally infeasible due to the size of \mathcal{X} . In this setting, called source-based selection ( Figure 3 ), the standard methods for output-based selection cannot be used. However, by distilling the output-based selection methods, we build predictors that only use the item input to predict the expected item difficulty or likelihood that it leads to diverse model outputs. We demonstrate the efficacy of our data selection approach with case studies on two typical natural language generation tasks: machine translation and summarization. Our key contributions include: • framing the task of informative subset selection for evaluation in two variants, • multiple evaluation subset selection methods, including cost- and document-aware selection, • selector distillation for source-based selection, • package subset2evaluate for budget-efficient test set construction for model evaluation. This paper is structured as follows. We formalize the problem in Section 2 , describe the output-based and source-based selectors in Sections 3 and 4 . We show our results on efficient machine translation evaluation in Section 5 and our results on evaluating summarization models in Section 6 . In particular, we find that in the annual WMT evaluation for machine translation models, our methods yield the same evaluation result (model ranking) as random sampling, but with only 70% of human annotations. Given the high cost of human evaluation, this is a non-trivial cost saving. 2 Problem Statement We are given a set of items \mathcal{X} and a set of models \mathcal{M} that we wish to rank according to \mathcal{X} . Here, each item x\in\mathcal{X} is an input text and m(x) is the output of the model m in the NLG task. For example, in machine translation, each x is the input in the source language and m(x) is the corresponding translation. Since |\mathcal{X}| is very large (exceeding the human evaluation budget B ), we seek a subset \mathcal{Y}\subseteq\mathcal{X} such that the ranking of models on \mathcal{Y} can be as close to that on \mathcal{X} as possible. An illustration of the problem is also shown in Figure 1 . Human evaluation set selection. In order to obtain a subset \mathcal{Y}\subseteq\mathcal{X} to be human-evaluated, we quantify the cost of human evaluations on \mathcal{Y} with \mathrm{Cost}(\mathcal{Y}) and the usefulness of the subset for evaluation with \mathrm{Utility}(\mathcal{Y}) . In the ideal case, the utility of the set \mathcal{Y} indicates how close the human ranking of models on items y\in\mathcal{Y} is to the human ranking of models on whole \mathcal{X} . We frame this as a subset selection problem as follows: \displaystyle\underset{\mathcal{Y}\subseteq\mathcal{X}}{\arg\max}\,\,\mathrm{Utility}(\mathcal{Y})\quad\text{s.t.}\quad\mathrm{Cost}(\mathcal{Y})\leq B (1) In our work, we make two simplifications. First, we note that the cost of evaluating a set of items is the sum of costs for evaluating individual items, i.e. \mathrm{Cost}(\mathcal{Y})=\sum_{y\in\mathcal{Y}}\mathrm{Cost}(y) . This assumption is generally true if the human evaluation of items y is carried out one by one. Second, we assume that the utility of the set \mathcal{Y} is the sum of the utilities of items y\in\mathcal{Y} . Note that this is generally not true, as similar datapoints may not offer an extra marginal utility for model ranking. However, in our experiments, we find that this assumption leads to good empirical results. Moreover, our preliminary experiments on modeling diversity of datapoints did not lead to better performance (see methods without this assumption in Appendix A ). With the two assumptions, we rewrite Equation 1 using 0/1 indicator variables z_{x} , which becomes a 0-1 knapsack problem. maximize \displaystyle\sum_{x\in\mathcal{X}}z_{x}\cdot\mathrm{Utility}(x) subject to \displaystyle\sum_{x\in\mathcal{X}}z_{x}\cdot\mathrm{Cost}(x)\leq B \displaystyle\mathrm{and}\,\forall x\in\mathcal{X}:z_{x}\in\{0,1\} (2) This problem is generally NP-complete ( Karp et al., 1975 ) . However, for constant \mathrm{Cost}(.) and positive item utilities, we can find an optimal solution taking the items with the highest utilities until the budget B is reached. For a non-constant \mathrm{Cost}(.) , we find an approximate solution using integer linear programming ( Huangfu and Hall, 2018 ) . Next in Section 3 ,

Read the complete text