ErrEval: Error-Aware Evaluation for Question Generation through Explicit Diagnostics
Weiping Fu, Bifan Wei, Jingyi Hao, Yushun Zhang, Jian Zhang, Jiaxin Wang, Bo Li, Yu He, Lingling Zhang, Jun Liu · 2026
Read ErrEval: Error-Aware Evaluation for Question Generation through Explicit Diagnostics by Weiping Fu, Bifan Wei, Jingyi Hao, Yushun Zhang, Jian Zhang, Jiaxin Wang, Bo Li, Yu He, Lingling Zhang, Jun Liu (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-10406.txt.
Introduction
Weiping Fu and co authors grade quiz items by their flaws, not by one score. Auto quiz makers can write items with deep flaws. A fact may be false. The marked answer may not fit the ask. Past grade tools give one whole score. That can hide a flaw and rate a bad item high.
The team built ErrEval, a grade tool that names flaws. It lists each flaw it finds in plain tags. The score then rests on those tags. Labs can see why an item lost marks. The work aims for grades that match how a human pro would fault an item.
Catalog record
| CREATOR | Weiping Fu, Bifan Wei, Jingyi Hao, Yushun Zhang, Jian Zhang, Jiaxin Wang, Bo Li, Yu He, Lingling Zhang, Jun Liu |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 50 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2601-10406
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
ErrEval: Error-Aware Evaluation for Question Generation through Explicit Diagnostics Weiping Fu, Bifan Wei, Jingyi Hao, Yushun Zhang, Jian Zhang, Jiaxin Wang, Bo Li, Yu He, Lingling Zhang, Jun Liu Abstract Automatic Question Generation (QG) often produces outputs with critical defects, such as factual hallucinations and answer mismatches. However, existing evaluation methods, including LLM-based evaluators, mainly adopt a black-box and holistic paradigm without explicit error modeling, leading to the neglect of such defects and overestimation of question quality. To address this issue, we propose ErrEval , a flexible and Err or-aware Eval uation framework that enhances QG evaluation through explicit error diagnostics. Specifically, ErrEval reformulates evaluation as a two-stage process of error diagnosis followed by informed scoring. At the first stage, a lightweight plug-and-play Error Identifier detects and categorizes common errors across structural, linguistic, and content-related aspects. These diagnostic signals are then incorporated as explicit evidence to guide LLM evaluators toward more fine-grained and grounded judgments. Extensive experiments on three benchmarks demonstrate the effectiveness of ErrEval, showing that incorporating explicit diagnostics improves alignment with human judgments. Further analyses confirm that ErrEval effectively mitigates the overestimation of low-quality questions 1 1 1 Codes and resources are available at: https://github.com/WeipingFu/ErrEval . 1 Introduction Question Generation (QG) is a fundamental task in Natural Language Generation (NLG) 8 , aiming to generate questions given a context and often with a target answer. Reliable evaluation of generated questions is essential for ensuring the quality of QG systems deployed in downstream applications such as question answering 15 , dialogue systems 32 , and educational assessments 6 . Traditional QG evaluation methods, including similarity-based metrics (e.g., BLEU 19 , BERTScore 33 ) and generation-based approaches (e.g., BARTScore 31 , GPTScore 4 ), provide efficient but coarse assessments, offering limited interpretability and weak support for fine-grained, multi-dimensional evaluation. In response to these limitations, large language models (LLMs), such as GPT-4 18 and LLaMA 24 , have recently emerged as powerful evaluators that support multi-dimensional evaluation, generate natural language explanations, and exhibit improved alignment with human judgments 12 ; 25 . Figure 1: Comparison between vanilla CoT and error-aware evaluation using LLMs, where the former overestimates a flawed question and the latter aligns more closely with human judgment. Despite these advancements, current LLM-based evaluators mainly follow a black-box and holistic evaluation paradigm, which maps generated questions directly to scalar ratings without modeling the underlying errors that lead to low-quality outputs. As a result, questions with critical defects are often overestimated. A natural way to address this limitation is to introduce error modeling into the evaluation process, which prior studies have shown to improve evaluation reliability 29 . Following this insight, we explore augmenting LLM-based QG evaluation with explicit error diagnostics. As illustrated in Figure 1 , LLM evaluators using a vanilla Chain-of-Thought (CoT) prompt 28 overrate a flawed question (a), but revise their scores toward the human judgment when explicit error information is provided (b). To substantiate this observation, we conduct a pilot study on 300 samples from QGEval 5 and find that providing LLM evaluator with human-labeled error information improves alignment with human judgments by an average of 56.6% across evaluation dimensions (Figure 2 ). Taken together, these findings indicate that explicit error modeling provides a useful diagnostic perspective for QG evaluation, enabling more fine-grained and calibrated judgments. Figure 2: Pilot experiment - Pearson correlation coefficients (%) between model scores and human scores. Flu.: Fluency; Clar.: Clarity; Conc.: Conciseness; Rel.: Relevance; Cons.: Consistency; Ans.: Answerability; AnsC.: Answer Consistency; Avg.: Average. To this end, we propose an error-aware evaluation framework named ErrEval that reformulates LLM-based QG evaluation by introducing explicit error diagnostics as an intermediate step. Specifically, we first design an error taxonomy which comprises 11 error types spanning structural, linguistic, and content-related aspects (Table 1 ), inspired by the Multidimensional Quality Metrics (MQM) framework in machine translation 3 . Each error type is aligned with the evaluation dimensions it directly affects (Appendix A ), enabling principled use of error diagnostics during evaluation. Based on the proposed error taxonomy, we develop a lightweight Error Identifier using an iterative refinement strategy to support accurate error identification. The predicted errors are finally converted into explicit diagnostic signals and incorporated into the evaluation process, guiding LLM evaluators to focus on dimension-relevant issues and produce more accurate judgments. We validate ErrEval through extensive experiments on three benchmarks and four LLM evaluators, spanning different evaluation settings and covering both open-source and closed-source models (Sec. 4.1 ). The results show that ErrEval improves the alignment between model and human judgments across various dimensions (Sec. 4.2 ). Further analyses indicate that accurate error identification contributes to error-aware evaluation and that incorporating explicit error diagnostics reduces the tendency to overestimate low-quality questions (Sec. 4.3 ). To summarize, our main contributions are threefold: • We propose ErrEval , an error-aware framework that enhances LLM-based evaluation through explicit error diagnosis. To the best of our knowledge, this is the first work to adopt such a diagnostic paradigm in the context of question generation. • To support error-aware evaluation in practice, we design an error taxonomy with 11 error types and develop a lightweight plug-and-play Error Identifier to provide diagnostic signals for LLM evaluators. • Extensive experiments on three benchmarks show that ErrEval consistently improves alignment with human judgments across different LLM evaluators and evaluation settings. Figure 3: Framework of ErrEval. Given a passage ( p ), answer ( a ), generated question ( q ), and evaluation criteria ( c ), ErrEval performs evaluation with explicit error diagnostics. An iteratively trained Error Identifier (EI) detects error types ( el ), which are organized as diagnostic information ( err ) to guide dimension-specific scoring by LLM evaluators. The EI functions as a lightweight and plug-and-play module. 2 Problem Formulation Our goal is to evaluate the quality of generated questions across multiple dimensions (e.g., Answerability ). We formalize the LLM-based evaluation paradigm as a dimension-specific scoring function: (s,r)=F(p,a,q,c,err) (1) where F is the evaluation function, p , a , and q represent the source passage, the reference answer, and the generated question, respectively. c denotes the evaluation criteria (i.e., a specific dimension with its scoring criteria). The optional input err provides error diagnosis information, which guides the evaluator toward a more focused and interpretable assessment. The output includes a score s for the specific dimension and a reason r that explains why the scoring decision is made. The scoring scales vary across evaluation settings (e.g., Likert-scale ratings, binary classification). In parallel, we formalize the error identification task as a multi-label classification problem, as a single question may contain multiple types of errors. Let \mathcal{L}=\{l_{1},l_{2},...,l_{K}\} denote the predefined set of K error types, including a special label No Error . Given a triplet (p,a,q) , the task is to predict which error types apply to the generated question. The model learns a function: el=f(p,a,q) (2) where f is a multi-label classifier parameterized by a neural network, and el\in\{0,1\}^{K} is a binary vector indicating the presence (1) or absence (0) of each error type in \mathcal{L} . These two components are tightly coupled in our proposed framework: the predicted error types el are used to form the diagnosis information err to guide evaluators for