MinosEval: Distinguishing Factoid and Non-Factoid for Tailored Open-Ended QA Evaluation with LLMs
Yongqi Fan, Yating Wang, Guandong Wang · 2025
Read MinosEval: Distinguishing Factoid and Non-Factoid for Tailored Open-Ended QA Evaluation with LLMs by Yongqi Fan, Yating Wang, Guandong Wang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-15215.txt.
Introduction
Yongqi Fan and co-authors grade open ended answers more fairly in this 2025 paper. Scoring free form answers from language models is hard: old text overlap metrics miss meaning, and judge models that score everything the same way ignore what kind of question was asked.
Their method, MinosEval, first sorts questions into factoid and non-factoid. Factoid answers get scored against adaptive key points drawn from a reference. Open questions get ranked as a list, with the ranking tuned to each instance. On several datasets, including home built ones with many candidate answers, the method matched human judgments better than rivals and explained its verdicts more clearly. The authors pitch it as a tailored judge for open ended question answering.
Catalog record
| CREATOR | Yongqi Fan, Yating Wang, Guandong Wang |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 54 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2506-15215
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
MinosEval: Distinguishing Factoid and Non-Factoid for Tailored Open-Ended QA Evaluation with LLMs Thanks: Corresponding authors. Yongqi Fan, Yating Wang, Guandong Wang Abstract Open-ended question answering (QA) is a key task for evaluating the capabilities of large language models (LLMs). Compared to closed-ended QA, it demands longer answer statements, more nuanced reasoning processes, and diverse expressions, making refined and interpretable automatic evaluation both crucial and challenging. Traditional metrics like ROUGE and BERTScore struggle to capture semantic similarities due to different patterns between model responses and reference answers. Current LLM-based evaluation approaches, such as pairwise or listwise comparisons of candidate answers, lack intuitive interpretability. While pointwise scoring of each response provides some descriptions, it fails to adapt across different question contents. Most notably, existing methods overlook the distinction between factoid and non-factoid questions. To address these challenges, we propose MinosEval , a novel evaluation method that first distinguishes open-ended questions and then ranks candidate answers using different evaluation strategies. For factoid questions, it applies an adaptive key-point scoring strategy, while for non-factoid questions, it uses an instance-aware listwise ranking strategy. Experiments on multiple open-ended QA datasets, including self-built ones with more candidate responses to complement community resources, show that MinosEval better aligns with human annotations and offers more interpretable results. Figure 1: Typical Samples of Factoid and Non-Factoid Open-Ended QA. 1 Introduction Open-ended question answering (QA) is a fundamental task type in comprehensive large language modeling (LLM) evaluation benchmarks and plateforms Srivastava et al. 2023 ; Contributors 2023 ; Liu et al. 2024b ; Myrzakhan et al. 2024 ; Liu et al. 2024a . Unlike closed-ended questions, which generally elicit brief, predefined responses (e.g., multiple-choice question Wang et al. 2024b , true/false question Luo et al. 2023 , and close question), open-ended questions Kantharaj et al. 2022 ; Tao et al. 2024 ; Lin et al. 2024 necessitate the generation of longer, more detailed answers that require complex reasoning, nuanced understanding, and diverse modes of expression. Consequently, the evaluation of open-ended QA has become a significant area of research, with an increasing focus on developing methods and benchmarks Amirizaniani et al. 2024 ; Yang et al. 2024b ; Wang et al. 2024c that not only measure the quality of the model responses but also provide interpretability and alignment with human annotation. Traditional evaluation metrics for free-text responses, such as BLEU Papineni et al. 2002 , ROUGE Lin 2004 , and BERTScore Zhang et al. , focus on lexical overlap or semantic similarity. While useful for tasks with clear answers, they are less effective for open-ended QA, where responses vary in style and content. These metrics also struggle with complex open-ended answers, involving reasoning, creativity, and entailment. In recent years, evaluation methods have shifted toward leveraging LLMs for automatic evaluation. These methods typically involve employing pairwise comparisons for Elo Boubdir et al. 2023 ; Harang 2024 , listwise comparisons where LLMs rank responses based on relevance or quality, and pointwise approaches Liang et al. ; Vu et al. 2024 that score responses on predefined dimensions. Meanwhile, some studies conducted pairwise and pointwise supervised fine-tuning to obtain dedicated LLMs for evaluation Kim et al. 2024 ; Li et al. 2023a . While promising, these methods have notable limitations. Pairwise methods involve multiple comparisons, leading to O(n^{2}) complexity for n responses. Pointwise methods, though scoring on predefined criteria, e.g., “Fluency” and “Truthfulness”, often fail to be adapted to each specific question context. Additionally, pairwise and pointwise methods suffer from the tie-breaking problem. In addition, pairwise and listwise methods lack intuitive explanations, making their rankings difficult for human evaluators to interpret. More notably, these LLM-based approaches overlook the distinction between factoid and non-factoid questions in open-ended QA. Factoid Jiang et al. 2019 ; Gaikwad and Patil 2023 and non-factoid Lakshmi and Sindhu 2023 ; Bolotova-Baranova et al. 2023a questions exhibit a clear distinction. Factoid questions aim to elicit answers based on objective, real-world facts or entities, with a relatively fixed and clear scope, e.g., “ List the five highest mountains on Earth and the average elevation of their peaks ”. Of course, additional sentences can be included in the answer to ensure semantic coherence. In contrast, non-factoid questions offer more creative freedom, with the primary requirement being to meet the demands of the question without necessarily including critical, fact-based information, e.g., “ Please help me write homophone joke ”. By failing to account for this distinction, such a one-size-fits-all strategy undermines the precision and effectiveness of evaluation. In this paper, we propose MinosEval , a novel two-stage approach for evaluating open-ended QA and providing corresponding interpretive information. It distinguishes between factoid and non-factoid questions based on their semantic and content differences, employing tailored evaluation strategies for each type of question. For factoid questions, we use an adaptive key-point scoring strategy that extracts key factoid points from a given reference answer. It then compares how each response entails these key points using a natural language inference (NLI) model, inspired by Bohnet et al. 2022 . For non-factoid questions, we apply an instance-aware listwise ranking approach, generating five levels of silver answer instances to enhance the LLM’s performance in directly ranking these more creative and diverse responses. Table 1: Dataset statistics and description. The columns indicate the dataset name, source, language, number of reference answers, samples, and candidate model responses to be evaluated. Datasets Source Language # Res # Ref # Samples Factoid Non-factoid Total ANTIQUE_S5 ANTIQUE Hashemi et al. 2020 English 2 to 4 2 96 \times 5 404 \times 5 500 \times 5 TREC-DL-NF_S5 TREC-DL-NF Craswell et al. 2020 ; Craswell et al. 2021 English 2 to 4 1 28 \times 5 27 \times 5 55 \times 5 AlignBench_Minos AlignBench Liu et al. 2023b Chinese 6 1 299 384 683 GaokaoBench_Minos GaokaoBench Zhang et al. 2023 Chinese 6 1 158 247 405 Our approach tackles the challenges of existing methods in open-ended QA evaluation and offers several advantages. First, it distinguishes between different types of open-ended questions, enabling a more tailored evaluation strategy. It also provides clear ranking guidelines based on the characteristics of factoid and non-factoid questions, including key points and silver instances to enhance interpretability. Moreover, the guidelines are adaptive to the specific questions, and the entire process is fully automated, requiring no manual intervention. We conducted extensive experiments on four datasets. Following the approach of Yang et al. 2024b , we performed five sets of sampling to construct ANTIQUE_S5 and TREC-DL-NF_S5 from the publicly available open quality assurance datasets ANTIQUE Hashemi et al. 2020 and TREC-DL-NF Craswell et al. 2020 ; Craswell et al. 2021 . Additionally, we created two self-built datasets, AlignBench_Minos and GaokaoBench_Minos, which contain a larger number of candidate responses based on AlignBench Liu et al. 2023b and GaokaoBench Zhang et al. 2023 to supplement the resources of the research community. The results demonstrate that our method outperforms existing LLM-based evaluation approaches for open-ended QA. We also present case studies in which the key points for factoid questions and the silver answer instances for non-factoid questions provide a valuable interpretive foundation. We hope this work contributes to the effective evaluation of LLMs’ performance in open-ended QA and promotes further research within the LLM community. Our datasets, evaluation results, and code