Consensus or Conflict? Fine-Grained Evaluation of Conflicting Answers in Question-Answering
Eviatar Nachshoni, Arie Cattan, Shmuel Amar, Ori Shapira, Ido Dagan · 2025
Read Consensus or Conflict? Fine-Grained Evaluation of Conflicting Answers in Question-Answering by Eviatar Nachshoni, Arie Cattan, Shmuel Amar, Ori Shapira, Ido Dagan (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-12355.txt.
Introduction
Eviatar Nachshoni and four co-authors present a 2025 paper on question answering when several valid answers clash. Large language models do well on standard questions, but real questions can have multiple right answers that conflict. Building test sets for such cases is slow and costly, and older sets often use fake data or simple yes-no questions. The team offers a cheaper way to build a realistic benchmark.
Their method reuses fact-checking data sets to create NATCONFQA, a benchmark with labels for which answer pairs clash. The task asks models both to list all valid answers and to name the conflicting pairs. Testing eight top models showed weak handling of conflicts and flawed habits for settling them. The authors present the benchmark as a tool for building more honest question-answering systems.
Catalog record
| CREATOR | Eviatar Nachshoni, Arie Cattan, Shmuel Amar, Ori Shapira, Ido Dagan |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 65 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2508-12355
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Consensus or Conflict? Fine-Grained Evaluation of Conflicting Answers in Question-Answering Eviatar Nachshoni, Arie Cattan, Shmuel Amar, Ori Shapira, Ido Dagan Abstract Large Language Models (LLMs) have demonstrated strong performance in question answering (QA) tasks. However, Multi-Answer Question Answering (MAQA), where a question may have several valid answers, remains challenging. Traditional QA settings often assume consistency across evidences, but MAQA can involve conflicting answers. Constructing datasets that reflect such conflicts is costly and labor-intensive, while existing benchmarks often rely on synthetic data, restrict the task to yes/no questions, or apply unverified automated annotation. To advance research in this area, we extend the conflict-aware MAQA setting to require models not only to identify all valid answers, but also to detect specific conflicting answer pairs , if any. To support this task, we introduce a novel cost-effective methodology for leveraging fact-checking datasets to construct NatConfQA , a new benchmark for realistic, conflict-aware MAQA, enriched with detailed conflict labels, for all answer pairs. We evaluate eight high-end LLMs on NatConfQA , revealing their fragility in handling various types of conflicts and the flawed strategies they employ to resolve them. 1 1 1 NatConfQA is publicly available at https://github.com/EN555/ContraQA . Figure 1: Deriving a conflict-aware MAQA instance from a fact-chcecking instance. The source instance is composed of a claim with supporting , refuting and neutral evidence. Annotators then create WH and Yes/No questions to surface these conflicts and label conflicting answer pairs. 1 Introduction Recent advances in Large Language Models (LLM) ( Fischer et al. 2024 ; OpenAI-Team 2024a ) have led to substantial performance improvement in various tasks, including Question Answering (QA) with one or multiple correct answers ( Voorhees 2004 ) . Although previous work on multi-answer QA (MAQA) largely assumes that the different answers are mutually consistent and complementary ( Zhong et al. 2022 ; Amouyal et al. 2023 ) , realistically, questions can be controversial and lack a definitive answer. In such cases, models should not only generate a response that incorporates several answers, but also detect the conflicts and communicate them to the reader. For example, when asked “What is the effect of aspartame?” , a comprehensive response should aggregate various effects reported in the available sources, while explicitly distinguishing between effects with consensus and those that remain contested or under debate. Collecting QA instances with naturally occurring contradictory answers is challenging, as knowledge conflicts are not always prevalent in arbitrary texts. As a result, only a few datasets handle knowledge conflicts while exhibiting some limitations, such as relying on LLMs to inject misinformation into other reliable texts, or focusing only on Yes/No questions (Section 6 ). Furthermore, although recent benchmarks aim to evaluate whether the entire response acknowledges the debatable nature of the question ( Xu et al. 2024 ; Hou et al. 2024 ) , they do not assess whether the response accurately reflects which answers are subject to disagreements (e.g., whether aspartame increases cancer risk). To foster research on this important challenge and to enable fine-grained evaluation, we create NatConfQA , the first conflict-aware MAQA dataset with annotations labeling individual answer pairs that are in conflict. To collect NatConfQA , we first leverage standard fact-checking datasets to identify sources with naturally occurring disagreements. Then, we ask human annotators to write various QA pairs and to label the relationship between pairs of conflicting answers. Finally, we verify the annotations for quality assurance. Figure 1 illustrates our general annotation scheme. NatConfQA is a high-quality dataset that covers Yes/No and WH- questions, and includes instances, based on reliable sources, with a mix of conflicting and non-conflicting answers for the same question. We evaluate the performance of eight LLMs on NatConfQA , including open-source and proprietary models, measuring both answer quality and conflict identification. In terms of answer quality, we show that while models generally achieve high precision, they fail to output all correct answers. Furthermore, our fine-grained evaluation of conflict identification reveals that models struggle to distinguish between conflicting and non-conflicting answer pairs. Further analysis of the model failures reveal insightful error patterns: models evade exposing conflicts by selecting a single answer, erroneously attempting to reconcile contradictory information, or refraining from answering the question altogether. Taken together, our work uncovers the behavior of strong LLMs when confronted with conflicting information, while providing suitable methodologies and data to investigate these challenges in future research. 2 Conflict-Aware Multi-Answer QA The Conflict-Aware Multi-Answer QA task is an extension of the traditional Multi-Answer QA Min et al. 2020 ; Amouyal et al. 2023 task, that considers potential conflicts between the different answers. We extend recent work in conflict-aware QA Xu et al. 2024 , which either focused on binary conflicting answers ( Hou et al. 2024 ) or addressed multiple answers without indicating which pairs conflict ( Jiayang et al. 2024 ) . Our generalized task formulation supports two or more answers per question and, importantly, pinpoints the answer pairs that are in conflict. Given a question q and a set of candidate passages P=\{p_{1},\dots,p_{n}\} , the task is to generate a free-text response y that satisfies two main requirements. First, similar to the traditional MAQA task, y should incorporate all answers that appear in reference A=\{a_{1},\dots,a_{m}\} . Second, the response y should indicate all conflicts, if any, between the answers within it. We assume that the response y is in natural language, as typically generated by large language models (LLMs), and not necessarily in a structured format. For example, consider the question “What climate degree change is caused by greenhouse gases?" . The different answers 0.45°C and 0.8°C cannot be simultaneously true, hence an ideal response should present both answers through contrastive language (e.g., by using the word “however”, both answers can be communicated, and the conflict is established). In contrast, for the question “Which domestic pets can potentially test positive for SARS-CoV-2?" , the answers dogs , cats , and ferrets are non-conflicting and an ideal response should enumerate the different answers cohesively. Since a question may elicit a mixture of conflicting and non-conflicting answers, we want to determine whether the model response y accurately reflects the conflict or non-conflict relations between the different answer pairs. Formally, for reference answers A and a respective set C=\bigl\{\{a_{i},a_{j}\}\mid a_{i}\text{ and }a_{j}\text{ conflict}\bigr\} , that lists the pairs of conflicting answers in A , the objective of response y is to accurately incorporate A and C . In accordance with the task definition, we define two evaluation criteria, adopted from related tasks ( Min et al. 2020 ; Hou et al. 2024 ) : Answer Quality , measuring how well the model covers the set of correct answers; and Conflict Identification , assessing the model’s ability to correctly identify conflicting answer pairs. We propose metrics for the evaluation criteria in § 4.3 . 3 Creating the NatConfQA Dataset Obtaining annotations for a conflict-aware MAQA dataset is challenging, because informational conflicts are infrequent in arbitrary sources from which answers can be collected. In this work, we approach this challenge by leveraging existing fact-checking datasets, a well-studied field with many datasets, which were annotated with large manual effort Thorne et al. 2018 ; Sarrouti et al. 2021 . Some fact-checking datasets contain claims for which some pieces