Evaluating LLMs’ Assessment of Mixed-Context Hallucination Through the Lens of Summarization

Siya Qi, Rui Cao, Yulan He, Zheng Yuan · 2025

Read Evaluating LLMs’ Assessment of Mixed-Context Hallucination Through the Lens of Summarization by Siya Qi, Rui Cao, Yulan He, Zheng Yuan (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-01670.txt.

Introduction

Siya Qi, Rui Cao, Yulan He, and Zheng Yuan test whether large language models can judge hallucinations in text when falsehoods of two kinds are mixed, in a 2025 paper. Using models as judges is now common, but earlier checks split errors into two tidy kinds: breaking faith with a source text, or breaking faith with the world. Real errors blend both.

The team uses summarization as the test bed and grades judges across direct and retrieval-backed models of several sizes. Three findings stand out. A model's own stored knowledge biases its judging. That bias hits world-fact errors hardest and caps performance there. And the core problem is balance: leaning on stored knowledge versus the given context at the right moments. The authors map where judge models stand on this mixed task.

Catalog record

CREATORSiya Qi, Rui Cao, Yulan He, Zheng Yuan
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE59 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-01670

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Evaluating LLMs’ Assessment of Mixed-Context Hallucination Through the Lens of Summarization Siya Qi, Rui Cao, Yulan He, Zheng Yuan Abstract With the rapid development of large language models (LLMs), LLM-as-a-judge has emerged as a widely adopted approach for text quality evaluation, including hallucination evaluation. While previous studies have focused exclusively on single-context evaluation (e.g., discourse faithfulness or world factuality), real-world hallucinations typically involve mixed contexts, which remains inadequately evaluated. In this study, we use summarization as a representative task to comprehensively evaluate LLMs’ capability in detecting mixed-context hallucinations, specifically distinguishing between factual and non-factual hallucinations. Through extensive experiments across direct generation and retrieval-based models of varying scales, our main observations are: (1) LLMs’ intrinsic knowledge introduces inherent biases in hallucination evaluation; (2) These biases particularly impact the detection of factual hallucinations, yielding a significant performance bottleneck; (3) The fundamental challenge lies in effective knowledge utilization, balancing between LLMs’ intrinsic knowledge and external context for accurate mixed-context hallucination evaluation. 1 Introduction Figure 1: Examples of our automated construction for mixed-context hallucination datasets, where "a city of the United Kingdom" is the correct description of "Belfast", constructed as factual hallucination. "A city in America" is an incorrect description of "Belfast", constructed as a non-factual hallucination. Large language models (LLMs) generate coherent text and follow instructions across diverse tasks Zhao et al. (2024) ; Minaee et al. (2024) . However, a critical challenge in scaling LLM applications is hallucination, where the generated content lacks factual grounding Lin et al. (2022) ; Li et al. (2023) or deviates from the intended discourse context Zhou et al. (2023) . This issue makes hallucination evaluation essential for improving LLM reliability and developing more robust models Huang et al. (2023) ; Zhang et al. (2023) ; Ji et al. (2023) ; Tonmoy et al. (2024) . Recent studies have explored the potential of using LLMs themselves to assess hallucinations, reporting strong performance in hallucination detection with LLM-based evaluators Gu et al. (2025) ; Min et al. (2023) ; Chern et al. (2023) . In practical applications, hallucinations typically derive from two primary sources: one originates from the discourse context, affecting faithfulness; while the other arises from external knowledge or the model’s inherent knowledge, influencing factuality Maynez et al. (2020) . Previous LLM-based evaluators for hallucinations have focused on assessing either faithfulness Manakul et al. (2023b) ; Wu et al. (2023) ; Zha et al. (2023) or factuality Chern et al. (2023) ; Min et al. (2023) ; Dhuliawala et al. (2024) ; Hu et al. (2024) of the text under evaluation. However, hallucinations can emerge from a combination of discourse context and external information – a mixed-context setting that remains under-explored in LLM-based evaluation. Many real-world scenarios operate within this mixed-context setting. For instance, in summarization, a system might fabricate details while still presenting factually correct information; in customer support chatbots, it may hallucinate from chat histories alongside accurate product information; and in image captioning, models might provide factually correct descriptions about non-existing objects in an image. Evaluating these mixed-context hallucinations requires simultaneous assessment of both faithfulness and factuality, which presents unique challenges for developing robust evaluation metrics and benchmarks. In this work, we focus on the summarization task to assess LLMs’ capability under this setting. With mixed-context considered, hallucinations in summarization can be categorized into two types 1 1 1 We focus on the news summarization task, where source articles are assumed to be factually correct, so a hallucinated summary cannot be both faithful and non-factual. : factual hallucinations , where the generated content is factually correct but inconsistent with the source (e.g., the green block in Figure 1 ); and non-factual hallucinations , where the output contains outright factual inaccuracies (e.g., the red block in Figure 1 ). Existing datasets for mixed-context hallucinations in summarization Maynez et al. (2020) ; Cao et al. (2022) ; Dong et al. (2022) are limited both in scale and exhibit unbalanced distributions of hallucination instances, making them inadequate to assess LLMs’ capability in handling mixed-context hallucinations. To address this limitation, this study introduces the first automated pipeline for constructing mixed-context hallucination evaluation datasets, and aims to investigate the following research questions: RQ1) How do LLM-based hallucination evaluators perform in the mixed-context setting with different prompting strategies? Given the rapid advancement of retrieval-augmented generation (RAG) Gao et al. (2024) , we then examine RQ2) How can LLMs better leverage external knowledge to detect hallucinations? Among the hallucination categories, we ask RQ3) Which category benefits the most across evaluation methods and models? Finally, a fundamental question in LLM research, RQ4) Does scaling up model size lead to better hallucination assessment? These questions collectively address the challenges in mixed-context hallucination assessment, with our findings laying the groundwork for scalable LLM self-evaluation and self-evolution. The contributions and key observations of this work are as follows: 1. We comprehensively evaluate LLM-as-a-judge approaches for mixed-context hallucination detection through summarization, introducing an effective, well-balanced, and easily scalable benchmark, FHSumBench 2 2 2 Data and code are available at https://github.com/cece00/FHSumBench . 2. Our extensive experiments across direct generation and retrieval-based LLMs of varying scales reveal that model scaling does not guarantee better performance. However, the main performance bottleneck lies in the detection of factual hallucinations. 3. We identify that effective knowledge utilization remains the primary challenge in mixed-context hallucination evaluation. Prompt engineering yields greater improvements on smaller LLMs. While external knowledge augmentation enhances accuracy, it requires carefully designed retrieval strategies. 2 Related work Mixed-Context Hallucination Context plays a crucial role in hallucination evaluation. Hu et al. (2024) examine the hallucination detection capabilities of LLMs in the question-answer task under different contextual settings. However, their evaluation is limited to isolated contexts. For summarization, while the source document serves as a context input, the factuality of the summary is equally critical. Maynez et al. (2020) first systematically annotated the faithfulness and factuality separately on the XSum Narayan et al. (2018) dataset, using summaries generated by small language models. Later, Cao et al. (2022) introduced another dataset with entity-level annotations, though it remained limited to XSum. The annotation process demands extensive effort from domain experts and heavily relies on annotator agreement, making it both time-consuming and resource-intensive. Hallucination Evaluation Methods Most previous works only focus on one aspect of hallucination evaluation, which is either faithfulness or factuality. For faithfulness evaluation, a natural approach is to assess the entailment between the text and the context document Wu et al. (2023) ; Zha et al. (2023) ; Goyal and Durrett (2020) or to extract and compare answers by querying them Manakul et al. (2023a) ; Durmus et al. (2020) ; Fabbri et al. (2022) . For factuality evaluation, LLMs intrinsic knowledge Chen et al. (2023) ; Huang et al. (2024) and external knowledge source Min et al. (2023) ; Chern et al. (2023) ; Dhuliawala et al. (2024) are both used for detection. With both aspects considered, EntFA Cao et al. (2022) utilizes entity original and context-conditioned features to detect factual and non-factual hallucinations. 3 FHSumBench To evaluate models’ ability to detect mixed-context hallucinations in summaries, an appropriate test set is essential. To the best of our knowledge, the XEnt dataset Cao et al. (2022) is the only one providing factual hallucination

Read the complete text