Beyond Facts: Evaluating Intent Hallucination in Large Language Models

Yijie Hao, Haofei Yu, Jiaxuan You · 2025

Read Beyond Facts: Evaluating Intent Hallucination in Large Language Models by Yijie Hao, Haofei Yu, Jiaxuan You (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-06539.txt.

Introduction

Yijie Hao, Haofei Yu, and Jiaxuan You name a failure mode of large language models in twenty twenty-five. When a query has many conditions, models often answer only part of it. They may skip some conditions or invent parts that were never asked. The authors call this intent hallucination, since the model's grasp of intent goes wrong.

They built a benchmark, FAITHQA, to measure the problem in a set way. It holds just over twenty thousand problems, across many topics and levels of difficulty. It covers plain queries and cases where the model also retrieves outside text. Early tests found intent hallucination to be common across models. Readers get a tool that looks past wrong facts to wrong aims.

Catalog record

CREATORYijie Hao, Haofei Yu, Jiaxuan You
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE63 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2506-06539

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Beyond Facts: Evaluating Intent Hallucination in Large Language Models Yijie Hao, Haofei Yu, Jiaxuan You Abstract When exposed to complex queries containing multiple conditions, today’s large language models (LLMs) tend to produce responses that only partially satisfy the query while neglecting certain conditions. We therefore introduce the concept of Intent Hallucination . In this phenomenon, LLMs either omit (neglecting to address certain parts) or misinterpret (responding to invented query parts) elements of the given query, leading to intent hallucinated generation. To systematically evaluate intent hallucination, we introduce FaithQA , a novel benchmark for intent hallucination that contains 20,068 problems, covering both query-only and retrieval-augmented generation (RAG) setups with varying topics and difficulty. FaithQA is the first hallucination benchmark that goes beyond factual verification, tailored to identify the fundamental cause of intent hallucination. By evaluating various LLMs on FaithQA , we find that (1) intent hallucination is a common issue even for state-of-the-art models, and (2) the phenomenon stems from omission or misinterpretation of LLMs. To facilitate future research, we introduce an automatic LLM generation evaluation metric, Constraint Score , for detecting intent hallucination. Human evaluation results demonstrate that Constraint Score is closer to human performance for intent hallucination compared to baselines. 1 Introduction Large Language Models (LLMs) have demonstrated utility across various applications ( OpenAI et al. 2024 ; Dubey et al. 2024 ) . However, hallucination remains a significant challenge ( Ji et al. 2023 ; Huang et al. 2023 ) . In particular, for complex queries containing multiple conditions (Figure 1 ), LLM outputs often deviate from the query, yielding unsatisfactory results. We term this phenomenon “Intent Hallucination”, which has received little attention in current research ( Min et al. 2023 ; Hou et al. 2024 ; Manakul et al. 2023 ) . Unlike factual hallucination ( Li et al. 2023 ; Cao et al. 2021 ) , which researchers can directly detect through search-based fact-checking ( Sellam et al. 2020 ; Min et al. 2023 ) , detecting and evaluating intent hallucination poses more challenges. This is because complex queries often contain duplicate intents, and LLMs often satisfy only a portion of them, making dissatisfaction difficult to detect or quantify. Furthermore, as LLMs continue to advance, users tend to provide these stronger LLMs with increasingly complicated queries, which even humans find difficult to understand. This trend highlights the need for LLMs to be not only factually correct but also intentionally aligned with human beings. Our paper addresses two under-explored questions: (1) Why do LLMs tend to exhibit Intent Hallucination? and (2) How can we detect Intent Hallucination? Answering these questions is vital for LLM applications that rely on both factual accuracy and faithful intent alignment. Figure 1: Examples of two types of intent hallucination (omission and misinterpretation) . For omission, GPT-4o omits "particularly from Spain", leading to factually accurate yet hallucinated outputs. For misinterpretation, GPT-4o misinterprets the missing article as provided, which leads to hallucinated outputs. For the first question, we propose that LLMs’ omission ( e.g. , ignoring query components) or misinterpretation ( e.g. , responding to invented query components) over word-level meaning constitutes the fundamental cause of intent hallucination. To further investigate, we introduce FaithQA , the first benchmark specifically designed to address intent hallucination’s two key scenarios: omission and misinterpretation. FaithQA consists of 20,068 queries, validated through extensive human evaluations to ensure quality. FaithQA covers a wide range of topics with varying levels of difficulty and proves challenging even for state-of-the-art models. Our benchmark reveals that increasing query complexity correlates with a higher rate of intent hallucination. To address the second question, we introduce Constraint Score , a new evaluation metric that focuses on detecting intent hallucination. Our approach involves two major steps: (1) decomposing the query by concepts and actions, then converting it into a series of short statements, each representing a specific requirement the generation must meet; and (2) assigning an importance-weighted binary label to each constraint, which enables fine-grained evaluation. Our human evaluation shows that Constraint Score significantly outperforms LLM-as-the-judge ( Manakul et al. 2023 ; Mishra et al. 2024 ; Sriramanan et al. 2024 ) , as LLM judgers tend to provide biased evaluations compared with human judgment. Taken together, our key contributions include: (1) We propose the concept of intent hallucination beyond the existing category of factual hallucination; (2) We develop FaithQA , the first benchmark that focuses on evaluating intent hallucination. Our result shows that intent hallucination represents a prevalent phenomenon even for state-of-the-art LLMs; (3) We introduce Constraint Score , a novel evaluation metric that automatically assesses LLM generations by breaking down the query into intent constraints and computing a weighted score. Our analysis shows that Constraint Score significantly outperforms pure LLM grading baselines, which tend to be biased. 2 Related Works Hallucinations in LLMs . In LLMs, "hallucination" refers to outputs that are nonfactual, irrelevant, or fabricated. This issue arises in tasks such as question answering ( Sellam et al. 2020 ) , translation ( Lee et al. 2018 ) , summarization ( Durmus et al. 2020 ) , and dialogue ( Balakrishnan et al. 2019 ) , as noted in several studies ( Ji et al. 2023 ; Azaria and Mitchell 2023 ; Huang et al. 2023 ; Cao et al. 2021 ) . To address this issue, many efforts aim to detect and mitigate hallucinations. Min et al. 2023 evaluate factual accuracy by checking core facts (atomic facts) in each sentence against reliable sources such as Wikipedia. Hou et al. 2024 propose a hidden Markov tree model that breaks statements into premises and assigns a factuality score based on the probability of all parent premises. Manakul et al. 2023 detects hallucinations by sampling multiple responses and using self-consistency to identify discrepancies. Despite these significant efforts, limitations remain. Most existing work either focuses solely on factual precision or on in-context recall, overlooking the role of the query in generation ( Li et al. 2023 ; Yang et al. 2023 ; Niu et al. 2024 ) ( e.g. , scoring both outputs equally in Figure 2 ), or treats the query as a whole ( Zhang et al. 2024a ) , which results in coarse-grained evaluation. Hallucination benchmarks . Recent work on hallucination detection for LLMs includes HaluEval ( Li et al. 2023 ) (synthetic and natural responses), FELM ( Chen et al. 2023 ) (natural responses across domains), RAGTruth ( Niu et al. 2024 ) (RAG hallucinations), and InfoBench ( Qin et al. 2024 ) (instruction-following via query decomposition). These benchmarks mainly focus on factual hallucinations or require manual annotation. In contrast, FaithQA is, to our knowledge, the first to assess non-factual hallucinations from a query-centric perspective. Although Zhang et al. 2024b also discusses a related topic, their work primarily explores the causes of intent hallucination from a training corpus perspective. In contrast, our paper provides a comprehensive evaluation metric along with an extensive benchmark for systematic testing. FaithEval ( Ming et al. 2025 ) investigates hallucination in RAG settings by evaluating whether model outputs remain faithful to externally retrieved contexts, particularly under conditions involving unanswerable or contradictory evidence. FaithQA adopts a similar RAG setup but shifts the focus from context alignment to query alignment.

Read the complete text