IUQ: Interrogative Uncertainty Quantification for Long-Form Large Language Model Generation
Haozhi Fan Jinhao Duan Kaidi Xu · 2026
Read IUQ: Interrogative Uncertainty Quantification for Long-Form Large Language Model Generation by Haozhi Fan Jinhao Duan Kaidi Xu (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-15109.txt.
Introduction
Fan, Duan, and Xu tackle uncertainty in long, free-form answers from large language models. Short-answer methods falter here, since fluent replies can still carry false claims across many tangled meanings. Their IUQ framework first interrogates a reply and then answers, combining agreement across samples with faithfulness within each sample. The result gives claim-level uncertainty measures for long-form generation, giving readers a steadier gauge of what to trust.
Catalog record
| CREATOR | Haozhi Fan Jinhao Duan Kaidi Xu |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 46047 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.15109
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
IUQ: Interrogative Uncertainty Quantification for Long-Form Large Language Model Generation Haozhi Fan Jinhao Duan Kaidi Xu Abstract Despite the rapid advancement of Large Language Models (LLMs), uncertainty quantification in LLM generation is a persistent challenge. Although recent approaches have achieved strong performance by restricting LLMs to produce short or constrained answer sets, many real-world applications require long-form and free-form text generation. A key difficulty in this setting is that LLMs often produce responses that are semantically coherent yet factually inaccurate, while the underlying semantics are multifaceted and the linguistic structure is complex. To tackle this challenge, this paper introduces Interrogative Uncertainty Quantification (IUQ) , a novel framework that leverages inter-sample consistency and intra-sample faithfulness to quantify the uncertainty in long-form LLM outputs. By utilizing an interrogate-then-respond paradigm, our method provides reliable measures of claim-level uncertainty and the model’s faithfulness. Experimental results across diverse model families and model sizes demonstrate the superior performance of IUQ over two widely used long-form generation datasets. The code is available at https://github.com/louisfanhz/IUQ . 1 Introduction Large Language Models (LLMs) have shown remarkable improvement across a diverse range of Natural Language Processing tasks ( 1 ; 4 ; 21 ). However, LLMs remain susceptible to hallucination, as they generate plausible answers that are factually incorrect ( 49 ; 16 ). Figure 1: An example of LLM generation on biography. The model incorrectly identifies the individual’s profession and fabricates a complete biography to maintain logical consistency. When multiple outputs are sampled, they exhibit a high level of similarity even though the biographies are based on false information. The model is then shown to be uncertain about the subject in a separate session asking specifically about the individual’s profession. Recent Uncertainty Quantification (UQ) methods effectively measure hallucination within a confined answer space, where the models are prompted to generate short responses or answer multiple-choice questions ( 23 ; 24 ; 5 ; 2 ; 42 ). These approaches leverage token-probabilities or semantic entailment between responses to construct uncertainty estimates. However, in the scenario of free-form generation, where the response lengthens and exhibits structure and logic, it can be difficult to evaluate the entailment relationships between long answers, and aggregating token-probabilities becomes less indicative of uncertainty. Current work on long-form UQ involves evaluating the semantic consistency between LLM responses. The long-form response is decomposed into sentences or claims, which are then compared against additional sampled responses to obtain uncertainty estimates ( 26 ; 48 ; 20 ; 43 ). However, the following observation requires a closer look at the fine-grained contextual dependence in LLM generations: long-form output may differ across samples but is rarely self-contradictory due to next-token conditioning on the preceding context. Therefore, a critical challenge occurs when models fabricate information for the sake of logical consistency. As illustrated in Fig. 1 , when the LLM is prompted to provide information on a historical figure, it mistakenly identifies the individual’s profession but continues to generate a plausible story to maintain logical consistency. On the other hand, when asked specifically about the individual’s profession, the model returns inconsistent answers, showing uncertainty about the subject. The tendency for LLM to fabricate information is not captured by current UQ methods, which only evaluate consistency across sampled generations without testing the models’ real knowledge on the subject. Recent studies reveal LLMs hallucinate on facts that are present in the training data ( 19 ). When the topic is underrepresented, LLMs may even be overconfident about false knowledge ( 22 ; 25 ; 33 ). These findings corroborates our observation that LLMs are not always faithful, and it has become increasingly difficult to identify incorrect information, as LLMs are more capable of formulating plausible responses ( 15 ; 17 ). To tackle this challenge, we first differentiate the claims in a generative context from the model’s knowledge. Specifically, we incorporate an interrogator LLM to construct tailored short questions for each factual claim made in the long-form output. As a result, the original long-form response is decomposed into atomic knowledge decoupled from the generative context. To demonstrate correct knowledge, the model needs to answer consistently to each independent question and not provide answers that contradict the original claims. Naturally, the amount by which the model returns contradictory answers constitutes a measure of the tendency to fabricate information. We propose a novel UQ framework: Interrogative Uncertainty Quantification (IUQ) to facilitate fine-grained probing of long-form LLM responses. Different from other UQ methods that only implement inter-sample consistency, IUQ also enforces intra-sample claims consistency through independent question-answering. The strategy is analogous to an interrogate-respond scenario in which the responder is continuously questioned by the interrogator to screen untruthfulness. Furthermore, since the short questions extracted from claims are independent of other sampled generations, IUQ presents a confidence landscape for each generation by viewing the quantified uncertainty of claims as data points in a time-series. We evaluate IUQ on various model families of diverse sizes: GPT4o ( 31 ) , Qwen2 47 , Gemma-3 ( 37 ) , Mistral ( 18 ) , LlaMA-3.1, and LlaMA-3.3 14 , with model sizes ranging from 24B up to 72B. We use two datasets tailored for long-form generation: FActScore ( 27 ) , which contains items of biography, and LongFact ( 43 ) , which contains prompt sets on topics of art, science, and so on. Our contribution is the following: • We propose the Interrogative Uncertainty Quantification (IUQ) workflow that evaluates long-form responses through fine-grained probing of claim-level knowledge. Extensive experiments have demonstrated the effectiveness of IUQ over diverse model families. • We highlight an under-explored phenomenon of long-form LLM generation where the models fabricate factual information to maintain logical consistency, and present a quantitative analysis of this tendency at the claim-level. Figure 2: The framework of Interrogative Uncertainty Quantification (IUQ): Responses are sampled from LLMs and decomposed into atomic claims. The model then answers questions that address the information contained in the claim in a separate session without context. The model is unfaithful about a claim if the claim contradicts the corresponding answers, which represent the model’s knowledge on the subject. 2 Related Work Uncertainty Quantification Existing UQ methods can be roughly categorized into white-box and black-box methods. White-box methods assume the model architecture is partially or completely visible ( 23 ; 30 ; 5 , 10 ; 7 ), whereas the black-box methods rely on the input prompts and LLM responses to measure uncertainties ( 46 ; 13 ; 41 ; 6 ). Our work follows the line of black-box methods. Among them, 38 utilizes a weighted ensemble of semantically equivalent prompts to compute output uncertainty, where the weights are obtained through Bayesian variational inference. 46 explores various strategies in prompting, sampling, and aggregating phases to acquire confidence scores from the model. 13 perturbs the prompts and measures the semantic variation in responses. IUQ is distinct from these methods that it applies to long-form LLM outputs and decouples the contextual dependence to examine the models’ real knowledge. Self-Consistency in LLMs Self-consistency based approaches are proven to be effective in diverse domains associated with LLMs ( 32 ) . 40 have shown significant improvement in Chain-of-thought prompting by sampling multiple paths and picking the most consistent answer. 35 robustly induces