Quantifying Uncertainty in Natural Language Explanations of Large Language Models for Question Answering

Yangyi Li, Mengdi Huai · 2025

Read Quantifying Uncertainty in Natural Language Explanations of Large Language Models for Question Answering by Yangyi Li, Mengdi Huai (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-15403.txt.

Introduction

Yangyi Li and Mengdi Huai ask how sure we can be of a model's own words. Large language models now answer questions and can explain their answers in plain text. Those explanations help users, even for closed models. Yet no one had set a firm bound on how much trust an explanation deserves. Noise in the input, such as in medical queries, makes this harder.

The pair build a method that puts valid bounds on doubt for such explanations. It works after the model has run and needs no access to its insides. A second, tougher version keeps its bounds even when the input is noisy. Tests on question answering tasks show both methods work as claimed. The work adds a check on model talk.

Catalog record

CREATORYangyi Li, Mengdi Huai
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE31 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2509-15403

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Quantifying Uncertainty in Natural Language Explanations of Large Language Models for Question Answering Yangyi Li, Mengdi Huai Abstract Large language models (LLMs) have shown strong capabilities, enabling concise, context-aware answers in question answering (QA) tasks. The lack of transparency in complex LLMs has inspired extensive research aimed at developing methods to explain large language behaviors. Among existing explanation methods, natural language explanations stand out due to their ability to explain LLMs in a self-explanatory manner and enable the understanding of model behaviors even when the models are closed-source. However, despite these promising advancements, there is no existing work studying how to provide valid uncertainty guarantees for these generated natural language explanations. Such uncertainty quantification is critical in understanding the confidence behind these explanations. Notably, generating valid uncertainty estimates for natural language explanations is particularly challenging due to the auto-regressive generation process of LLMs and the presence of noise in medical inquiries. To bridge this gap, in this work, we first propose a novel uncertainty estimation framework for these generated natural language explanations, which provides valid uncertainty guarantees in a post-hoc and model-agnostic manner. Additionally, we also design a novel robust uncertainty estimation method that maintains valid uncertainty guarantees even under noise. Extensive experiments on QA tasks demonstrate the desired performance of our methods. 1 Introduction Large language models (LLMs) such as GPT-4 have recently achieved impressive gains in natural - language understanding and generation, demonstrating near - human fluency across a wide range of tasks Achiam et al. 2023 . When adapted for open - domain question answering (QA), these models exploit their vast parametric knowledge to deliver concise, context - aware answers that surpass traditional retrieval pipelines in reasoning depth and coverage Kwiatkowski et al. 2019 ; Chen et al. 2025b . However, in LLM-based QA systems, the underlying LLMs are complex models where their inner working mechanisms are not yet fully understood. This lack of interpretability poses a significant barrier to their deployment in high-stakes decision-making applications, where inappropriate guidance can have severe consequences. Given the importance of explaining LLMs’ behaviors, many interpretation methods have been proposed Zhu et al. 2024 . Among them, natural language explanations Kumar and Talukdar 2020 ; Camburu et al. 2018 ; Wadhwa et al. 2024 are appealing for their self-contained insights, even when the models are closed-source. However, there is no work studying rigorous uncertainties behind these explanations, making it difficult to understand the confidence level associated with them. Traditional uncertainty estimation methods (e.g., perturbation and Bayesian-based methods) Tanneru et al. 2024 ; Xiong et al. 2023 ; Liu et al. 2024 face significant limitations when applied to natural language explanations. Specifically, they either fail to provide valid uncertainty guarantees, or require the access of model logits and extensive model retraining. In this work, we aim to provide rigorous, post - hoc, and model - agnostic uncertainty guarantees for natural - language explanations in QA. Conformal prediction offers a distribution - free framework with provably valid coverage Shafer and Vovk 2008 ; Su et al. 2024 ; Campos et al. 2024 ; Qian et al. 2024b ; Li et al. 2024 ; Lidder et al. 2025 ; Zhao et al. 2025 ; Angelopoulos et al. 2024 . However, traditional conformal prediction methods cannot be directly applied to natural language explanations. The reason is that in conformal prediction, the underlying models are typically trained in a supervised manner, where the uncertainty sets correspond to predefined class labels. In contrast, LLMs are trained in an auto-regressive fashion, generating text one token at a time, with each token conditioned on the previously generated tokens. Additionally, real - world QA queries often contain noise, such as ambiguous phrasing and typographical errors. Such noise can violate the underlying exchangeability assumption required by conformal prediction Shafer and Vovk 2008 . Therefore, these generated uncertainties could become invalid, posing substantial challenges for generating reliable uncertainty estimates of natural language explanations. Although several robust conformal methods have been proposed Yan et al. 2024 ; Ghosh et al. 2023 ; Wang et al. 2024 ; Jeary et al. 2024 , they typically assume well-structured datasets and fail to account for discrete and token-level noise that is inherent in natural language explanations generated by LLMs. Such noise complicates valid uncertainty guarantees for natural language explanations in QA. To address the above challenges, in this work, we propose ULXMQA , a novel u ncertainty method for natural l anguage e x planations for m edical q uestion a nswering, which can generate valid uncertainty guarantees in a post-hoc and model-agonistic way. Specifically, in our method, we first design prompts, which can assign each input token an importance score. Then, we design an uncertainty set construction function, which selects explanation tokens based on their assigned importance scores. For the constructed uncertainty sets, we provide theoretical guarantees by proving that the expected fraction of ground-truth tokens included in these uncertainty sets is theoretically guaranteed. Additionally, to address noisy data that may undermine the validity of the generated uncertainty sets, we also design a r obust u ncertain estimation method for these generated natural l anguage e x planations ( RULX ), which can provide robust valid uncertainty guarantees under discrete and token-level noise in questions. We further conduct extensive experiments to verify the desired performance of our proposed methods across different question answering tasks. 2 Methodology Here, we first introduce our valid uncertainty method for natural language explanations in LLM-based QA systems. Then, we present the proposed robust uncertainty method, designed to mitigate the effects of noise. Without loss of generality, in this paper, we consider a vision-language model based QA system, which can output accurate answers to medical questions about the input medical image. Let \mathcal{Q} represent the question space and \mathcal{A} the answer space. We denote a language model as \mathcal{M}:\mathcal{Q}\rightarrow\mathcal{A} , which takes a sequence of k question tokens Q=(q_{1},q_{2},\ldots,q_{k}) , and produces a sequence of m answer tokens A=(a_{1},a_{2},\ldots,a_{m}) . A designed prompt P augments the input and instructs \mathcal{M} to emit a natural language explanation E , which we model as a subset of question tokens deemed essential for predicting A . Modeling uncertainty in natural language explanations. Here, we propose a post - hoc, model - agnostic method that assigns provably valid uncertainty to natural language explanations E . Let \mathcal{D}^{\text{cal}}=\{(P_{i},Q_{i},E_{i}^{*},A_{i})\}_{i=1}^{n} denote the calibration data, and Q_{n+1} denote the test-time question. Here, E_{i}^{*} denotes the ground - truth explanation supplied by human annotators. Each example comes with a gold explanation sentence, and annotators mark the question tokens they judge essential in light of that sentence Aggarwal et al. 2021 . Specifically, we construct an uncertainty set of Q_{n+1} that provides theoretical guarantees on the inclusion ratio of ground-truth natural language explanations. The challenge in providing guarantees is that the language model \mathcal{M} generates text auto - regressively. To address this, for each question Q_{i}\in\{Q_{i}\}_{i=1}^{n+1} , we propose a confidence-aware prompt P_{i} , which enables \mathcal{M} to output an importance score \mathcal{S}(P_{i},q_{i,j};\mathcal{M})\in[0,1] for each word in Q_{i} and simultaneously obtain the final answer. These scores reflect how essential each token is for predicting the final answer. Then, for Q_{i} , we construct its uncertainty explanation set as follows \displaystyle\mathcal{C}_{\lambda}(Q_{i};\mathcal{M})= \displaystyle\quad\{q_{i,j}\in

Read the complete text