Enhancing Uncertainty Estimation in LLMs with Expectation of Aggregated Internal Belief

Zeguan Xiao, Diyang Dou, Boya Xiong, Yun Chen, Guanhua Chen · 2025

Read Enhancing Uncertainty Estimation in LLMs with Expectation of Aggregated Internal Belief by Zeguan Xiao, Diyang Dou, Boya Xiong, Yun Chen, Guanhua Chen (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-01564.txt.

Introduction

Language models often sound sure even when they are wrong, and that overconfidence sharpens after training on human feedback. Zeguan Xiao, with four co-authors, proposes EAGLE, a self-evaluation method that reads the model's internal states to score its certainty more honestly. Instead of trusting the final output, it gathers beliefs from several middle layers while the model checks its own answer.

Those layer-wise beliefs are aggregated, and the expected value of the resulting confidence distribution becomes the refined score. Wide experiments across data sets and models show EAGLE improves calibration over existing methods, meaning stated confidence tracks real accuracy more closely. Xiao and the team argue a model's hidden layers hold a truer sense of its own doubt than its polished final words suggest.

Catalog record

CREATORZeguan Xiao, Diyang Dou, Boya Xiong, Yun Chen, Guanhua Chen
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE34 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2509-01564

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Enhancing Uncertainty Estimation in LLMs with Expectation of Aggregated Internal Belief Zeguan Xiao, Diyang Dou, Boya Xiong, Yun Chen, Guanhua Chen Abstract Large Language Models (LLMs) have achieved remarkable success across a wide range of natural language tasks, but often exhibit overconfidence and generate plausible yet incorrect answers. This overconfidence, especially in models undergone Reinforcement Learning from Human Feedback (RLHF), poses significant challenges for reliable uncertainty estimation and safe deployment. In this paper, we propose EAGLE (Expectation of AGgregated internaL bEief), a novel self-evaluation-based calibration method that leverages the internal hidden states of LLMs to derive more accurate confidence scores. Instead of relying on the model’s final output, our approach extracts internal beliefs from multiple intermediate layers during self-evaluation. By aggregating these layer-wise beliefs and calculating the expectation over the resulting confidence score distribution, EAGLE produces a refined confidence score that more faithfully reflects the model’s internal certainty. Extensive experiments on diverse datasets and LLMs demonstrate that EAGLE significantly improves calibration performance over existing baselines. We also provide an in-depth analysis of EAGLE , including a layer-wise examination of uncertainty patterns, a study of the impact of self-evaluation prompts, and an analysis of the effect of self-evaluation score range. 1 1 1 Our code is public at https://github.com/sustech-nlp/EAGLE . 1 Introduction Large Language Models (LLMs) have demonstrated impressive capabilities across a wide spectrum of natural language understanding and generation tasks ( Brown et al. 2020 ; Wolf et al. 2020 ; Achiam et al. 2023 ; Touvron et al. 2023 ) . Their ability to generate coherent and contextually relevant text has led to their increasing deployment in various real-world applications. However, a critical challenge remains: LLMs often produce plausible-sounding but incorrect answers, a phenomenon known as hallucination ( Bai et al. 2022 ) . This challenge is further exacerbated by the lack of reliable uncertainty estimation for the outputs of LLMs that have undergone Reinforcement Learning from Human Feedback (RLHF) ( Kadavath et al. 2022 ; Tian et al. 2023 ) , undermining their trustworthiness and limits their safe deployment. To mitigate this issue, recent research has introduced calibration techniques, which aim to ensure that LLMs’ confidence of predictions accurately reflect their likelihood of correctness ( Kadavath et al. 2022 ; Chen and Mueller 2023 ; Xiong et al. 2023 ) . Among the various research directions, a particularly important approach is leveraging the LLMs’ inherent capabilities for self-reflection and self-evaluation as a means of uncertainty estimation ( Lin, Hilton, and Evans 2022 ; Kadavath et al. 2022 ; Tian et al. 2023 ; Chen and Mueller 2023 ; Huang et al. 2024 ) . These methods often involve prompting the LLM to explicitly assess the uncertainty and/or correctness of its own generated responses, usually by eliciting verbalized confidence scores. The motivation of our method stems from the observation that LLMs’ internal representations, particularly the hidden states at different layers, contain rich information about LLMs’ internal belief and can be used to reveal truthfulness of statements, knowledge discovery and behavior monitoring ( Burns et al. 2022 ; Azaria and Mitchell 2023 ; Zou et al. 2023 ) . Inspired by this, we propose a novel self-evaluation-based calibration method, EAGLE (Hidden Layer Aggregation), for uncertainty estimation. Unlike previous methods that rely on superficial verbalized confidence scores, our approach derives the confidence score by leveraging the internal representations of LLMs. Specifically, EAGLE first begins by prompting the LLM to assess the uncertainty of its own answer by providing a numerical confidence score. It then extract the hidden representations corresponding to the self-evaluation token from multiple layers of the model. These hidden states are projected into the vocabulary space to obtain layer-wise logits, which are subsequently aggregated by averaging to form a refined logit vector. By applying a softmax over the logits associated with the candidate confidence scores, EAGLE produces a probability distribution that reflects the model’s internal belief about its answer’s correctness. The final confidence score is computed as the expectation over this distribution. This layer-wise aggregation strategy enables EAGLE to capture richer and more nuanced confidence signals than methods based solely on the model’s final output, leading to improved calibration performance across diverse tasks and model architectures. The main contributions of this work are as follows: • We propose a novel training-free self-evaluation-based uncertainty estimation method, EAGLE , which leverages layer-wise hidden states to enhance calibration in LLMs. • We conduct extensive empirical evaluations on three diverse datasets with state-of-the-art LLMs, showing that our proposed methods achieve superior calibration performance compared to existing baselines. • We provide an in-depth analysis of EAGLE , including a layer-wise examination of uncertainty patterns, a study of the impact of self-evaluation prompts, and an analysis of the effect of self-evaluation score range. Figure 1: The EAGLE pipeline. First, the LLM generates an answer. Then, EAGLE prompts the LLM to self-evaluate the correctness of its answer, extracts the hidden states corresponding to the self-evaluation token from multiple layers, projects these hidden states to logits, and aggregates them. Finally, a softmax function is applied to the aggregated logits to compute the final confidence score. 2 Related Work Language Models Calibration. Model calibration—the alignment of a model’s predicted probabilities with the true likelihood of correctness ( Hendrycks and Gimpel 2016 ; Guo et al. 2017 ) —is crucial for user trust and informed decision-making in AI systems. In traditional supervised learning, post-hoc calibration techniques such as temperature scaling ( Guo et al. 2017 ) , Platt scaling ( Platt et al. 1999 ) , and isotonic regression ( Niculescu-Mizil and Caruana 2005 ) are used to improve the reliability of model confidence estimates. However, directly applying these methods to LLMs, particularly those undergone RLHF ( Kadavath et al. 2022 ; Tian et al. 2023 ) , poses significant challenges. Calibration in LLMs has been approached primarily through two lines of research. The first involves analyzing the diversity of multiple generated outputs, where techniques such as clustering-based metrics are used to examine the variance in the model’s responses ( Kuhn, Gal, and Farquhar 2023 ; Lin, Trivedi, and Sun 2024 ) . While effective in some scenarios, these methods are often computationally intensive due to the need for multiple generations per input, and the diversity observed does not always reliably indicate the correctness of individual outputs. A second, increasingly prominent direction leverages the self-evaluation capabilities of LLMs. In this approach, the model is prompted to assess the quality or correctness of its own generated responses, typically by providing a verbalized confidence score or categorical judgment ( Lin, Hilton, and Evans 2022 ; Kadavath et al. 2022 ; Tian et al. 2023 ; Chen and Mueller 2023 ; Huang et al. 2024 ) . Self-evaluation methods offer a more direct means of accessing the model’s internal assessment of its outputs, and have shown promise for improving calibration. However, most existing self-evaluation approaches rely on superficial verbalized confidence scores, which may fail to capture the nuanced confidence signals present in the model’s internal representations. Internal Representations of LLMs. Analyzing the internal representations of LLMs has emerged as a crucial direction for understanding their inner workings and decision-making processes. A promising line of research involves interpreting these hidden states by projecting

Read the complete text