A Head to Predict and a Head to Question: Pre-trained Uncertainty Quantification Heads for Hallucination Detection in LLM Outputs

Artem Shelmanov, Ekaterina Fadeeva, Akim Tsvigun, Ivan Tsvigun, Zhuohan Xie Igor Kiselev Nico Daheim Caiqi Zhang Artem Vazhentsev, Mrinmaya Sachan Preslav Nakov Timothy Baldwin · 2025

Read A Head to Predict and a Head to Question: Pre-trained Uncertainty Quantification Heads for Hallucination Detection in LLM Outputs by Artem Shelmanov, Ekaterina Fadeeva, Akim Tsvigun, Ivan Tsvigun, Zhuohan Xie Igor Kiselev Nico Daheim Caiqi Zhang Artem Vazhentsev, Mrinmaya Sachan Preslav Nakov Timothy Baldwin (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-08200.txt.

Introduction

Artem Shelmanov and a large team build ready-made heads that gauge a language model's doubt, to catch its made-up text. Their paper is from two thousand twenty-five.

Models can state false things with full ease, and users have few tools to spot them. Gauges of doubt give a frame to judge a reply's trust. Past gauges need no training, but they are weak. The team's heads are small add-on parts, taught in advance on marked data. They latch onto the model and read its inner state. Their strength comes from that taught base. Tests show they catch false output far better than the untaught gauges.

Catalog record

CREATORArtem Shelmanov, Ekaterina Fadeeva, Akim Tsvigun, Ivan Tsvigun, Zhuohan Xie Igor Kiselev Nico Daheim Caiqi Zhang Artem Vazhentsev, Mrinmaya Sachan Preslav Nakov Timothy Baldwin
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE42 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-08200

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

A Head to Predict and a Head to Question: Pre-trained Uncertainty Quantification Heads for Hallucination Detection in LLM Outputs Artem Shelmanov, Ekaterina Fadeeva, Akim Tsvigun, Ivan Tsvigun, Zhuohan Xie Igor Kiselev Nico Daheim Caiqi Zhang Artem Vazhentsev, Mrinmaya Sachan Preslav Nakov Timothy Baldwin Abstract Large Language Models (LLMs) have the tendency to hallucinate, i.e., to sporadically generate false or fabricated information. This presents a major challenge, as hallucinations often appear highly convincing and users generally lack the tools to detect them. Uncertainty quantification (UQ) provides a framework for assessing the reliability of model outputs, aiding in the identification of potential hallucinations. In this work, we introduce pre-trained UQ heads: supervised auxiliary modules for LLMs that substantially enhance their ability to capture uncertainty compared to unsupervised UQ methods. Their strong performance stems from the powerful Transformer architecture in their design and informative features derived from LLM attention maps. Experimental evaluation shows that these heads are highly robust and achieve state-of-the-art performance in claim-level hallucination detection across both in-domain and out-of-domain prompts. Moreover, these modules demonstrate strong generalization to languages they were not explicitly trained on. We pre-train a collection of UQ heads for popular LLM series, including Mistral, Llama, and Gemma 2. We publicly release both the code and the pre-trained heads. 1 1 1 https://github.com/IINemo/llm-uncertainty-head Figure 1: The architecture of uncertainty quantification heads. The example represents a text generated using an LLM, containing the hallucination 20 Grammy Awards highlighted in red. 1 Introduction Uncertainty quantification (UQ) Gal and Ghahramani 2016 ; Baan et al. 2023 ; Geng et al. 2024 ; Zhang et al. 2024a has become an increasingly important topic in natural language processing (NLP), particularly for addressing challenges with hallucinations Huang et al. 2025 and low-quality outputs of large language models (LLMs) Malinin and Gales 2021 ; Kuhn et al. 2023 ; Fadeeva et al. 2024 . UQ offers the potential to improve the safety and reliability of LLM-based applications by flagging highly uncertain generations. Such generations could be discarded or marked as untrustworthy, thus reducing the risk of misleading information reaching users ( Zhang et al. 2024a ; Zhang et al. 2024b ; Huang et al. 2024 ) . Contrary to other methods for detecting hallucinations that rely on external knowledge bases or additional LLMs Manakul et al. 2023 ; Min et al. 2023 ; Chen et al. 2023 , UQ methods assume that LLMs naturally encode information about their own limitations, and this self-knowledge can be efficiently accessed. There are many existing UQ techniques for well-defined tasks such as classification and regression Zhang et al. 2019 ; He et al. 2020 ; Xin et al. 2021 ; Wang et al. 2022 ; Vazhentsev et al. 2023 ; He et al. 2024a . However, applying UQ to text generation has unique challenges including ( i ) an infinite number of possible generations, which complicates the normalization of the uncertainty scores, ( ii ) potentially multiple correct answers with different surface forms Kuhn et al. 2023 , ( iii ) need to aggregate uncertainties across multiple interdependent predictions corresponding to generated tokens Zhang et al. 2023 , ( iv ) generated tokens not contributing to uncertainty equally, as some tokens represent auxiliary words Duan et al. 2024 , and ( v ) some sources of uncertainty being irrelevant for hallucination detection Fadeeva et al. 2024 . These challenges hinder the performance of classical unsupervised UQ techniques, and addressing them explicitly in a single method is quite difficult. Recently, researchers have proposed learning the aforementioned intricacies from the annotated data and developed supervised methods for UQ and hallucination detection Azaria and Mitchell 2023 ; Li et al. 2024 ; He et al. 2024b ; Chuang et al. 2024 . We continue this line of work by introducing pre-trained UQ heads: supervised auxiliary modules for LLMs that substantially enhance their ability to capture uncertainty compared to unsupervised UQ methods. Their strong performance stems from the powerful Transformer architecture in their design and informative features derived from LLM attention maps. These heads do not require re-training of the entire LLM and do not alter its outputs. In addition to their high performance, these methods maintain a relatively small memory and computational footprint, ensuring practical usability. Experimental evaluation shows that our uncertainty heads are highly robust and achieve state-of-the-art performance in claim-level hallucination detection across both in-domain and out-of-domain prompts, outperforming other supervised and unsupervised techniques. Moreover, these modules demonstrate strong generalization to languages they were not explicitly trained on. Training uncertainty quantification heads requires annotated hallucinations in LLM outputs. For constructing training data, we created an automatic pipeline for the annotation of hallucinations of LLM outputs, which allows us to scale our experiments and to pre-train uncertainty heads for various LLMs. We release a collection of pre-trained UQ heads for popular open-source instruction-following LLMs, including Llama series Dubey et al. 2024 , Gemma 2 Team et al. 2023 , and Mistral-v0.2 Jiang et al. 2023a . The contributions of this work are as follows: • We design a pre-trained uncertainty quantification head: a supplementary module for an LLM that yields substantially better performance for claim-level hallucination detection than classical unsupervised UQ methods and state-of-the-art supervised techniques. • We conduct a vast empirical investigation and find that uncertainty heads show good generalization across various domains and languages. We also compare various feature sets used for building supervised UQ modules. • We build and release a collection of pre-trained uncertainty quantification heads for popular series of open-source instruction-tuned LLMs: Llama, Gemma 2, Mistral. These modules could be seamlessly integrated into text generation code and be used as off-the-shelf hallucination detection tools. 2 Related Work Unsupervised methods. UQ for LLMs has recently experienced a surge of work, with early efforts focusing on unsupervised techniques such as information-based approaches Kuhn et al. 2023 ; Farquhar et al. 2024 , density-based scores Vazhentsev et al. 2022 , self-consistency methods Lin et al. 2023 ; Zhang et al. 2024a ; Qiu and Miikkulainen 2024 , and verbalized (reflexive) strategies Tian et al. 2023 . While unsupervised approaches have shown some potential, they still fall short of offering a strong solution to the problem of LLM hallucinations Vashurin et al. 2024 . Supervised methods. Recently, researchers have started exploring supervised UQ methods that leverage the internal states of LLMs during generation as features Azaria and Mitchell 2023 ; Slobodkin et al. 2023 ; Su et al. 2024 ; CH-Wang et al. 2024 ; He et al. 2024b . These methods achieve substantial performance gains over unsupervised approaches, especially for in-domain data. Azaria and Mitchell 2023 proposed one of the first methods of this kind called SAPLMA, where they trained a perceptron using activations from various layers to detect when the LLM “agrees” with false statements. Slobodkin et al. 2023 trained a linear model on hidden states to detect question “answerability”, effectively identifying unanswerable questions that typically lead to hallucinations. Factoscope He et al. 2024b implemented a Siamese model with a rich feature set that incorporates activation maps, token ranks, and probabilities from unembedding matrices across layers. They reported performance improvements over SAPLMA within the training domain, but encountered challenges with generalization to other domains. CH-Wang et al. 2024

Read the complete text