Calibrating Verbal Uncertainty as a Linear Feature to Reduce Hallucinations
Ziwei Ji, Lei Yu, Yeskendir Koishekenov, Yejin Bang, Anthony Hartshorn, Alan Schelten, Cheng Zhang, Pascale Fung, Nicola Cancedda · 2025
Read Calibrating Verbal Uncertainty as a Linear Feature to Reduce Hallucinations by Ziwei Ji, Lei Yu, Yeskendir Koishekenov, Yejin Bang, Anthony Hartshorn, Alan Schelten, Cheng Zhang, Pascale Fung, Nicola Cancedda (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-14477.txt.
Introduction
Ziwei Ji and eight co-authors study how language models express doubt in this 2025 paper. Models often state false claims in the same confident tone as true ones, and such overconfident hallucinations mislead users. The authors ask whether the model's words about its own uncertainty can be measured and adjusted.
They find that verbal uncertainty, how unsure the wording sounds, is steered by a single linear feature inside the model's representation space. That feature matches the model's real, semantic uncertainty only moderately. The gap between the two turns out to predict hallucinations better than semantic uncertainty alone. By nudging the verbal feature at run time, the team cut confident hallucinations on short answers by about thirty percent on average. The result suggests a cheap control knob for making models sound as unsure as they actually are.
Catalog record
| CREATOR | Ziwei Ji, Lei Yu, Yeskendir Koishekenov, Yejin Bang, Anthony Hartshorn, Alan Schelten, Cheng Zhang, Pascale Fung, Nicola Cancedda |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 73 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-14477
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Calibrating Verbal Uncertainty as a Linear Feature to Reduce Hallucinations In this paper, we employ the term ‘verbal’ to mean ‘pertaining to words rather than meaning or substance,’ as opposed to ‘spoken rather than written’ (refer to Merriam-Webster’s definitions: https://www.merriam-webster.com/dictionary/verbal ). Readers may substitute ‘verbal uncertainty’ with ‘expressed uncertainty’ throughout the text if they find it preferable. The code is available at https://github.com/facebookresearch/verbal_uncertainty_feature_calibration . Ziwei Ji, Lei Yu, Yeskendir Koishekenov, Yejin Bang, Anthony Hartshorn, Alan Schelten, Cheng Zhang, Pascale Fung, Nicola Cancedda Abstract LLMs often adopt an assertive language style also when making false claims. Such “overconfident hallucinations” mislead users and erode trust. Achieving the ability to express in language the actual degree of uncertainty around a claim is therefore of great importance. We find that “verbal uncertainty” is governed by a single linear feature in the representation space of LLMs, and show that this has only moderate correlation with the actual “semantic uncertainty” of the model. We apply this insight and show that (1) the mismatch between semantic and verbal uncertainty is a better predictor of hallucinations than semantic uncertainty alone and (2) we can intervene on verbal uncertainty at inference time and reduce confident hallucinations on short-form answers, achieving an average relative reduction of ~30%. † † affiliation: Meta FAIR † † affiliation: Meta GenAI † † affiliation: Hong Kong University of Science and Technology † † affiliation: University of Toronto † † contribution: Equal Contribution † † contribution: Work done during Internship at Meta FAIR † † date: August 11, 2026 † † correspondence: [email protected]; [email protected]; {pascalefung; ncan}@meta.com 1 Introduction Despite their remarkable capability in utilizing their internal knowledge, LLMs often suffer from hallucinations, stating or implying facts that are not supported neither by their input nor by their training data ( Ji et al. 2022 ; Xiao and Wang 2021 ; Bang et al. 2023 ; Xiong et al. 2023 ) . The issue is exacerbated when models produce hallucinations using language that suggests high confidence. Such overconfidence can cause users to rely too heavily on these responses ( Zhou et al. 2024a ; Kim et al. 2024 ) , possibly resulting in harm, loss of trust in the model, or both. While enhancing a model’s ability to generate accurate knowledge is important, it is inevitable that models have knowledge gaps. In such cases, it is important for models to express uncertainty about their knowledge or altogether abstain from answering ( Tomani et al. 2024 ; Feng et al. 2024 ; Zhou et al. 2023 ; Zhang et al. 2024b ) . We refer to this expression as ”verbal uncertainty (VU)” (see § 2.2 ). When faced with questions close to their knowledge boundary, they should qualify their answers with expressions such as: “I am not sure but …”, and when the answer is squarely beyond such boundary, they should reply: “I don’t know”. However, LLMs lack reliable mechanisms to convey their intrinsic confidence in the correctness of generated content by means of the degree of doubt expressed in their outputs ( Zhou et al. 2024b ) . Figure 1: Illustration of our framework. We discover a linear verbal uncertainty feature (VUF) that controls model expression of uncertainty, and apply this insight to: (1) detect hallucinations arising from the miscalibration between high semantic uncertainty (SU) and low verbal uncertainty (VU); (2) mitigate hallucinations by intervening on activations along the VUF direction at inference time to make VU more aligned with model’s SU. For example, when asked ”What is the 29th largest city in England?”, the model initially responds with ”It’s Bournemouth”, exhibiting high SU and VU. By applying the VUF to intervention, we improve the VU to better align with the SU and the response becomes ”Hmm, maybe Bournemouth?” – demonstrating a nuanced expression of uncertainty. Figure 2: Evidence of verbal-semantic uncertainty miscalibration. This plot presents the Kernel Density Estimation (KDE) for samples from TriviaQA, categorized into four classes. These classes are based on the correctness of the answers generated by Llama3.1 and the consistency in abstaining. Miscalibration is indicated by high Semantic Entropy (proxy for SU) & low VU in hallucinated answers (red), and low SU & high VU in consistently abstained answers (blue). Our work begins with the analysis of the model representation space. We show that, similarly to refusal ( Arditi et al. 2024 ) and other behaviours ( Zou et al. 2023 ) , the degree of uncertainty expressed by a model is mediated by a single direction, which we call the “Verbal Uncertainty Feature” (VUF). Specifically, we show that the hidden states of input questions answered with low and high verbal uncertainty can be linearly separated. This allows us to use contrastive pairs ( Burns et al. 2023 ; Panickssery et al. 2023 ) to identify a single difference-in-means direction that can be intervened upon to control model expression of uncertainty. We next leverage our findings to study hallucinations through the lens of uncertainty features. As illustrated in Figure 2 , we highlight the misalignment between verbal uncertainty and the uncertainty about what meaning to convey in model outputs, i.e. semantic uncertainty, as an important factor contributing to hallucinations. We propose a novel method to detect model hallucinations that incorporates both verbal and semantic uncertainty information, and show that it outperforms detection methods that rely solely on semantic uncertainty. Next, we propose a method, M echanistic U ncertainty C alibration (MUC), that steers LLM activations using VUF to calibrate verbal uncertainty with the model semantic uncertainty. We demonstrate that MUC effectively reduces model confident hallucinations, achieving an average relative reduction of 29.6% in short-form QA tasks. It also induces nuances expressions of uncertainty and achieves a 28.4% improvement in the alignment between verbal and semantic uncertainty. Our main contributions are therefore threefold: 1. We discovered that verbal uncertainty is mediated by a single direction in representation space, i.e. a linear Verbal Uncertainty Feature (VUF) (§ 3 ). 2. We detect hallucinations arising from the misalignment between high semantic and low verbal uncertainty by integrating both types of uncertainty (§ 4.1 ). 3. We introduce Mechanistic Uncertainty Calibration (MUC), an inference-time intervention mechanism using VUFs to calibrate verbal uncertainty with semantic uncertainty, thereby mitigating hallucinations (§ 4.2 ). In addition, we introduce methods to quantify verbal uncertainty and metrics that help characterize the calibration between the two kinds of uncertainty without requiring the model to output numerical confidence estimates. Overall, this work contributes to a better understanding of LLMs, shows how to reduce hallucinations and thereby make LLMs more trustworthy. 2 Background and Motivation The miscalibration of semantic and verbal uncertainty triggers overconfident hallucinations. To bring the discussion in a quantitative framework, we introduce in this section some definitions and measures. 2.1 Semantic Uncertainty Semantic Uncertainty (SU) refers to the intrinsic uncertainty of an agent in the semantic meaning expressed by a statement. It reflects the confidence level of a model’s prediction, focusing on its meaning and disregarding paraphrastic variations ( Lin et al. 2022 ; Kadavath et al. 2022 ; Mielke et al. 2022 ) . We follow Kuhn et al. 2023 in operationalizing this concept in terms of Semantic Entropy that can be calculated as follows: for a given question,