Retrieval-Augmented Linguistic Calibration

Yi-Fan Yeh, Linwei Tao, Minjing Dong, Tao Huang, Jialin Yu, Philip Torr, Chang Xu · 2026

Read Retrieval-Augmented Linguistic Calibration by Yi-Fan Yeh, Linwei Tao, Minjing Dong, Tao Huang, Jialin Yu, Philip Torr, Chang Xu (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-19344.txt.

Introduction

Yi-Fan Yeh and colleagues study linguistic calibration: the trust words language models use, 'I believe' and 'probably.' Plain scores hide how readers take such hedges, and mixed cues add to the puzzle. They treat trust talk as a spread of likely odds for a claim, and add faithfulness, scored by a new gap measure, as a second test. Their rewrite pipeline, fed by retrieved examples, puts tuned cues back into plain words, lifting faithfulness by up to 66 percent and tuning by up to 58 percent on quiz sets.

Catalog record

CREATORYi-Fan Yeh, Linwei Tao, Minjing Dong, Tao Huang, Jialin Yu, Philip Torr, Chang Xu
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE77584
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.19344

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Retrieval-Augmented Linguistic Calibration Yi-Fan Yeh, Linwei Tao, Minjing Dong, Tao Huang, Jialin Yu, Philip Torr, Chang Xu Abstract Linguistic cues such as “I believe” and “probably” offer an intuitive interface for communicating confidence, yet a generalisable, principled calibration framework for linguistic confidence expressions remains underexplored. In particular, co-occurring linguistic cues, contextual variation, and subjective audience interpretation pose unique challenges. We therefore model linguistic confidence as a distribution over plausible perceived probability values that a statement is correct, capturing interpretation variability that scalar representations discard. Within this distributional framework, we introduce faithfulness as a complementary evaluation dimension and present Faithfulness Divergence (FD), an information-theoretic metric quantifying the surprise induced in audience beliefs upon truth revelation. Building on these foundations, we present Retrieval-Augmented Linguistic Calibration (RALC), a lightweight post-hoc pipeline that propagates calibrated confidence signals back into natural language via retrieval-augmented rewriting. Across three QA benchmarks and five LLM families, RALC improves in-domain faithfulness and calibration up to 66% and 58%, respectively, outperforming black-box and grey-box calibration baselines. 1 Introduction Reliable confidence estimation is fundamental to the trustworthy deployment of large language models (LLMs) in human decision-making pipelines [ 29 ] . Without well-calibrated confidence signals, users risk over-relying on model outputs that hallucinate or fail silently [ 13 ] , underscoring the need for confidence frameworks that are both scientifically rigorous and interpretable by human users. Existing confidence estimation methods represent confidence as scalar probability values, including token-level probability [ 16 ] , semantic uncertainty [ 5 ] , and verbalised scores [ 18 , 34 ] . However, humans struggle to reason accurately with numerical probabilities [ 41 ] , motivating the use of linguistic markers such as “may” or “likely” as more natural confidence interfaces. Prior work demonstrates that such markers retain evaluative signal [ 38 , 31 ] ; however, treating them as scalars discards the inherent subjectivity of linguistic interpretation: different readers map the same expression to different perceived probability values [ 32 ] . We address this gap by modelling linguistic confidence as a distribution over plausible perceived probability values that a statement is correct, where perception arises from readers’ interpretations of the full linguistic content of the statement rather than from a discrete mapping of individual vocabulary items. Treating linguistic confidence as a surrogate for statement correctness induces a binary classification view, in which confidence corresponds to the predictive probability of the true class. Drawing a parallel to evidential deep learning, which models class probabilities with Dirichlet distributions to capture second-order predictive uncertainty [ 27 ] , we formalise linguistic confidence for binary correctness using its natural binary special case: a Beta distribution over perceived confidence scores that a statement is correct, where the mean captures the central tendency of perceived confidence across readers and the concentration encodes the strength of agreement. The standard measure of confidence quality is calibration, assessed through population-level expected calibration error (ECE) [ 8 , 35 ] , quantifying the alignment between confidence and accuracy in expectation. Instance-level metrics such as the Brier score [ 3 ] and negative log likelihood offer pointwise assessment in the classical scalar setting, yet their distributional generalisations still fail to encode variance as the strength of agreement by readers. We therefore introduce faithfulness as a complementary dimension of confidence evaluation and present Faithfulness Divergence (FD), a concentration-weighted Bayesian updating cost that quantifies the information-theoretic surprise to confidence beliefs upon truth revelation. Calibration in linguistic space remains crucial yet largely unsolved. Classical post-hoc calibration methods, including temperature scaling [ 8 ] , Platt scaling [ 25 ] , histogram binning [ 39 ] , isotonic regression [ 40 ] , Beta calibration [ 14 ] , and distribution-matching approaches [ 28 , 20 ] , operate entirely in numerical space and provide no mechanism for propagating calibrated signals back into language. Prompt-conditioned hedging strategies offer a linguistic alternative, yet function as black-box procedures with no principled control over the output [ 38 ] . The closest related work performs discrete hedging word confidence profiling and remapping at the word level in a specialised domain [ 35 ] , overlooking the co-occurrence of multiple linguistic cues within a statement and their contextual interactions. Consequently, a generalisable, continuous, and lightweight post-hoc framework that provides principled guidance for hedging expressions remains underexplored. To address this gap, we introduce Retrieval-Augmented Linguistic Calibration (RALC), a post-hoc pipeline that operates directly in linguistic space to transform raw LLM responses into calibrated and faithful outputs. The pipeline applies Platt scaling [ 25 ] on confidence distribution means whilst preserving distributional concentration, and propagates calibrated distributions to language through retrieval-augmented LLM rewriting, employing the retrieval-augmented generation paradigm [ 17 ] . Furthermore, RALC is compatible with diverse upstream confidence signals beyond linguistic confidence, including token probability and semantic uncertainty. Our contributions are as follows: 1. We formalise linguistic confidence as a distribution over plausible perceived probability values that a statement is correct, capturing the interplay of linguistic cues and contexts beyond discrete expression mapping and scalar quantification. 2. We introduce faithfulness as a new dimension of confidence evaluation and present Faithfulness Divergence (FD), an instance-level metric that quantifies, as information-theoretic surprise, how faithfully a confidence distribution accounts for the ground-truth correctness outcome. 3. We introduce a generalisable retrieval-augmented linguistic confidence calibration pipeline that effectively improves faithfulness and calibration in linguistic space and is compatible with diverse confidence estimation signals. We evaluate the framework across multiple LLM families and QA benchmarks, including MMLU [ 9 ] , SQuAD 2.0 [ 26 ] , and TruthfulQA [ 19 ] . Results demonstrate near-lossless information transfer through the calibration pipeline and substantial improvements in both calibration and faithfulness across models and benchmarks, outperforming the prompt-based calibration baselines. 2 Related work LLM confidence estimation Existing confidence estimation methods predominantly represent confidence as scalar probability values, including token-level probability aggregation [ 16 , 4 ] and consistency-based approaches that infer confidence from the semantic support landscape across repeated samples [ 36 , 5 ] . Whilst recent work demonstrates that linguistic cues in model responses preserve evaluative signal as confidence surrogates [ 38 , 32 ] , their scalar quantification overlooks the inherently subjective nature of linguistic interpretation. 35 take a step towards distributional representations by mapping individual discrete hedging words to confidence distributions; however, their approach targets word-level remapping rather than statement-level confidence, where multiple linguistic cues co-occur and interact with context. 10 jointly model confidence and correctness as distributions over ambiguous long-form generation contexts, which is orthogonal to our objective of binary classification with predictive probability distributions. Confidence evaluation Expected Calibration Error (ECE) is the dominant metric for evaluating confidence, measuring alignment between scalar confidence scores and accuracy in both classical [ 8 ] and language model settings [ 42 ] . Extensions based on entropy [ 30 ] , variance [ 33 ] , and distributional generalisation [ 35 ] remain ECE-based and rely on local aggregation, discarding full distributional information at the instance level. Instance-level scoring such as the Brier score [ 3 ] and negative log likelihood similarly do not capture variance-scaled misalignment. Confidence calibration Classical post-hoc calibration methods, including Platt scaling [ 25 ] , histogram binning [ 39 ] , isotonic

Read the complete text