Calibrating LLMs with Semantic-level Reward

Fengfei Yu, Ruijia Niu, Dongxia Wu, Yian Ma, Rose Yu · 2026

Read Calibrating LLMs with Semantic-level Reward by Fengfei Yu, Ruijia Niu, Dongxia Wu, Yian Ma, Rose Yu (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-15588.txt.

Introduction

Fengfei Yu, Ruijia Niu, Dongxia Wu, Yian Ma, and Rose Yu fix calibration in large language models. Binary right-wrong rewards ignore confidence, so bold errors pass. Past fixes train models to voice confidence scores, but token scores clash across same-meaning wordings. Their CSR frame calibrates in meaning space, with no voiced channel. It spurs accord among right rollouts and damps accord among wrong ones. On four question sets it cuts calibration error by up to 40 percent and lifts ranking quality by up to 31 percent over voiced rivals.

Catalog record

CREATORFengfei Yu, Ruijia Niu, Dongxia Wu, Yian Ma, Rose Yu
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE69490
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.15588

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Calibrating LLMs with Semantic-level Reward Fengfei Yu, Ruijia Niu, Dongxia Wu, Yian Ma, Rose Yu Abstract As large language models (LLMs) are deployed in consequential settings such as medical question answering and legal reasoning, the ability to estimate when their outputs are likely to be correct is essential for safe and reliable use, requiring well-calibrated uncertainty. Standard reinforcement learning with verifiable rewards (RLVR) trains models with a binary correctness reward that is indifferent to confidence, providing no penalty for confident but wrong predictions and thereby degrading calibration. Recent work addresses this by training models to produce verbalized confidence scores alongside answers and rewarding agreement with correctness. However, verbalized confidence is calibrated at the token level and thus exhibits inconsistency across textual variations with same semantic meaning. We propose Calibration with Semantic Reward (CSR) , a framework that calibrates language models directly in semantic space without a verbalized confidence interface. CSR combines the correctness reward with a novel semantic calibration reward that encourages exploitation among correct rollouts by promoting semantic agreement, and exploration among incorrect ones by discouraging spurious consistency. Experiments across three model families on HotpotQA (in-distribution) and TriviaQA, MSMARCO, and NQ-Open (out-of-distribution) show that CSR consistently achieves lower ECE and higher AUROC than verbalized-confidence baselines across nearly all settings, reducing ECE by up to 40 % 40\% and improving AUROC by up to 31 % 31\% over verbalized-confidence baselines, with calibration behavior generalizing robustly across all four evaluation settings. 1 1 footnotetext: Department of Computer Science and Engineering, University of California San Diego, La Jolla, California, USA 2 2 footnotetext: Halıcıoğlu Data Science Institute, University of California San Diego, La Jolla, California, USA 3 3 footnotetext: Department of Statistics, Stanford University, Stanford, California, USA † † footnotetext: ∗ Equal contribution. Correspondence to: Rose Yu \langle [email protected] \rangle . 1 Introduction Reinforcement learning with verifiable rewards (RLVR) has become the dominant training paradigm for reasoning language models, achieving strong performance across question answering, mathematical reasoning, and multi-step inference ( 23 ) . The standard RLVR objective uses a binary correctness reward: a rollout receives a positive signal when the generated answer matches the ground truth and zero otherwise.While effective for improving accuracy, this formulation does not account for model confidence in the generated answer. A model that guesses incorrectly with high confidence receives the same signal as one that abstains. This indifference structurally incentivizes overconfident guessing, as expressing certainty carries no additional cost. In high-stakes deployment settings such as medical question answering and legal reasoning, a reliable system must not only produce correct answers but also express calibrated uncertainty to support sound downstream decisions. Empirically, RL training is known to worsen calibration and increase hallucination rates, with reasoning models exhibiting substantially greater overconfidence than their base counterparts ( 9 ; 3 ) . Recent work addresses this by training models to produce an explicit confidence score alongside each answer and rewarding agreement between the score and empirical correctness. Methods such as Rewarding Doubt (RD) ( 2 ) , RLCR ( 3 ) , SaySelf ( 26 ) , ConfTuner ( 14 ) , and LACIE ( 20 ) operate within this paradigm, attaching the training signal to a verbalized confidence interface that the model generates in response to explicit prompting. (a) Single-sample distribution (b) Inconsistency (c) Variability Figure 1: Verbal confidence is not always stable. (a) For a single sampled answer, the decoded confidence is inherently uncertain, with alternative values appearing with substantial probability. (b) Multiple responses expressing the same answer yield inconsistent confidence scores. (c) Grouping rollouts by semantic equivalence count, verbalized confidence still varies widely within each group. Experimental details are provided in Appendix B.1 . However, existing work has identified verbalized confidence as an unreliable proxy for model uncertainty ( 25 ; 6 ; 27 ) . The reported score depends on the answer text and prompt format rather than the distribution of meanings produced under free sampling. Consequently, the score can shift with prompt phrasing or instruction style without corresponding change in answer behavior. Fig. 1 illustrates this concretely under RLCR. Panel (a) shows that even for a single sampled answer, the decoded confidence is not a stable quantity: alternative values appear in the output distribution with substantial probability. Panel (b) shows that multiple responses producing the same answer report inconsistent confidence scores, confirming that verbalized confidence varies across semantically equivalent outputs. Panel (c) quantifies this effect at scale: grouping rollouts by semantic equivalence count, confidence still varies widely within each group, failing to track the degree of semantic agreement. Together, these observations indicate that existing methods calibrate a reporting interface rather than the model’s underlying uncertainty over answer meanings. In this work, we propose to calibrate language models directly in semantic space, bypassing the verbalized confidence interface. We frame calibration in terms of semantic-level uncertainty : the distribution of meanings the model produces when sampled repeatedly from the same question. A model is semantically well-calibrated when correct predictions form a concentrated semantic cluster and incorrect predictions yield dispersed, semantically diverse outputs. Uncertainty can then be estimated post hoc from the answer distribution without any confidence-reporting instruction. To this end, we introduce Calibration with Semantic Reward (CSR) , a fine-tuning framework that aligns pairwise semantic agreement across sampled rollouts with per-rollout correctness. CSR shapes the model’s answer distribution such that semantic entropy ( 4 ) , computed by clustering semantically equivalent outputs, becomes a reliable uncertainty signal. A curriculum learning schedule stabilizes the joint optimization of accuracy and calibration. To summarize, our contributions include: • Semantic calibration , a formulation of LLM calibration grounded in semantic-level uncertainty rather than verbalized confidence, providing a principled alternative to prompt-dependent confidence-reporting interfaces. • CSR , a reinforcement learning framework that aligns pairwise semantic agreement across sampled rollouts with correctness, enabling models to learn calibrated semantic uncertainty without explicit confidence supervision. • Empirical validation across three model families showing that CSR reduces ECE by up to 42% and improves AUROC by up to 16% over the base model, while maintaining competitive accuracy and generalizing to out-of-distribution datasets. 2 Related Work Sampling-based uncertainty estimation. A prominent family of post-hoc uncertainty methods probes the model’s output distribution without modifying the model or adding a confidence interface. Token-level proxies derive confidence from generation likelihoods, such as normalized entropy ( 16 ) , perplexity ( 4 ) , or the P(\text{true}) score obtained by asking the model to self-evaluate a candidate answer ( 12 ) . Sampling-based proxies instead treat uncertainty as dispersion across repeated draws: self-consistency ( 22 ) uses agreement among independently sampled reasoning chains as an implicit correctness signal, while semantic entropy ( 4 ) refines this by clustering rollouts by semantic equivalence , providing an uncertainty estimate invariant to paraphrastic variation. RL for verbalized uncertainty calibration. Prompting models to verbalize their confidence is an appealing approach to uncertainty estimation ( 15 ; 21 ) , but verbalized scores are unreliable in practice: they depend on prompt format and answer phrasing, exhibit systematic overconfidence, and need not reflect the distribution of answer meanings produced under free sampling ( 24 ; 27 ; 6 ) . Standard RLVR training further degrades calibration, as binary correctness rewards are indifferent to confidence and structurally

Read the complete text