Why Semantic Entropy Fails: Geometry-Aware and Calibrated Uncertainty for Policy Optimization

Zheyuan Zhang, Kaiwen Shi, Han Bao, Zehong Wang, Tianyi Ma, Yanfang Ye · 2026

Read Why Semantic Entropy Fails: Geometry-Aware and Calibrated Uncertainty for Policy Optimization by Zheyuan Zhang, Kaiwen Shi, Han Bao, Zehong Wang, Tianyi Ma, Yanfang Ye (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-21801.txt.

Introduction

Zheyuan Zhang and co-authors give the first principled account, to their knowledge, of uncertainty signals as controls of gradient variance and signal quality in critic-free post-training. Entropy-based gauges for group methods like GRPO work unstably, and the authors trace the trouble to two gaps: an anisotropic gap and a calibration gap. Their Geometric-aware Calibrated Policy Optimization framework pairs geometry-aware measures of semantic disagreement with reward-based calibration tying uncertainty to strength. Experiments show it tracks gradient shifts well and improves post-training performance.

Catalog record

CREATORZheyuan Zhang, Kaiwen Shi, Han Bao, Zehong Wang, Tianyi Ma, Yanfang Ye
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE78386
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.21801

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Why Semantic Entropy Fails: Geometry-Aware and Calibrated Uncertainty for Policy Optimization Zheyuan Zhang, Kaiwen Shi, Han Bao, Zehong Wang, Tianyi Ma, Yanfang Ye Abstract Post-training has become central to improving reasoning and alignment in large language models, where critic-free models enable scalable learning from model-generated outputs but lack principled mechanisms to distinguish informative from noisy signals. Recent approaches leverage response-level measures as uncertainty signals to regulate group-based optimization methods such as GRPO. Yet their empirical success remains unstable and unclear in how they influence optimization dynamics. In this paper, we provide, to our knowledge, the first principled formulation that interprets uncertainty signals as mechanisms for characterizing and regulating gradient variance and learning signal quality. Based on both empirical and theoretical analysis, we identify two critical gaps of current entropy-based estimators: The anisotropic gap and The calibration gap . Motivated by this analysis, we propose Geometric-aware Calibrated Policy Optimization (GCPO) , a novel framework integrating geometry-aware measures to capture semantic disagreement with reward-based calibration to align uncertainty with learning signal strength. Experiments on multiple benchmarks show that our approach more faithfully tracks gradient variability and consistently improves post-training performance. Our results highlight the importance of designing uncertainty signals that are aligned with optimization dynamics, offering a principled perspective for robust post-training. 1 Introduction Despite the remarkable success of large-scale pretraining, recent Large Language Model (LLM) development increasingly shifts toward post-training to elicit reasoning, alignment, and task-specific behaviors [ 54 , 43 , 36 , 4 , 42 , 44 , 27 ] . In this regime, the central challenge is not only how to scale model capacity, but how to reliably extract learning signals from environments or model-generated outputs. A growing line of work, represented by Group Relative Policy Optimization (GRPO) [ 39 ] , adopts a critic-free, group-based formulation, where multiple responses are sampled for each input and optimized using relative rewards [ 29 , 31 , 1 ] . While this paradigm significantly simplifies supervision and improves scalability, it introduces a key limitation: all queries are treated uniformly, despite the fact that different queries can induce vastly different levels of uncertainty and learning signal quality [ 10 , 9 ] . To address this issue, a promising direction is to introduce uncertainty signals as a mechanism to select high quality model-generated outputs and regulate learning accordingly. Early approaches rely on token-level entropy to quantify uncertainty in the model distribution [ 13 , 12 , 45 ] . More recent work extends uncertainty estimation to the response level, forming a family of semantic entropy-type measures , including semantic entropy [ 5 , 57 ] , predictive entropy [ 11 ] , kernel-based variants [ 33 ] , and self-consistency scores [ 6 , 47 ] . These methods capture uncertainty over meaning rather than surface form, showing empirical benefits in improving reasoning performance and training stability. However, despite their empirical success, the role of such uncertainty signals in optimization remains poorly understood. Existing approaches largely treat uncertainty as a heuristic proxy, without explicitly modeling how it interacts with gradient updates and learning dynamics. As a result, it remains unclear whether these signals faithfully reflect gradient variance and learning signal quality. If poorly handled, such methods may simply suppress high-variance informative samples, thereby collapsing gradient signal and potentially leading to suboptimal or failed policy learning. In this paper, we provide, to the best of our knowledge, the first principled analysis that explicitly formulates uncertainty signals as mechanisms for characterizing and regulating gradient variance and learning signal quality. Based on both theoretical and empirical statistical analysis, we reveal two fundamental gaps of existing methods under this formulation: 1) The anisotropic gap , where entropy-based measures capture probability dispersion but ignore the geometric magnitude of semantic disagreement, although different errors may induce substantially different gradient directions. 2) The calibration gap , where entropy is decoupled from reward informativeness, and therefore cannot distinguish uninformative uncertainty from useful reward-differentiated variation. Motivated by the analysis, we then propose G eometric-aware reward- C alibrated P olicy O ptimization (GCPO) . Instead of treating uncertainty as purely distributional dispersion, our framework explicitly models uncertainty as a mechanism for characterizing and calibrating gradient variance with respect to learning signal quality . Specifically, we introduce geometry-aware measures, including Cosine Dispersion (CD) and Barycentric Transport (BoT) , to capture semantic disagreement beyond entropy, and further incorporate a Reward Dispersion (RD) module to align update strength with reward informativeness. Extensive experiments on QA and math reasoning benchmarks show that our proposed measures better align with sample-level gradient variance and consistently improve GRPO-style optimization over entropy-based baselines. These findings provide empirical evidence for our formulation, suggesting a principled guideline for designing uncertainty signals that preserve informative gradients while suppressing spurious variance in LLM post-training. Overall, our contributions can be summarized as follows. • We provide, to our best knowledge, the first principled analysis that explicitly formulates entropy-type uncertainty signals as mechanisms for characterizing and regulating gradient variance and learning signal quality, revealing two fundamental limitations: the anisotropic gap and the calibration gap , which explain both their effectiveness and failure modes. • We propose G eometric-aware reward- C alibrated P olicy O ptimization (GCPO) , a framework that models uncertainty as a mechanism for calibrating gradient variance , combining geometry-aware semantic structure with reward-based calibration to selectively suppress noisy variance while preserving informative learning signals. • Extensive experiments show that our method achieves stronger alignment and consistently improves GRPO-style training across multiple tasks, supporting a principled guideline for designing uncertainty signals preserving informative gradients while avoiding collapse. 2 Why Semantic Entropy-based Measures Fail? 2.1 Preliminary GRPO. Group Relative Policy Optimization (GRPO) is a critic-free policy gradient method for reasoning-oriented RL [ 39 ] . For each input x , it samples a group of responses Y=\{y_{i}\}_{i=1}^{G} and estimates the gradient using normalized rewards: \hat{g}(x)=\frac{1}{G}\sum_{i=1}^{G}\hat{A}_{i}\,\nabla_{\theta}\log\pi_{\theta}(y_{i}\mid x),\quad\hat{A}_{i}=\frac{R(y_{i})-\mu_{R}}{\sigma_{R}+\epsilon}. (1) Why Focus on GRPO? GRPO is known for its strong and widely adopted optimization paradigm and its critic-free, rollout-based formulation makes it particularly well-aligned with the study of both token-level and response-level uncertainty . Unlike actor-critic methods that rely on learned value functions for variance reduction, GRPO exposes gradient variance at the query level and provides a natural setting for analyzing how response-level uncertainty influences optimization. Semantic Entropy-type Measures. Token-level entropy measures uncertainty over surface forms, and therefore is sensitive to paraphrastic variation that does not affect meaning. As a result, it often overestimates uncertainty in reasoning tasks where multiple responses differ only in wording but share the same underlying semantics [ 5 , 33 ] . To address this limitation, semantic entropy [ 23 ] measures uncertainty at the response level by aggregating semantically equivalent outputs, thereby focusing on variability across distinct meanings rather than surface realizations. This formulation has led to a broader class of response-level entropy-like methods, including predictive entropy, kernel-based variants, and other consistency-based scores, all of which quantify uncertainty via grouped responses rather than token-level distributions. Formally, we denote this family by H_{sem} . For instance, in semantic entropy, H_{\mathrm{sem}}(x)=-\sum_{k=1}^{K}C_{k}\log C_{k} , where C_{k} denotes the probability mass of semantic cluster k . In predictive entropy, H_{\mathrm{sem}}(x)=-\sum_{k=1}^{K}q_{k}\log q_{k} , where q_{k} represents the mass of prediction groups. In this work, for simplicity,

Read the complete text