Unsupervised Confidence Calibration for Reasoning LLMs from a Single Generation

\Name Thomas Zollo \Email [email protected] \addr Columbia University \Name Jimmy Wang \Email [email protected] \addr Columbia University \Name Richard Zemel \Email [email protected] \addr Columbia University · 2026

Read Unsupervised Confidence Calibration for Reasoning LLMs from a Single Generation by \Name Thomas Zollo \Email [email protected] \addr Columbia University \Name Jimmy Wang \Email [email protected] \addr Columbia University \Name Richard Zemel \Email [email protected] \addr Columbia University (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-19444.txt.

Introduction

Zollo, Wang, and Zemel introduce Binary Agreement for Self-consistency Calibration, a method for calibrating confidence in reasoning language models without correctness labels. Such models often lack steady scores for reliable use, notably with small sampling budgets. The key insight is that agreement between an anchor response and one independent probe is a Bernoulli observation whose mean is the anchor's latent self-consistency. Pooling these cases across unlabeled queries supports both fixing current scores and learning sharper scores first, using as few as two responses per query.

Catalog record

CREATOR\Name Thomas Zollo \Email [email protected] \addr Columbia University \Name Jimmy Wang \Email [email protected] \addr Columbia University \Name Richard Zemel \Email [email protected] \addr Columbia University
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE68077
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.19444

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Unsupervised Confidence Calibration for Reasoning LLMs from a Single Generation \Name Thomas Zollo \Email [email protected] \addr Columbia University \Name Jimmy Wang \Email [email protected] \addr Columbia University \Name Richard Zemel \Email [email protected] \addr Columbia University Abstract Reasoning language models can solve increasingly complex tasks, but struggle to produce the calibrated confidence estimates necessary for reliable deployment. Existing calibration methods usually depend on labels or repeated sampling at inference time, making them impractical in many settings. We introduce a method for unsupervised confidence calibration of reasoning LLMs when only a single generation is available at inference time. Our approach uses offline sampling on unlabeled data to derive a self-consistency-based proxy target, then distills this signal into a lightweight deployment-time confidence predictor. In a broad evaluation across 5 math and question-answering tasks using 9 reasoning models, our method substantially outperforms baselines, including under distribution shift, and improves downstream performance in selective prediction and simulated downstream decision-making. 1 Introduction Large language models (LLMs) with reasoning abilities have achieved high accuracy in tasks such as mathematical problem solving and scientific question answering. In consequential settings, however, reliable deployment also requires confidence estimates that reflect the probability that a model’s answer is correct ( Bommasani et al. 2022 ; Yang et al. 2026 ) . This property, known as confidence calibration ( Guo et al. 2017 ) , is widely regarded as a key ingredient for making deep learning systems reliable in practice. Despite its importance, modern LLMs often remain poorly calibrated, making reliable confidence estimation an important unresolved challenge ( Kadavath et al. 2022 ; Leng et al. 2024 ; Damani et al. 2025 ) . Although progress has been made, calibration for reasoning LLMs remains difficult to obtain in many of the settings where it would be most useful. Most existing methods, whether based on post-hoc recalibration or RL-style training, require labeled data ( Zadrozny and Elkan 2001 ; Guo et al. 2017 ; Kumar et al. 2020 ; Band et al. 2024 ; Leng et al. 2024 ; Damani et al. 2025 ) , while many uncertainty estimation methods rely on repeated generations or other expensive inference-time procedures ( Chen et al. 2024 ; Kuhn et al. 2023 ; Duan et al. 2024 ) . In practice, however, labeled calibration data is often unavailable or too expensive to obtain ( Liu et al. 2022 ) . Repeated-sampling-based confidence estimation is also often impractical: edge devices face strict resource constraints ( Wang et al. 2025a ) , while cloud-served systems may be unable to incur the cost or delay of generating many samples per query. These limitations arise naturally in real applications. Consider a multilingual math tutor running on students’ phones (see Figure 1 ): the system needs calibrated confidence to decide whether to answer directly, offer a hint, or defer to a teacher. In such a setting, labeled data may be difficult to obtain, since collecting correctness labels across many languages, curricula, and student populations would require substantial expert annotation. At the same time, deployment on consumer devices imposes memory, latency, and battery usage constraints, limiting both parallel decoding and the ability to generate multiple samples sequentially. Similar challenges arise in mobile health triage tools, workplace assistants, or embedded accessibility systems, where trustworthy confidence estimates matter but labeled data collection and expensive test-time computation are often out of reach. Further, as LLMs are increasingly deployed as components of agentic systems, lightweight confidence estimates will also be crucial for deciding when to act, defer, invoke tools, or escalate. Figure 1 : We study calibration for reasoning LLMs in a practical deployment regime where labeled calibration data is unavailable and repeated sampling at test time is too costly. In our motivating example of a multilingual on-device tutor, the system must decide whether to answer directly or defer, but cannot rely on expert-labeled data or many generations per query. Our approach instead uses unlabeled questions offline: we sample multiple responses, compute answer self-consistency, and use this signal to train a confidence calibrator. At deployment, the model produces only a single response for a new question, and the calibrator maps that response to a calibrated confidence estimate that can support downstream decisions. In this work, we propose an approach to confidence calibration for reasoning LLMs in a challenging, underserved deployment regime: no labeled calibration data available and only one generation possible. Our key idea is to treat repeated sampling not as a deployment-time inference procedure, but as an offline source of unlabeled supervision. Concretely, we use repeated sampling on unlabeled examples to estimate a self-consistency signal, then amortize that signal into a lightweight predictor that maps a single generated response to an answer-level confidence estimate. This yields a practical route to calibration when labels are unavailable and multi-sample inference is not possible. Requiring only offline access to model outputs, our method is also applicable in black-box settings, for example when a proprietary model is held behind an API. We evaluate this approach in comprehensive experiments across 5 math and question-answering tasks using 9 models from the Qwen, DeepSeek, and Nemotron families. Empirically, we show three main results: 1. Self-consistency is generally well-calibrated with respect to correctness in reasoning LLMs. 2. This signal can be distilled into a single-generation confidence predictor that substantially outperforms relevant unsupervised baselines, including under distribution shift and in black-box settings. 3. The resulting confidence estimates improve downstream outcomes in applications such as selective prediction (without producing any tokens) and enhancing the calibration of stronger models (e.g., GPT-4o-mini). Despite the stringent constraint of one-generation deployment with no labeled data, our method often narrows much of the gap to stronger supervised or multi-sample approaches; it also works robustly across different temperatures and with a relatively small number of samples (e.g., 5-10). Overall, our results show that calibration for reasoning LLMs can be learned from unlabeled data and deployed with only one generation, expanding the range of settings where calibrated reasoning systems can be used. 2 Related Work Most prior work on calibration in deep learning has focused on classification ( Naeini et al. 2015 ; Guo et al. 2017 ; Ovadia et al. 2019 ; Kumar et al. 2020 ; Minderer et al. 2021 ; Wang et al. 2021 ; Zollo et al. 2024 ) , and many of these ideas extend naturally to LLM settings with single-token outputs from a finite discrete set ( Desai and Durrett 2020 ; Xiao et al. 2022 ) . Calibration in LLMs becomes substantially more difficult when moving from classification to open-ended generation, where outputs are multi-token, often admit many valid surface forms, and may be supported by reasoning paths ( Kuhn et al. 2023 ; Yang et al. 2026 ) . As a result, work in this area has explored a range of answer-level confidence signals for LLMs, including token-probability-based scores ( Kadavath et al. 2022 ) , self-consistency ( Lyu et al. 2025 ; Si et al. 2023 ) , and verbalized confidence expressions ( Tian et al. 2023 ; Xiong et al. 2024 ; Yoon et al. 2025 ) . Recent work suggests that post-training with reinforcement learning can induce undesirable side effects related to calibration, including overconfidence and poor abstention behavior ( Leng et

Read the complete text