Balancing Classification and Calibration Performance in Decision-Making LLMs via Calibration Aware Reinforcement Learning
Duygu Nur Yaldiz, Evangelia Spiliopoulou, Zheng Qi, Siddharth Varia, Srikanth Doss, Nikolaos Pappas · 2026
Read Balancing Classification and Calibration Performance in Decision-Making LLMs via Calibration Aware Reinforcement Learning by Duygu Nur Yaldiz, Evangelia Spiliopoulou, Zheng Qi, Siddharth Varia, Srikanth Doss, Nikolaos Pappas (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-13284.txt.
Introduction
Duygu Nur Yaldiz and co-authors teach decision models to be right and to know when they may be wrong. Large language models now sit in decision chains. In such chains a right pick is not all that counts. The trust score that comes with the pick must be sound too. A sound score lets a down stream tool know when to trust the model and when to pass to a back up.
Past teach can lift pick skill while it bends trust scores, or the next way round. Their teach is calibration aware reward learning. The reward pays for both a right pick and a trust score that fits the true odds. For teams that put models in care, funds, or law chains, that pair is the trust base.
Catalog record
| CREATOR | Duygu Nur Yaldiz, Evangelia Spiliopoulou, Zheng Qi, Siddharth Varia, Srikanth Doss, Nikolaos Pappas |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 50 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2601-13284
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Balancing Classification and Calibration Performance in Decision-Making LLMs via Calibration Aware Reinforcement Learning Duygu Nur Yaldiz, Evangelia Spiliopoulou, Zheng Qi, Siddharth Varia, Srikanth Doss, Nikolaos Pappas Abstract Large language models (LLMs) are increasingly deployed in decision-making tasks, where not only accuracy but also reliable confidence estimates are essential. Well-calibrated confidence enables downstream systems to decide when to trust a model and when to defer to fallback mechanisms. In this work, we conduct a systematic study of calibration in two widely used fine-tuning paradigms: supervised fine-tuning (SFT) and reinforcement learning with verifiable rewards (RLVR). We show that while RLVR improves task performance, it produces extremely overconfident models, whereas SFT yields substantially better calibration, even under distribution shift, though with smaller performance gains. Through targeted experiments, we diagnose RLVR’s failure, showing that decision tokens act as extraction steps of the decision in reasoning traces and do not carry confidence information, which prevents reinforcement learning from surfacing calibrated alternatives. Based on this insight, we propose a calibration-aware reinforcement learning formulation that directly adjusts decision-token probabilities. Our method preserves RLVR’s accuracy level while mitigating overconfidence, reducing ECE scores up to 9 points. 1 Introduction Large Language Models (LLMs) are increasingly deployed in high-stakes decision-making systems such as content moderation, medical assistance, financial services, and legal frameworks 4 ; 8 . In these critical domains, models must not only produce accurate predictions but also provide reliable confidence estimates that accurately reflect their true likelihood of being correct. For example, a guard model must flag unsafe content with high confidence while remaining uncertain on ambiguous cases, and a clinical assistant must express uncertainty when multiple diagnoses are plausible. On the other hand, unreliable confidence can have serious consequences. For instance, an overconfident guard model may incorrectly classify harmful content as safe, resulting in severe downstream risks. This issue, commonly referred to as the confidence calibration problem, arises when the model’s predicted confidence levels do not align with its actual accuracy 39 ; 5 ; 32 . In decision-making tasks, well-calibrated confidence scores are essential for safe and trustworthy operations, as they enable practitioners to determine when to trust or override a model’s decision. Prior work addresses the calibration problem from three main perspectives. The first focuses on uncertainty estimation: developing strategies to quantify model confidence or uncertainty reliably 2 . The second centers on post-hoc calibration, which adjusts model outputs after training to better align predicted probabilities with empirical accuracy 9 . The third explores calibration-aware fine-tuning, where the training objective is modified to produce calibrated models for a chosen type of confidence estimation 5 . Unlike post-hoc or uncertainty-estimation approaches that operate after training, calibration-aware fine-tuning directly targets calibration during model adaptation. However, most existing methods in this category focus on verbalized calibration 15 ; 36 for open-ended generations, which requires prompting the model to express confidence in its own output, a costly and often impractical setup for large-scale or low-latency applications. In contrast, our work studies probability-based confidence in decision-making tasks, specifically, the probability of the final decision token, and analyzes how different fine-tuning paradigms affect its calibration. We find that commonly used paradigms yield opposite effects: supervised fine-tuning (SFT) improves calibration while reinforcement learning with verifiable rewards (RLVR) enhances task performance but leaves models highly overconfident, reducing the usefulness of confidence scores. Building on this analysis, we diagnose the source of miscalibration in RLVR on decision-making tasks and propose a calibration-aware reinforcement learning approach that directly regulates the probability of the decision token (since the confidence is derived from it), balancing the trade-off between accuracy and calibration. We summarize our contributions as follows: • We conduct a systematic empirical study comparing SFT and RLVR across multiple decision-making benchmarks. Our results reveal a consistent trade-off: both SFT and RLVR improve performance over the base model, but while RLVR achieves higher accuracy gains, SFT yields better-calibrated confidence estimates (Section 3 ). • We diagnose the source of RLVR’s miscalibration. Through targeted experiments, we show that the decision token inherits overconfidence from reasoning traces and that reinforcement learning cannot achieve calibration, since there are no calibrated paths to reinforce from the base model (Section 4 ). • We propose a calibration-aware reinforcement learning formulation that directly adjusts decision-token probabilities to improve calibration along with accuracy. Our method consistently mitigates extreme overconfidence, produces more reliable confidence scores, and generalizes well to out-of-distribution settings (Section 5 ). 2 Preliminaries Decision-Making with LLMs In decision-making tasks, an LLM is prompted to make a decision given a question q and a finite set of options \{c_{0},c_{1},...,c_{K}\} . Formally, we denote {\bm{x}} as the full input to the model containing the instructions, question q , and the set of choices. The model, parameterized by \theta , outputs a set of tokens {\bm{y}}=\{y_{0},y_{1},...,y_{T}\} conditioned on {\bm{x}} . For models without explicit reasoning, the output reduces to a single decision token {\bm{y}}=\{y_{d}\} . For reasoning-enabled models, the output consists of intermediate reasoning tokens followed by a final decision token: {\bm{y}}=\{y_{0},y_{1},\dots,y_{T-1},y_{T}\}=\{y_{0},y_{1},\dots,y_{T-1},y_{d}\} . We consider the model’s decision correct if the final decision token y_{d} matches the ground-truth label \hat{c} . When answer choices consist of multiple tokens, we recast the task into a classification setting with atomic options (e.g., A, B, C, D). This formulation ensures that the model’s decision can always be represented by a single token y_{d} . Confidence Estimation There are several algorithmic approaches to estimating confidence in generative models 2 , including verbalized confidence expression 15 ; 36 , sampling-based methods 20 ; 16 , and logit-based probability estimation 23 . In this work, we focus on probability-based confidence estimation. Unlike sampling-based methods, it does not require generating multiple outputs, nor does it rely on prompting the model to express verbalized confidence 44 . Instead, it leverages the probabilities of the output tokens, which is mostly accessible even for API-based models 28 . Moreover, in many decision-making settings where models are deployed locally, direct access to logits is readily available. Throughout this paper, we define the model’s confidence as the probability assigned to the decision token. For example, in a binary decision task (e.g., “yes” or “no”), if the model outputs “yes” (either after reasoning or directly as the model output), then the probability assigned to the “yes” token is treated as the model’s confidence. Formally, we define confidence as follows: C({\bm{x}},{\bm{y}};\theta)=P(y_{d}|{\bm{x}},y_{<d};\theta) where y_{<d} denotes the tokens generated before y_{d} . Confidence Calibration is the alignment between model’s predicted confidence and its actual accuracy. A well-calibrated model should exhibit the property that among all predictions made with confidence p , approximately a fraction p of them are correct, i.e: P(y_{d}=\hat{c}\,\,|\,\,C({\bm{x}},{\bm{y}};\theta)=p)\cong p . Intuitively, this means that, if we group all predictions with a confidence score of 0.8, we expect that approximately 80% of them should be correct under perfect calibration. A common tool to visualize calibration quality is the reliability diagram 9 . It plots the observed accuracy against the predicted confidence by binning predictions into discrete confidence intervals. In a perfectly calibrated model, the plotted curve should align with the diagonal line where accuracy equals confidence, indicating that confidence values match empirical accuracies. Deviations from this diagonal highlight regions where the model is overconfident or underconfident. It is also