Task-Aware Calibration: Provably Optimal Decoding in LLMs
Tim Tomov Dominik Fuchsgruber Rajeev Verma Stephan Günnemann [0.5em] {tim.tomov,d.fuchsgruber,s.guennemann}@tum.de & [email protected], [0.8em] School of Computation, Information & Technology, Technical University of Munich · 2026
Read Task-Aware Calibration: Provably Optimal Decoding in LLMs by Tim Tomov Dominik Fuchsgruber Rajeev Verma Stephan Günnemann [0.5em] {tim.tomov,d.fuchsgruber,s.guennemann}@tum.de & [email protected], [0.8em] School of Computation, Information & Technology, Technical University of Munich (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-10202.txt.
Introduction
Tim Tomov and three colleagues address a quiet flaw in how language models choose their answers. Decoding usually follows the model's predicted distribution, so any mismatch with the true distribution yields suboptimal choices. Calibrating over all possible free-form outputs is intractable, since the space of word sequences is combinatorially vast. Their 2026 paper introduces task-aware calibration as a way around that wall.
Many tasks let free-form outputs be read as outcomes in a smaller latent space, such as class labels, integers, or sets. The authors calibrate the model's distribution in that task-induced space instead. They prove that Minimum Bayes Risk decoding on the calibrated latent distribution is the optimal strategy given the model's latent beliefs. Across tasks and baselines the method consistently improved generation quality. They also propose Task Calibration Error, a metric for the loss caused by miscalibration.
Catalog record
| CREATOR | Tim Tomov Dominik Fuchsgruber Rajeev Verma Stephan Günnemann [0.5em] {tim.tomov,d.fuchsgruber,s.guennemann}@tum.de & [email protected], [0.8em] School of Computation, Information & Technology, Technical University of Munich |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 100 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.10202
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Task-Aware Calibration: Provably Optimal Decoding in LLMs Tim Tomov Dominik Fuchsgruber Rajeev Verma Stephan Günnemann [0.5em] {tim.tomov,d.fuchsgruber,s.guennemann}@tum.de & [email protected], [0.8em] School of Computation, Information & Technology, Technical University of Munich Abstract LLM decoding often relies on the model’s predictive distribution to generate an output. Consequently, misalignment with respect to the true generating distribution leads to suboptimal decisions in practice. While a natural solution is to calibrate the model’s output distribution, for LLMs, this is ill-posed at the combinatorially vast level of free-form language. We address this by building on the insight that in many tasks, these free-form outputs can be interpreted in a semantically meaningful latent structure, for example, discrete class labels, integers, or sets. We introduce task calibration as a paradigm to calibrate the model’s predictive distribution in the task-induced latent space. We apply a decision-theoretic result to show that Minimum Bayes Risk (MBR) decoding on the task-calibrated latent distribution is the optimal decoding strategy on latent model beliefs. Empirically, it consistently improves generation quality across different tasks and baselines. We also introduce Task Calibration Error (TCE), an application-aware calibration metric that quantifies the excess loss due to miscalibration. Our work demonstrates that task calibration enables more reliable model decisions across various tasks and applications. * * footnotetext: Equal contribution. 1 Introduction The quality of an LLM’s output depends heavily on the decoding strategy used [ 42 , 16 ] . Decoding can be viewed as a decision problem in which the LLM provides a distribution p({\textnormal{Y}}\mid{\textnormal{X}}) over responses Y given a query X , and the goal is to select the best token sequence according to some loss. Even though the model is trained to approximate the true data-generating distribution p^{*}({\textnormal{Y}}\mid{\textnormal{X}}) , LLMs are often miscalibrated [ 27 , 3 ] and exhibit systematic biases [ 10 ] . Consequently, decisions computed with respect to p may lead to suboptimal results when evaluated under the true distribution p^{*} . While calibrating the distribution p to match p^{*} is an appealing remedy to this issue [ 36 ] , an LLM’s output distribution p is defined over the infinite space of all token sequences \mathcal{V}^{\infty} . Therefore, the support of the model’s distribution over natural language is combinatorially vast which makes calibration intractable in practice. Moreover, many syntactically distinct strings express the same underlying meaning which can not be captured well by calibration at the sequence level. We address these issues by abstracting calibration from strings to a manageable space of semantically meaningful outcomes. In particular, we leverage the recent idea that for many downstream tasks, strings in \mathcal{V}^{\infty} often implicitly correspond to representations in a latent space \mathcal{L} that captures the semantics that are relevant to the problem [ 45 ] . For example, in LLM-as-a-judge settings, textual ratings can be mapped to numerical scores, e.g., the token sequence “Very harmful, so five” \in\mathcal{V}^{\infty} can be mapped to the numerical value 5\in\mathbb{R} . Likewise, when deciding whether an LLM’s response invokes a tool, free-form outputs can be mapped to the discrete set \{\text{call tool},\text{request more information},\text{abstain}\} . Expressing an LLM’s predictive distribution in a discrete task-dependent space enables applying established notions of calibration [ 24 ] . We refer to calibration in this task space \mathcal{L} as task calibration . Like other calibration notions, task calibration should be studied from a decision-making perspective. While the literature often conflates calibration with uncertainty quantification (UQ) [ 49 ] , calibration does not aid decision-making at the level of individual instances, such as in hallucination detection or other surrogate tasks studied in UQ. Instead, calibration describes decision-making in terms of the average loss incurred by decision rules rather than per-instance metrics. Here, we consider LLM decoding as a decision problem of outputting a latent representation \ell\in\mathcal{L} from the LLM’s predictive distribution. Specifically, we leverage task calibration in the following two ways: First, we apply a known decision-theoretic result that shows that Minimum Bayes Risk (MBR) decoding on the task-calibrated latent distribution incurs the minimum expected loss with respect to the true data generation distribution p^{*} in terms of any task-dependent loss. Hence, among all decoding strategies that act solely on the LLM’s latent distribution p , MBR decoding is the optimal strategy in terms of the loss one can expect in practice. This gives rise to a simple decoding paradigm for a given task: first calibrating an LLM’s latent distribution, and then acting optimally by outputting the MBR action, as shown in Figure 1 . Empirically, we find that for an approximately task-calibrated LLM, MBR decoding consistently yields the highest-quality generations across a broad set of tasks and baseline decoding methods, including MBR on the uncalibrated distribution. Second, we introduce the notion of task-calibration error (TCE). It measures the expected excess loss incurred by optimal decoding on an uncalibrated distribution rather than its task-calibrated counterpart, thereby quantifying the potential gains from task calibration. Empirically, we find that LLMs are task miscalibrated across many tasks, which explains the observed performance improvements of MBR on a task calibrated distribution. Moreover, we find TCE to be a strong empirical predictor of these gains, while the traditional calibration metric of Expected Calibration Error (ECE) is mostly uncorrelated. Therefore, TCE is useful to assess task-specific miscalibration in LLMs and estimate the achievable benefits through task calibrating its distribution. Figure 1 : LLM responses Y get mapped into a task-dependent latent space \mathcal{L} (here: \mathcal{L}\subset\mathbb{N} ). Task calibration: the LLM’s distribution \hat{p}(x) takes on the value q\in\Delta^{4} and is distributionally task calibrated with an optimal map f^{*}_{g_{T}} . Minimum Bayes Risk decoding on the recalibrated distribution \smash{\delta^{\mathrm{MBR}}(f^{*}_{g_{T}}(q))} is, on average, the best strategy based on q ( Theorem 3.4 ) and yields a different output rating (4) compared to MBR on the uncalibrated distribution. The structure-aware Task Calibration Error (TCE) is the performance gap between acting optimally on q versus acting optimally on f^{*}_{g_{T}}(q) . 2 Background Task-Awareness in LLMs through Latent Structure. Our work builds on the insight of 45 that for many tasks, the output of an LLM can be modeled as an element in a task-specific latent structure \mathcal{L} . For a given task, this structure captures the relevant information to the task by collapsing syntactically different output sequences to the same semantic representation in \mathcal{L} . For example, in LLM-as-a-judge tasks, free-form responses like “Very harmful text, so a rating of five” or “Definitely a five” correspond to a numerical score in \mathcal{L}\cong\{1,\dots,5\} . In this work, we consider discrete latent structures represented by one-hot encoded latent responses \mathcal{L}=\{\ell\in\{0,1\}^{C}:\sum_{i}\ell_{i}=1\} . Formally, let {\textnormal{X}},{\textnormal{Y}}\in\mathcal{V}^{\infty} denote the random variable for a query and corresponding free-form LLM response over a vocabulary \mathcal{V} respectively. For a given task T , we represent the correspondence between responses and latent representations with a task-specific map g_{T}:\mathcal{V}^{\infty}\mapsto\mathcal{L} that induces the random variable {\textnormal{L}}=g_{T}({\textnormal{Y}})\in\{0,1\}^{C} of latent responses. As argued by 45 , many downstream applications already explicitly require post-processing free-form responses Y by applying g_{T} . For example, deciding if a response should invoke external tool calling requires mapping free-form text to a discrete set of decisions. For many tasks, extracting the semantics from a response, i.e. implementing g_{T} , typically is an easy problem that can be reliably solved using