When are likely answers right? On Sequence Probability and Correctness in LLMs
Johannes Zenn, Jonas Geiping, [0.5em] Max Planck Institute for Intelligent Systems · 2026
Read When are likely answers right? On Sequence Probability and Correctness in LLMs by Johannes Zenn, Jonas Geiping, [0.5em] Max Planck Institute for Intelligent Systems (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-27359.txt.
Introduction
Zenn and Geiping ask a deceptively simple question about large language models: when the model finds an answer likely, is it usually right? Decoding methods, from low-temperature sampling to best-of-N selection, shift probability toward likely outputs. Their success hinges on probability tracking correctness. The relationship is measured at four levels: across decoding methods, across settings within a method, across prompt-answer pairs, and across repeated answers to one prompt. Benchmarks and models vary widely, in a language model study published in 2026.
Within a fixed dataset, higher sequence probability does predict correctness across different prompts. That comfort ends there. Raising probability by changing decoding settings or methods does not reliably raise accuracy. For repeated answers to the same prompt, probability is a poor correctness signal. The findings draw a firm boundary around a popular intuition.
Catalog record
| CREATOR | Johannes Zenn, Jonas Geiping, [0.5em] Max Planck Institute for Intelligent Systems |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 80 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2606.27359
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
When are likely answers right? On Sequence Probability and Correctness in LLMs Johannes Zenn, Jonas Geiping, [0.5em] Max Planck Institute for Intelligent Systems Abstract Many decoding methods for large language models can be understood as shifting probability mass toward outputs that are more likely under the model, either locally at the token level or globally at the sequence level. Therefore, their success depends on a fundamental question: when does sequence probability, that is, the conditional probability of a continuation given a prompt, actually align with correctness? In this paper, we set out to quantify this relationship across decoding methods, models, and benchmarks at four levels: across decoding methods, across hyperparameters within a method, across prompt-answer pairs within a dataset, and across repeated responses to the same prompt. We find that higher sequence probability is often predictive of correctness across prompt-answer pairs within a fixed dataset. However, this relationship does not generally transfer to decoding decisions: increasing sequence probability by changing hyperparameters or methods does not reliably improve accuracy. Further, sequence probability is not a good indicator of correctness for responses to the same prompt. These findings clarify when decoding can and cannot be expected to improve correctness, and provide practical guidance for decoding, self-consistency, and verifier-free self-improvement. 1 Introduction Large language models (LLMs) are autoregressive models that assign probabilities to sequences of tokens. At inference time, decoding methods use the autoregressive next-token distribution to exploit this distribution in different ways. Many widely used methods can be viewed as shifting probability mass toward outputs that are more likely under the model. Decoding methods, like low-temperature next-token sampling and truncation methods (e.g., top- k sampling ( Fan et al. 2018 ) ) suppress unlikely tokens locally . Best-of- N (Bo N ) keeps only the highest-probability samples among a set of N sequences approximating the global mode (for N\to\infty ) similar to beam search ( Lowerre 1976 ; Sutskever et al. 2014 ) . Power sampling ( Karan and Du 2025 ; Ji et al. 2026 ; Azizi et al. 2026 ) biases the distribution over full sequences towards high-probability regions over-weighting high-probability sequences and under-weighting low-probability sequences. Viewing decoding through this lens raises a fundamental question: when does sequence probability align with correctness? An answer to this question has two important implications. Firstly, it helps explain when particular decoding methods improve performance and opens up a way to guide the development of new decoding methods. Secondly, if higher-probability sequences are more likely to be correct, then model probability itself may provide a verifier-free approach to self-improvement by using higher-probability samples for self-consistency ( Wang et al. 2023 ; Wang et al. 2024a ; Taubenfeld et al. 2025 ) and self-distillation ( Hinton et al. 2015 ; Zhang et al. 2019 ) . Since LLMs are trained on data which is predominantly correct, it seems natural to expect correct continuations to receive higher probability than incorrect ones. However, this intuition is too simple: it has been observed in various contexts that beam search, which approximately finds the most likely sequence, often degrades performance ( Vijayakumar et al. 2016 ; Cohen and Beck 2019 ; Koehn and Knowles 2017 ; Yang et al. 2018 ; Su et al. 2022 ) . This suggests that the question is less whether there is a general probability-correctness relationship, but under which conditions probability and correctness align and which decoding methods are able to exploit the relationship. In this paper, we quantify the relationship between sequence probability and correctness across 8 decoding methods ( 2 methods targeting the power distribution, 2 methods targeting the mode of the distribution, and 4 local methods), 14 models (from the Qwen2.5 , Qwen3 , Olmo3 families), and 6 benchmark datasets. We analyze the relationship on three different levels: We define across-method correlation which fixes model and dataset and computes correlation across methods with a representative hyperparameter ( Figure 1 , left). For each method, within-method correlation considers the correlation across hyperparameters ( Figure 1 , left, non-transparent lines). Zooming in on one hyperparameter ( Figure 1 , middle) we define the within-dataset correlation . Finally, within-sample correlation is the most granular correlation for an individual sample from a dataset ( Figure 1 , right). Our results show that there is no uniform probability-correctness relationship for LLM generations. We find a consistent correlation within a dataset depending on the model family but not the method ( Section 3.1 ). This indicates that a model is often able to tell correct prompt-answers pairs from incorrect prompt-answer pairs. However, this signal does not generally transfer to the settings directly used for decoding. We find that tuning the hyperparameters of a decoding method, while producing sequences of higher log-probability, does not result in more correct sequences ( Section 3.2 ). Further, methods that produce higher-probability sequences are not consistently more accurate ( Section 3.3 ). For a single prompt, there is no consistent correlation within the corresponding responses. However, more correct samples also show larger within-sample correlations ( Section 3.4 ). Figure 1: The correlation between log-probability and correctness across three levels. Left: Across-method-correlation : log-probability plotted against accuracy for various decoding methods and hyperparameters (connected by lines). Within-method correlation : plotted without transparency. Middle: Within-dataset correlation : for one method (Bo N with N=32 ) we bin correct (green dots, top) and incorrect data (red dots, bottom) in 10 bins and fit a Pearson correlation. The correlation is positive. Right: Within-sample correlation : for one sample of the dataset, a Pearson rank correlation is computed (also, binned Pearson corelation is shown). The correlation is positive. The plot shows data of Qwen3-8B on MATH500. See Section 1 and Section 3 for details. Summarily, the main contributions of this work are as follows: • We frame decoding methods as approaches to maximize sequence probability: we distinguish local methods from global sequence-level methods and give a variational objective for global methods. • We study the relationship between sequence probability and correctness across 12 models, 6 datasets, and 8 decoding methods. We analyze correlation on various levels: across-method correlation , within-dataset correlation , within-method correlation , and within-sample correlation . • We find that within-dataset correlations between log-probability and correctness are often consistent and largely determined by the dataset and model variant, rather than by the decoding method. We show that increasing sequence probability by changing a decoding method or tuning its hyperparameters does not reliably improve correctness. • We connect these findings to verifier-free self-improvement methods. We find that probability can be informative when the model already has sufficient task accuracy, but within-sample correlations are often weak or symmetric around zero, limiting the reliability of probability-weighted aggregation. 2 Decoding Methods Maximize Sequence Probability in LLMs In this section, we view decoding methods in large language models (LLMs) as sampling mehods to produce high-probability sequences. We highlight two classes of commonly used decoding methods: local decoding methods in Section 2.1 and global decoding methods in Section 2.2 . Let p denote an autoregressive LLM. Further, let the distribution over continuations {\mathbf{s}}=(s_{1},\dots,s_{T}) given a prompt \bar{\mathbf{s}} under p be defined as follows \displaystyle p({\mathbf{s}}\mid\bar{\mathbf{s}})=\prod_{t=1}^{T}p(s_{t}\mid\bar{\mathbf{s}},{\mathbf{s}}_{<t}). (1) 2.1 Local Decoding Methods Figure 2: Within-dataset correlation: log-probability and correctness are correlated for