Certain but not Probable? Differentiating Certainty from Probability in LLM Token Outputs for Probabilistic Scenarios

Autumn Toney-Wails, Ryan Wails · 2025

Read Certain but not Probable? Differentiating Certainty from Probability in LLM Token Outputs for Probabilistic Scenarios by Autumn Toney-Wails, Ryan Wails (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-00620.txt.

Introduction

Autumn Toney-Wails and Ryan Wails test whether a language model's confidence means what probability says it should. Certainty is often read from token logits, with probabilities and entropy standing in for trust. In gambling-like scenarios with known odds, though, a well-behaved model's token probabilities should track the theoretical distribution of outcomes, a stricter demand than mere accuracy.

The authors probe GPT-4.1 and DeepSeek-Chat with ten probability prompts, such as rolling a six-sided die, with and without fairness cues. Both models answered every scenario validly and perfectly. Yet their token-level probabilities and entropy values consistently diverged from the theoretical distributions. Certainty signals, the study warns, should not be confused with calibrated probability in high-stakes decision support.

Catalog record

CREATORAutumn Toney-Wails, Ryan Wails
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE31 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2511-00620

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Certain but not Probable? Differentiating Certainty from Probability in LLM Token Outputs for Probabilistic Scenarios Autumn Toney-Wails, Ryan Wails Abstract Reliable uncertainty quantification (UQ) is essential for ensuring trustworthy downstream use of large language models, especially when they are deployed in decision-support and other knowledge-intensive applications. Model certainty can be estimated from token logits, with derived probability and entropy values offering insight into performance on the prompt task. However, this approach may be inadequate for probabilistic scenarios, where the probabilities of token outputs are expected to align with the theoretical probabilities of the possible outcomes. We investigate the relationship between token certainty and alignment with theoretical probability distributions in well-defined probabilistic scenarios. Using GPT-4.1 and DeepSeek-Chat, we evaluate model responses to ten prompts involving probability (e.g., roll a six-sided die), both with and without explicit probability cues in the prompt (e.g., roll a fair six-sided die). We measure two dimensions: (1) response validity with respect to scenario constraints, and (2) alignment between token-level output probabilities and theoretical probabilities. Our results indicate that, while both models achieve perfect in-domain response accuracy across all prompt scenarios, their token-level probability and entropy values consistently diverge from the corresponding theoretical distributions. 1 Introduction As large language models (LLMs) are increasingly integrated into decision-support and knowledge-intensive applications, uncertainty quantification (UQ) is essential to ensure reliable downstream use Xiong et al. 2024 ; Vashurin et al. 2025 . Prior work has focused on leveraging the token logits—numerical representations encoding the model’s output probabilities—for UQ methods applied to natural language generation tasks Malinin and Gales 2021 ; Kuhn et al. 2023 ; Gupta et al. 2024 ; Lin et al. 2024 ; Duan et al. 2024 ; Fadeeva et al. 2024 ; Lovering et al. 2025 . Token logits can be transformed into probabilities using activation functions (e.g., softmax or sigmoid), enabling entropy computation over the token distribution. These probability and entropy values are often used to quantify model certainty, providing token-level or response-level confidence scores to users. For prompts involving randomness, risk, or chance, traditional UQ alone may be insufficient to capture the true confidence a user should have in implementing a model. Key insight: Specifically, for prompts involving probabilistic scenarios with inherent aleatoric uncertainty (e.g., “flip a fair coin”), a model’s behavior is trustworthy only when its distribution over possible outputs matches the intended theoretical distribution (which may be only implicitly specified). Hence, in these probabilistic scenarios, we argue there are two considerations for uncertainty quantification that are particularly important: 1. Whether a response is a valid output under the specified scenario constraints (e.g., if the prompt is “roll a six-sided dice”, the output response should not be “7”). 2. Whether the response probability aligns with the underlying theoretical distribution. These considerations are somewhat in contrast to traditional UQ settings, where model accuracy typically corresponds to low uncertainty when a model is well-calibrated. In this work, we explore the relationship between token certainty and theoretical probability in well-defined probabilistic scenarios. We prompt GPT-4.1 and DeepSeek with 10 well-known scenarios (e.g., roll a six-sided die or pick a card from a deck of 52 cards). Our selected scenarios allow us to measure the models’ response certainty (was the response valid?) and to compare the token output probability and entropy of possible output tokens to theoretical distributions (does the response probability align to real-world distributions?). We provide an example of our experimental design framework in Figure 1 . We experiment with a second series of prompting, in which we include language that reflects the distribution the model should use (e.g., roll a fair six-sided die, pick a card from a deck of 52 cards uniformly at random ). Figure 1: Probabilistic scenario prompting and response evaluation design. Our findings suggest that although GPT-4.1 and DeepSeek exhibit appropriate contextual understanding and high response certainty, their token-level output probabilities do not reliably represent true probabilities in scenarios that require statistical reasoning or random sampling. Specifically, we find that they respond with valid outputs with 100% accuracy (i.e., they understand the constraints of the prompted scenario) but their probability and entropy values never align with the corresponding theoretical values. Based on our findings, we explore three research questions for further discussion: RQ1 Can LLMs accurately reason about the theoretical probabilities of our prompt scenarios? RQ2 Are LLMs appropriate and reliable tools for probability-oriented tasks where usability depends on alignment with theoretical distributions? RQ3 How can uncertainty quantification methods be adapted to jointly evaluate response validity and distribution alignment for probability-driven tasks? 2 Background and Related Work 2.1 Uncertainty Quantification for LLMs Uncertainty quantification (UQ) measures the confidence associated with model output, providing information to guide user decisions on model deployment, refinement, or rejection. UQ for LLMs spans a wide range of methodologies for varying granularity levels and certainty dimensions Liu et al. 2025 ; Shorinwa et al. 2025 . Depending on the task, UQ can be computed at the token level (a single word), the span level (a contiguous sequence of tokens representing the response segment of interest), or the response level (the entire generated output). To organize the wide range of UQ methods, Liu et al. 2025 and Shorinwa et al. 2025 present surveys that synthesize the related literature and characterize the state of the field. Liu et al. 2025 proposed a taxonomy to organize sources of uncertainty beyond the general aleatoric (uncertainty stemming from the randomness or variability in the data) and epistemic (uncertainty stemming from lack of knowledge) categorization. The authors provide four dimensions of uncertainty for finer-grained analysis: (1) input uncertainty, (2) reasoning uncertainty, (3) parameter uncertainty, and (4) prediction uncertainty. This taxonomy supports their survey design, which focuses on the relationships between model scale, open-ended generation, and uncertainty dynamics. The authors further outline research directions, emphasizing the need for enhanced UQ methods for natural language generation (NLG) tasks that extend beyond traditional binary formulations. Shorinwa et al. 2025 provide an in-depth overview on UQ, starting with the application in machine learning tasks before focusing on applications in LLMs. The authors frame their survey around the characteristics of the transformer architecture and the auto-regressive, token-by-token generation process underlying NLG. The survey is organized by UQ methods: (1) token-level, (2) self-verbalized, (3) semantic-similarity, and (4) mechanistic interpretability. In conclusion, Shorinwa et al. provide 5 directions for future work that recognize the common mistakes and unique characteristics of UQ for LLM-generated output (e.g., distinguishing consistency from factuality and recognizing that entropy does not equate to factuality). 2.2 Task-Dependent UQ The choice of UQ granularity and methodology is task-dependent—relevant certainty dimensions should be determined by the user’s objectives and appropriately inform reasoning about model outputs. Prior work has investigated methods to measure certainty based on semantic similarity Kuhn et al. 2023 , fact-checking information claims Fadeeva et al. 2024 , and the calibration of probability distributions Lovering et al. 2025 . These certainty dimensions provide user insight into model performance on common aspects of generated output; for example, whether multiple valid phrasings exist for an idea, whether model confidence reflects factual accuracy, or whether responses to probabilistic scenarios align with formally defined theoretical distributions. Kuhn et al. 2023 defined semantic entropy, a UQ measurement that captures semantic meaning, to

Read the complete text