Beyond Reproducibility: Token Probabilities Expose Large Language Model Nondeterminism
Tairan Fu, Gonzalo Martínez, Javier Conde, Carlos Arriaga, Pedro Reviriego, Xiuyuan Qi, Shanshan Liu · 2026
Read Beyond Reproducibility: Token Probabilities Expose Large Language Model Nondeterminism by Tairan Fu, Gonzalo Martínez, Javier Conde, Carlos Arriaga, Pedro Reviriego, Xiuyuan Qi, Shanshan Liu (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-06118.txt.
Introduction
Same prompt, same settings, different results: Tairan Fu and 6 co-authors dig into language model nondeterminism in a 2026 paper. Even models set for deterministic output vary on graphics processors. Tiny arithmetic rounding differences depend on execution order, which shifts with whatever else shares the chip.
Earlier work watched the generated text. This team instead tracks token probabilities, the model's confidence in each next word. Variation was large for middle-range probabilities and small near certainty or impossibility. All models tested showed similar trends and values. The findings imply benchmark differences may stem from probabilities or answer length, and that one run's probabilities may estimate nondeterminism's impact without many repeats.
Catalog record
| CREATOR | Tairan Fu, Gonzalo Martínez, Javier Conde, Carlos Arriaga, Pedro Reviriego, Xiuyuan Qi, Shanshan Liu |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 42 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2601-06118
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Beyond Reproducibility: Token Probabilities Expose Large Language Model Nondeterminism Tairan Fu, Gonzalo Martínez, Javier Conde, Carlos Arriaga, Pedro Reviriego, Xiuyuan Qi, Shanshan Liu Abstract The execution of Large Language Models (LLMs) has been shown to produce nondeterministic results when run on Graphics Processing Units (GPUs), even when they are configured to produce deterministic results, for example, by setting the temperature to zero. This is due to the finite precision effects of the arithmetic operations, which depend on the order in which they are executed. This order, in turn, depends on the processes that are running concurrently on the GPU. Previous studies have focused on the impact of nondeterminism on the text generated by the LLMs or on proposing mechanisms to achieve deterministic execution. This work takes a closer look at nondeterminism by analyzing the variations on the token probabilities, not on the generated text. This provides additional information that enables a deeper analysis. Interestingly, all the models evaluated have similar results in both the trends and the actual values of the variations of the probabilities. In particular, the results show that the effects of nondeterminism are significant for token probabilities that are in the range of 0.1 to 0.9, while they are much smaller when the probabilities are close to 0 or 1. Variations are largely independent of the magnitude of the probability, provided that they are in the 0.1 to 0.9 range. This has significant implications for our understanding of nondeterminism. The first is that nondeterminism will likely have a non-negligible impact on generated text when the temperature is not zero, as it introduces significant variations in the token probabilities except when they are close to 0 or 1. Secondly, it suggests that all models have similar non deterministic variations at the token probability level. Therefore, different variations in the performance of the generated text, for example, when measuring accuracy on a benchmark, seem to come from different token probabilities or response lengths. In the first case, models that have more token probabilities close to 0 or 1 will be less affected by nondeterminism, while in the case of length, models that produce shorter answers are also potentially less likely to be affected by nondeterminism. A third implication is that we may be able to estimate the impact of nondeterminism by running a single inference and analyzing the token level probabilities, instead of having to run the same inference many times. Finally, the data produced in the evaluation is shared in a public repository to facilitate further analysis by other researchers, which may lead to additional observations. K eywords Artificial intelligence \cdot Large language models (LLMs) \cdot Determinism 1 Introduction The adoption of large language models (LLMs) has grown rapidly driven by their ability to perform a wide range of natural language processing tasks, including text generation, summarization, question answering, and code completion 4 ; 22 . To perform inference, LLMs are commonly deployed on Graphic Processing Units (GPUs), which provide massive parallelism and optimized hardware for the tensor operations achieving the high performance needed to generate text at reasonable speed 26 . Major vendors supplying GPUs for LLM workloads include NVIDIA, which offers professional GPUs such as the A100 and H200, as well as high-end consumer GPUs like the RTX A6000, capable of supporting large-scale language model inference, and Huawei, with their Ascend series designed for AI workloads. These hardware platforms have become the standard backbone for deployment of LLMs, enabling models with billions to trillions of parameters to perform inference efficiently. To maximize throughput, LLMs are often run with multiple concurrent inferences on the same GPU, a practice commonly referred to as batching. By processing several prompts simultaneously, the GPU can better use its massive parallelism more efficiently, utilizing tensor cores and memory bandwidth to perform operations across multiple sequences in parallel 25 , 15 . Batching improves utilization of GPU resources and reduces per - prompt overhead, enabling faster response times in production environments and making it a widely adopted strategy for handling high query volumes with efficient resource usage. When multiple prompts are processed together in a batch, the GPU executes operations across sequences in parallel. The exact order in which computations are carried out can vary depending on thread scheduling, memory access patterns, or kernel optimizations. The evaluation of large language models (LLMs) is essential for understanding their capabilities, limitations, and suitability for real - world applications. As these models are increasingly used in domains such as healthcare, education, and finance, systematic assessment of metrics like accuracy, coherence, relevance, and safety has become a central concern in AI research 29 . Without rigorous evaluation, apparent advances may mask subtle limitations in reasoning, factuality, or domain - specific understanding, hampering the effective and safe integration of LLMs into practical systems 14 . In the evaluation of large language models (LLMs), reproducible results are often required 2 . To ensure reproducible results, it is commonly assumed that setting the temperature to zero and fixing random seeds is enough. However, recent research shows that LLMs can remain non deterministic even in this case 3 . For example, 1 provides a large-scale empirical study of this phenomenon, showing that inference runs with these settings can produce different outputs. The differences are attributed to floating-point rounding errors 7 and non-deterministic kernel behaviors, especially in fused attention and normalization operations. The role of floating-point non-associativity as a fundamental source of nondeterminism in high-performance computing and deep learning applications was studied in 24 , showing that mathematically identical computations can yield different results due to different orders in the operations. To improve reproducibility, the use of higher precision-floating point formats was proposed in 28 , showing that more stable results can be obtained when using them both for the parameters and arithmetic operations or only for the latter. However, this comes at the cost of, for example, doubling the memory because now each parameter needs 32 bits instead of 16 bits, and the arithmetic operations are also more complex. Nondeterminism is not only an issue for reproducibility; it can also be exploited to extract information from LLM providers as demonstrated in 5 which relies on deterministic output comparisons to detect if a model has been changed to a lower cost or lower precision version. Potentially, an attacker could use nondeterminism to extract information about the hardware, batch size, or load of the provider. In fact, nondeterminism can be more broadly considered as a shift in the model’s estimated token probabilities that has many potential implications beyond reproducibility, for example, it may affect performance when the temperature is larger than zero. Recent studies have primarily focused on observing differences in output sequences across inference runs, but less attention has been paid to token-level probability distributions and their variability. In this paper, a more general approach to the issue of nondeterminism is pursued, considering changes at the token-level probability rather than directly on the LLM generated text. Measuring the logits or log-probabilities of each token across repeated runs provides a more fine-grained view of nondeterminism, revealing effects that may not immediately result in a token change. Metrics such as the standard deviation and range of token probabilities variations across runs allow quantification of