Large Language Models Are Bad Dice Players: LLMs Struggle to Generate Random Numbers from Statistical Distributions
Minda Zhao Yilun Du Mengyu Wang · 2026
Read Large Language Models Are Bad Dice Players: LLMs Struggle to Generate Random Numbers from Statistical Distributions by Minda Zhao Yilun Du Mengyu Wang (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-05414.txt.
Introduction
Minda Zhao and 2 co-authors test, in a 2026 audit, whether large language models can produce truly random numbers on request. Random sampling matters when models are used inside systems that need fair draws from a stated probability distribution. The team benchmarked eleven frontier models across fifteen distributions.
Two protocols were compared: asking for a large batch in one reply, and making many separate stateless calls. Batch replies passed statistical checks only rarely. Separate calls performed far worse, with most models passing none of the distributions. Performance also fell as distributions grew more complex. Downstream tests showed biased quiz answer positions and missed demographic targets, arguing for external sampling tools.
Catalog record
| CREATOR | Minda Zhao Yilun Du Mengyu Wang |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 44 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2601-05414
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Large Language Models Are Bad Dice Players: LLMs Struggle to Generate Random Numbers from Statistical Distributions Minda Zhao Yilun Du Mengyu Wang Abstract As large language models (LLMs) transition from chat interfaces to integral components of stochastic pipelines across domains like educational assessment and synthetic data construction, the ability to faithfully sample from specified probability distributions has become a functional requirement rather than a theoretical curiosity. We present the first large-scale, statistically powered audit of native probabilistic sampling in frontier LLMs, benchmarking 11 models across 15 distributions. To disentangle failure modes, we employ a dual-protocol design: Batch Generation, where a model produces N = 1000 N=1000 samples within one response, and Independent Requests, comprising N = 1000 N=1000 stateless calls. We observe a sharp protocol asymmetry: batch generation achieves only modest statistical validity, with a 13% median pass rate, while independent requests collapse almost entirely, with 10 of 11 models passing none of the distributions. Beyond this asymmetry, we reveal that sampling fidelity degrades monotonically with distributional complexity and aggravates as the requested sampling horizon N N increases. Finally, we demonstrate the propagation of these failures into downstream tasks: models fail to enforce uniform answer-position constraints in MCQ generation and systematically violate demographic targets in attribute-constrained text-to-image prompt synthesis. These findings indicate that current LLMs lack a functional internal sampler, necessitating the use of external tools for applications requiring statistical guarantees. 1 Introduction As Large Language Models (LLMs) transition from open-ended dialogue agents toward core components of complex application pipelines 4 ; 3 ; 40 ; 28 ; 23 , their capacity for statistically faithful probabilistic sampling has emerged as a critical functional requirement 11 . The prominence of LLMs in synthetic data generation 23 has underscored the critical need for robust sampling mechanisms across diverse application scenarios 35 . For instance, in educational material generation, the need for faithful sampling is particularly acute. Automatic question generation has emerged as a promising application of LLMs, with the potential to reduce instructor workload and enable personalized learning at scale 20 . A critical requirement in Multiple Choice Question (MCQ) construction is that correct answers be uniformly distributed across positions (A, B, C, D) to prevent test-takers from exploiting positional patterns 12 . Yet prior work has shown that LLMs exhibit strong positional preferences when selecting among options 46 ; 41 . Whether analogous biases emerge during generation, when models must produce MCQs adhering to uniformity constraints, remains unexplored. Similarly, in text-to-image generation pipelines, LLMs are increasingly employed to automatically generate diverse prompt sets 14 ; 31 . When constructing synthetic image datasets, practitioners often require prompts with controlled attribute distributions, such as demographic balance across gender and ethnicity, to ensure representational fairness in downstream applications 32 . The effectiveness of this approach hinges entirely on the LLM’s ability to faithfully sample from specified distributions; if native sampling is biased, the resulting prompts will systematically deviate from target specifications. Currently, to ensure statistical rigor, the mainstream practice involves prompting LLMs to generate Python code that calls external numerical libraries such as numpy.random and this reliance on code-based workarounds is not incidental but systematic 8 ; 5 ; 33 . However, the ability to internalize and simulate world dynamics is increasingly viewed as a prerequisite for general intelligence 21 . Just as the community has pursued making LLMs solve mathematical problems without external calculators 42 ; 22 ; 34 , we ask whether models can generate samples from specified distributions without relying on external libraries. If a model must rely on an external calculator to generate even basic distributions, it suggests the model has learned to produce linguistic descriptions of randomness without acquiring the underlying functional competence 25 . Recent studies have begun to examine the native sampling capabilities of LLMs, yielding valuable but fragmented insights. 15 identified systematic biases toward “favorite” numbers in random integer generation, while 43 revealed persistent deviations from target probabilities in coin-flip tasks. As the most extensive empirical study to date, 11 evaluated five probability distributions within behavioral simulation contexts. However, their study is constrained by a sample size of N=100 insufficient for reliable convergence assessment, and an experimental scope limited to five simple distributions. Moreover, their single-prompt protocol generates all samples in one response rather than through independent calls, this design cannot determine whether models possess genuine independent sampling capabilities. To address these gaps, we present the first large-scale, systematic evaluation of native probabilistic sampling capabilities in frontier LLMs. Our benchmark characterizes the stochastic fidelity of 11 state-of-the-art models across a taxonomy of 15 probability distributions. Distinguishing our work from prior small-scale studies, we evaluate each configuration at a high-resolution sample size of N=1000 , enabling a statistically powered assessment of distributional convergence. Central to our methodology is a dual-protocol experimental design intended to disentangle distinct failure modes: (1) Batch Generation, where the model generates a sequence of samples within a single context window, and (2) Independent Requests, where each sample is produced via an independent call. Beyond abstract distributional benchmarks, we provide the first systematic evidence that native sampling failures carry downstream consequences: in MCQ generation, models exhibit severe positional bias despite explicit uniformity instructions; in attribute-constrained prompt synthesis, demographic specifications are systematically violated. Our contributions are summarized as follows: (1) We demonstrate that current LLMs lack a functional internal mechanism for probabilistic sampling. (2) We reveal that sampling performance is bounded by distributional complexity. (3) We identify that increasing the sampling budget ( N ) degrades distributional adherence. Figure 1: Overview of the Evaluation Pipeline. We systematically benchmark 11 frontier LLMs across 15 probability distributions spanning three complexity tiers. The evaluation employs a dual-protocol design to disentangle failure modes: Protocol A (Batch) produces samples sequentially within a single response, while Protocol B (Independent) produces samples via stateless single-sample calls. Distributional fidelity is rigorously quantified using statistical validity tests (KS, \chi^{2} ) and geometric metrics ( \mathcal{W}_{1} ) against high-precision numpy reference samples. 2 Related Work Sampling in LLM Applications. LLMs now serve as core components in applications demanding statistical fidelity. In agent-based systems, 28 demonstrated that LLM-powered agents can simulate believable human behaviors in interactive environments, where behavioral diversity is essential for realistic social dynamics. Similarly, 13 showed that LLMs can function as world simulators, predicting environment state transitions to enable multi-step planning. These paradigms rely on the model’s ability to faithfully sample from complex probability spaces, as precise stochasticity is essential for maintaining behavioral diversity and modeling the inherent uncertainty of environment transitions. Beyond agents, a broad class of generative applications requires LLMs to produce outputs conforming to explicit distributional constraints. In synthetic data generation, researchers rely on LLMs to create diverse training sets ( 23 ; 16 ) , yet biased sampling produces coverage gaps that propagate into downstream model failures ( 35 ) . In educational material generation, LLMs are deployed to automatically construct assessments, exercises, and personalized learning materials at scale ( 20 ; 19 ; 44 ) ; applications requiring randomized test construction depend critically on the model’s ability to honor uniformity constraints ( 12 ) . In text-to-image pipelines, LLMs increasingly serve as prompt generators and parsers for diffusion models ( 14 ; 29 ) , producing prompt sets