Transformers in Pseudo-Random Number Generation: A Dual Perspective on Theory and Practice
Ran Li, Lingshu Zeng · 2025
Read Transformers in Pseudo-Random Number Generation: A Dual Perspective on Theory and Practice by Ran Li, Lingshu Zeng (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-01134.txt.
Introduction
Ran Li and Lingshu Zeng published this study in 2025 on an unlikely pairing: transformer networks and random number generation. Pseudo-random generators are deeply nonlinear machines, and transformers are built to model complex nonlinear patterns. The authors ask whether a transformer can learn to be a generator itself. They approach the question from theory first. With step-by-step reasoning, a decoder-only transformer can simulate two classic generators, the linear congruential generator and the Mersenne Twister.
That simulation result places transformer computation inside a known class of simple circuits. Experiments then backed the theory. Numbers produced by transformer-based generators passed most of the standard NIST randomness tests. Their statistical patterns looked properly random in visual maps. The authors also probed prediction attacks, where an adversary tries to guess future outputs from past ones. They present the work as a two-way bridge between the two fields.
Catalog record
| CREATOR | Ran Li, Lingshu Zeng |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 58 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2508-01134
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Transformers in Pseudo-Random Number Generation: A Dual Perspective on Theory and Practice Ran Li, Lingshu Zeng Abstract Pseudo-random number generators (PRNGs) are high-nonlinear processes, and they are key blocks in optimization of Large language models. Transformers excel at processing complex nonlinear relationships. Thus it is reasonable to generate high-quality pseudo-random numbers based on transformers. In this paper, we explore this question from both theoretical and practical perspectives, highlighting the potential benefits and implications of Transformer in PRNGs. We theoretically demonstrate that decoder-only Transformer models with Chain-of-Thought can simulate both the Linear Congruential Generator (LCG) and Mersenne Twister (MT) PRNGs. Based on this, we conclude that the log-precision decoder-only Transformer can represent non-uniform AC 0 \text{AC}^{0} . Our simulative theoretical findings are validated through experiments. The random numbers generated by Transformer-based PRNGs successfully pass the majority of NIST tests, whose heat maps exhibit clear statistical randomness. Finally, we assess their capability in prediction attacks. 1 Introduction Pseudo-random numbers are essential components in scientific and technological applications, from cryptography to statistical sampling and numerical computing. Modern applications require PRNGs that can generate high-quality random sequences. A PRNG uses a deterministic algorithm to produce reproducible sequences that appear random, based on an initial seed value. Gentle 2003 ; Press 2007 . In recent years, Transformer-based large language models (LLMs) have shown exceptional performance across multiple domains. While these models excel in language processing and knowledge tasks, their theoretical foundations remain unclear. Specifically, how Transformers simulate complex nonlinear functions is still not fully understood. Recent theoretical advances have provided significant insights into the expressive power of Transformers. Notably, it has been demonstrated that Transformers with constant depth and precision, when enhanced with Chain-of-Thought (CoT) prompting Wei et al. 2022 of polynomial length O(\text{poly}(n)) , possess sufficient expressive power to capture the complexity class P/poly (Theorem 3.3 in Li et al. 2024 ). This theoretical foundation suggests that Transformers might be particularly well-suited for processing nonlinear functions, with PRNGs serving as a classic and practical example of such functions. The application of Transformers as PRNGs offers several key characteristics: (1) Transformer’s advanced feature learning capabilities enable effective extraction and learning of complex statistical patterns in random sequences, (2) The sophisticated position encoding mechanisms systematically capture temporal dependencies and sequential relationships between numerical elements Wu et al. 2025b , (3) The multi-head attention architecture facilitates parallel computation across diverse representation subspaces, enabling simultaneous analysis of multiple sequence characteristics. Building on these advantages, our research explores a fundamental question: What are the advantages of Transformers as PRNGs or tools for evaluating PRNGs security? To address this, we conduct a comprehensive study combining theoretical analysis with empirical validation of Transformer models’ capabilities in pseudo-random number generation. Our result This paper makes several significant contributions to understand the expressiveness of Transformer models in the context of PRNGs. Our work encompasses both theoretical analysis and empirical validation. Specifically, we demonstrate that Transformers can be utilized for prediction attacks, where in a Transformer model is trained on a sequence of generated pseudo-random numbers to predict subsequent values, thereby providing a mechanism to assess the security of the PRNGs. First, we provide a comprehensive theoretical analysis of Transformer models’ capability to express two widely-used PRNGs: the Linear Congruential Generator (LCG) and the Mersenne Twister (MT) algorithm. We demonstrate that Transformer architectures equipped with CoT 1 1 1 In this paper, we define Chain of Thought as a step-by-step reasoning process, aligning with the interpretation in Feng et al. 2024 , rather than as a specialized prompting technique. can precisely simulate these PRNGs using only a constant number of attention heads and layers, while maintaining O(n) parameter complexity in the hidden layer ( Theorem 3.5 and Theorem 3.6 ). Our proofs are constructive in nature, providing explicit mechanisms for simulating each computational step of both algorithms. To validate these theoretical findings, we conduct comprehensive experiments across multiple bit-widths (8-bit, 12-bit, and 16-bit) for pseudo-random number generation based on the MT algorithm, achieving near-perfect accuracy in all cases. Significantly, during this analysis, we establish a non-uniform \text{AC}^{0} lower bound for Decoder-only Transformers with \log(n) precision ( Corollary 3.7 ),contributing to the theoretical understanding of Transformer limitations. Second, we perform a comprehensive empirical evaluation of Transformer-based pseudo-random number generators (PRNGs) through three critical experiments: • We investigate the potential of Transformer architectures as novel implementations for PRNGs. Through empirical analysis utilizing heat map visualizations, we clearly observe that sequences generated by Transformers exhibit significant statistical randomness properties. • To rigorously analyze the capabilities of Transformers as pseudo-random number generators, we focus on their inherent non-linear transformations, which enhance the unpredictability of the generated sequences and potentially improve security. Consequently, we implement a Transformer model that simulates the MT algorithm for generating 16-bit pseudo-random numbers. The generated sequences successfully pass the majority of NIST statistical test suites. • While Transformers can effectively learn complex statistical patterns from large-scale data, this capability has significant security implications for PRNGs, as their security fundamentally depends on the unpredictability of output sequences. Leveraging this insight, we propose utilizing Transformer models for security analysis of PRNGs through prediction attacks. Specifically, we investigate whether a Transformer model, after training on a sequence of generated pseudo-random numbers, can accurately predict subsequent values. Our experimental results demonstrate that Transformers has the potential to as a tool for evaluating PRNGs security by identifying potential statistical vulnerabilities in their output sequences. To the best of our knowledge, this work represents the first comprehensive study of Transformer models in the context of pseudo-random number generation, establishing both theoretical foundations and practical viability. Our findings explore new possibilities of PRNGs based on transformer. 2 Preliminaries We introduce the key notations used throughout the paper: n denotes the number of tokens in a sample sequence, d represents the dimension of embedding vectors, H is the number of attention heads in the model, d_{k}=\frac{d}{H} defines the dimension per attention head, and L indicates the number of layers in the Transformer model. boldface letters are used to denote vectors, and normal letters are used to denote scalars. Unless otherwise specified, all vectors are represented as row vectors. 2.1 Decoder-only Transformers Transformer models, introduced in 2017 Vaswani 2017 , revolutionized natural language processing through their self-attention mechanism, which effectively captures long-range sequence dependencies. The auto-regressive Transformer, also known as the Decoder-Only Transformer Radford et al. 2019 , is a sequence-to-sequence neural network model as follows defined by the following equations: first, the input tokens s_{i} are embedded into a d -dimensional vector \mathbf{x}_{i}^{0}=\text{Embed}(s_{i})+\text{pos}(i)\in\mathbb{R}^{d} for i=1,\ldots,n , where pos(i) is the position embedding of the i -th token. Then, the Transformer model processes the sequence of vectors through a series of attention layers. the l -th layer of the Transformer is defined as \displaystyle\mathbf{x}_{i}^{l}=\mathrm{Attention}^{l}(\mathbf{x}_{i}^{l-1})+\mathrm{FFN}^{l}(\mathrm{Attention}^{l}(\mathbf{x}_{i}^{l-1})) (1) \displaystyle\mathrm{Attention}^{l}(\mathbf{x}_{i}^{l-1})=\mathrm{Attention}^{l}(\mathbf{x}_{i}^{l-1})+\mathbf{x}_{i}^{l-1} (2) \displaystyle\mathrm{FFN}^{l}(\mathbf{x})=\mathrm{GeLU}(\mathbf{x}\mathbf{W}_{1}^{l}+\mathbf{b}_{1}^{l})\mathbf{W}_{2}^{l} (3) where \mathrm{Attention} is the residual multi-head attention layer and \mathrm{FFN} is the feed-forward network, and \mathbf{W}_{1}^{l},\mathbf{W}_{2}^{l}\in\mathbb{R}^{d\times d} , \mathbf{b}_{1}^{l},\mathbf{b}_{2}^{l}\in\mathbb{R}^{d} are the weights and biases of the l -th layer. \mathrm{Attention}^{l}(\mathbf{x}_{i}^{l-1}) is given by \displaystyle\sum_{h=1}^{H}\mathrm{softmax}\left(\frac{(\mathbf{x}_{i}^{l-1}W_{q}^{(l,h)})(\mathbf{X}^{l-1}W_{k}^{(l,h)})^{T}}{\sqrt{d_{k}}}\right)(\mathbf{X}^{l-1}W_{v}^{(l,h)}), where \mathbf{W}_{q}^{(l,h)},\mathbf{W}_{k}^{(l,h)},\mathbf{W}_{v}^{(l,h)}\in\mathbb{R}^{d\times d_{k}} are the weights of the query, key, and value vectors, \mathbf{q}_{i}^{(l,h)}=\mathbf{x}_{i}^{l-1}\mathbf{W}_{q}^{(l,h)} , \mathbf{k}_{j}^{(l,h)}=\mathbf{x}_{j}^{l-1}\mathbf{W}_{k}^{(l,h)} , \mathbf{v}_{j}^{(l,h)}=\mathbf{x}_{j}^{l-1}\mathbf{W}_{v}^{(l,h)}\in\mathbb{R}^{d_{k}} are the query, key, and value vectors, \mathbf{X}^{l-1}=[(\mathbf{x}_{1}^{l-1})^{T},\ldots,(\mathbf{x}_{n}^{l-1})^{T}]^{T}\in\mathbb{R}^{n\times d} is the matrix of the key vectors. 2.2 Pseudo-random Number Generator Random Number