Do Large Language Models (Really) Need Statistical Foundations?
Weijie Su · 2025
Read Do Large Language Models (Really) Need Statistical Foundations? by Weijie Su (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-19145.txt.
Introduction
Weijie Su wrote a paper in two thousand twenty-five asking whether large language models really need statistical foundations. Language models are a new way to process unstructured data across many fields. Su argues the answer is yes, on two grounds that together make a case for the statistics community to engage.
First, the models are statistical at their core: they depend deeply on data and generate text by chance, so ideas about variability and uncertainty apply naturally. Second, their sheer scale and black-box nature make closed-form analysis hopeless, which leaves flexible statistical tools as the practical path. Su lists areas where such tools are already needed, from alignment and watermarking to evaluation and data mixing.
Catalog record
| CREATOR | Weijie Su |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 50 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-19145
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Do Large Language Models (Really) Need Statistical Foundations? Weijie Su Abstract Large language models (LLMs) represent a new paradigm for processing unstructured data, with applications across an unprecedented range of domains. In this paper, we address, through two arguments, whether the development and application of LLMs would genuinely benefit from foundational contributions from the statistics discipline. First, we argue affirmatively, beginning with the observation that LLMs are inherently statistical models due to their profound data dependency and stochastic generation processes, where statistical insights are naturally essential for handling variability and uncertainty. Second, we argue that the persistent black-box nature of LLMs—stemming from their immense scale, architectural complexity, and development practices often prioritizing empirical performance over theoretical interpretability—renders closed-form or purely mechanistic analyses generally intractable, thereby necessitating statistical approaches due to their flexibility and often demonstrated effectiveness. To substantiate these arguments, the paper outlines several research areas—including alignment, watermarking, uncertainty quantification, evaluation, and data mixture optimization—where statistical methodologies are critically needed and are already beginning to make valuable contributions. We conclude with a discussion suggesting that statistical research concerning LLMs will likely form a diverse “mosaic” of specialized topics rather than deriving from a single unifying theory, and highlighting the importance of timely engagement by our statistics community in LLM research. † † Email: [email protected] . 1 Introduction Consider a thought experiment where an octopus under the seabed connects to a submarine cable and then eavesdrops on human conversations without any prior knowledge of human language. All it has access to are the utterances of one speaker and the corresponding responses of the other. One may ask: can the octopus ever learn to understand human language, the content of these conversations, and even possess some level of intelligence, purely based on human conversations by passively listening to arbitrarily large amounts of these paired observations? This thought experiment illustrates how large language models (LLMs) are essentially developed ( Kottke 2023 ) . LLMs are massive neural networks, predominantly based on the Transformer architecture ( Vaswani et al. 2017 ) , trained on immense text corpora—including human-written content, code, and various other forms of text—to predict the next word (formally called a “token”) given the preceding sequence of tokens ( Radford et al. 2018 ; Brown et al. 2020 ) . By design, an LLM is an autoregressive model that attempts to learn language purely by learning statistical patterns in human-generated text, rather than explicit linguistic rules or semantic grounding. Interestingly, when Ilya Sutskever and his colleagues at OpenAI proposed next-token prediction as a training strategy in 2018 ( Radford et al. 2018 ) , very few researchers believed that such a simple training paradigm would yield capabilities resembling “understanding” of language ( Bender et al. 2021 ; Matthew and Toner 2024 ) . 1 1 1 In this paper, we exclude non-GPT models such as BERT from our discussion, which use a masked-language-modeling objective ( Devlin et al. 2019 ) . In stark contrast, when browser-based ChatGPT launched in late 2022, it created a global sensation as the public marveled at its ability to generate human-like text and handle wide-ranging tasks ( Bubeck et al. 2023 ) . 2 2 2 Although the NLP community had already been impressed by GPT-3 in 2020, ChatGPT’s web interface made these advances far more widely accessible. Not only did ChatGPT impress users with its near Turing test level conversational abilities ( Jones and Bergen 2025 ) , but it also demonstrated elementary reasoning skills—enough to carry out basic statistical analyses and data visualization ( Tu et al. 2024 ; Lin and Zhu 2025 ) . These capabilities have continued to advance at a rapid pace. Given that LLMs involve many statistical aspects, a burgeoning body of statistical research on LLMs has emerged ( Ji et al. 2025 ) . Nonetheless, the question we pose here is whether LLMs would genuinely benefit from statistical foundations that our community develops and could develop. Furthermore, we ask if such statistical contributions would lead to improved development, deployment, and application, particularly to guide and enhance their real-world use? In other words, we move beyond the discussion of how LLMs can be used to enhance statistical analysis and statistical education to focus on whether statistical methodology and insights can improve LLMs themselves. 3 3 3 For clarity, this paper does not address how LLMs can be used to enhance statistical analysis or statistical education. Interested readers are referred to Tu et al. 2024 . A straightforward argument supporting the need for statistics in LLMs comes from Richard Sutton’s influential essay, The Bitter Lesson ( Sutton 2019 ) . The recent Turing Award laureate observed that enduring progress in artificial intelligence (AI) over decades has primarily stemmed from leveraging increased computational power and data scale, rather than incorporating hard-coded human knowledge or intricate model design. Statistical approaches are well suited to leveraging massive data via computation, as they are designed to let the data speak for themselves. Indeed, Sutton articulated in his essay that “statistics and computation came to dominate the field” of AI. Given the repeated validation of the bitter lesson ( Yousefi and Collins 2024 ) , there are compelling reasons to believe that statistics will undoubtedly benefit LLMs. However, we should recognize that the term “statistics” in Sutton’s context is broad. It encompasses both purely predictive algorithms—such as neural networks, boosting, and random forests—and what might be termed inferential statistics. This latter category roughly corresponds to what Bradley Efron termed “estimation and attribution” methods ( Efron 2020 ) or what Leo Breiman called “data-modeling” methods in his Two Cultures paper ( Breiman 2001 ) . These approaches focus on interpretable parameters and uncertainty quantification to make inferences from noisy observations. In this paper, we ask whether LLMs, as they become more powerful and widely deployed, would benefit from the interpretable, inferential, and uncertainty-aware techniques that statistics in this narrower sense can offer. We argue affirmatively that LLMs require more statistical contributions for their continued advancement. Although LLMs by design are purely predictive algorithms, they differ significantly from prior methods in this category, including even pre-Transformer neural networks ( Krizhevsky et al. 2012 ; Simonyan and Zisserman 2014 ) . This distinction emerges in two key ways: First, LLMs operate on an unprecedented diversity of data types, including almost all possible forms of text data. While earlier models could process text ( Manning and Schutze 1999 ) , it is the first time a single model can integrate and process text almost as seamlessly as numbers, enabling a unified approach to handling tasks as diverse as code generation, language translation, and data analysis, marking a substantial departure from previous predictive algorithms primarily focused on structured or image data. This strong data dependency positions LLMs as compressors of vast human data, as suggested by the influential metaphor “ChatGPT is a blurry JPEG of the web” ( Chiang 2023 ) . Second, the generative and stochastic nature of LLMs, which arises from next-token prediction, makes the models themselves random, with outputs necessarily involving variability and uncertainty ( Huang et al. 2025 ) . Together, these characteristics lead to numerous statistical challenges and opportunities for principled treatment of variability, uncertainty, calibration, and