Inferring the Size of Large Language Models From Popular Text Memorization
Ivica Nikolić · 2026
Read Inferring the Size of Large Language Models From Popular Text Memorization by Ivica Nikolić (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-29223.txt.
Introduction
Ivica Nikolic proposes a black-box method inferring lower bounds on model size from generated text. Parameter counts of models are withheld, leaving size largely undisclosed. The method rests on popular texts present in every pretraining corpus. Next-word prediction accuracy across fragments signals memorization, which total parameter count limits. That signal becomes one profile per model, feeding a pairwise larger test and a scaling estimator mapping an index to a count. Validated on open models, it recovers closed hierarchies and reveals strategies, some growing fast, others capped.
Catalog record
| CREATOR | Ivica Nikolić |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 62004 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.29223
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Inferring the Size of Large Language Models From Popular Text Memorization Ivica Nikolić Abstract The parameter counts of the most widely used large language models (LLMs) are often withheld by their developers, leaving model size—a primary reference point for interpreting capabilities and costs—largely undisclosed. We propose a black-box method to infer conservative lower bounds on LLM size from generated text outputs alone, requiring nothing beyond the ability to submit text fragments and observe next-token predictions. Our approach is grounded in a key observation: popular, widely-circulated texts—such as classical literature, religious texts, and foundational documents—are present in virtually every large-scale pretraining corpus, and how accurately a model predicts the next word across text fragments of varying length is a reliable signal of how much it has memorized them, which in turn is fundamentally limited by its total parameter count. We aggregate this memorization signal across a diverse corpus of texts and fragment lengths into a single accuracy profile vector per model, and build two complementary inference methods on top of it: a pairwise statistical test that determines which of two models is larger, and a scaling-law estimator that extracts a one-dimensional latent index from these vectors via Principal Component Analysis (PCA) to map the aggregated signal to a parameter count. Validated on a broad set of open-weight models, both methods produce accurate and reliable lower bounds. When applied to popular closed-weight models, our framework recovers internal product hierarchies and reveals a clear divergence in industry scaling strategies: while some developers yield significantly higher bounds indicative of large generational parameter growth, others operate under strict parameter ceilings, demonstrating that hidden design choices can be systematically probed even under strict API limitations. I Introduction Large language models (LLMs) have become deeply integrated into many applications and services. The development of the most advanced systems is heavily driven by major tech companies such as OpenAI, Google, Meta, and Anthropic. The most widely used are general-purpose models, which are trained on vast amounts of text to adapt flexibly to a wide variety of open-ended user instructions and tasks. Despite their widespread use, the most capable models remain closed-weight. Providers share very little about how they are built. While keeping specific architectural choices private is an understandable way to protect commercial interests, it is notable that even the most basic descriptive metric—the total parameter count—is almost always kept secret. This omission has a large impact because model size is a primary variable for understanding, comparing, and deploying language models. If parameter count cannot be used as a control variable, comparing closed-weight and open-weight models becomes ambiguous. Without knowing the size of a model, it is impossible to determine whether a performance advantage stems from algorithmic breakthroughs, higher quality training data, or simply a massive increase in parameter count. Furthermore, size dictates economics. Parameter count tightly bounds the compute and memory required to serve a model, directly driving API pricing and latency. Providers may therefore have strong commercial incentives to hide this number, as it prevents independent verification of the computational cost and resource requirements associated with running their systems. Attempting to infer model size directly from a model’s text outputs raises substantial methodological challenges. Model providers typically restrict interaction to APIs where a user can submit a prompt and inspect the returned text, but cannot obtain token probabilities, logits, hidden states, architectural metadata, or gradients. This constrained interaction is a limited form of model property inference—narrower than full model extraction, where an adversary reconstructs weights or a functional replica, but relying on the same premise: that a private attribute can be extracted through strategic black-box querying. Estimating model size purely from returned text requires isolating a signal that correlates strongly with parameter count. However, returned text is sensitive to numerous unknown variables, including training data, formatting constraints, system prompts, the extent of instruction tuning, and the hyperparameters used during deployment. Separating the intrinsic effect of model size from these unobserved variables is non-trivial. For this reason, inferring the size of a closed-weight language model from black-box interaction is both a practically relevant and well-posed problem formulation. It asks what can be learned about a black-box model from its returned text alone, while acknowledging that any such inference must contend with multiple unobserved variables and limited access to the system. Contributions. We make the following contributions: 1. Memorization-based size signal. We show that aggregating a model’s accuracy at predicting the next word of incomplete sentences drawn from popular, widely-circulated texts, across many such texts and sentence lengths yields a high-dimensional vector—an accuracy profile —that contains a strong signal for model size. 2. Dual lower-bound inference framework. We derive two complementary methods for translating accuracy profiles into conservative lower bounds on model size: (i) a scaling-law estimator , a PCA projection onto a one-dimensional latent index combined with an exponential scaling law calibrated on open-weight reference models; and (ii) a pairwise test , a statistical hypothesis test that establishes which of two models is larger with a controlled false positive rate. The final bound for any target model is the maximum of the two. 3. Validation across open- and closed-weight models. We calibrate and validate our methods on 19 dense open-weight models, where the scaling-law estimator achieves R^{2}=0.95 and the pairwise test achieves around 90% precision and recall. We then test the calibrated framework on 16 open-weight MoE models—producing empirically valid lower bounds in all evaluated cases—and on 21 popular closed-weight models from Anthropic, Google, OpenAI, and Alibaba, obtaining lower bound estimates that vary depending on the provider. II Problem Model size of a large language model f is measured by its parameter count and denoted with |f| . It is one of the primary factors shaping model capabilities and a key reference point for interpreting performance comparisons. For closed-weight models, however, this quantity is often not disclosed. The most direct form of the problem is to estimate the size of an unknown model from its behavior alone. Problem 1 (Absolute Size Inference). Let f be a language model with unknown parameter count |f| . Given black-box query access to f , estimate |f| . This is the strongest of the problems we study, and obtaining the model size is always the ultimate goal. However, size inference may not be possible, or may not be precise. We therefore also study a simpler problem that asks only whether one model is larger than another. Problem 2 (Relative Size Inference). Let f and g be two language models with unknown parameter counts |f| and |g| . Given a threshold \tau>0 , determine whether |f|>(1+\tau)\,|g| . Relative size inference is weaker than absolute estimation but remains useful in practice. If two models perform similarly on a benchmark but one turns out to have more parameters, the smaller model is arguably preferable—it achieves comparable results at lower computational cost. Drawing this conclusion requires knowing which model is larger, even without knowing exact counts. The problem is formulated with a multiplicative threshold \tau rather than asking simply whether |f|>|g| , because the differences that matter in practice are not marginal ones. Distinguishing a 30 B model from a 32 B model is of limited consequence—models so close in parameter count carry essentially the