Predictable Confabulations: Factual Recall by LLMs Scales with Model Size and Topic Frequency

Matthew L. Smith, Jonathan P. Shock, Samuel T. Segun, Iyiola E. Olatunji, Tegawendé F. Bissyandé · 2026

Read Predictable Confabulations: Factual Recall by LLMs Scales with Model Size and Topic Frequency by Matthew L. Smith, Jonathan P. Shock, Samuel T. Segun, Iyiola E. Olatunji, Tegawendé F. Bissyandé (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-18732.txt.

Introduction

Matthew Smith and four co-authors test, in a 2026 study, how factual recall scales with size and topic frequency. They evaluate 38 models on over 8,900 references. Recall quality follows a sigmoid in the log-linear mix of parameter count and topic share. Those two variables alone explain 60 percent of variance across 16 dense models. Within families, the share rises to 74 to 94 percent. The pattern fits a superposition account where signal-to-noise gating ties signal to concept frequency and noise floor to capacity.

Catalog record

CREATORMatthew L. Smith, Jonathan P. Shock, Samuel T. Segun, Iyiola E. Olatunji, Tegawendé F. Bissyandé
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE51357
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.18732

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Predictable Confabulations: Factual Recall by LLMs Scales with Model Size and Topic Frequency Matthew L. Smith, Jonathan P. Shock, Samuel T. Segun, Iyiola E. Olatunji, Tegawendé F. Bissyandé Abstract While scaling laws govern aggregate large language model performance, no scaling law has linked factual recall to both model size and training-data composition. We evaluated 38 models on over 8,900 scholarly references evaluated by an automated reference verification system. Recall quality follows a sigmoid in the log-linear combination of model parameter count and topic representation in training data. These two variables alone explain 60% of the variance across 16 dense models from four families, rising to 74–94% within individual families. The form matches a superposition-inspired account in which recall is gated by a signal-to-noise ratio: signal strength scales with concept frequency and the noise floor with model capacity. 1 Introduction Factual recall errors, or confabulations, remain one of the most consequential failure modes of large language models (LLMs) 9 . Prior work has linked these failures to both model scale and the distribution of knowledge in training data, but no single framework combines both factors into a predictive scaling law for factual recall quality. We measured recall quality across 38 LLMs and 24 topics spanning five orders of magnitude in training-data representation. Using academic references as the factual recall being measured, we find that recall quality scales with the log of a signal-to-noise ratio (SNR), where signal strength is set by how frequently a concept appears in training data and the noise floor is set by model capacity. Under a superposition-inspired account, both contributions enter the SNR multiplicatively, yielding a sigmoid in their log-sum: \text{quality}=\sigma\!\left(\alpha\,\log_{10}P+\beta\,\log_{10}S+\gamma\right) (1) where P is model parameter count, S is a proxy for concept frequency in training data, and \sigma is the logistic function. Fitted to 16 dense models across 24 topics ( N=384 model–topic observations from 3,661 evaluated references; R^{2}=0.599 ), the relationship holds across independent model families. Across all 38 models and 8,913 evaluated scholarly references, topic representation frequency is positively associated with recall quality independently of architecture or parameter count. The remainder of this paper is organised as follows. Section 2 reviews the relevant literature. Section 3 develops the theoretical framework that motivates the sigmoid functional form. Section 4 describes the experimental design, verification pipeline, and statistical methods. Section 5 presents the empirical findings, and Section 6 interprets them. Section 8 summarises our contributions and outlines directions for future work. 2 Related Work Scaling laws for language models. Kaplan et al. 11 established that aggregate language model performance improves as a power law in model size, dataset size, and compute. Hoffmann et al. 8 refined these relationships to show that model and dataset size should be scaled in tandem for compute-optimal training. These aggregate scaling laws describe loss on held-out text but do not decompose performance into specific capabilities such as factual recall. Knowledge capacity and superposition. Allen-Zhu and Li 1 showed that knowledge capacity scales at roughly two bits per parameter, providing a quantitative link between model size and the number of facts that can be stored. Mechanistically, Elhage et al. 6 introduced the concept of superposition—the encoding of more features than a network has dimensions—and demonstrated it in toy models. Bricken et al. 3 and Templeton et al. 20 confirmed that superposition operates in production language models via sparse autoencoder analysis. Bereska et al. 2 formalised superposition as lossy compression. Liu et al. 12 provided empirical evidence across four open-source LLM families that models operate in a strong-superposition regime where representation vectors form near-equiangular tight frames with squared overlaps scaling as 1/d (where d is model width). Knowledge frequency and the long tail. A complementary line of work has established that rarer concepts are disproportionately susceptible to recall failure. Kandpal et al. 10 showed that LLMs struggle to learn long-tail knowledge, with question-answering accuracy strongly correlated with the number of training documents containing the relevant entity. Mallen et al. 14 found that parametric memory is reliable only for popular entities, while retrieval augmentation is needed for the long tail. Sun et al. 19 confirmed this head-to-tail degradation across a range of LLMs, and Lu et al. 13 derived scaling laws for fact memorisation as a function of model size and training epochs. Yin et al. 24 formalised the phenomenon as a “law of knowledge overshadowing,” in which high-frequency knowledge suppresses retrieval of related low-frequency facts. Yuan et al. 23 developed holistic benchmarks for factual knowledge recall. Moayeri et al. 16 demonstrated systematic geographic biases in LLM factual recall, consistent with frequency-driven disparities. Memorisation and training data. Carlini et al. 4 quantified memorisation across neural language models, showing that larger models memorise more training data and that memorisation increases with the number of times a sequence appears in training. Hernandez et al. 7 established scaling laws for learning from repeated data, finding that data repetition and model size interact in predictable ways. Xiong et al. 21 provide a comprehensive survey of memorisation mechanisms, measurement, and mitigation in LLMs. This work. While these strands have established model size and knowledge frequency as independent drivers of factual recall, no prior work has combined them into a single predictive functional form. We provide this unification by showing that a sigmoid in the log-linear combination of both factors—motivated by a superposition-inspired SNR account—fits the data across four independent model families. 3 Theoretical Framework The functional form fitted in Section 5 is motivated by two ingredients: the geometry of superposition in neural representations, and the statistics of concept frequencies in natural-language training corpora. We present this as a superposition-inspired latent signal-to-noise model rather than a derivation: each step involves an explicit assumption that we flag below, and the data are consistent with the resulting sigmoid form rather than uniquely selecting it. 3.1 Setup: superposition, interference, and the recall threshold We use the following notation throughout. On the model side , P is the total parameter count and N_{\mathrm{eff}} the effective number of dimensions available for factual encoding, with N_{\mathrm{eff}}\ll P and N_{\mathrm{eff}}\propto P^{\gamma_{e}} for some \gamma_{e}\leq 1 . M is the number of factual concepts encountered in training. On the data side , f_{i} is the training-corpus frequency of concept i . The encoded signal strength of concept i is s_{i}(f_{i}) , a monotonically increasing function of f_{i} for which we adopt the simplest scale-free form s(f)=f^{\beta_{s}} . (Note that \beta_{s} and \gamma_{e} are theoretical exponents distinct from the fitted parameters \beta and \gamma .) S denotes the OpenAlex scholarly work count for a topic, serving as an empirical proxy for training-data representation at the topic level, with individual concept frequencies scaling as f\propto S^{\delta} for some \delta\leq 1 . We note that S also reflects the number of distinct facts associated with a topic (its inventory size), so the f\propto S^{\delta} assumption with \delta<1 partially absorbs the breadth-versus-reinforcement ambiguity but does not formally separate signal strength from concept count; the framework should accordingly be read as a coarse-grained model in which S acts as a single composite proxy. When M>N_{\mathrm{eff}} , the network must encode concepts in overlapping directions, a phenomenon termed superposition 6 and confirmed by sparse-autoencoder studies 3 ; 20 as well as work formalising

Read the complete text