Pre-training Limited Memory Language Models with Internal and External Knowledge
Linxi Zhao, Sofian Zalouk, Christian K. Belardi, Justin Lovelace, Jin Peng Zhou · 2025
Read Pre-training Limited Memory Language Models with Internal and External Knowledge by Linxi Zhao, Sofian Zalouk, Christian K. Belardi, Justin Lovelace, Jin Peng Zhou (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-15962.txt.
Introduction
Linxi Zhao and co-authors introduce limited memory language models that store facts outside the model, in a research paper from 2025. Standard models entangle language patterns and facts across billions of opaque weights. That tangle makes single facts hard to inspect, verify, or update when they change or prove wrong.
Their pre-training masks retrieved fact values out of the training loss. The model thus learns to look facts up in an external database instead of memorizing them. Tests show scores that rival much larger standard models on common benchmarks. The payoff is a knowledge base that is explicit, editable, and checkable. The design suits fields where facts must stay current.
Catalog record
| CREATOR | Linxi Zhao, Sofian Zalouk, Christian K. Belardi, Justin Lovelace, Jin Peng Zhou |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 84 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-15962
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Pre-training Limited Memory Language Models with Internal and External Knowledge Linxi Zhao, Sofian Zalouk, Christian K. Belardi, Justin Lovelace, Jin Peng Zhou Abstract Neural language models are black-boxes – both linguistic patterns and factual knowledge are distributed across billions of opaque parameters. This entangled encoding makes it difficult to reliably inspect, verify, or update specific facts. We introduce Limited Memory Language Models ( LmLm ) 2 2 2 LmLm stands for L imited M emory L anguage M odel and is pronounced “LamLam”. a new class of language models that externalizes factual knowledge to external database during pre-training rather than memorizing them. Our pre-training approach strategically masks externally retrieved factual values from the training loss, thereby teaching the model to perform targeted lookups rather than relying on memorization in model weights. Our experiments demonstrate that LmLm s achieve competitive performance compared to significantly larger LLMs on standard benchmarks, while offering the advantages of explicit, editable, and verifiable knowledge bases. 3 3 footnotetext: We will open-source our code and models at https://github.com/kilian-group/LMLM . 1 Introduction Many challenges with deploying LLMs in real-world applications originate from the fact that training on vast text corpora exposes LLMs to vast amounts of knowledge ( Kaddour et al., 2023 ; Allen-Zhu & Li, 2024a ) . Ideally, the knowledge in language models should be fully controllable—a customer service agent for a restaurant chain shouldn’t have the ability to answer questions about historical facts, prescription medicine, real estate law, etc. Unfortunately, current LLM pre-training procedures entangle the learning of knowledge with language competency in their neural weights. This has significant implications for both training and inference. During pre-training, facts must be observed many times for reliable memorization ( Allen-Zhu & Li, 2024c ; Kandpal et al., 2023 ) . During inference, it is difficult to unlearn knowledge that may be outdated or inappropriate for a particular application ( Longpre et al., 2021 ; Maini et al., 2024 ; Ovadia et al., 2024 ; Deeb & Roger, 2025 ) . This tight coupling of knowledge and linguistic ability makes updating one without affecting the other extremely challenging. Figure 1: A schematic illustration of LmLm . Unlike RAG (left), which exclusively adds knowledge from external sources, LmLm offloads knowledge from the LLMs to the external database during pre-training. Our vision is a pre-trained language model where knowledge is fully modular and controllable, and can be added or deleted easily for different use cases. This leads us to conclude that storing knowledge inside the model weights during pre-training should be limited as much as possible. This gives rise to the following research question: Can factual memorization be disentangled from language understanding in language models? To realize this vision, we introduce Limited Memory Language Models ( LmLm ), a new class of language models with a pre-training recipe that teaches the model to query an external database for entity-level facts rather than memorizing them. We provide an integrated solution for LmLm s spanning data preparation, pre-training, and inference. To prepare the training data, we annotate the pre-training corpus with database lookups to offload factual content, using a small, cost-effective LM fine-tuned for this task. During pre-training, the returned facts are masked from the loss , systematically separating factual knowledge from the neural weights. During inference, instead of recalling memorized facts, the model queries the database . Unlike the dominant paradigm of post-training and inference-time approaches such as retrieval-augmented generation (RAG) ( Lewis et al., 2021 ) that maximize access to external information, our approach takes the perspective of limiting knowledge stored inside the model parameters during pre-training (Figure 1 ). This frees up model capacity, disentangles knowledge from language competency, and enables precise control over factual knowledge. During inference , LmLm is naturally compatible with RAG systems, tool calling ( Schick et al., 2023 ) , or task-specific fine-tuning ( Wei et al., 2022 ) . LmLm s always rely on lookups for factual information, so attempts to access out-of-scope knowledge trigger detectable lookup failures. In contrast, traditional RAG models without LmLm pre-training typically fall back to their internal knowledge , which is beyond the control of the user and may lead to hallucinations or misinformation ( Longpre et al., 2021 ; Mallen et al., 2023 ; Sun et al., 2025 ) . We compare our LmLm s against LLM counterparts of the same size, pre-trained on the same corpus without external memory. Our experiments demonstrate that LmLm s achieve lower perplexity than comparable LLMs (§ 4.2 ). Crucially, by externalizing knowledge from model weights, LmLm s achieve significant parameter efficiency—our 382M-parameter model matches the factual precision of a LLaMA2-7B (§ 4.4 ). Knowledge externalization also enables instant, verifiable updates and unlearning through simple database operations (§ 4.3 ). Additionally, we present findings regarding the knowledge decoupling achieved by LmLm and examine the trade-off between internal memorization and external offloading (§ 5 ). LmLm s offer a pathway toward language models with substantially reduced dependency on parameter count for factual accuracy. LmLm s have potential for integration with knowledge representation ( Pan et al., 2024 ) , symbolic reasoning ( Chaudhuri et al., 2021 ) , and mechanistic interpretability ( Bereska & Gavves, 2024 ) , potentially transforming how language models store, access, and maintain knowledge. 2 Related Work Table 1: Comparison of selected related work. ✓ = supported, ✗ = not supported. Full version in Table 11 in Appendix. Knowledge Storage Performance Model Internal External Integration PPL \downarrow Factuality \uparrow Unlearning Goal Inference kNN-LM ✓ DS kNN search + prob. interp. ✓ ✓ ✗ Neighbor interpolation RAG ✓ Docs Retrieved docs in prompt ✗ ✓ ✗ Improve factuality Post-training Toolformer ✓ APIs Tool calls ✗ ✓ ✗ Extend via tools Pretraining RETRO ✓ Docs Cross-attn. to retrieved chunks ✓ ✓ ✗ Scale with retrieval MemSinks ✓ (sink neurons) – Memorization isolation ✓ ✗ ✓ Localize memorization LMLM (ours) ✓ (partial) KB Explicit lookup ✓ ✓ ✓ Decouple facts from weights Parametric vs Non-Parametric Knowledge. Language models represent knowledge in two main ways: parametric, encoded in model weights during training, and non-parametric, retrieved from external sources during pretraining or inference ( Mallen et al., 2023 ) . Pretrained models have been shown to function as implicit knowledge bases, storing millions of facts in parameters ( Devlin et al., 2019 ; Petroni et al., 2019 ) . Scaling model size and training data can further increase this knowledge capacity and improve factual precision ( Brown et al., 2020 ; Izacard & Grave, 2022 ; Allen-Zhu & Li, 2024c ) . Yet, recent findings suggest that parametric knowledge may remain under-trained if the knowledge is observed less than a few hundred times during training ( Allen-Zhu & Li, 2024c ) . Broadly, parametric knowledge suffers from well-documented issues, including hallucination, staleness, weak attribution, and limited adaptability ( Peng et al., 2023a ; Huang et al., 2024 ; Kandpal et al., 2023 ; Bi et al., 2025 ; Allen-Zhu & Li, 2024b ) , motivating the shift toward non-parametric approaches that explicitly externalize factual knowledge. Retrieval-Augmented LLMs. A large body of work improves factuality by incorporating external knowledge at inference or training time. Retrieval-augmented generation (RAG) retrieves relevant passages ( Lewis et al.,