To Memorize or to Retrieve: Scaling Laws for RAG-Considerate Pretraining
Karan Singh, Michael Yu, Varun Gangal, Zhuofu Tao, Sachin Kumar, Emmy Liu, Steven Y. Feng · 2026
Read To Memorize or to Retrieve: Scaling Laws for RAG-Considerate Pretraining by Karan Singh, Michael Yu, Varun Gangal, Zhuofu Tao, Sachin Kumar, Emmy Liu, Steven Y. Feng (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-00715.txt.
Description
Singh and colleagues study scaling laws for retrieval-aware pretraining: memorise or retrieve? Retrieval-augmented generation improves language models by supplying context at test time for hard knowledge tasks. The team studies the training-retrieval trade-off, training OLMo-2-based models from 30 million to 3 billion parameters on 100 billion DCLM tokens, varying data scale, store size, and source across three QA benchmarks. Retrieval gains depend on capacity and pretraining history and come mostly early, with a median 91 percent of the largest improvement realised by one token per parameter.
Catalog record
| CREATOR | Karan Singh, Michael Yu, Varun Gangal, Zhuofu Tao, Sachin Kumar, Emmy Liu, Steven Y. Feng |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 50391 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.00715