Parametric Retrieval Augmented Generation

Weihang Su, Yichen Tang, Qingyao Ai, Junxi Yan, Changyue Wang, Hongning Wang, Ziyi Ye, Yujia Zhou, Yiqun Liu · 2025

Read Parametric Retrieval Augmented Generation by Weihang Su, Yichen Tang, Qingyao Ai, Junxi Yan, Changyue Wang, Hongning Wang, Ziyi Ye, Yujia Zhou, Yiqun Liu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-15915.txt.

Introduction

Language models answer better when given relevant documents, but stuffing documents into the prompt is slow and can muddy hard reasoning. Weihang Su and co-authors propose parametric retrieval-augmented generation, which bakes each document into the model's own weights instead. A document becomes a small add-on to the feed-forward layers, so the knowledge sits where the model stores its own.

Because the add-ons are built ahead of time, answering a question no longer needs a long prompt packed with documents. The authors report gains in both answer quality and speed over the usual prompt-based method. The two styles can also be combined for still better results. Code, data, and models are released with the paper.

Catalog record

CREATORWeihang Su, Yichen Tang, Qingyao Ai, Junxi Yan, Changyue Wang, Hongning Wang, Ziyi Ye, Yujia Zhou, Yiqun Liu
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE56 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2501-15915

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Parametric Retrieval Augmented Generation Weihang Su, Yichen Tang, Qingyao Ai, Junxi Yan, Changyue Wang, Hongning Wang, Ziyi Ye, Yujia Zhou, Yiqun Liu Abstract . Retrieval-augmented generation (RAG) techniques have emerged as a promising solution to enhance the reliability of large language models (LLMs) by addressing issues like hallucinations, outdated knowledge, and domain adaptation. In particular, existing RAG methods append relevant documents retrieved from external corpus or databases to the input of LLMs to guide their generation process, which we refer to as the in-context knowledge injection method. While this approach is simple and often effective, it has inherent limitations. Firstly, increasing the context length and number of relevant documents can lead to higher computational overhead and degraded performance, especially in complex reasoning tasks. More importantly, in-context knowledge injection operates primarily at the input level, but LLMs store their internal knowledge in their parameters. This gap fundamentally limits the capacity of in-context methods. To this end, we introduce Parametric retrieval-augmented generation (Parametric RAG), a new RAG paradigm that integrates external knowledge directly into the parameters of feed-forward networks (FFN) of an LLM through document parameterization. This approach not only saves online computational costs by eliminating the need to inject multiple documents into the LLMs’ input context, but also deepens the integration of external knowledge into the parametric knowledge space of the LLM. Experimental results demonstrate that Parametric RAG substantially enhances both the effectiveness and efficiency of knowledge augmentation in LLMs. Also, it can be combined with in-context RAG methods to achieve even better performance 1 1 1 We have open-sourced all the code, data, and models in the following anonymized GitHub link: https://github.com/oneal2000/PRAG . Conference: 2025; Under Review; 1. Introduction Figure 1. An illustration of the comparison of in-context RAG and our proposed Parametric RAG paradigms: In-context RAG combines the tokens of relevant documents and the query in the input, using the original LLM \theta to answer the question without modifying its parameters. Our proposed Parametric RAG updates the LLM’s parameters \theta^{\prime}=\theta+\Delta\theta based on the retrieved documents, temporarily integrating relevant knowledge into LLM’s parameters to answer the question. Large Language Models (LLMs) have demonstrated remarkable capabilities across a wide range of information retrieval (IR) and natural language processing (NLP) tasks ( Brown et al., 2020 ; Fang et al., 2024 ; Chowdhery et al., 2022 ; Touvron et al., 2023 ; Scao et al., 2022 ; Yang et al., 2024 ) . Despite these successes, a critical limitation remains: once training is complete, an LLM’s internal knowledge becomes effectively static, making it challenging to incorporate newly emerging information or knowledge not included in its pre-training data. To address this challenge, retrieval-augmented generation (RAG) has emerged as a prominent solution. RAG enables LLMs to dynamically access and utilize information beyond their pre-trained parameters by retrieving relevant information from an external corpus, thus improving their adaptability and performance ( Borgeaud et al., 2022 ; Su et al., 2024c ; Lewis et al., 2020 ; Su et al., 2024b ; Guu et al., 2020 ; Izacard and Grave, 2020 ; Jiang et al., 2022 ; Su et al., 2024d ) . Existing studies have explored various aspects of the RAG pipeline, considering factors such as retrieval timing ( Asai et al., [n. d.] ; Su et al., 2024c ; Jiang et al., 2022 ; Su et al., 2024b ) , document selection ( Yu et al., 2024 ; Ke et al., 2024 ) , and external knowledge organization ( Edge et al., 2024 ; Hu et al., 2024 ; Peng et al., 2024 ) . While these innovations improve different stages of the pipeline, all RAG methods, regardless of their variations, share a common characteristic: they inject external knowledge by directly adding passages or documents into the input context of LLMs, which we refer to as the in-context knowledge injection. Although this in-context knowledge injection approach is straightforward and often effective, recent studies have highlighted several limitations of this paradigm. First, injecting knowledge through input prompts will inevitably increase the context length. Long context not only introduces extra computational overhead and latency for LLM inference, but also hurts the performance of LLMs in understanding and utilizing external knowledge, especially in tasks that involve complex reasoning ( Liu et al., 2024a ; Levy et al., 2024 ) . Second, more importantly, the way LLMs process information in context is fundamentally different from the way they utilize internal knowledge stored in their parameters. Studies have shown that LLMs store most of their knowledge within the parameters of their neural network architecture (e.g., the parameters of their feed-forward network layers) ( Yu and Ananiadou, 2024 ; Nanda et al., 2023 ) . Adding passages or documents in the input context could only affect the online computation of key-value (KV) pairs in the attention networks of LLMs, but not the model’s stored parameters, where its knowledge is encoded ( Yu and Ananiadou, 2024 ) . This means that LLMs may never be able to utilize external knowledge as effectively as they use their internal knowledge in in-context RAG methods. A straightforward solution to this problem is to conduct supervised fine-tuning (SFT) with retrieved documents, thereby incorporating relevant knowledge directly into the LLM’s parameters. However, such SFT-based methods are considered suboptimal as they require substantial computational resources and GPU memory, making it impractical to inject relevant documents online for every query. In addition, they can negatively affect the original ability of the LLM to follow instructions ( Wu et al., 2024 ; Dong et al., 2023b ) and lack the flexibility of in-context methods, which allow external knowledge to be added or removed on the fly. The observations above inspire us to raise the following research question: Is it possible to inject external knowledge into LLM parameters effectively, efficiently, and flexibly for retrieval-augmented generation? To this end, we introduce a new RAG paradigm, Parametric Retrieval Augmented Generation (Parametric RAG) , which directly injects the external knowledge into the Feed-Forward Networks (FFN) of an LLM. Specifically, our approach begins with an offline preprocessing phase that parameterizes each document from the external corpus, transforming them into a small set of parameters (usually a few MB per document) that can be directly integrated into the downstream LLM. We refer to this set of parameters as the parametric representation of the document. In the inference phase, we conduct retrieval augmented generation following a Retrieve-Update-Generate (RUG) workflow as shown in Figure 1 . The Retrieve step retrieves top-n documents from the external corpus based on the input prompt following the same procedure used by the existing RAG pipeline. Then, the Update step uses the parametric representations of the retrieved documents to update the LLM. Finally, in the Generate step, we use the updated LLMs to conduct inference directly based on the original input prompt. Theoretical and empirical analysis show that our Parametric RAG method has superior inference efficiency and outperforms state-of-the-art in-context methods on several RAG benchmarks that involve tasks with complex reasoning. While the preprocessing phase of Parametric RAG introduces an offline computational overhead, this cost is affordable and even neglectable compared to the online cost of large-scale inference requests, leading to long-term savings in terms of power and carbon footprints. Also, similar

Read the complete text