Tuning LLMs by RAG Principles: Towards LLM-native Memory
Jiale Wei Shuchi Wu Ruochen Liu Xiang Ying Jingbo Shang Fangbo Tao · 2025
Read Tuning LLMs by RAG Principles: Towards LLM-native Memory by Jiale Wei Shuchi Wu Ruochen Liu Xiang Ying Jingbo Shang Fangbo Tao (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-16071.txt.
Introduction
Jiale Wei and co-authors present a way to bake retrieval skill into a language model itself in 2025. Assistants need memory beyond their training. Long-context models read whole memory at high cost, while retrieval systems fetch only matched pieces and can miss the big picture.
Their method fine-tunes a modest model on data built by retrieval principles. The tuned model keeps retrieval sharpness for detail queries and gains whole-memory grasp for broad ones. Tests on three data sets show it beats both long-context models and standard retrieval setups across query types. The approach points toward models with native memory skill.
Catalog record
| CREATOR | Jiale Wei Shuchi Wu Ruochen Liu Xiang Ying Jingbo Shang Fangbo Tao |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 45 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-16071
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Tuning LLMs by RAG Principles: Towards LLM-native Memory Jiale Wei Shuchi Wu Ruochen Liu Xiang Ying Jingbo Shang Fangbo Tao Abstract Memory, additional information beyond the training of large language models (LLMs), is crucial to various real-world applications, such as personal assistant. The two mainstream solutions to incorporate memory into the generation process are long-context LLMs and retrieval-augmented generation (RAG). In this paper, we first systematically compare these two types of solutions on three renovated/new datasets and show that (1) long-context solutions, although more expensive, shall be easier to capture the big picture and better answer queries which require considering the memory as a whole; and (2) when the queries concern specific information, RAG solutions shall be more competitive especially when the keywords can be explicitly matched. Therefore, we propose a novel method RAG-Tuned-LLM which fine-tunes a relative small (e.g., 7B) LLM using the data generated following the RAG principles, so it can combine the advantages of both solutions. Extensive experiments on three datasets demonstrate that RAG-Tuned-LLM can beat long-context LLMs and RAG methods across a wide range of query types. 1 Introduction Figure 1: Overview of our RAG-Tuned-LLM method. Stage 1 : RAG provides the foundation for synthesizing training data (query-answer pairs) for fine-tuning. Stage 2 : The synthesized data is used to fine-tune a large language model (LLM) via LoRA. Stage 3 : Inference is performed exclusively with LLM-native memory, eliminating the need for external memory. The RAG-Tuned-LLM combines the strengths of LLM-native solutions and RAG methods. Memory, additional information beyond the training of large language models (LLMs), is crucial to various real-world applications, such as personal assistant ( Mai et al., 2023 ) . The most intuitive solution to enable long memory into the generation process is long-context LLM, for example, 128K-token GPT-4o ( Achiam et al., 2023 ) , 1M-or 10M-token Gemini 1.5 ( Reid et al., 2024 ) , or an LLM with “unlimited” context lengths by length extrapolation ( Peng et al., 2023 ; Xiao et al., 2023 ; Han et al., 2023 ; Zhang et al., 2024a ) and position bias ( Liu et al., 2024 ; Peysakhovich and Lerer, 2023 ; An et al., 2024 ) . Retrieval-augmented generation (RAG) Lewis et al. (2020) ; Kočiskỳ et al. (2018) ; Pang et al. (2022) ; Trivedi et al. (2022) ; Edge et al. (2024) is another popular approach to incorporate memory in a plug-in manner: a retriever identifies a small number of query-relevant contexts from a large corpus, and then feeds them into an LLM to answer the query. Compared with long-context LLMs, RAG’s serving cost is more affordable, and therefore, RAG is potentially more popular than long-context LLMs in real-world applications. In this paper, we first systematically compare these two types of methods on three renovated/new datasets. We start with two public datasets, namely news articles Tang and Yang (2024) and podcast transcripts Scott (2024) , following the general ideas mentioned in Edge et al. (2024) to generate queries and references. On these two datasets, we use the entire corpus as the memory. We categorize the queries into two types, local and global . Specifically, local queries target specific information and concrete answers from small chunks of memory. Global queries, on the other hand, require considering memory as a whole to generate high-level answers. We further introduce a new proprietary dataset containing journaling articles and user-provided local / global queries and their expected answers from our journaling app 1 1 1 Me.bot: https://app.me.bot/ . Intuitively, (1) long-context solutions, although more expensive, shall be easier to capture the big picture and better answer global queries; and (2) when the queries concern local information, RAG solutions shall be more competitive especially when the keywords can be explicitly matched. Based on these three datasets, we run competitions between a vanilla RAG Lewis et al. (2020) and Gemini 1.5 Reid et al. (2024) , with the win rate results shown in Table 2 , confirming our intuitions. It is worth mentioning that RAG surpasses long-context LLMs when handling local queries, yet underperforms in addressing global ones. Following our findings, we propose a novel LLM-native method RAG-Tuned-LLM which fine-tunes a relatively small (e.g., 7B) LLM using the data generated following the RAG principles, so it can combine the advantages of RAG and long-context solutions. We call it LLM-native because it maintains the same speed as directly prompting an LLM with only the question –— without requiring long contexts or retrieval from a knowledge base. It enables the LLM to parameterize knowledge in a way that allows it to maintain contextual coherence and handle different types of queries more naturally and efficiently. Specifically, as illustrated in Figure 1 , we follow the GraphRAG Edge et al. (2024) principles to extract useful information from plain text documents. We then generate data from both local and global perspectives: (1) local data synthesis concentrates on generating content-specific query and answer pairs, and (2) global data synthesis focuses on producing query-answer pairs that integrate insights across entities and relationships. And finally, we employ the widely adopted LoRA technique Hu et al. (2021) to fine-tune the LLM. Our experiments then demonstrate that RAG-Tuned-LLM can beat long-context LLMs and RAG methods on both local and global queries, and further case studies show that RAG-Tuned-LLM excels in providing insightful and user-friendly responses. Our codes have been released to the public at Github 2 2 2 https://github.com/mindverse/rag-tuned-llm . Our contributions are summarized as follows. • We create three datasets with local and global queries with their references, and then systematically compare LLM-native and (vanilla) RAG solutions, showing their respective unique advantages. It is worth mentioning that on one dataset, both the queries and references are manually created by human users. • We follow the comparison results and propose a novel LLM-native method RAG-Tuned-LLM to combine the advantages of RAG and long-context solutions. • Extensive experiments on three datasets demonstrate that RAG-Tuned-LLM can indeed outperform long-context LLMs and (advanced) RAG methods on both local and global queries. 2 Long-context vs. RAG To motivate our work, we systematically compare long-context and RAG solutions and discuss their respect strengths in this section. 2.1 Settings Datasets. We consider three datasets for comparison, as detailed in Table 1 . For the two public datasets— News articles Tang and Yang (2024) and Podcast transcripts Scott (2024) —we follow Edge et al. (2024) to generate 125 local queries and 125 global queries for each, along with their corresponding references. The Journaling dataset, newly introduced by us, is proprietary and derived from our journaling app. It contains 45 local queries and 15 global queries designed by users, accompanied by their expected answers. Users were informed to craft queries aimed at complex and nuanced scenarios, prioritizing reasoning capabilities over simple retrieval. It is designed to robustly evaluate models’ ability to handle intricate reasoning tasks in diverse real-world scenarios. It extends beyond basic fact retrieval to assess how well models can retrieve specific details while performing higher-order reasoning. Please refer to Table 1 for detailed statistics. Methods. For the long-context LLM, we choose Gemini-1.5-pro-001 due to its remarkable 2-million-token context window, which stands out as one of the longest among widely recognized and authoritative LLMs.