M-RAG: Making RAG Faster, Stronger, and More Efficient

Xu Sun, Tongkai Xu, Baiheng Xie, Li Huang, Qiang Gao, Kunpeng Zhang · 2026

Read M-RAG: Making RAG Faster, Stronger, and More Efficient by Xu Sun, Tongkai Xu, Baiheng Xie, Li Huang, Qiang Gao, Kunpeng Zhang (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-26667.txt.

Introduction

Sun and co-authors propose M-RAG, a semantic key-value index layer for retrieval-augmented generation under a token budget. Most pipelines use the same text chunks for lookup and for evidence, though the two jobs pull apart. Small chunks split key facts, while large ones waste context on noise. M-RAG pulls meta-markers from whole documents, each holding a search key, a value payload, and provenance links. Lookup runs on the compact keys, and matched values fill the prompt in a position-aware order.

Catalog record

CREATORXu Sun, Tongkai Xu, Baiheng Xie, Li Huang, Qiang Gao, Kunpeng Zhang
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE47828
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.26667

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

M-RAG: Making RAG Faster, Stronger, and More Efficient Xu Sun, Tongkai Xu, Baiheng Xie, Li Huang, Qiang Gao, Kunpeng Zhang Abstract Retrieval-Augmented Generation (RAG) has become a widely adopted paradigm for enhancing the reliability of large language models (LLMs). However, RAG systems are sensitive to retrieval strategies that rely on text chunking to construct retrieval units, which often introduce information fragmentation, retrieval noise, and reduced efficiency. Recent work has even questioned the necessity of RAG, arguing that long-context LLMs may eliminate multi-stage retrieval pipelines by directly processing full documents. Nevertheless, expanded context capacity alone does not resolve the challenges of relevance filtering, evidence prioritization, and isolating answer-bearing information. To this end, we proposed M-RAG , a novel Chunk-free retrieval strategy. Instead of retrieving coarse-grained textual chunks, M-RAG extracts structured, k k - v v decomposition meta-markers, with a lightweight, intent-aligned retrieval key for retrieval and a context-rich information value for generation. Under this setting, M-RAG enables efficient and stable query-key similarity matching without sacrificing expressive ability. Experimental results on the LongBench subtasks demonstrate that M-RAG outperforms chunk-based RAG baselines across varying token budgets, particularly under low-resource settings. Extensive analysis further reveals that M-RAG retrieves more answer-friendly evidence with high efficiency, validating the effectiveness of decoupling retrieval representation from generation and highlighting the proposed strategy as a scalable and robust alternative to existing chunk-based methods. 1 Introduction As a promising approach for improving the factual reliability and adaptability of large language models (LLMs), Retrieval-Augmented Generation (RAG) has recently attracted increasing attention. By dynamically incorporating external, domain-specific knowledge during the generation process, RAG alleviates the inherent static-knowledge limitations of LLMs, reduces hallucination, and has demonstrated strong applicability in knowledge-intensive domains 21 ; 35 ; 37 . Typically, a RAG system follows a multi-stage pipeline: given an input query, the system first retrieves a set of candidate documents or passages from an external knowledge base using dense or sparse retrieval models. These candidates are then segmented into fixed-length or semantically defined chunks , followed by optional re-ranking to filter relevant sets. Consequently, selected chunks with queries are fed into the LLMs to conduct the generation process 18 ; 11 ; 15 ; 9 . Figure 1: Insight from our concerns: information fragmentation and structural loss caused by RAG chunking. (Figure generated by Nano Banana 28 ) Although effective, the designation of RAG tightly couples retrieval granularity with generation context construction, making RAG systems sensitive to the choice of chunking strategy. For instance, fixed-length segmentation often fragments coherent semantic units and introduces irrelevant noise. While adaptive chunking methods such as semantic-based approaches like PIC 33 or trained mixtures like MoC 38 attempt to dynamically aggregate sentences, they remain fundamentally constrained by local context windows and struggle to preserve document-level causal, temporal, or hierarchical structures. More critically, chunking constitutes an inherently lossy preprocessing operation. By predefining information boundaries and granularity prior to retrieval, it limits the capacity of LLMs to reason over complete contextual dependencies, as illustrated in Figure 1 . Recent studies, such as DOS RAG 17 , suggest that retaining the original document structure and abandoning complex chunking strategies can lead to more robust performance, which is increasingly feasible with the availability of long-context LLMs. Thus, a critical issue is naturally raised: if model "see" the entire document, is the RAG process still necessary? The answer should be affirmative: long-context capability does not eliminate the need for RAG, but rather shifts its role from overcoming memory extension limitations to enabling efficient, relevance-aware knowledge utilization 14 ; 30 ; 1 ; 23 ; 31 . While extended context windows relax hard token constraints for LLMs, they do not inherently provide mechanisms for relevance filtering or evidence prioritization. Recent work CFIC 24 supports this view by proposing a chunk-free, in-context retrieval approach that identifies evidence text via autoregressive decoding over document representations. Although our method adopts a different problem-solving perspective, both lines of work acknowledge that RAG explicitly structures the interaction between external knowledge and generation, allowing selective access to relevant information and controlled knowledge injection. As a result, RAG remains essential as a principled framework, but asking for more efficient and robust knowledge-retrieval mechanisms. To this end, we propose M-RAG, a novel Chunk-free retrieval strategy within the RAG framework that fundamentally departs from text chunking. Instead, M-RAG leverages off-the-shelf LLMs (e.g., DeepSeek-V3.2 6 ) to extract structured marker entries directly from complete documents. Each marker entry, a.k.a. meta-marker, is decoupled into two complementary components: a lightweight retrieval key k , which serves as an efficient "anchor" for similarity matching when retrieved, and a context-rich information value v , which preserves the corresponding content for future generation. Unlike existing paragraph- or chunk-level retrieval, this key-value decomposition enables lightweight retrieval while providing stable, intent-aligned access to contextual information. Our main contributions are summarized as: • We propose M-RAG, a novel and principled Chunk-free retrieval strategy within the RAG framework that fundamentally departs from text chunking and rethinks knowledge injection in the long-context era. • We design a decoupled K-V Marker that enables lightweight and efficient retrieval without sacrificing contextual fidelity and, to our knowledge, is the first retrieval strategy to explicitly separate retrieval representations from generation content within the RAG system. • Experiments on LongBench subtasks demonstrate strong performance, and in-depth analyses indicate that the proposed retrieval strategy offers a scalable and robust alternative to chunk-based RAG methods. 2 Methodology Figure 2: The overall architecture of M-RAG. In this section, we introduce M-RAG, a novel retrieval strategy that eliminates traditional text chunking in conventional RAG. As shown in Figure 2 , the whole workflow consists of two stages: 1) Marker Extractor, which drives concise, semantically and retrieval-optimized meta-markers from documents, is the main contribution of this work, and 2) meta-marker retrieval and response generation, where the most query-relevant meta-markers are selected under a fixed token budget to guide response generation. We next detail the design of each module. 2.1 Marker Extractor With simplicity and effectiveness in mind, M-RAG is designed as a model-agnostic, drop-in replacement for existing RAG systems. Conception. Intuitively, a user query reflects an underlying information need associated with their core intent. Considering the same information need may be expressed through many semantically equivalent queries, effective retrieval must be robust to substantial surface-form variability. In this context, aligning queries to fixed or semantically aggregated text chunks is unnecessarily heavy. We therefore argue that, rather than requiring alignment with textual chunks, effective retrieval should match user queries to lightweight semantic cues that capture the core intent of the information need. This perspective motivates the design of M-RAG, which operates intent-aligned semantic cues as retrieval units in place of textual chunks. Specifically, we first derive a set of meta-markers \boldsymbol{m}_{i} to form an n -markers entry M_{\mathcal{D}}=\{\boldsymbol{m}_{1},\cdots,\boldsymbol{m}_{n}\} , where each \boldsymbol{m}_{i} is a self-contained textual string with its length l_{i} , and both the n and l_{i} are unconstrained. Herein, each meta-marker \boldsymbol{m}_{i} is decoupled into two complementary components: • \boldsymbol{k}_{i} , a lightweight retrieval key , serves as an intent-aligned semantic cue optimized for matching anticipated queries and maximizing retrieval recall. • \boldsymbol{v}_{i} , a context-rich information value , preserves the substantial factual or relational content and provides grounding for generation. Thus, a complete meta-marker

Read the complete text