MODE: Mixture of Document Experts for RAG
Rahul Anand · 2025
Read MODE: Mixture of Document Experts for RAG by Rahul Anand (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-00100.txt.
Introduction
Rahul Anand, working alone, presents MODE in 2025, a light retrieval design for small, focused document sets. Retrieval-augmented generation grounds a language model in fetched text. Standard setups lean on huge vector databases and rerankers tuned for vast corpora, which is overkill for a small domain set.
MODE groups documents into clusters by meaning and stores each cluster's centroid, its center point. At query time it routes to the top clusters and fetches only within them, with no external database or rerank step. On two question sets with a few hundred chunks, answer quality matches or beats a dense baseline, with faster end-to-end time. Cluster size and multi-cluster routing tune the recall and precision balance.
Catalog record
| CREATOR | Rahul Anand |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 17 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2509-00100
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
MODE: Mixture of Document Experts for RAG Rahul Anand Abstract Retrieval-Augmented Generation (RAG) augments language models by integrating retrieval of external knowledge. However, traditional RAG pipelines depend heavily on large vector databases and complex retrieval mechanisms, making them inefficient for small to medium-sized datasets. We propose MODE (Mixture of Document Experts) , a lightweight and efficient alternative framework for RAG. MODE reimagines the retrieval process by replacing fine-grained vector search with a cluster-and-route mechanism. It organizes documents into semantically coherent clusters—our ”document experts”—and uses fast centroid-based matching to route a query to the most relevant cluster for context retrieval. This design eliminates the need for dedicated vector databases and re-rankers, drastically reducing infrastructure complexity and query latency. Experiments on HotpotQA and SQuAD demonstrate that MODE not only provides significant efficiency gains but also matches or exceeds the generation quality of traditional RAG systems, largely by improving the topical relevance of retrieved context. Corpus sizes. We categorize datasets by the number of retrieval chunks : small = 100 chunks, medium = 200 chunks, and large = 500 chunks. A “chunk” is a contiguous text segment used for retrieval (typically 400–1,000 tokens with metadata). These thresholds reflect common domain-specific deployments: small (e.g., a single handbook or wiki), medium (a team’s mixed documents), and large (a multi-project knowledge base). For intuition, assuming ∼ \sim 800–1,000 tokens per chunk, these categories correspond to roughly 80k–100k tokens (small), 160k–200k tokens (medium), and 400k–500k tokens (large). [email protected] Keywords: Retrieval-Augmented Generation, RAG, Document Clustering, Expert Models, Centroid-Based Retrieval, Language Models 1 Introduction Retrieval-Augmented Generation (RAG) has become a cornerstone for building factually grounded and knowledgeable language model applications [ 8 ] . The dominant paradigm involves an ”index-retrieve-rerank” pipeline: a corpus is chunked, embedded, and indexed into a specialized vector database. At inference time, a query is used to retrieve a set of candidate passages, which are often re-ranked by a more powerful cross-encoder model before being fed to the LLM [ 7 , 12 ] . While powerful, this architecture is optimized for web-scale corpora and can be a poor fit for the thousands of smaller, domain-specific applications common in enterprise settings. In these regimes, the operational burden of maintaining a vector database, the added latency from re-ranking, and the potential for web-trained dense retrievers to underperform on specialized content present significant practical barriers. In this work, we ask: can we design a more resource-efficient RAG architecture for this small-to-medium data regime without sacrificing quality? We introduce MODE (Mixture of Document Experts) , a framework that shifts the retrieval paradigm from searching over individual items to routing between thematic topics. Instead of indexing every document chunk, MODE first partitions the corpus into semantically coherent clusters. Retrieval then becomes a two-stage process: first, a fast comparison of the query embedding against pre-computed cluster centroids identifies the most relevant topic; second, context is drawn exclusively from the documents within that cluster. In our framework, an ”expert” is therefore not a complex neural model, but a specialized, static collection of documents that collectively covers a distinct topic. This cluster-and-route approach is motivated by the long-standing ”Cluster Hypothesis” in information retrieval: documents that cluster together tend to be relevant to the same information needs [ 17 ] . By treating clusters as the primary retrieval unit, MODE makes a deliberate trade-off. It sacrifices the ability to find an exact nearest-neighbor document in favor of finding a thematically-focused context, which can reduce noise and improve the signal-to-noise ratio of the information provided to the LLM. This makes retrieval exceptionally fast—a single pass over a small set of centroids—and eliminates the need for a re-ranking stage entirely. We summarize our contributions as follows: • A Lightweight RAG Framework: We present MODE, an architectural alternative to standard RAG that replaces vector databases and re-rankers with a cluster-and-route mechanism, optimized for efficiency and simplicity in low-to-medium resource settings. • Principled and Practical Design: We ground our method in the principles of cluster-based retrieval and provide a practical, end-to-end recipe with robust defaults for chunking, clustering (HDBSCAN+KMeans), and centroid management. • Empirical Validation: Through experiments on HotpotQA and SQuAD, we demonstrate that MODE is not only more efficient but also achieves competitive or superior performance in generation quality compared to a traditional RAG baseline, especially at smaller corpus scales. We include ablations that explore the impact of cluster granularity and routing strategy. 2 Related Work Our work builds on three main pillars of research: Retrieval-Augmented Generation (RAG) and cluster-based retrieval. 2.1 Retrieval-Augmented Generation (RAG) The dominant RAG paradigm, popularized by Lewis et al. [ 8 ] , enhances LLMs by retrieving relevant passages from an external corpus. A significant body of subsequent work has focused on improving the components of this pipeline, such as developing more powerful dense retrievers [ 7 ] , sophisticated re-ranking models to improve precision [ 12 , 9 ] , and dynamic systems that adapt the retrieval process [ 3 ] . However, these advances often assume a large-scale setting and introduce computational and infrastructural costs that are misaligned with the needs of smaller, domain-specific applications. MODE offers a direct alternative to this standard retrieval architecture. 2.2 Mixture of Experts and Modular Architectures The ”Mixture of Experts” (MoE) concept in machine learning involves routing an input to one of several specialized sub-models, or ”experts,” to increase model capacity efficiently [ 16 , 2 ] . This principle of modularity has been extended to LLMs for domain adaptation and efficiency [ 5 , 1 ] . MODE adapts this philosophy metaphorically: instead of routing to neural experts, it routes a query to a document expert —a semantically coherent cluster of documents—thereby leveraging specialization at the data level to provide focused, relevant context. 2.3 Cluster-Based Retrieval The idea that semantically related documents tend to be relevant to the same queries, known as the Cluster Hypothesis in Information Retrieval [ 17 ] , is foundational to our work. This principle has been historically used to accelerate search by first selecting promising clusters and then searching within them [ 4 ] . More recently, methods like ColBERTv2 have used clustering for efficient indexing [ 15 ] . While these methods use clustering to optimize a step within a larger pipeline, MODE’s novelty lies in using the cluster-and-route mechanism as a complete replacement for the vector database lookup and re-ranking stages, creating a fundamentally simpler and more lightweight RAG architecture. 3 Background: The Cost of Traditional RAG Systems To motivate our design, we first analyze the standard RAG pipeline and its associated costs, particularly in the context of small-to-medium corpora. The process typically involves [ 8 ] : 1. Text Chunking & Embedding: Documents are split into chunks and converted into dense vector embeddings. 2. Indexing in a Vector Database: Embeddings are stored in a specialized database (e.g., FAISS, Pinecone) optimized for Approximate Nearest Neighbor (ANN) search. 3. ANN Retrieval: A query embedding is used to retrieve the top- k most similar chunks from the database. 4. Re-ranking: A powerful but slow cross-encoder model re-scores the top- k candidates to improve relevance. 5. LLM Synthesis: The final, re-ranked passages are passed as context to an LLM for generation. Latency and Infrastructure Overheads. For corpora with