The Coverage Illusion: From Pre-retrieval Routing Failure to Post-retrieval Cascades in a Production RAG System
Zafar Hussain, Kristoffer Nielbo · 2026
Read The Coverage Illusion: From Pre-retrieval Routing Failure to Post-retrieval Cascades in a Production RAG System by Zafar Hussain, Kristoffer Nielbo (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-27220.txt.
Introduction
Hussain and Nielbo study retrieval in a live question-answering system on the Danish National Encyclopedia. Synthetic queries suggest over 90 percent of queries need language-model augmentation for high coverage, but under the live deferral policy only 27.8% of real queries do. The authors call this gap the Coverage Illusion: the need for augmentation shows only after searching, so pre-search routing cannot fix it. Their cheapest-first cascade escalates only when a step returns nothing, lifting quality, cutting latency by about a third, and serving most real queries without augmentation.
Catalog record
| CREATOR | Zafar Hussain, Kristoffer Nielbo |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 63524 |
| LICENSE | http://creativecommons.org/licenses/by-sa/4.0/ |
| RIGHTS | CC BY SA |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.27220
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
The Coverage Illusion: From Pre-retrieval Routing Failure to Post-retrieval Cascades in a Production RAG System Zafar Hussain, Kristoffer Nielbo Abstract In modern RAG pipelines, query augmentation methods such as HyDE and query expansion are applied to every query, resulting in substantial LLM inference costs and increased end-to-end latency. The empirical justification for this overhead in real production traffic remains largely unexplored. We present a case study of the Danish National Encyclopedia, evaluating five retrieval workflows over 20,000 query–workflow pairs from production traffic and synthetic conditions. In this system, synthetic queries suggest that LLM augmentation is needed for over 90% of queries to achieve high retrieval coverage. However, under our production deferral policy, only 27.8% of real user queries need LLM augmentation. We call this gap the Coverage Illusion and attribute it to a structural mismatch between synthetic and real query distributions. Pre-retrieval routing cannot resolve this gap, as the need for LLM augmentation is only revealed after searching the index, a result confirmed by our evaluation of four machine learning paradigms. The coverage gap, undetectable from the query alone, motivates a post-retrieval cascade that runs workflows in cheapest-first order and escalates to LLM augmentation only when a step returns no documents. Operating entirely without training overhead or secondary serving infrastructure, the cascade improves quality by +0.140 Composite Overall points over Always-HyDE, reduces latency by 31.8%, and serves 72.2% of real user queries without LLM augmentation. 1 Introduction Modern Retrieval-Augmented Generation (RAG) systems increasingly use Large Language Models (LLMs) not just for generation but within the retrieval process itself. Techniques such as HyDE 8 and query expansion 12 ; 18 rewrite or enrich queries before the index is searched, often yielding substantial gains in retrieval coverage but at a high cost in compute and latency. Despite this overhead, both are applied uniformly to every query in production systems 31 ; 13 , with no per-query assessment of whether augmentation is necessary. We study the Danish National Encyclopedia, a production RAG system serving real users across a corpus of ~240,000 curated articles. Two features of this deployment make the question of uniform LLM augmentation consequential. The system defers rather than speculates when retrieval returns no relevant documents, so retrieval coverage is the dominant quality factor. Furthermore, real users submit short keyword phrases, quite unlike the verbose conversational queries that dominate English RAG benchmarks 6 . Both features motivate the question of whether LLM augmentation can be applied selectively rather than uniformly. A natural solution is pre-retrieval query routing, in which a lightweight classifier assigns each query to the most likely successful workflow 21 ; 39 . However, we demonstrate that pre-retrieval routing fails because the need for LLM augmentation depends on the retrieval index rather than the query itself. Our analysis yields three main findings. 1. The Coverage Illusion. Our synthetic queries indicate LLM augmentation is necessary for over 90% of queries. On real user queries from the same deployment, only 27.8% require LLM augmentation under our deferral policy, as retrieval-only Hybrid already returns sources for the remaining 72.2%. 2. Pre-retrieval Routing Fails. Across four Machine Learning (ML) paradigms (rule-based heuristics, classification, neural fine-tuning, and regression), every approach fails on real user queries. In an entity-heavy reference corpus, whether a query needs augmentation is a property of the index, not of the query text. 3. Post-retrieval Cascade. Since the need for augmentation is revealed only after retrieval runs, the natural response is a cascade in which retrieval proceeds in cheapest-first order and escalates only when a step returns no documents. The post-retrieval cascade improves quality by +0.140 Composite Overall points over the best static baseline, reduces latency by 31.8%, and invokes LLM augmentation for only 27.8% of queries. 2 Related Work Synthetic vs. Real Query Evaluation. Synthetic queries have become the standard benchmarking tool for RAG systems 28 ; 11 ; 2 ; 33 . This benchmarking practice introduces a fundamental structural mismatch. Synthetic queries tend to be contextually verbose and grammatically complete, whereas real user interactions are characterized by brevity and extreme vocabulary sparsity 35 ; 24 ; 14 . How far this gap skews system-level conclusions, particularly about when retrieval augmentation is necessary, remains unanswered 34 . Production RAG Systems and Case Studies. Despite increasing research on RAG evaluation and adaptive routing 29 ; 33 , empirical analyses grounded in production settings remain relatively rare. Most studies rely on benchmark datasets that may not reflect real-world query distributions 32 ; 27 . Our work addresses this gap by analyzing a production system in which query distributions, retrieval characteristics, and quality incentives are drawn from live traffic, enabling us to measure how far synthetic evaluation diverges from real deployment behavior. Query Routing and Adaptive RAG. Adaptive RAG systems are designed to steer each query toward the most appropriate retrieval path. Frameworks like LangChain 15 and LlamaIndex 17 use hand-crafted heuristics; Gorilla 23 and RouteLLM 20 learn routing policies from data; Self-RAG 1 decides dynamically whether to retrieve at all. All these systems commit to a routing decision before retrieval, based solely on the input query 36 . To our knowledge, no prior work has questioned whether the query alone provides sufficient signal for this, nor explored routing conditioned on what retrieval returns 29 . Advanced Retrieval Strategies for RAG. HyDE 8 and LLM-based query expansion 10 ; 7 ; 18 ; 12 improve retrieval for underspecified queries by rewriting or enriching them. Both methods add substantial inference overhead 16 ; 30 , yet most prior work applies LLM augmentation uniformly to all queries, treating it as universally beneficial 3 . 3 Experimental Setup Our testbed is the Danish National Encyclopedia (lex.dk), a corpus of approximately 240,000 professionally curated articles, semantically chunked, embedded with multilingual-e5-large 37 , and indexed in a vector database. Three properties suit this deployment for the study. It carries real production traffic, its entity-heavy content produces clear retrieval outcomes, and its stable curation ensures performance differences are attributable to the retrieval workflow rather than corpus noise. Five retrieval workflows operate on this index under four query conditions of 1,000 queries each; each pair is scored by an automated LLM judge, yielding 20,000 evaluated pairs (Figure 4 , Appendix). From these, we construct an oracle dataset that identifies the best-performing workflow for each query, which serves as the benchmark for pre-retrieval routing and the cascade. 3.1 Query Conditions We evaluated four query conditions, each with 1,000 queries. One condition draws from production logs (Real User), while the remaining three are synthetic variants spanning a range of formulation styles, from raw LLM output to natural conversational language (Synth-Polluted, Synth-Keywords, and Synth-Conversational). Full details of each condition, including sources and query statistics, are provided in Appendix B (Tables 6 – 7 ). Real User queries come from production search logs and are the primary reference condition for all deployment-relevant conclusions. They have a median length of 5.5 words, typically consist of a bare topic fragment or keyword phrase, as in Menneskets anatomi (‘Human anatomy’). Users name a topic rather than describe it, a pattern that maps naturally to BM25 26 lexical matching but gives dense retrieval little semantic signal to work with. Their brevity and lexical sparsity make them the most