Context-Efficient Retrieval with Factual Decomposition
Yanhong Li, David Yunis, David McAllester, Jiawei Zhou · 2025
Read Context-Efficient Retrieval with Factual Decomposition by Yanhong Li, David Yunis, David McAllester, Jiawei Zhou (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-19574.txt.
Introduction
Yanhong Li and co-authors make retrieval cheaper for language models by chopping documents into atomic facts first. Models that fetch outside text can answer questions about fresh events, but fetching long passages bloats the prompt and slows the model down. The team instead pre-splits the corpus into small, semi-structured fact units.
With the same tight budget of retrieved text, fact-based retrieval beats passage-based retrieval on several question-answering tasks. Less context means faster inference and lower cost. The authors frame the fact store as a kind of episodic memory for the model: compact, searchable, and easy to grow as new text arrives. The method suits any setting where answers must stay current without wasting tokens.
Catalog record
| CREATOR | Yanhong Li, David Yunis, David McAllester, Jiawei Zhou |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 53 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-19574
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Context-Efficient Retrieval with Factual Decomposition Yanhong Li, David Yunis, David McAllester, Jiawei Zhou Abstract There has recently been considerable interest in incorporating information retrieval into large language models (LLMs). Retrieval from a dynamically expanding external corpus of text allows a model to incorporate current events and can be viewed as a form of episodic memory. Here we demonstrate that pre-processing the external corpus into semi-structured ‘‘atomic facts’’ makes retrieval more efficient. More specifically, we demonstrate that our particular form of atomic facts improves performance on various question answering tasks when the amount of retrieved text is limited. Limiting the amount of retrieval reduces the size of the context and improves inference efficiency. 1 1 1 Code available at https://github.com/yanhong-lbh/DecompFacts . 1 Introduction Figure 1: Example of datastore used for knowledge retrieval in our approach compared with typical fixed-size text chunks in RAG. We retrieve much shorter contexts. Although large language models (LLMs) demonstrate remarkable capabilities across various tasks, their inability to continuously adapt to dynamic or domain-specific knowledge without parameter updates remains a substantial limitation. To address this limitation, retrieval-augmented generation (RAG) supplements models with some external knowledge source during inference ( Lewis et al. 2020 ; Borgeaud et al. 2022a ; Ram et al. 2023 ; Li et al. 2025 ) . Typically these models treat the external source as a set of arbitrarily segmented blocks of raw text. However, there has also been interest in using more structured external knowledge sources such as knowledge graphs ( Edge et al. 2024 ; Peng et al. 2024 ) , compressed documents ( Xu et al. 2024 ) or document trees ( Sarthi et al. 2024 ) . In each case one can identify a “unit of retrieval” where one retrieves some set of such units, such as a set of documents or a set of knowledge graph triples. Various candidates for units of retrieval, such as different forms of “atomic facts,” have been formulated ( Chen et al. 2023 ; Jiang et al. 2024c ; Min et al. 2023 ; Gunjal and Durrett 2024 ) . In designing units of retrieval there is a tension between concise but brittle logical representations, such as knowledge graph triples, and highly expressive and nuanced, but verbose and unstructured, chunks of raw text. We propose an intermediate retrieval unit that we call an entity-description pair (EDP). This is a pair of an ‘‘entity’’ 2 2 2 Here we take a very liberal notion of “entity” not to be confused with the narrow notion of entity used in named entity recognition. and some form of description of that entity. For example the entity might be “Alan Turing’s contributions” and the factual description could be “Pioneered modern computing, laid the foundation of computer science and engineering.” Each EDP is a structured piece of information, like in structured databases, but also enjoys the flexibility of natural language. See Figure 1 . We use a three-step language model prompting protocol to decompose a chunk of free text into a collection of EDPs and use the EDPs as the unit of retrieval in the resulting EDP knowledge base (KB). Our approach, referred to as FADER ( F actual A tomic D ecomposition for E fficient R etrieval), aims to reduce RAG inference overhead by pre-processing external corpora into atomic facts for efficient, high-precision retrieval. Our main result is a demonstration that on various challenging question answering benchmarks EDP KB retrieval with FADER achieves better accuracy when the amount of retrieval (the number of retrieved tokens) is limited. This can be phrased as improving the “context-efficiency” of RAG. We are also optimistic that our formulation of EDP KBs is a significant step toward more structured yet expressive internal representations of knowledge. 2 Related Work Context-Efficient Retrieval As we will see in experiments, our approach achieves superior performance in context-efficient retrieval , which we define as RAG methods aiming to reduce retrieved contexts for cost-effective LLM generations. Previous related work involves various compression methods. Some focus on vector-based compression, where models learn to compress long contexts into compact memory slots through end-to-end training ( Ge et al. 2024 ; Cheng et al. 2024 ) . Others are text-based compression, which includes training rerankers Pradeep et al. 2023 , applying extractive summarization Xu et al. 2024 , or training abstractive summarizers to compress the retrieved context ( Xu et al. 2024 ; Jiang et al. 2024b ) . We also reduce retrieved contexts, but rather than compressing them post-retrieval , we achieve context efficiency from the outset through improved knowledge representation pre-retrieval . Post-retrieval context compression methods still rely on token chunks as coarse units of knowledge for retrieval, whereas we structure the knowledge more efficiently with clear, well-defined representations that maintain high expressivity. Knowledge Representation for Retrieval Most previous works directly segment source documents into equal-length text chunks, each containing hundreds of tokens ( Lewis et al. 2020 ; Ram et al. 2023 ; Borgeaud et al. 2022a ) . Recent research has explored alternative formats for knowledge representation, such as indexing source documents using knowledge graphs ( Edge et al. 2024 ) , hierarchical tree structures Sarthi et al. 2024 , or more relaxed versions of knowledge graphs Liang et al. 2024 . Unlike these works, we don’t rely on any explicit relational structures; our knowledge datastore consists of flat, semi-structured entity-description pairs . The work most similar to ours is Chen et al. 2023 , which decomposes each sentence in Wikipedia into individual propositions for retrieval. However, we propose a novel method where speculated queries generated by the LM guide the fact extraction, and repeated samples of factual decompositions enhance our database. These techniques yield significant performance improvements by enabling more targeted information extraction and increased coverage of constructed facts, resulting in more relevant and concise data being incorporated during inference. 3 3 3 More general background of RAG is in Appendix A . 3 Methodology Figure 2: Overview of the FADER pipeline. As shown in Figure 2 , FADER consists of three main steps: (1) question speculation, (2) atomic fact extraction, and (3) knowledge base (KB) augmentation. This approach allows to decompose long documents into concise, factual entity-description pairs (EDPs), building up a semi-structured KB for RAG, reducing retrieval overhead on the lengths of contexts to improve inference efficiency. 3.1 Question Speculation Let D represent a long document, which is split into N equal-length chunks D=\{D_{1},D_{2},\dots,D_{N}\} , where each chunk D_{i} contains approximately the same number of tokens. For each chunk D_{i} , we prompt a LM to speculate a set of possible questions Q_{i}=\{q_{i1},q_{i2},\dots,q_{iJ}\} , where q_{ij} represents a potential question one might ask about the chunk D_{i} . This question speculation process helps direct the extraction of relevant knowledge and facilitates targeted information retrieval later. Formally, given a document chunk D_{i} , we define the question speculation process as: Q_{i}=\text{LM}_{\text{speculate}}(D_{i}) , where \text{LM}_{\text{speculate}} denotes the question-speculation language model. The result is a set of speculative questions Q_{i} for each document chunk D_{i} . Prompts for \text{LM}_{\text{speculate}} are shown in Table 5 and Table 6 in Appendix G . 3.2 Query-Guided Factual Decomposition Once we have the speculative questions Q_{i} , we feed both the set of