MDKeyChunker : Single-Call LLM Enrichment with Rolling Keys and Key-Based Restructuring for High-Accuracy RAG

Bhavik Mangla · 2026

Read MDKeyChunker : Single-Call LLM Enrichment with Rolling Keys and Key-Based Restructuring for High-Accuracy RAG by Bhavik Mangla (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-23533.txt.

Introduction

Mangla asks what one language model call per text chunk buys a search pipeline. Many RAG setups pay that call to add titles, summaries, and questions at index time. His preregistered tests compare paid enrichment with free structure and stronger retrievers. The verdict: upgrade the retriever first and chunk at headers. Enrichment helps most under word search, and ties with contextual retrieval at far lower token cost. Under strong retrievers, a free section prefix matches or beats the paid route. Late chunking, by contrast, lowers recall.

Catalog record

CREATORBhavik Mangla
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE28356
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.23533

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

MDKeyChunker : Single-Call LLM Enrichment with Rolling Keys and Key-Based Restructuring for High-Accuracy RAG Bhavik Mangla Abstract RAG pipelines typically rely on fixed-size chunking, which ignores document structure, fragments semantic units across boundaries, and requires multiple LLM calls per chunk for metadata extraction. We propose MDKeyChunker , a three-stage pipeline for Markdown documents that (1) performs structure-aware chunking treating headers, code blocks, tables, and lists as atomic units; (2) enriches each chunk via a single LLM call extracting title, summary, keywords, typed entities, hypothetical questions, and a semantic key, while propagating a rolling key dictionary to maintain document-level context; and (3) restructures chunks by merging those sharing the same semantic key via bin-packing, co-locating related content for retrieval. The single-call design extracts all seven metadata fields in one LLM invocation, eliminating the need for separate per-field extraction passes. Rolling key propagation replaces hand-tuned scoring with LLM-native semantic matching. An empirical evaluation on 30 queries over an 18-document Markdown corpus shows Config D (BM25 over structural chunks) achieves Recall@5 = 1.000 and MRR = 0.911, while dense retrieval over the full pipeline (Config C) reaches Recall@5 = 0.867. MDKeyChunker is implemented in Python with four dependencies and supports any OpenAI-compatible endpoint. 1 1 1 Code: https://github.com/bhavik-mangla/MDKeyChunker . Keywords: Retrieval-Augmented Generation, document chunking, LLM enrichment, semantic keys, chunk restructuring, Markdown parsing 1 Introduction Retrieval-Augmented Generation (RAG) combines a retriever with a large language model (LLM) to ground generated text in external knowledge sources ( Lewis et al. 2020 ; Borgeaud et al. 2022 ) . Standard RAG pipelines index document chunks —typically fixed-length segments of 256–512 tokens—and retrieve the top- k most relevant chunks to augment the LLM’s context window. While effective for factoid queries, this approach exhibits three systematic failure modes. Chunk boundary fragmentation. Fixed-size splitting ruptures document coherence, scattering related content across segments ( Zhao et al. 2024 ) . A table may be separated from its caption; a code block may be split from its explanatory paragraph. Zhao et al. 2024 demonstrates that such fragmentation degrades both retrieval recall and downstream answer quality. Furthermore, the chunk size trade-off is well documented: small chunks improve vector search recall but provide insufficient context to the generator, while large chunks dilute embedding specificity ( Barnett et al. 2024 ) . Metadata extraction cost. Prior work shows that structured metadata (summaries, entities, keywords, and questions) improves retrieval precision ( Granata et al. 2025 ) . Industry systems further adopt similar strategies. However, traditional pipelines chain separate extraction steps—NER, keyword extraction, summarization, question generation—each requiring an independent model inference pass. For a corpus of n chunks, this scales as O(n\cdot m) LLM invocations where m is the number of extraction stages, creating significant cost and latency barriers at scale. Contextual isolation. Even when metadata is extracted, each chunk is processed independently. Without inter-chunk context propagation, independently extracted keys suffer from synonym proliferation (e.g., “admissions timeline” vs. “application deadlines” for the same subtopic). MDKeyChunker addresses all three failure modes through a unified three-stage pipeline: 1. A Markdown structural parser splits documents along semantic boundaries (headers, code fences, tables, lists), preserving atomic content units. 2. A single-call LLM enrichment module extracts all metadata fields simultaneously—including a semantic key —while receiving a rolling key dictionary accumulated from prior chunks. 3. A key-based restructuring module merges chunks sharing the same semantic key via bin-packing, subject to a maximum merged size constraint. The design extends contextual retrieval ideas with explicit key propagation and post-enrichment restructuring. Contributions. 1. A single-call LLM enrichment protocol that extracts seven metadata fields per chunk in one invocation, replacing separate per-field extraction passes and producing internally coherent metadata (§ 3.2 ). 2. A rolling key propagation mechanism that maintains document-level topical context without scoring formulas or threshold tuning (§ 3.2 ). 3. A key-based chunk restructuring algorithm that merges semantically related chunks globally via bin-packing (§ 3.3 ). 4. Formal complexity analysis and empirical evaluation on 30 queries over an 18-document corpus (§ 3.6 , § 5 ). 5. A fully open-source Python implementation with 76 unit tests (§ 3.5 ). 2 Related Work 2.1 Chunking Strategies for RAG Fixed-size chunking remains the default in production RAG systems ( Barnett et al. 2024 ) . Zhao et al. 2024 proposed Meta-Chunking, which identifies split points via perplexity-based boundary detection. Hierarchical alternatives construct offline summary trees, while frameworks such as LlamaIndex provide sentence-level indexing and dynamic context expansion in practice. MDKeyChunker follows the offline-enrichment paradigm of TreeRAG but replaces summary trees with key-based merging, operating at the section level rather than the sentence level. 2.2 Semantic Enrichment Granata et al. 2025 integrate knowledge graphs with hybrid ranking. Gao et al. 2023 demonstrate that pre-generating hypothetical questions bridges the query-document length mismatch in embedding space. Anthropic 2024 prepend a document-context sentence to each chunk, yielding substantial retrieval improvements. MDKeyChunker extracts commonly used metadata fields in a single LLM call, extending contextual retrieval with an explicit rolling key dictionary. 2.3 Graph-Based and Structured RAG GraphRAG ( Edge et al. 2024 ) constructs a knowledge graph for corpus-level reasoning. RAPTOR ( Sarthi et al. 2024 ) builds recursive summarization trees. HippoRAG ( Gutiérrez et al. 2024 ) draws on hippocampal memory models. These approaches emphasize global structure and corpus-level reasoning, but typically introduce additional preprocessing and indexing overhead. In contrast, MDKeyChunker operates at the chunk level via rolling keys and key-based merging, avoiding explicit graph construction while preserving inter-chunk topical links through implicit edges in related_keys . 2.4 Comparison of Approaches Table 1 : Comparison of chunking and enrichment approaches for RAG. System Chunking Enrichment Inter-Chunk Restructuring Calls/Chunk Fixed-size Token window None None None 0 Meta-Chunking ( Zhao et al. 2024 ) Perplexity None None None 0 GraphRAG ( Edge et al. 2024 ) Sent./passage Entity ext. KG None Multiple RAPTOR ( Sarthi et al. 2024 ) Clustering Rec. summaries Tree hier. Cluster merge Multiple Contextual Ret. Any Context prepend None None 1 MDKeyChunker MD structure Single-call (7) Rolling keys Key merge 1 3 Methodology MDKeyChunker processes a Markdown document D through three sequential stages. We denote the initial chunk sequence as C=[c_{1},c_{2},\ldots,c_{n}] and the final restructured sequence as C^{\prime}=[c^{\prime}_{1},c^{\prime}_{2},\ldots,c^{\prime}_{n^{\prime}}] where n^{\prime}\leq n . 3.1 Stage 1: Markdown Structural Splitting Block parsing. The parser identifies six block types: header , code (fenced or indented), table , list , blockquote , and paragraph . Each block records its type, raw content, line range, and (for headers) heading level \ell\in\{1,\ldots,6\} . A header stack H tracks the current section hierarchy: when a header of level \ell is encountered, all entries with level \geq\ell are popped and the new header is pushed. The current section path is the concatenation of all entries in H . Atomicity constraints. The following elements are never split across chunks: fenced code blocks (delimited by ‘‘‘ or ~~~ ), tables (including header rows and separator lines), list items, and blockquotes. Size management. Blocks are grouped into chunks subject to two thresholds: \tau_{\min} (minimum chunk size in characters, default 100) and \tau_{\max} (soft maximum, default 1500). The overall pipeline is illustrated in Figure 1 . Each chunk c_{i} is represented as: c_{i}=\bigl(\text{text}_{i},\;\text{section\_title}_{i},\;\text{content\_types}_{i},\;\text{start\_line}_{i},\;\text{end\_line}_{i}\bigr). Figure 1 : MDKeyChunker three-stage pipeline: structural Markdown splitting (Stage 1), single-call LLM enrichment with rolling

Read the complete text