LFRAG: Layout-oriented Fine-grained Retrieval-Augmented Generation on Multimodal Document Understanding
Yifan Zhu, Yu Mi, Yue Lu, Yanchu Guan, Zhixuan Chu* · 2026
Read LFRAG: Layout-oriented Fine-grained Retrieval-Augmented Generation on Multimodal Document Understanding by Yifan Zhu, Yu Mi, Yue Lu, Yanchu Guan, Zhixuan Chu* (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-22829.txt.
Introduction
Zhu and four colleagues develop LFRAG, a retrieval system for visually rich documents that works at the level of layout blocks rather than whole pages. Page-level retrieval misses fine detail and drags in redundant context, so their method segments documents into coherent blocks and fuses local meaning with global layout. On their new LFDocQA benchmark it led baselines on retrieval, lifted answer accuracy by over seven percent, and cut generation tokens sharply.
Catalog record
| CREATOR | Yifan Zhu, Yu Mi, Yue Lu, Yanchu Guan, Zhixuan Chu* |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 65930 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.22829
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
LFRAG: Layout-oriented Fine-grained Retrieval-Augmented Generation on Multimodal Document Understanding Yifan Zhu, Yu Mi, Yue Lu, Yanchu Guan, Zhixuan Chu* Abstract Multimodal Retrieval-Augmented Generation (RAG) has emerged as an effective paradigm for enhancing Large Language Models (LLMs) with external knowledge. However, existing multimodal RAG systems predominantly rely on coarse-grained page-level retrieval, which fails to capture fine-grained semantic and layout structures in visually rich documents, thereby compromising retrieval accuracy and leading to redundant context in downstream tasks. To address these issues, we propose Layout-oriented Fine-grained Retrieval-Augmented Generation (LFRAG), a novel framework that advances multimodal RAG from page-level to block-level retrieval. We perform layout segmentation to construct semantically coherent fine-grained retrieval units and design a semantic–layout fusion encoder that integrates local semantics with global context via cross-attention. With block-level late interaction retrieval, LFRAG enables precise query–content alignment and reduces irrelevant content for downstream generation. To enable rigorous evaluation, we construct LFDocQA, a large-scale benchmark with block-level annotations spanning diverse document types, designed to assess both multimodal document retrieval and question answering with greater granularity than existing datasets. Extensive experiments on LFDocQA demonstrate that LFRAG achieves state-of-the-art performance on retrieval tasks, outperforms the best baseline by 7.20% in answer accuracy, and reduces token consumption by 73.07% in generation tasks, confirming LFRAG as an accurate and efficient framework for multimodal RAG over visually rich documents. Our code and datasets will be released soon. Figure 1: Comparison between page-level and layout-oriented block-level retrieval. Our LFRAG significantly improves retrieval accuracy and generation efficiency by reducing irrelevant context and token consumption compared to page-level RAG. 1 Introduction Retrieval-Augmented Generation (RAG) empowers Large Language Models (LLMs) with access to external knowledge resources [ 8 ] . While significant progress has been made in text-based retrieval [ 11 ] , a wealth of real-world knowledge resides in visually rich documents such as scientific papers, financial reports, and infographics, whose intricate layout structures and multimodal content cannot be fully represented by plain text alone. Effectively capturing and fusing semantic content with visual structural information thus makes multimodal RAG an active research frontier at the intersection of NLP and CV. Existing multimodal RAG methods fall into two categories. OCR-based methods [ 26 , 15 , 4 ] convert documents to plain text before retrieval, inevitably discarding visual and structural information such as table row-column relationships, chart trends, and figure-text layout logic. VLM-based methods, represented by ColPali [ 7 ] , encode page images directly into multi-vector embeddings and retrieve by vector similarity. However, they introduce three key limitations: (1) page-level retrieval treats an entire page as the minimal retrieval unit, ignoring intra-page hierarchical semantics; (2) fixed-resolution image patching forces high-resolution pages to be downsampled, causing fine-grained information loss; and (3) the absence of layout-oriented modeling leads to poor performance on complex-layout and structured tabular content. The root cause of these limitations is page-level retrieval granularity. Returning entire pages inevitably introduces substantial irrelevant content, increasing the computational overhead of downstream generation and introducing distracting context that elevates the risk of hallucination. Precise localization of query-relevant content is therefore central to reliable multimodal RAG. A natural solution is to shift the minimal retrieval unit from entire pages to semantically coherent document blocks. To this end, we propose block-level retrieval, which treats each such block as an independent retrieval unit, enabling precise localization of query-relevant content, filtering out redundant context, and fundamentally improving retrieval accuracy and mitigating hallucination in generation. Nevertheless, block-level retrieval for multimodal documents faces two key challenges: how to segment documents into semantically complete blocks while preserving layout information, and how to design a VLM-based encoding and retrieval framework that effectively integrates fine-grained block features, global page context, and inter-block spatial relationships. To tackle these challenges, we propose Layout-oriented Fine-grained Retrieval-Augmented Generation (LFRAG), a novel multimodal RAG framework that advances the retrieval paradigm from the page-level to the block-level. The core idea of LFRAG is to integrate document layout analysis into the retrieval pipeline. First, a dedicated document layout model is used to perform initial page segmentation. To mitigate the issue of excessive fragmentation, we design a semantic block aggregation strategy that merges fine-grained regions into semantically coherent blocks. Second, we design a semantic-layout feature fusion encoder that jointly encodes block-level semantic features and global page layout context, incorporating block-type tags to enhance structural awareness and enable layout-oriented representation learning. Finally, a late interaction mechanism is employed for fine-grained block-level retrieval, and the retrieved top- k relevant blocks are fed into a VLM for augmented generation. This design equips LFRAG with both fine-grained information mining capabilities and holistic layout understanding capabilities, enabling precise matching between user queries and multimodal document content. Meanwhile, it significantly reduces the irrelevant context passed to the generation model, thereby achieving more accurate evidence localization and answer generation. To enable rigorous fine-grained evaluation, we construct LFDocQA, the first multimodal document benchmark with block-level bounding box annotations for retrieval and QA beyond page-level metrics to precisely measure retrieval quality. Experiments show that LFRAG consistently outperforms page-level baselines in both retrieval accuracy and generation quality, and remains competitive on two public page-level benchmarks when block-level results are fused to the page-level, demonstrating strong generalization. Our contributions are summarized as follows: • We design LFRAG, a novel multimodal RAG framework integrating layout segmentation, semantic-layout fusion encoding and block-level late interaction to enable precise retrieval and efficient generation. • We construct LFDocQA, the first multimodal document block-level retrieval and question answering benchmark, filling the gap of existing page-level evaluation and supporting rigorous fine-grained multimodal retrieval and QA assessment. • Extensive experiments demonstrate that LFRAG achieves state-of-the-art retrieval and generation performance, significantly reduces generation overhead, and exhibits strong generalization across diverse document types and evaluation settings. Figure 2: The framework of LFRAG. Given a query, LFRAG retrieves relevant information from a document corpus by segmenting pages into structured semantic blocks and leveraging layout-oriented representation learning for fine-grained matching, enabling more accurate evidence localization for downstream generation. 2 Related Works 2.1 Multimodal Retrieval-Augmented Generation Retrieval-Augmented Generation (RAG) enhances large language models (LLMs) by retrieving relevant external knowledge to ground generation and improve factuality [ 9 , 16 , 6 , 8 , 11 ] . With the increasing prevalence of visually rich documents, retrieval has extended from pure text to multimodal settings. Early multimodal RAG typically adopt OCR-based pipelines, which convert document pages into plain text before applying text retrieval methods such as BM25 [ 26 ] , dense retrievers like NV-Embed-V2 [ 15 ] , or unified embedding models such as BGE-M3 [ 4 ] . However, these approaches inevitably lose rich visual information, especially for tables, figures, and complex layouts. With the advancement of vision-language models (VLMs), recent research [ 21 , 7 , 39 ] shift toward directly encoding document images for retrieval. Methods such as ColPali [ 7 ] employ VLMs to encode document pages as visual tokens and perform late interaction retrieval, enabling alignment between queries and image patches. Some works [ 39 , 31 , 30 , 35 ] further integrates VLM-based retrieval into end-to-end RAG pipelines. Despite their success, most existing methods operate at the coarse page level, retrieving entire pages as atomic units, which introduces substantial irrelevant content [ 7 , 22 ] . Recent works