Enhancing Document VQA Models via Retrieval-Augmented Generation
Eric López, Artemis Llabrés, Ernest Valveny · 2025
Read Enhancing Document VQA Models via Retrieval-Augmented Generation by Eric López, Artemis Llabrés, Ernest Valveny (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-18984.txt.
Introduction
Eric López, Artemis Llabrés, and Ernest Valveny answer questions over long documents by fetching proof first. Their 2025 paper deals with document question answering across dozens of pages, where top systems stuff all pages in or lean on huge vision models. Both paths eat memory. Retrieval-augmented generation picks a small proof set first, then answers from it.
Two fetch styles were tested across models and benchmarks: text fetch over OCR words, and pure visual fetch with no text step. The text style lifted scores sharply over the stuff-all baseline, and the visual style gained too. Fetch and rerank steps drove the gains, while layout chunking added nothing.
Catalog record
| CREATOR | Eric López, Artemis Llabrés, Ernest Valveny |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 29 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2508-18984
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Enhancing Document VQA Models via Retrieval-Augmented Generation Eric López, Artemis Llabrés, Ernest Valveny Abstract Document Visual Question Answering (Document VQA) must cope with documents that span dozens of pages, yet leading systems still concatenate every page or rely on very large vision-language models, both of which are memory-hungry. Retrieval-Augmented Generation (RAG) offers an attractive alternative, first retrieving a concise set of relevant segments before generating answers from this selected evidence. In this paper, we systematically evaluate the impact of incorporating RAG into Document VQA through different retrieval variants—text-based retrieval using OCR tokens and purely visual retrieval without OCR—across multiple models and benchmarks. Evaluated on the multi-page datasets MP-DocVQA, DUDE, and InfographicVQA, the text-centric variant improves the "concatenate-all-pages" baseline by up to +22.5 ANLS, while the visual variant achieves +5.0 ANLS improvement without requiring any text extraction. An ablation confirms that retrieval and reranking components drive most of the gain, whereas the layout-guided chunking strategy—proposed in several recent works to leverage page structure—fails to help on these datasets. Our experiments demonstrate that careful evidence selection consistently boosts accuracy across multiple model sizes and multi-page benchmarks, underscoring its practical value for real-world Document VQA. Project page: https://pikurrot.github.io/RAG-DocVQA/ 1 Introduction Document VQA aims to enable models to answer questions about the contents of documents, requiring multimodal reasoning over textual, spatial, and visual elements. While early research mainly addressed single-page scenarios through popular benchmarks such as DocVQA, the majority of real-world documents—including manuals, scientific papers, and technical reports—extend across multiple pages. Addressing this gap, MP-DocVQA was recently proposed as a multi-page extension of the original DocVQA dataset, posing natural language questions over sequences of up to twenty pages, accompanied by OCR-extracted text and page images. Existing state-of-the-art multi-page models, such as Hi-VT5, leverage hierarchical transformer architectures to summarize individual pages before synthesizing the final answer. However, their performance deteriorates as the document length grows, since cross-page reasoning significantly increases complexity. While Large Visual Language Models (LVLMs) such as Qwen2.5-VL can handle long-context inputs, they require substantial computational resources and GPU memory, making deployment costly and inefficient. A promising alternative is Retrieval-Augmented Generation (RAG), which has recently demonstrated effectiveness in open-domain question answering by first retrieving compact subsets of relevant evidence, thus drastically reducing the inference cost and noise introduced by irrelevant content. In the context of multi-page Document VQA, a RAG pipeline involves segmenting documents into concise textual chunks or visual patches, retrieving the most pertinent segments based on the question, and subsequently generating answers conditioned solely on these retrieved segments. In this paper, we hypothesize that augmenting existing small and medium-sized Document VQA models with RAG can significantly mitigate the challenges posed by multi-page scenarios. Specifically, we argue that this approach yields performance improvements in both accuracy and computational efficiency across various models and datasets. To validate this hypothesis, our contributions are as follows: • We propose and implement textual RAG variants for two representative Vision Language Models: VT5 and Qwen2.5-VL-7B-Instruct, along with a visual counterpart for Pix2Struct, an OCR-free vision transformer model. • Through comprehensive experiments on MP-DocVQA, DUDE, and InfographicVQA, we demonstrate consistent improvements in question-answering accuracy, ANLS scores, and retrieval metrics, thereby validating our hypothesis. 2 Related Work 2.1 Datasets and Task Evolution Document Visual Question Answering was initially formalized with the DocVQA dataset [ 20 ] , comprising 50,000 questions posed over more than 12,000 industry-related single-page document images, where answers are explicitly present in the OCR-extracted text. InfographicVQA [ 19 ] expanded the task visually, introducing infographic-rich images accompanied by questions that occasionally require numerical or categorical reasoning beyond mere text extraction. Subsequently, DocCVQA [ 22 ] introduced a retrieval-oriented scenario, where questions are answered by retrieving relevant evidence from a collection of 14,362 single-page documents. Recognizing the limitations of single-page scenarios, MP-DocVQA [ 23 ] extended the original DocVQA dataset to multi-page contexts by augmenting each document-question pair with preceding and following pages, forming sequences of up to 20 pages. Concurrently, DUDE [ 15 ] provided diverse multi-page documents from various domains, featuring questions that require multi-step reasoning and both extractive and abstractive answers. Additionally, the recent MMLongBench-Doc dataset [ 18 ] features 1,091 questions over 135 long PDF documents with rich layouts, where 33% of the questions require cross-page reasoning and 22.5% are unanswerable, which tests model robustness. 2.2 From OCR-based to OCR-free Approaches Initial Document VQA models largely relied on OCR-extracted text, adapting pre-trained language models such as BERT [ 8 ] with a classification head to predict the start and end indices where the answer appeared in the text, and T5 [ 21 ] for generative extractive question answering from OCR tokens. LayoutLM [ 28 ] introduced spatial understanding by augmenting textual embeddings from BERT with 2-dimensional positional information derived from OCR bounding boxes. LayoutLMv2 [ 27 ] then integrated visual embeddings from the document image and added relative positional biases to the self-attention mechanism. LayoutLMv3 [ 12 ] further refined this by encoding images as patch-level embeddings. To further leverage visual modalities, DocFormer [ 1 ] combined textual, visual, and spatial features via a novel multi-modal self-attention mechanism, using a ResNet [ 10 ] backbone for image features. Later, DocFormerv2 [ 2 ] replaced the ResNet CNN with a linear projection of image patches and adopted a T5-based encoder-decoder architecture. Addressing multi-page reasoning, Hi-VT5 [ 23 ] introduced a hierarchical transformer where the encoder summarizes each page independently, and the decoder attends to these summaries to generate the answer. GRAM proposed inserting additional document-level transformer layers between page-level encodings, using document tokens to propagate information across pages, achieving state-of-the-art performance when combined with DocFormerv2. More recently, OCR-free models emerged, eliminating explicit text extraction. Pix2Struct [ 16 ] proposed a fully end-to-end approach without explicit OCR, pretraining a vision transformer encoder to transcribe masked web screenshots into simplified HTML. Nevertheless, handling multi-page scenarios remained challenging due to growing computational complexity. 2.3 Generic Vision-Language Models Recently, generic vision-language models (VLMs), such as Qwen2.5-VL [ 3 ] , have started being explored for Document VQA tasks. These models can directly process combined visual and textual inputs thanks to extensive pre-training on large multimodal datasets. However, a major issue with these models is their high computational cost and GPU memory usage, making practical deployment difficult. To mitigate such limitations, retrieval-based strategies like ColPali [ 9 ] have been adopted, which extends the PaliGemma [ 4 ] vision-language model with a ColBERT-style [ 14 ] late interaction mechanism, generating visual page embeddings optimized for retrieval tasks, thus improving evidence selection prior to answer generation. 3 Methodology We adapt Retrieval-Augmented Generation (RAG) to multi-page Document VQA through a three-stage pipeline comprising indexing, retrieval, and generation. In particular, we propose two complementary variants: a textual RAG, applied to two language-based models—a smaller encoder–decoder (VT5) and a larger vision–language model (Qwen2.5-VL-7B)—and a visual RAG, where retrieval and generation operate directly on image inputs without explicit OCR, using Pix2Struct as our baseline model. 3.1 Textual RAG Figure 1: Overview of the Textual RAG pipeline. Offline (pink panel): A multi-page document is segmented into chunks consisting of OCR (text and boxes) and the image crop of the chunk. A Bi-Encoder converts the chunk text into a dense embedding for later retrieval. Online (green