Are We on the Right Way for Assessing Document Retrieval-Augmented Generation?

Wenxuan Shen, Mingjia Wang, Yaochen Wang, Dongping Chen, Junjie Yang,, Yao Wan, Weiwei Lin, South China University of Technology, Huazhong University of Science, Technology, University of Maryland · 2025

Read Are We on the Right Way for Assessing Document Retrieval-Augmented Generation? by Wenxuan Shen, Mingjia Wang, Yaochen Wang, Dongping Chen, Junjie Yang,, Yao Wan, Weiwei Lin, South China University of Technology, Huazhong University of Science, Technology, University of Maryland (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-03644.txt.

Introduction

Wenxuan Shen and co-authors wrote a 2025 paper that asks whether the benchmarks for document retrieval tools are fit for the job. Such tools fetch texts, then have a large model summarize them. Their benchmarks often measure one part of the chain, on synthetic data with thin ground truth. Real bottlenecks go unseen.

The team built a new benchmark set: over three thousand documents, in six languages and four document types, with over five thousand questions. The evidence pages for each question were scanned exhaustively and verified by hand. Runs of nine retrieval tools, four large models, and four full systems fill the study. Retrieval tools that read images close the gap on text ones. One flaw stood out: systems often answer with no evidence in hand, too confident in themselves.

Catalog record

CREATORWenxuan Shen, Mingjia Wang, Yaochen Wang, Dongping Chen, Junjie Yang,, Yao Wan, Weiwei Lin, South China University of Technology, Huazhong University of Science, Technology, University of Maryland
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE75 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2508-03644

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Are We on the Right Way for Assessing Document Retrieval-Augmented Generation? Wenxuan Shen, Mingjia Wang, Yaochen Wang, Dongping Chen, Junjie Yang,, Yao Wan, Weiwei Lin, South China University of Technology, Huazhong University of Science, Technology, University of Maryland Abstract Retrieval-Augmented Generation (RAG) systems using Multimodal Large Language Models (MLLMs) show great promise for complex document understanding, yet their development is critically hampered by inadequate evaluation. Current benchmarks often focus on specific part of document RAG system and use synthetic data with incomplete ground truth and evidence labels, therefore failing to reflect real-world bottlenecks and challenges. To overcome these limitations, we introduce Double-Bench : a new large-scale, multilingual, and multimodal evaluation system that is able to produce fine-grained assessment to each component within document RAG systems. It comprises 3,276 documents (72,880 pages) and 5,168 single- and multi-hop queries across 6 languages and 4 document types with streamlined dynamic update support for potential data contamination issues. Queries are grounded in exhaustively scanned evidence pages and verified by human experts to ensure maximum quality and completeness. Our comprehensive experiments across 9 state-of-the-art embedding models, 4 MLLMs and 4 end-to-end document RAG frameworks demonstrate the gap between text and visual embedding models is narrowing, highlighting the need in building stronger document retrieval models. Our findings also reveal the over-confidence dilemma within current document RAG frameworks that tend to provide answer even without evidence support. We hope our fully open-source Double-Bench provide a rigorous foundation for future research in advanced document RAG systems. We plan to retrieve timely corpus and release new benchmarks on an annual basis. https://double-bench.github.io † † ∗ Equal Contribution, ‡ Project Lead, † Corresponding Authr 1 Introduction Retrieval-Augmented Generation (RAG) ( Lewis et al. 2020 ) has emerged as a transformative technique in textual information retrieval, enhancing Large Language Models (LLMs) by retrieving the most relevant content from knowledge bases in response to queries. This approach has driven significant advances in context engineering, particularly for knowledge-intensive NLP tasks. Besides text-only scenarios, vision documents—including scanned files ( Breci et al. 2024 ) , charts ( Masry et al. 2022 ) , and slides ( Tanaka et al. 2023 ) —serve as rich information sources that have traditionally required substantial manual effort to examine. These documents are now being efficiently enhanced and handled through multimodal document RAG systems, enabling advanced document understanding ( Faysse et al. 2024 ; Wang et al. 2025 ; Cho et al. 2024 ) . Despite the growing importance of document RAG systems ( Mortaheb et al. 2025a ; Yu et al. 2024 ; Mortaheb et al. 2025b ) , effectively evaluating them in detail presents significant challenges. Existing document RAG evaluation benchmarks ( Friel et al. 2024 ) suffer from four critical shortcomings as we identified through pilot experiments shown in Figure 1 : (1) Limited evaluation scope : Current benchmarks only focus on specific parts such as embedding model or VQA model, failing to reveal the bottlenecks of current RAG system in a holistic and comprehensive way. (2) Unrealistic prior knowledge assumptions: Many VQA-style benchmarks ( Li et al. 2024 ; Wu et al. 2025 ) presuppose that the target page or document is already known, making queries inappropriate for evaluating real-world global retrieval scenarios. (3) Ambiguous or non-unique evidence: Synthetic queries are often crafted from a single page and assume a one-to-one mapping between query and evidence, neglecting cases where multiple pages could be relevant. (4) Unlinked multi-hop compositions: Synthesized multi-hop queries are frequently constructed from loosely connected single-hops, failing to evaluate retrieval models’ ability on multi-hop reasoning across documents and modalities. Figure 1: Existing document RAG benchmarks suffer from ambiguous queries that are insufficiently specified to retrieve relevant results, failing to authentically evaluate current document retrieval models and systems. To overcome these limitations, we introduce Double-Bench ( DO cument U nified B road-coverage L ogical hops E valuation Benchmark), consisting of 5,168 human-validated single- and multi-hop queries and 3,276 documents in 6 languages and 4 types of document data. First, we assemble and preprocess a large diverse document corpus spanning PDFs, scanned documents, slides and web pages by a two-stage filtering and modality decomposition. Next, we synthesize and rigorously label both single- and multi-hop queries ( Zhang et al. 2025b ; Tang & Yang 2024 ) , using a combination of iterative clarity-oriented refinement, knowledge-graph-guided composition and exhaustive evidence search. Finally, expert annotators review ( Chiang et al. 2024 ; Chen et al. 2024a ) and correct machine-assigned evidence to ensure high-precision ground truth for large-scale realistic multimodal retrieval evaluation. As shown in Figure 2 , Double-Bench contains high quality queries with low ambiguity. To avoid potential data contamination issues, Double-Bench is also designed to support streamlined dynamic updates with minimal human intervention. Based on Double-Bench , we conduct extensive experiments across 9 state-of-the-art textual, visual and multimodal embedding models, 4 MLLMs and 4 advanced document RAG frameworks. Our findings demonstrate that text embedding models are narrowing the gap with visual embedding models, and Colqwen2.5-3b achieve a strong performance with an 0.795 averaged hit@5 score. Regarding different languages, retrieval models perform generally better on high-resource language than low-resource like Arabic and French. Regarding different document types, clean and structured documents, such as PDFs and HTML pages, are generally easier for models to inspect. Moreover, MLLMs’ low accuracy across both single- and multi-hop queries demonstrates the inherent challenges in multimodal long document understanding. Multi-hop queries prove particularly challenging for current document RAG frameworks, achieving only 0.655 accuracy even when ground truth pages are directly provided. In summary, our contributions are three-fold: • We diagnose several major limitations in existing document RAG evaluation, including incomplete scope, unrealistic prior knowledge assumptions, ambiguous or non-unique evidence, and non-grounded multi-hop query design. • We introduce Double-Bench , the first-of-its-kind live evaluation system for multilingual and multimodal document RAG system, featuring a diverse document corpus, fine-grained page decomposition, and high-quality single- and multi-hop QA pairs with manually labeled evidence. • Our experiments across several state-of-the-art embedding models, MLLMs, and document RAG system uncover critical limitations in current RAG frameworks, providing insights and findings for the research community. Figure 2: Our pilot study reveals critical limitations in existing document RAG system benchmarks, which make them unable to reliably assess current system in a fine-grained and realistic manner. 2 Limitations of Existing Document RAG System Evaluation: A Pilot Study 2.1 Task Formulation Let C be a large corpora consisting of documents \{d_{1},d_{2},...,d_{n}\} . Each document d_{i} is stored by page images \{p_{1}^{i},p_{2}^{i},...,p_{m}^{i}\} . Given a query Q , the objective is to retrieve top- k possible evidence pages E_{r} from the entire corpora to formulate the answer A . For single-hop queries Q_{s} , answer A can be found if one or more evidence pages from the evidence set E_{q} is successfully retrieved. For multi-hop queries Q_{m} , the requirement extends to having one or more evidence page for every evidence set E_{q,j} of each query hop j . These conditions are formated in Equation ( 1 ) and ( 2 ). (E_{r}\cap E_{q}\neq\varnothing)\implies\text{Enable}(A|Q_{s},E_{r}) (1) \displaystyle\left(\bigwedge_{j=1}^{k}(E_{r}\cap E_{q,j}\neq\varnothing)\right)\implies\text{Enable}(A|Q_{m},E_{r}) (2) 2.2 Four Major Overlooked Issues for Document RAG Evaluation Though practical document RAG scenarios typically involve queries that clearly state an informational need, the core

Read the complete text