MRAG-Suite: A Diagnostic Evaluation Platform for Visual Retrieval-Augmented Generation

Yuelyu Ji · 2025

Read MRAG-Suite: A Diagnostic Evaluation Platform for Visual Retrieval-Augmented Generation by Yuelyu Ji (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-24253.txt.

Introduction

Systems that answer questions from both images and text can fail in ways that flat accuracy scores hide. Yuelyu Ji and 2 co-authors built MRAG-Suite, a diagnostic platform for visual retrieval-augmented generation. It unites several multimodal benchmarks and filters questions by difficulty and ambiguity, two factors past evaluations ignored.

Under hard or ambiguous queries, accuracy dropped sharply and hallucinations stood out. A companion tool, MM-RAGChecker, checks claims one by one to locate the faults. The platform thus points builders to the exact stage where a system breaks. The authors offer it as a guide for improving visual question answering with retrieved evidence.

Catalog record

CREATORYuelyu Ji
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE49 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2509-24253

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

MRAG-Suite: A Diagnostic Evaluation Platform for Visual Retrieval-Augmented Generation Yuelyu Ji Abstract Multimodal Retrieval-Augmented Generation (Visual RAG) significantly advances question answering by integrating visual and textual evidence. Yet, current evaluations fail to systematically account for query difficulty and ambiguity. We propose MRAG-Suite, a diagnostic evaluation platform integrating diverse multimodal benchmarks (WebQA, Chart-RAG, Visual-RAG, MRAG-Bench). We introduce difficulty-based and ambiguity-aware filtering strategies, alongside MM-RAGChecker, a claim-level diagnostic tool. Our results demonstrate substantial accuracy reductions under difficult and ambiguous queries, highlighting prevalent hallucinations. MM-RAGChecker effectively diagnoses these issues, guiding future improvements in Visual RAG systems. 1 Introduction Retrieval-Augmented Generation (RAG) has emerged as a foundational paradigm for enhancing the factuality and reliability of large language models (LLMs) by grounding their outputs in external evidence. With recent advancements in multimodal capabilities, visual RAG systems—capable of leveraging textual and visual information simultaneously have garnered significant interest, promising substantial improvements in answering complex questions grounded in diverse modalities such as charts, plots, figures, images, and documents. Despite these advancements, current evaluation methodologies have not kept pace with these multimodal developments, primarily focusing either on purely textual contexts or closed-domain multimodal setups that provide pre-selected visual evidence. Consequently, the research community lacks rigorous evaluation benchmarks capable of systematically assessing multimodal retrieval accuracy, reasoning fidelity, and robustness to ambiguity and distractors. Figure 1: Illustration of our multimodal claim verification process. Given a long-form answer generated by a retrieval-augmented generation (RAG) system, we extract atomic factual claims and assess whether each is supported by retrieved visual evidence. Claims are categorized as Entailed if supported by at least one image. In this example, the model correctly describes several physical traits and habitat features of a marmot , but omits the gold claim about “high-altitude environments” , which remains unsupported by the retrieved evidence. Existing multimodal benchmarks like ScienceQA ( Lu et al. 2022 ) , MMMU ( Yue et al. 2023 ) , and MMMU-Pro ( Yue et al. 2024 ) typically provide necessary visual contexts as part of the inputs, neglecting the retrieval component critical in real-world scenarios. Meanwhile, retrieval-centric evaluations like Visual Haystacks ( Wu et al. 2024 ) emphasize challenging retrieval tasks but limit their scope to a single domain without fine-grained error diagnostics. Recent text-focused diagnostic frameworks such as RAGChecker and RefChekcer ( Ru et al. 2024 ; Hu et al. 2024b ) offer detailed claim-level analysis but remain restricted to textual contexts, unable to handle the additional complexity introduced by visual evidence. To address these critical gaps, we propose MRAG-Suite , a comprehensive diagnostic evaluation platform designed explicitly for visual RAG systems. MRAG-Suite integrates eight heterogeneous datasets, spanning a wide range of real-world modalities, including charts, plots, scanned documents, slides, and open-web images. Our benchmark systematically introduces difficulty-based filtering and ambiguity-aware query variants, coupled with controlled distractor scenarios, enabling robust assessment of a model’s capability to handle ambiguity, retrieve precise evidence, and produce faithful, evidence-grounded answers. Specifically, MRAG-Suite contributes: 1. Multi-domain multimodal benchmark : We consolidate and normalize datasets across diverse domains, providing unified evaluation standards and enabling direct comparisons of model performance across various multimodal tasks. 2. Difficulty- and ambiguity-aware filtering : By filtering trivial or easily guessable questions and introducing ambiguity through carefully constructed query variations, MRAG-Suite rigorously evaluates robustness against ambiguity and query complexity. 3. Claim-level multimodal diagnostic tool (MM-RAGChecker) : Extending claim-level diagnostics to multimodal contexts, MM-RAGChecker systematically verifies each claim within generated answers against textual and visual evidence, enabling precise identification of hallucinations, modality biases, and retrieval accuracy. All code, prompts, and diagnostics are available at: https://anonymous.4open.science/status/MRAGChecker-B33D 2 Related Work Retrieval-Augmented Generation Evaluation Traditional evaluation of RAG systems primarily relied on metrics like exact match and ROUGE scores, which are insufficiently fine-grained to diagnose specific failures. RAGChecker ( Ru et al. 2024 ) introduced claim-level diagnostics for textual RAG, evaluating retrieval accuracy, factual consistency, and generation hallucinations. However, these tools remain limited to text-only contexts, lacking the capability to assess visual evidence integration and multimodal reasoning. Multimodal QA and Reasoning Benchmarks Benchmarks such as ScienceQA ( Lu et al. 2022 ) , MMMU ( Yue et al. 2023 ) , and their advanced variants MMMU-Pro ( Yue et al. 2024 ) evaluate models’ abilities to reason jointly over provided image and text inputs but exclude retrieval challenges. In contrast, Visual Haystacks ( Wu et al. 2024 ) targets multimodal retrieval tasks but is constrained to single-domain visual reasoning, omitting extensive claim-level diagnostics and failing to capture multimodal reasoning intricacies across diverse domains Li et al. 2025b ; Li et al. 2025a . Biases in Multimodal Retrieval-Augmented Systems Recent studies reveal significant biases in multimodal RAG systems. For example, Yao et al. ( Yao et al. 2025 ) demonstrated strong positional and modality biases, with models disproportionately relying on initial evidence items or textual over visual modalities. However, existing benchmarks have not systematically quantified these biases across multiple domains under controlled experimental conditions. MRAG-Suite specifically addresses this shortcoming by incorporating varied distractor and ambiguity scenarios to measure these biases explicitly Wang et al. 2025 ; Liang et al. 2025 ; Jiang et al. 2024 ; Dan et al. 2024 ; Lu et al. 2025 ; Li et al. . Domain-specific Multimodal QA Prior work on multimodal QA has predominantly focused on specific domains or modalities, such as ChartMRAG ( Yang et al. 2025 ) and VDocRAG ( Tanaka et al. 2025 ) , demonstrating the feasibility and utility of retrieval-augmented QA systems in targeted contexts. Nonetheless, these domain-specific evaluations lack a unified framework for cross-domain comparisons and fine-grained diagnostics across modalities, a gap that MRAG-Suite explicitly aims to fill. 3 MRAG-Suite Benchmark Construction 3.1 Task Definition We evaluate multimodal Retrieval-Augmented Generation (RAG) for open-ended QA. Given a query q , the system retrieves textual ( E^{\text{txt}} ) and visual ( E^{\text{img}} ) evidence and must produce (i) a concise short answer and (ii) a grounded long answer: (a^{\text{short}},a^{\text{long}})=f\!\big(q,E^{\text{txt}},E^{\text{img}}\big). 3.2 Corpora Overview MRAG-Suite merges eight heterogeneous sources spanning charts, natural photos, open-web images, scanned PDFs, and slides. Each source contributes distinct modalities and reasoning demands: • WebQA Chang et al. 2022 : open-web questions with both images (captions/descriptions) and short text snippets; often multi-hop and open-ended. • Chart-RAG Yang et al. 2025 (from ChartQA Masry et al. 2022 ): chart understanding (e.g., “When did X peak?”); evidence includes the chart image plus OCR/metadata; answers are numeric/categorical values. • Visual-RAG Wu et al. 2025 : queries that require visual knowledge (object/attribute recognition), drawn from Visual7W/FVQA/MRAG-Bench-style scenarios. • MRAG-Bench Hu et al. 2024a : 1,353 vision-centric MCQ converted to open-ended by taking the correct option as a^{\text{short}} ; emphasizes fine-grained and multi-view recognition. • VisRAG Yu et al. 2024 subsets: Arxiv Li et al. 2024 (scholarly PDF pages), Plot Methani et al. 2019 (technical plots), Slide Tanaka et al. 2023 (slide decks), and Document Tito et al. 2022 (scanned manuals/forms). Each page is treated as a visual document with OCR text. Dataset statistics. Table 1 summarizes domains, filtering counts, image resolutions, and text length statistics. Table 1: Overview of the eight sources in MRAG-Suite, covering domains, filtering counts, image resolution, and question/answer lengths. The suite spans open-web images, charts/plots, scanned documents, slides, and scholarly PDFs to enable cross-domain visual RAG evaluation. Dataset Domain

Read the complete text