MAVIS: A Benchmark for Multimodal Source Attribution in Long-form Visual Question Answering
Seokwon Song, Minsu Park, Gunhee Kim · 2025
Read MAVIS: A Benchmark for Multimodal Source Attribution in Long-form Visual Question Answering by Seokwon Song, Minsu Park, Gunhee Kim (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-12142.txt.
Introduction
Seokwon Song, Minsu Park and Gunhee Kim present MAVIS, a benchmark for judging whether AI answers to visual questions cite their sources well. Good attribution lets users check each claim, but past work focused on text alone. MAVIS tests systems that must grasp the intent behind a visual question, find multimodal evidence, and answer at length with citations.
The dataset holds a large set of visual question cases, each answer tagged with fact-level links to text or image documents. New measures score how informative, grounded and fluent answers are, and they match human judgments well. Findings show multimodal search helps fluency, but image sources are grounded more weakly than text.
Catalog record
| CREATOR | Seokwon Song, Minsu Park, Gunhee Kim |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 52 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2511-12142
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
MAVIS: A Benchmark for Multimodal Source Attribution in Long-form Visual Question Answering Seokwon Song, Minsu Park, Gunhee Kim Abstract Source attribution aims to enhance the reliability of AI-generated answers by including references for each statement, helping users validate the provided answers. However, existing work has primarily focused on text-only scenario and largely overlooked the role of multimodality . We introduce MAVIS, the first benchmark designed to evaluate multimodal source attribution systems that understand user intent behind visual questions, retrieve multimodal evidence, and generate long-form answers with citations. Our dataset comprises 157K visual QA instances, where each answer is annotated with fact-level citations referring to multimodal documents. We develop fine-grained automatic metrics along three dimensions of informativeness, groundedness, and fluency, and demonstrate their strong correlation with human judgments. Our key findings are threefold: (1) LVLMs with multimodal RAG generate more informative and fluent answers than unimodal RAG, but they exhibit weaker groundedness for image documents than for text documents, a gap amplified in multimodal settings. (2) Given the same multimodal documents, there is a trade-off between informativeness and groundedness across different prompting methods. (3) Our proposed method highlights mitigating contextual bias in interpreting image documents as a crucial direction for future research. The dataset and experimental code are available at https://github.com/seokwon99/MAVIS 1 Introduction Figure 1: Example of the system’s response in our MAVIS benchmark. Given a user question paired with an input image, the system must generate a long-form answer supported by sentence-level citations that reference multimodal documents. Highlighted texts can be verified using the corresponding colored documents. † † nicematrix-placeholder: NiceTabular (nicematrix) Table 1: Comparison of long-form question answering (LFQA) benchmarks. Q , I , and A denote the number of unique questions, images, and answers, respectively. D_{T}/Q and D_{I}/Q are the average number of text and image documents per question. The number in parentheses in our dataset indicates the statistics of the human-annotated set. The answer length is the average word count of answers. Fact-level citation indicates whether supporting documents are available for each verifiable fact. The dash (–) indicates that the corresponding feature is not covered in that benchmark. {\dagger} denotes the absence of an annotated document for each question; however, an external knowledge base is utilized for retrieval. Since the advent of open-domain generative models, detecting and mitigating potentially inaccurate or fabricated information has become a critical challenge ( Ye et al. 2023 ; Zhang et al. 2023 ) . This so-called hallucination problem arises from the models’ ability to generate highly fluent, human-like responses, making inaccuracies difficult to detect ( Huang et al. 2025 ) . This issue is particularly problematic in long-form generation, where the generated content frequently includes numerous pieces of information that are a mixture of true and false ( Min et al. 2023 ) . These challenges suggest a growing need that not only generates factually accurate answers but also supports them with verifiable evidence. One such approach includes source attribution, in which models enhance the verifiability of their long-form answers by including citations for each statement ( Bohnet et al. 2022 ; Gao et al. 2023 ) . Existing works in this area focus on the text-only setting, where models interpret user queries expressed in natural language and retrieve evidence solely from textual sources. However, text-only attribution may be insufficient, as many real-world scenarios require multimodal understanding. First, images are inherently compact yet rich , making them suitable for conveying dense and detailed information intuitively. For instance, as shown in document [1] of Figure 1 , the image simultaneously represents multiple layers of information—such as “the structure of an induction coil”, “the generation of a magnetic field by current flowing through the coil”, and “the magnetic field inducing electric currents in metal”—in a compact and intuitive manner. Second, user-provided images play a critical role in understanding user intent and enabling precise responses. It is often difficult to grasp the intent of a question based solely on a natural language query—for example, “what’s this metal gear assembly for?”—without considering the accompanying image. To bridge the gap between existing attribution benchmarks and real-world scenarios, we introduce MAVIS ( M ultimodal A ttribution for Vis ual Question Answering), a benchmark designed to evaluate models on their ability to: (1) comprehend questions involving visual inputs, (2) generate effective search queries to retrieve multimodal documents, and (3) provide long-form answers with appropriate citations. We automatically construct a dataset of 157K instances, each containing a visual question and a ground-truth answer citing multimodal documents at the sentence level, and manually annotate 1K instances for evaluation. As shown in Table 1 , our dataset uniquely includes both visual questions and multimodal supporting documents, and provides fact-level citation. To ensure fine-grained evaluation of long-form responses and citation quality, we employ three metrics. (1) Informativeness measures how thoroughly the answer covers necessary information without unnecessary content; (2) Groundedness assesses how well the answer is supported by the citations; and (3) Fluency evaluates how fluent and coherent the output is. Our human evaluation results show that this automatic evaluation highly correlates with human judgments, making it a reliable evaluation method. Through extensive experiments, we also find multiple intriguing observations. 1. Multimodal RAG generates more informative and fluent answers than unimodal RAG (i.e., text-only or image-only). However, LVLMs primarily rely on text when generating answers, making them less attentive to image-based documents. 2. Given the same multimodal documents, there is a trade-off between informativeness and groundedness across different prompting methods, indicating the challenge of improving both simultaneously. 3. LVLMs fabricate information more frequently from image documents than from text documents. We introduce a knowledge extraction step before the final answer generation, effectively addressing this issue by mitigating contextual bias. Figure 2: An illustration of the task formulation in our benchmark. 2 Related Works Source attribution. Open-domain generative systems often produce plausible yet inaccurate content, known as hallucinations ( Ye et al. 2023 ; Zhang et al. 2023 ) . To mitigate this, attribution techniques have been introduced, enabling models to provide supporting evidence in the form of citations to improve verifiability ( Rashkin et al. 2023 ; Gao et al. 2023 ; Li et al. 2024b ; Sun et al. 2023 ; Slobodkin et al. 2024 ; Huang et al. 2024 ; Li and Ng 2024 ) . However, these methods primarily address text-based inputs, with limited attention to the multimodal settings. We bridge this gap by adapting attribution approaches to multimodal contexts and identifying key factors for enhancing performance. Long-form question answering (LFQA). LFQA datasets are widely used in AQA tasks, as long-form responses are prone to including inaccurate information ( Liu, Zhang, and Liang 2023 ) . ELI5 ( Fan et al. 2019 ) and HowSumm ( Boni et al. 2021 ) are popular LFQA datasets based on Reddit and WikiHow, respectively, and are evaluated by reference-based metrics such as ROUGE ( Lin 2004 ) and BLEU ( Papineni et al. 2002 ) . To better address the open-ended nature of LFQA, LongFact ( Wei et al. 2024b ) verifies atomic claims via web search, while RAG-QA Arena ( Han et al. 2024 ) employs pairwise preference evaluations. However, most existing datasets are still limited to text-only inputs, overlooking the