Multimodal Iterative RAG for Knowledge-Intensive Visual Question Answering
Changin Choi, Wonseok Lee, Jungmin Ko, Wonjong Rhee · 2025
Read Multimodal Iterative RAG for Knowledge-Intensive Visual Question Answering by Changin Choi, Wonseok Lee, Jungmin Ko, Wonjong Rhee (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-00798.txt.
Introduction
Changin Choi, with three co-authors, proposes PMSR for visual question answering. Some questions about images need outside knowledge beyond the picture. Existing retrieval-based approaches often search only once. They can gather too little knowledge that way. They also lack a way to revise reasoning that goes astray.
PMSR builds a structured reasoning path step by step. Queries draw on both the latest record and the whole path so far. They retrieve varied knowledge from mixed sources. Retrieved evidence is then merged into compact records. This loop refines reasoning in a controlled way. It also limits how errors spread through the chain. Tests span six diverse benchmarks. PMSR improves both retrieval recall and final answer accuracy throughout.
Catalog record
| CREATOR | Changin Choi, Wonseok Lee, Jungmin Ko, Wonjong Rhee |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 56 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2509-00798
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Multimodal Iterative RAG for Knowledge-Intensive Visual Question Answering Changin Choi, Wonseok Lee, Jungmin Ko, Wonjong Rhee Abstract Recent advances in Multimodal Large Language Models (MLLMs) have significantly enhanced the ability of these models in multimodal understanding and reasoning. However, the performance of MLLMs for knowledge-intensive visual questions, which require external knowledge beyond the visual content of an image, still remains limited. While Retrieval-Augmented Generation (RAG) has become a promising solution to provide models with external knowledge, its conventional single-pass framework often fails to gather sufficient knowledge. To overcome this limitation, we propose MI-RAG, a Multimodal Iterative RAG framework that leverages reasoning to enhance retrieval and incorporates knowledge synthesis to refine its understanding. At each iteration, the model formulates a reasoning-guided multi-query to explore multiple facets of knowledge. Subsequently, these queries drive a joint search across heterogeneous knowledge bases, retrieving diverse knowledge. This retrieved knowledge is then synthesized to enrich the reasoning record, progressively deepening the model’s understanding. Experiments on challenging benchmarks, including Encyclopedic VQA, InfoSeek, and OK-VQA, show that MI-RAG significantly improves both retrieval recall and answer accuracy, establishing a scalable approach for compositional reasoning in knowledge-intensive VQA. 1 Introduction The emergence of Multimodal Large Language Models has driven significant progress in multimodal understanding and reasoning. Nonetheless, the performance of MLLMs remains constrained in knowledge-intensive Visual Question Answering (VQA) tasks, which require external knowledge beyond the visual content in the image. Answering such questions is particularly challenging as it involves: (1) grounding visual entities in the image to the textual question, (2) collecting text knowledge relevant to both the entities and the question, and (3) synthesizing the visual entities and corresponding textual knowledge to derive the answer. To address this challenge, multimodal Retrieval-Augmented Generation has emerged as a promising solution. The conventional retrieve-then-read process starts with a retriever using the input image and question to find relevant image-text pairs from a knowledge base (KB). A reader model then synthesizes this information to create the final answer. Research on optimizing this single-step process has largely advanced along two fronts: improving initial retrieval with high-quality embeddings Lin et al. 2024 ; Wei et al. 2024 ; Jiang et al. 2025 ; Lin et al. 2025 ; Jiang et al. 2024b ; Zhang et al. 2024b , and refining retrieval results by selection and reranking Cocchi et al. 2025 ; Zhang et al. 2024a ; Ling et al. 2025 ; Chen et al. 2024 ; Liu et al. 2024b ; Yan & Xie 2024 ; Yang et al. 2025 . Figure 1: An overview of conventional multimodal RAG and our MI-RAG framework. Conventional multimodal RAG employs a retrieve-then-read process using only a multimodal KB. In contrast, MI-RAG utilizes an iterative framework that cyclically refines its query and reasoning. At each iteration, MI-RAG formulates a multi-query with accumulated reasoning records and retrieves diverse knowledge from heterogeneous KBs. It then performs compositional reasoning to synthesize newly acquired knowledge into a reasoning record and appends it to the reasoning records to improve the subsequent iteration. Although prior work has achieved notable improvements within the retrieve-then-read process, the process itself has inherent limitations for knowledge-intensive VQA. A single retrieval step with an initial query is often insufficient, as imperfect retrievers may fail to gather all necessary knowledge at once Shi et al. 2023 ; Zhang et al. 2023 ; Cuconasu et al. 2024 . Furthermore, a single reasoning step to ground visual entities and synthesize knowledge is problematic, as distracting or irrelevant content can mislead synthesis and influence the final answer Petroni et al. 2020 ; Wu et al. 2024 ; Amiraz et al. 2025 ; Huang et al. 2025 ; Wu et al. 2025 . Motivated by these limitations, the text domain has shifted to iterative RAG , a paradigm that leverages an LLM’s reasoning through multiple rounds of retrieval and reasoning. In this iterative cycle, the model leverages its reasoning to guide intermediate steps, which transform queries to collect more relevant knowledge and synthesize newly retrieved knowledge to progressively update the reasoning itself. Shao et al. 2023 ; Trivedi et al. 2022 ; Xiong et al. 2024 ; Liu et al. 2024a ; Yue et al. 2025 ; Gao et al. 2025 . While previous work has shown promise with this approach in the text-only domain, it has largely focused on refining a single subquery of decomposing multi-hop questions at each step. Extending this to the more complex multimodal domain remains an open challenge. This challenge arises because a naive adaptation of the iterative RAG is insufficient for the multimodal domain, which introduces two distinct considerations: (i) it requires retrieving diverse knowledge by exploring multiple facets of a visually grounded entity and its related textual knowledge across modalities, and (ii) it demands compositional reasoning to synthesize the visual entity with its corresponding knowledge by integrating a diverse set of visual-to-visual, visual-to-text, and text-to-text factual links. This highlights the need for a more specialized approach that can effectively handle the complexities of both visual and textual knowledge. In response to these multimodal domain considerations, we propose Multimodal Iterative Retrieval-Augmented Generation (MI-RAG). As illustrated in Figure 1 , MI-RAG is an iterative framework that cyclically refines its query and reasoning by alternately performing reasoning-guided query transformation and retrieval-augmented reasoning. Our framework makes two primary contributions. First, we employ a reasoning-guided multi-query that dynamically searches multiple facets across modalities. Second, these queries drive a joint search across heterogeneous KBs to find a diverse set of factual links, allowing MI-RAG to compose visually-grounded knowledge and related textual knowledge. Through the synergy of multi-query and heterogeneous KBs, MI-RAG is able to retrieve sufficient knowledge and enable robust compositional reasoning over diverse knowledge across modalities. Through iterations, MI-RAG collects diverse knowledge progressively with accumulating a reasoning record, where each record is the result of compositional reasoning over newly retrieved knowledge, thereby refining the model’s understanding to guide the subsequent iteration step. 2 Related Work 2.1 Multimodal RAG for Knowledge-Intensive VQA The dominant paradigm for multimodal RAG follows the retrieve-then-read process. Initial work focused on enhancing the retrieval phase, primarily by learning more effective multimodal embeddings. Pioneering approaches aligned queries and text documents using contrastive learning Luo et al. 2023 , while others fused similarity scores from pretrained CLIP Wei et al. 2024 . To improve retrieval, subsequent methods employed fine-grained late interaction between visual and textual tokens Lin et al. 2023 ; Lin et al. 2024 or leveraged MLLMs with instruction tuning to produce multimodal embeddings Jiang et al. 2025 ; Liu et al. 2024b ; Zhang et al. 2024b . A separate line of research has explored generative retrieval, using document identifiers Long et al. 2024 or enhancing generative retrieval via reinforcement learning Long et al. 2025 . Another line of research improves the read phase, typically by reranking or refining the retrieved candidates. Hierarchical systems first perform a coarse, image-based search and then refine the results with text-based or multimodal rerankers Caffagni et al. 2024 ; Yan & Xie 2024 ; Yang et al. 2025 . Others leverage the reasoning capabilities of MLLMs to self-reflect on retrieval needs and evaluate relevance Cocchi et al. 2025 , generate semantic tags for consistency and reranking based on relevance scores Ling et