Windsock is Dancing: Adaptive Multimodal Retrieval-Augmented Generation

Shu Zhao, Tianyi Shen, Nilesh Ahuja, Omesh Tickoo, Vijaykrishnan Narayanan · 2025

Read Windsock is Dancing: Adaptive Multimodal Retrieval-Augmented Generation by Shu Zhao, Tianyi Shen, Nilesh Ahuja, Omesh Tickoo, Vijaykrishnan Narayanan (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-22694.txt.

Introduction

Systems that fetch facts before they reply can fetch at the wrong time. They can also fetch the wrong kind of fact. Shu Zhao wrote a 2025 paper on that flaw. His co-authors are Tianyi Shen and Nilesh Ahuja. Omesh Tickoo and Vijaykrishnan Narayanan are co-authors too. Their module is called Windsock.

Windsock picks if a fetch is needed at all. It picks which mode to fetch, text or image. A training plan called DANCE helps the model use fetched facts. It also helps the model resist noise in those facts. The team made training sets from question sets with model help. Tests show quality rose a lot. Fetch calls fell at the same time.

Catalog record

CREATORShu Zhao, Tianyi Shen, Nilesh Ahuja, Omesh Tickoo, Vijaykrishnan Narayanan
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE50 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2510-22694

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Windsock is Dancing: Adaptive Multimodal Retrieval-Augmented Generation Shu Zhao, Tianyi Shen, Nilesh Ahuja, Omesh Tickoo, Vijaykrishnan Narayanan Abstract Multimodal Retrieval-Augmented Generation (MRAG) has emerged as a promising method to generate factual and up-to-date responses of Multimodal Large Language Models (MLLMs) by incorporating non-parametric knowledge from external knowledge bases. However, existing MRAG approaches suffer from static retrieval strategies, inflexible modality selection, and suboptimal utilization of retrieved information, leading to three critical challenges: determining when to retrieve, what modality to incorporate, and how to utilize retrieved information effectively. To address these challenges, we introduce Windsock , a query-dependent module making decisions on retrieval necessity and modality selection, effectively reducing computational overhead and improving response quality. Additionally, we propose D yn a mic N oise-Resistan ce ( DANCE ) Instruction Tuning, an adaptive training strategy that enhances MLLMs’ ability to utilize retrieved information while maintaining robustness against noise. Moreover, we adopt a self-assessment approach leveraging knowledge within MLLMs to convert question-answering datasets to MRAG training datasets. Extensive experiments demonstrate that our proposed method significantly improves the generation quality by 17.07 % 17.07\% while reducing 8.95 % 8.95\% retrieval times. 1 Introduction While Large Language Models (LLMs) have demonstrated remarkable capabilities gpt4 ; gemini ; llama3 , they often struggle with factual accuracy parallelsearch ; expandsearch , leading to the development of Retrieval-Augmented Generation (RAG) as a solution by incorporating external knowledge into the generation process racm3 . In a typical RAG pipeline, given a query, the system first retrieves relevant documents from an external knowledge base and then concatenates them with the query as context, enabling the model to generate responses grounded in the retrieved information. While RAG has shown significant success with text-based knowledge crag ; hipporag , real-world tasks inherently require multimodal understanding, like visual analysis or spatial reasoning spatial_vlm . This fundamental need for multimodal comprehension has led to the development of multimodal RAG (MRAG) systems that can seamlessly integrate both textual and visual knowledge in their reasoning process mrag_survey . However, existing MRAG methods suffer from three significant limitations. First, they adopt a retrieve-for-all strategy that indiscriminately retrieves information for every query and cannot determine when to retrieve, even when the model’s parametric knowledge is sufficient to generate accurate responses surf ; ragvl . It leads to unnecessary computational overhead and may introduce noise from unreliable retrievers that degrade response quality. Second, current approaches either consistently retrieve images surf ; ragvl or exclusively access text-based knowledge bases like Wikipedia mr2ag ; reflectiva without considering the specific informational needs of each query. This inflexible approach fails to recognize that different queries may require different types of information for optimal response generation, underscoring the necessity for an adaptive retrieval mechanism to make query-dependent decisions about what modality to select. Third, Multimodal Large Language Models (MLLMs) gpt4o ; llava may struggle to know how to utilize retrieved information effectively and are sensitive to irrelevant documents surf ; mragbench . Either statistic-based bm25 or vector-based vbge ; clip retrievers may wrongly understand the query and return irrelevant documents, leading to reduce the reliability of responses and the risk of factual errors and hallucinations ralm ; raat . To address these limitations, we propose Windsock, a query-aware decision module that dynamically orchestrates the retrieval process for the when and what challenges. Specifically, Windsock decides if a retrieval operation is necessary and what modality should be retrieved based on user queries. For example, queries about historical events are more likely to benefit from textual documents, while visual documents contribute to queries regarding artist’s painting styles. This adaptive approach reduces computational overhead and improves response accuracy by ensuring retrieved information aligns with query intentions. Furthermore, Windsock’s modular design allows it to be seamlessly integrated into open-source and proprietary MLLMs. To address the how challenge, we devise D yn a mic N oise-Resistan ce (DANCE), an adaptive training strategy to dynamically select a modality during instruction tuning in response to retrieval noises, further enhancing MLLMs to improve their utilization of retrieved information while improving their robustness against irrelevant information. Furthermore, we propose an automatic self-assessment approach to convert any question-answering dataset to support RAG training. Specifically, we assess the inherent ability of MLLMs to identify their incomplete knowledge by providing or withholding retrievals with different modalities. Then, we can evaluate responses to identify the best retrieval strategy and the challenging samples for training. To summarize our contributions, we propose Windsock, a lightweight module that makes query-dependent decisions on retrieval necessity and modality selection, effectively reducing computational overhead while improving response quality. We develop DANCE, an adaptive training strategy that enhances MLLMs’ ability to utilize retrievals while maintaining robustness against noise. Additionally, we introduce a self-assessment pipeline that automatically converts question-answering datasets for RAG training without relying on proprietary models, enabling efficient identification of optimal retrieval strategies and instruction tuning dataset construction. Comprehensive experiments demonstrate that our proposed method significantly improves both efficiency and response quality. Figure 1: Framework overview of our proposed method. Given a query, Windsock adaptively selects between direct answering (no retrieval) or retrieving from either visual or textual knowledge bases, followed by MLLM instruction tuned by DANCE, generating the final response based on the query and retrieved documents. 2 Related Work Multimodal Retrieval-Augmented Generation Multimodal Retrieval-Augmented Generation (MRAG) has emerged as a promising approach to enhance multimodal large language models (MLLMs) by incorporating external knowledge sources. Early works focused on optimizing retrieval mechanisms through end-to-end training. RA-CM3 racm3 and MuRAG murag pioneered simultaneous optimization of retrievers and generators, while RagVL ragvl introduced reranking through MLLM fine-tuning. Subsequently, WikiLLaVA wikillava and EchoSight echosight advanced entity-centric retrieval by identifying and leveraging relevant contextual information. SURf surf introduced selective information utilization to improve knowledge integration and enhance noise rejection. RIR rir innovated by incorporating real-time information from online search engines. VisRAG visrag and VisDoM visdom specialized in multimodal document retrieval, and Re-ViLM revilm enhanced image captioning by integrating external knowledge into transformer cross-attention layers. Domain-specific applications have demonstrated MRAG’s practical value, with systems like FactMM-RAG sun2024fact improving medical radiology report generation. Recent benchmarks, including MRAG-Bench mragbench and SnapNTell snapntell , have highlighted both the potential and limitations of MRAG systems, particularly in handling entity-centric queries and utilizing retrieved visual knowledge effectively. Most approaches employ a retrieval-for-all strategy, leading to computational overhead and potential noise injection. While recent works like ReflectiVA reflectiva and mR 2 AG mr2ag have introduced adaptive retrieval and reranking through special tokens, they rely on expensive human or GPT-4 annotations and overlook the importance of modality selection based on query requirements. Our approach addresses these limitations by query-dependent decisions for both retrieval necessity and modality selection, thereby optimizing computational efficiency and response quality. Multimodal Large Language Models Recent advances in Multimodal Large Language Models (MLLMs) have enabled systems to process and generate both visual and textual information seamlessly. Leading commercial models such as GPT-4 gpt4o and Gemini gemini have demonstrated remarkable capabilities across various tasks. LLaVA llava and PaLM-E palme have further advanced the field by introducing efficient training methods that effectively leverage pre-trained language models and visual encoders. Particularly noteworthy is the emergence of open-source alternatives like LLaMA-Vision llama3 , InternVL internvl , and QwenVL qwen2vl , opening access to multimodal capabilities while

Read the complete text