Efficient Multimodal Planning Agent for Visual Question-Answering

Zhuo Chen, Xinyu Geng, Xinyu Wang, Yong Jiang, Zhen Zhang, Pengjun Xie, Kewei Tu · 2026

Read Efficient Multimodal Planning Agent for Visual Question-Answering by Zhuo Chen, Xinyu Geng, Xinyu Wang, Yong Jiang, Zhen Zhang, Pengjun Xie, Kewei Tu (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-20676.txt.

Introduction

Zhuo Chen and six co-authors build a smarter agent for answering questions about images. Their 2026 paper tackles visual question answering, or VQA. A VQA system must join what it sees with what it reads. Only then can it answer well. The team adds a planning step to guide that work.

One known boost for VQA is retrieval-augmented generation. The system fetches extra image and text evidence before it answers. But the standard recipe chains many stages, and each stage waits on the last. The authors propose a planning agent to run this process with less waste. The paper speaks to the push for AI that answers hard, knowledge-heavy questions about pictures.

Catalog record

CREATORZhuo Chen, Xinyu Geng, Xinyu Wang, Yong Jiang, Zhen Zhang, Pengjun Xie, Kewei Tu
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE67 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2601-20676

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Efficient Multimodal Planning Agent for Visual Question-Answering Zhuo Chen, Xinyu Geng, Xinyu Wang, Yong Jiang, Zhen Zhang, Pengjun Xie, Kewei Tu Abstract Visual Question-Answering (VQA) is a challenging multimodal task that requires integrating visual and textual information to generate accurate responses. While multimodal Retrieval-Augmented Generation (mRAG) has shown promise in enhancing VQA systems by providing more evidence on both image and text sides, the default procedure that addresses VQA queries, especially the knowledge-intensive ones, often relies on multi-stage pipelines of mRAG with inherent dependencies. To mitigate the inefficiency limitations while maintaining VQA task performance, this paper proposes a method that trains a multimodal planning agent, dynamically decomposing the mRAG pipeline to solve the VQA task. Our method optimizes the trade-off between efficiency and effectiveness by training the agent to intelligently determine the necessity of each mRAG step. In our experiments, the agent can help reduce redundant computations, cutting search time by over 60% compared to existing methods and decreasing costly tool calls. Meanwhile, experiments demonstrate that our method outperforms all baselines, including a Deep Research agent and a carefully designed prompt-based method, on average over six various datasets. Code will be released at https://github.com 1 Introduction Visual Question-Answering (VQA) is a fundamental task in multimodal artificial intelligence that requires the ability to understand and integrate both visual and textual information to produce correct answers ( 8 ; 21 ; 4 ) . Recent studies have demonstrated advancements in this area, focusing on improving model performance across different types of VQA queries. These include knowledge-intensive questions that require external factual information ( 26 ) as well as dynamic queries where answers may change over time ( 17 ) . Various methods have been studied to integrate multimodal Retrieval-Augmented Generation (mRAG) to better solve various types of VQA queries. These studies typically enhance the capabilities of models by incorporating retrieved evidence from both visual and textual sources ( 7 ; 29 ; 27 ) , and further advance Multimodal Large Language Models’ (MLLMs) potential in real-world applications. However, a key limitation constrains the practical efficiency and scalability of existing mRAG systems. Current implementations typically employ rigid, multi-stage pipelines, possibly involving image grounding ( 1 ) , image retrieval ( 14 ) , and query rewriting (potentially using retrieved contexts) ( 30 ; 19 ) , followed by text passage retrieval ( 17 ; 1 ) . Besides, these steps may also exhibit inherent potential dependencies. For instance, effective query rewriting often necessitates prior image retrieval to provide additional information about the image content, while text retrieval has a critical dependency on query rewriting ( 22 ) . These static workflows are inefficient and remain data-agnostic, often lacking dynamic selection mechanisms between processing stages. Also, redundant retrieval steps introduce overly long input length. Consequently, valuable computational resources are expended even when the original input query might be sufficiently answered using readily available cues alone, or when certain steps provide marginal benefit for a relatively simple query. Figure 1 : Workflow of our agent on solving VQA with dynamic mRAG strategies. The agent selects a sub-path based on different VQA inputs, which may require image search , query search , neither , or both . To mitigate inefficiency without compromising performance, this paper introduces a multimodal planning agent designed to enhance the efficiency of mRAG pipelines in VQA tasks by dynamically adapting to diverse queries. The agent takes necessary steps given a VQA query on a workflow as illustrated in Fig. 1 . In general, facing various VQA queries at test time, the agent optimizes computational resource allocation by intelligently omitting redundant operations. Specifically, for queries necessitating external knowledge or specialized tools, the agent strategically decomposes the mRAG workflow, selectively executing only those components essential for generating accurate responses (path \bm{2} or \bm{3} ), thereby departing from rigid pipeline architectures (path \bm{4} ). In addition, for simpler queries resolvable via the model’s intrinsic capabilities, the agent learns to bypass extraneous processing steps entirely (path \bm{1} ). Through experiments across six diverse VQA datasets, we demonstrate the efficiency and effectiveness of our method. The agent helps achieve substantial gains in inference efficiency compared to a Deep Research agent WebWatcher ( 10 ) and a designed prompt-based method OmniSearch ( 17 ) . Compared to WebWatcher , we achieve better performance with significantly lower tool-call latency, being 3× faster than WebWatcher-7B and 4.5× faster than WebWatcher-32B. Notably, compared to OmniSearch , we reduce the search time by 60%+ on average. Furthermore, this significant efficiency improvement is attained while enhancing the VQA task performance on average over six datasets compared to the default complete mRAG setting and all other baseline methods. To sum up, our contributions are as follows 1. The paper proposes a multimodal planning agent that dynamically optimizes mRAG pipelines while maintaining VQA performance with higher efficiency. 2. Experimental results across diverse VQA datasets show that the agent significantly reduces search time and costly retrieval operations compared to baseline and reference methods. In addition, we obtain improved performance on average over six test datasets. 2 Method Figure 2 : Proposed data annotation method. We propose a method that initially performs data annotation via VQA query decomposition, followed by fine-tuning of an MLLM agent. This section begins by establishing the requisite mathematical notations. Subsequently, we elaborate on the automated annotation process and the procedures for agent training and inference. The fine-tuned agent operates in alignment with the workflow illustrated in Fig. 1 . 2.1 Notations Let \bm{q}=(\bm{i},\bm{t}) represent a VQA query composed of image input \bm{i} and a textual question component \bm{t} , and let \bm{a} denote the corresponding ground truth answer. In general VQA tasks, the original textual query \bm{t} may require reformulation into an optimized query \bm{q}_{g} (henceforth may be referred to as gold query) to more accurately characterize the information needs expressed in \bm{q} . For example, in a situation where the query \bm{q} asks “When did this sorority established a chapter at American University” , the gold query \bm{q}_{g} should be “When was \langle Name \rangle established at American University?” , and \langle Name \rangle refers to the actual sorority name in the image. Let \bm{k_{i}} denote the set of multimodal contextual elements retrieved using visual input \bm{i} . Let \bm{k_{t}} denote textual contexts obtained through the optimized query \bm{q}_{g} . For an MLLM M_{\theta} parameterized by \theta , the answer generation process, when relying solely on the MLLM, can be formally characterized by: \displaystyle\bm{y}_{n} \displaystyle=M_{\theta}(\bm{y}|\bm{q}) (1) Generation with retrieval information from image retrieval, text retrieval, and both sides can be formulated as: \displaystyle\bm{y}_{i} \displaystyle=M_{\theta}(\bm{y}|\bm{q},\bm{k}_{i}) (2) \displaystyle\bm{y}_{t} \displaystyle=M_{\theta}(\bm{y}|\bm{q},\bm{k}_{t}) \displaystyle\bm{y}_{i,t} \displaystyle=M_{\theta}(\bm{y}|\bm{q},\bm{k}_{i},\bm{k}_{t}) 2.2 Agent Training Data Visual Query Decomposition To construct the training data of the agent, given one (\bm{q},\bm{a}) pair as an example, we further expand it into two derived queries \bm{q}_{i} and \bm{q}_{g} . \bm{q}_{i} refers to a image query that queries what is in the image (e.g., asking the entity in the image). \bm{q}_{g} refers to a gold query that combines \bm{i} and \bm{t} and more comprehensively describes the required information. Accordingly, we also need the corresponding gold answer for image query \bm{q}_{i} and gold query \bm{q}_{g} . The answer to \bm{q}_{i} is basically the image entity or a detailed description

Read the complete text