VisBrowse-Bench: Benchmarking Visual-Native Search for Multimodal Browsing Agents

Zhengbo Zhang 1,2,∗,† , Jinbo Su 3,2,∗,† , Zhaowen Zhou 2,∗ , Changtao Miao 2,∗ , Yuhan Hong 4 , Qimeng Wu 5 , Yumeng Liu 6 , Feier Wu 5 , Yihe Tian 5 , Yuhao Liang 7 , Zitong Shan 2 , Wanke Xia 5 , Yi-Fan Zhang 1 , Bo Zhang 2 , Zhe Li 2 , Shiming Xiang 1,‡ , Ying Yan 2,‡ 1 CASIA, 2 Ant Digital Technologies, Ant Group, 3 RUC, 4 FZU, 5 THU, 6 USTB, 7 PKU · 2026

Read VisBrowse-Bench: Benchmarking Visual-Native Search for Multimodal Browsing Agents by Zhengbo Zhang 1,2,∗,† , Jinbo Su 3,2,∗,† , Zhaowen Zhou 2,∗ , Changtao Miao 2,∗ , Yuhan Hong 4 , Qimeng Wu 5 , Yumeng Liu 6 , Feier Wu 5 , Yihe Tian 5 , Yuhao Liang 7 , Zitong Shan 2 , Wanke Xia 5 , Yi-Fan Zhang 1 , Bo Zhang 2 , Zhe Li 2 , Shiming Xiang 1,‡ , Ying Yan 2,‡ 1 CASIA, 2 Ant Digital Technologies, Ant Group, 3 RUC, 4 FZU, 5 THU, 6 USTB, 7 PKU (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-16289.txt.

Introduction

Zhang and co-authors build VisBrowse-Bench, a test for browsing agents that search with images. Older benchmarks under-test visual reasoning and skip the native images on web pages. Their set holds 169 expert-built question tasks across many domains. Scoring cross-checks text and image evidence during search. The team also gives a workflow that pushes agents to gather visual evidence. Even the best model scored only 47.6%. A top Deep Research model reached 41.1%.

Catalog record

CREATORZhengbo Zhang 1,2,∗,† , Jinbo Su 3,2,∗,† , Zhaowen Zhou 2,∗ , Changtao Miao 2,∗ , Yuhan Hong 4 , Qimeng Wu 5 , Yumeng Liu 6 , Feier Wu 5 , Yihe Tian 5 , Yuhao Liang 7 , Zitong Shan 2 , Wanke Xia 5 , Yi-Fan Zhang 1 , Bo Zhang 2 , Zhe Li 2 , Shiming Xiang 1,‡ , Ying Yan 2,‡ 1 CASIA, 2 Ant Digital Technologies, Ant Group, 3 RUC, 4 FZU, 5 THU, 6 USTB, 7 PKU
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE51962
LICENSEhttp://creativecommons.org/licenses/by-sa/4.0/
RIGHTSCC BY SA
SOURCEarXiv

Source: https://arxiv.org/abs/2603.16289

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

VisBrowse-Bench: Benchmarking Visual-Native Search for Multimodal Browsing Agents Zhengbo Zhang 1,2,∗,† , Jinbo Su 3,2,∗,† , Zhaowen Zhou 2,∗ , Changtao Miao 2,∗ , Yuhan Hong 4 , Qimeng Wu 5 , Yumeng Liu 6 , Feier Wu 5 , Yihe Tian 5 , Yuhao Liang 7 , Zitong Shan 2 , Wanke Xia 5 , Yi-Fan Zhang 1 , Bo Zhang 2 , Zhe Li 2 , Shiming Xiang 1,‡ , Ying Yan 2,‡ 1 CASIA, 2 Ant Digital Technologies, Ant Group, 3 RUC, 4 FZU, 5 THU, 6 USTB, 7 PKU Abstract The rapid advancement of Multimodal Large Language Models (MLLMs) has enabled browsing agents to acquire and reason over multimodal information in the real world. But existing benchmarks suffer from two limitations: insufficient evaluation of visual reasoning ability and the neglect of native visual information of web pages in the reasoning chains. To address these challenges, we introduce a new benchmark for visual-native search, VisBrowse-Bench. It contains 169 VQA instances covering multiple domains and evaluates the models’ visual reasoning capabilities during the search process through multimodal evidence cross-validation via text-image retrieval and joint reasoning. These data were constructed by human experts using a multi-stage pipeline and underwent rigorous manual verification. We additionally propose an agent workflow that can effectively drive the browsing agent to actively collect and reason over visual information during the search process. We comprehensively evaluated both open-source and closed-source models in this workflow. Experimental results show that even the best-performing model, Claude-4.6-Opus only achieves an accuracy of 47.6%, while the proprietary Deep Research model, o3-deep-research only achieves an accuracy of 41.1%. The code and data can be accessed at: https://github.com/ZhengboZhang/VisBrowse-Bench \reportnumber 001 1 1 footnotetext: Equal contribution. 2 2 footnotetext: Work done during internship in Ant Group. 3 3 footnotetext: Corresponding to [email protected], [email protected] 1 Introduction Driven by the rapid advancements in large language models (LLMs) and agent technologies, a plethora of high-quality works has emerged in the deep research domain [ 6 , 7 , 13 , 23 ] . However, existing deep research benchmarks predominantly focus on the textual modality, thereby neglecting the multimodal demands inherent in real-world retrieval scenarios. Concurrently, the evolution of multimodal large language models (MLLMs) has inspired a series of works on Multimodal Browsing Agents [ 22 , 21 ] . Nevertheless, existing multimodal benchmarks still exhibit significant limitations, as shown in Figure 1 . Figure 1 : Existing benchmarks have two limitations in evaluating multimodal browsing agents: 1. the semantic information of visual queries can be easily obtained through image search tools; 2. real-world browsing environments contain a wealth of multimodal information, which most benchmarks overlook. VisBrowse-Bench is designed to fuse multimodal information during the search process and ensure that visual capabilities are essential for completing the task. Specifically, most current benchmarks (e.g., MMSearch [ 8 ] and BrowseComp-VL [ 5 ] ) merely test models’ ability to invoke tools for solving text-image queries. These tasks typically introduce an image search tool, where models simply feed the image into the tool for retrieval. Such tasks do not require fine-grained understanding of multimodal information and thus fail to sufficiently challenge models’ multimodal comprehension capabilities in deep research scenarios, instead primarily emphasizing tool-calling abilities. Moreover, even though some benchmarks (e.g., MMSearch-Plus [ 15 ] and VDR-Bench [ 25 ] ) require initial visual perception of text-image queries, the subsequent information gathering process degenerates into single-modal text traversal. Existing benchmarks structure their search space such that once the query image yields an entity name or caption, all downstream reasoning can be completed through textual document retrieval and synthesis. The search trajectory never necessitates grounding, parsing, or reasoning over additional visual information discovered during the search process. The task thus degenerates into text-only browsing, failing to assess whether models can dynamically acquire and integrate visual information when it is not provided upfront but must be actively sought across web pages containing multimodal information. This ability to conduct visual search and understanding throughout the task is vital in real-world retrieval, but existing benchmarks fail to evaluate this capability, limiting further development. Figure 2 : (a) Overall performance of MLLMs on VisBrowse-Bench. (b) Performance of four MLLMs on seven categories. To address these challenges, we introduce VisBrowse-Bench , a challenging benchmark to comprehensively evaluate the reasoning and search capability of multimodal browsing agents. VisBrowse-Bench relies on two core design principles: the integration of multimodal information within the reasoning chain, and an inherent dependence on visual capabilities. We constructed our benchmark through a multi-stage, expert-guided pipeline, resulting in 169 VQA instances that cover seven distinct domains. Based on a seed entity with inherent visual ambiguity, domain experts recursively construct multi-hop reasoning chains, which require locating and reasoning novel visual evidence that cannot be paraphrased textually, alongside textual sources that provide complementary but insufficient information. Furthermore, we propose an efficient agentic workflow for visual reasoning and visual information retrieval during the search process. In this workflow, the agent is driven to actively perform visual reasoning and visual information retrieval using a rich set of tools. The preview performance of the MLLMs using our workflow on VisBrowse-Bench is shown in Figure 2 . The best performance in VisBrowse-Bench is achieved by Claude-4.6-Opus, with an accuracy of 47.6%, while most models achieve an accuracy of around 30%. Our key contributions can be summarized as follows: • We formalize the task of multimodal browsing, identifying two critical challenges in existing benchmarks: insufficient evaluation of visual reasoning ability and the neglect of visual-native information in the reasoning chains. • We propose VisBrowse-Bench, a new benchmark comprising 169 rigorously validated instances constructed by human experts. This benchmark aims to jointly evaluate the search and visual reasoning capabilities of multimodal browsing agent systems. • We introduce a multi-turn visual information retrieval-reasoning agentic workflow to solve multimodal browsing problems in the real world. Compared to direct answer, the models’ performance are significantly improved using our workflow, but performance limitations still exist. The results demonstrate that the agentic workflow is effective and our benchmark is challenging. 2 Related Work 2.1 Multimodal Browsing Agents Early systems built upon LLMs demonstrated the feasibility of tool-augmented web navigation, employing search engines and web browsing tools to retrieve and integrate textual information in response to user queries [ 9 , 24 , 19 , 18 , 3 ] . These agent systems established the foundational architecture for iterative retrieval and reasoning, but remained fundamentally constrained by their inability to perceive and process visual content, reducing rich multimodal web environments to text-only representations. The advent of MLLMs drives a paradigm shift toward multimodal browsing. MMSearch-R1 [ 20 ] is the first end-to-end reinforcement learning framework that drives MLLMs to perform multi-turn searches on demand in real-world internet environments. WebWatcher [ 5 ] can leverage more external tools and design a data synthesis pipeline to produce high-quality multimodal data for training. DeepMMSearch-R1 [ 11 ] , Skywork-R1V4 [ 26 ] and SenseNova-MARS [ 4 ] introduce the image cropping tool and enhance the reasoning ability for query images. Despite their strong performance on current multimodal deep research benchmarks, a rigorous benchmark dedicated to fine-grained image understanding is still missing. 2.2 Multimodal Browsing Benchmarks Multimodal browsing benchmarks are crucial for

Read the complete text