HRScene: How Far Are VLMs from Effective High-Resolution Image Understanding?

Yusen Zhang, Wenliang Zheng, Aashrith Madasu, Peng Shi, Ryo Kamoi, Hao Zhou · 2025

Read HRScene: How Far Are VLMs from Effective High-Resolution Image Understanding? by Yusen Zhang, Wenliang Zheng, Aashrith Madasu, Peng Shi, Ryo Kamoi, Hao Zhou (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-18406.txt.

Introduction

Yusen Zhang and co-authors published a 2025 paper testing how well AI vision models handle very large images. Some images hold over a million pixels: tissue scans, farm fields seen from the air, telescope views. Models that mix vision and language claim to cope with such pictures. Yet no test measured that claim across many kinds of scenes.

The team's benchmark, HRScene, gathers twenty-five datasets plus two made-up test sets. Image sizes run from about a thousand pixels square to more than thirty-five thousand on a side. Ten annotators collected and re-labeled the material. The scenes range from microscope views to street photos and scanned documents. The made-up sets hide a target image among distractors to probe how models use each region. Testing twenty-eight models showed how far current models fall short.

Catalog record

CREATORYusen Zhang, Wenliang Zheng, Aashrith Madasu, Peng Shi, Ryo Kamoi, Hao Zhou
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE65 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2504-18406

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

HRScene: How Far Are VLMs from Effective High-Resolution Image Understanding? Yusen Zhang, Wenliang Zheng, Aashrith Madasu, Peng Shi, Ryo Kamoi, Hao Zhou Abstract High-resolution image (HRI) understanding aims to process images with a large number of pixels, such as pathological images and agricultural aerial images, both of which can exceed 1 million pixels. Vision Large Language Models (VLMs) can allegedly handle HRIs, however, there is a lack of a comprehensive benchmark for VLMs to evaluate HRI understanding. To address this gap, we introduce HRScene, a novel unified benchmark for HRI understanding with rich scenes. HRScene incorporates 25 real-world datasets and 2 synthetic diagnostic datasets with resolutions ranging from 1,024 × \times 1,024 to 35,503 × \times 26,627. HRScene is collected and re-annotated by 10 graduate-level annotators, covering 25 scenarios, ranging from microscopic to radiology images, street views, long-range pictures, and telescope images. It includes HRIs of real-world objects, scanned documents, and composite multi-image. The two diagnostic evaluation datasets are synthesized by combining the target image with the gold answer and distracting images in different orders, assessing how well models utilize regions in HRI. We conduct extensive experiments involving 28 VLMs, including Gemini 2.0 Flash and GPT-4o. Experiments on HRScene show that current VLMs achieve an average accuracy of around 50% on real-world tasks, revealing significant gaps in HRI understanding. Results on synthetic datasets reveal that VLMs struggle to effectively utilize HRI regions, showing significant Regional Divergence and lost-in-middle, shedding light on future research. 1 Introduction High-resolution image (HRI) understanding aims to process images with a large number of pixels [ 7 ] . It plays an important role in numerous scenarios, such as pathology [ 16 ] , autonomous driving [ 79 ] , and large document understanding [ 27 , 52 , 28 ] . With the development of Vision Large Language Models (VLMs), automatic processing of HRIs has been a promising direction [ 64 ] . As shown in Figure 1(c) , Gemini [ 59 ] , Claude [ 4 ] , and GPT [ 3 ] can support images exceeding 1k resolution, enabling a wide range of real-world applications, such as 24/7 street monitoring [ 79 ] , galaxy research [ 35 ] , and radiology analysis [ 40 ] . (a) (b) (c) Figure 1 : (a) Overview taxonomy of the HRScene. (b) Performance of some VLMs on HRScene. (c) Comparison between the benchmarks that the mainstream VLMs are evaluated on and HRScene. The y-axis is the \sqrt{\text{total pixel}} . The boxes/icons indicate the image resolution they contain/support. The black lines inside each box show the average resolutions. However, even though existing VLMs can allegedly handle inputs of high-resolution images, there is a lack of comprehensive HRI benchmark, hindering objective calibration and measurement of progress on the effectiveness of HRI understanding. First, HRI evaluation is often missing from the official reports of mainstream VLMs. Figure 1(c) lists most of the vision-based benchmarks that VLMs are evaluated on, such as MMMU [ 76 ] , VQAv2 [ 23 ] , and AI2D [ 37 ] . Their average resolution is typically below 1k, making them unsuitable for HRI evaluation. Moreover, there is no comprehensive real-world or diagnostic benchmark for HRIs. As shown in Table 1 , the existing real-world datasets with HRIs tend to either focus on specific scenarios, like long-range images [ 65 ] , or particular resolutions, such as 8k [ 79 ] . The current diagnostic evaluation, namely, Multi-modal Neelde-in-the-Haystack (NIAH), primarily focuses on long text [ 48 ] or a mixture of low-resolution images [ 67 ] . To address this gap, we introduce HRScene, a unified benchmark for HRI understanding, covering diverse real-world scenes. HRScene incorporates 25 real-world tasks with resolutions ranging from 1024 \times 1024 to 35,503 \times 26,627, and 2 synthetic diagnostic datasets with 1k to 4k resolution. As shown in Figure 1(a) , we propose a task taxonomy to guide the development of HRScene : (1) we identify 8 categories of HRI tasks: Daily pictures, Urban planning, Paper scanned images, Artworks, Multi-subimages, Remote sensing, Medical Diagnosing, and Research understanding. (2) We focus on the 25 real-world scenes distributed across these categories, such as street monitoring and medical image understanding. (3) Each scene evaluates diverse capabilities of LLMs, such as counting, temporal and semantic reasoning, holistic image judgment, visual retrieval, spatial relations, and small object detection. Table 1 : Comparison with existing real-world benchmarks and Multi-modal NIAH diagnosis. Benchmark Scenes Easy Eval Highest Res Avg Res MME [ 79 ] 5 ✓ 5304 \times 7952 2000 \times 1500 HR-Bench [ 65 ] 1 ✗ 8000 \times 8000 8000 \times 8000 HRScene (ours) 25 ✓ 35503 \times 26627 4828 \times 4078 NIAH High-Res Multi-Res Real Img Needle in MM-NIAH [ 67 ] ✗ ✗ ✓ text MileBench [ 58 ] ✗ ✗ ✓ text Visual Haystack [ 71 ] ✗ ✗ ✗ image HRScene (ours) ✓ ✓ ✓ image HRScene is collected from 25 existing data resources, and 8 of them are re-annotated by 10 graduate-level annotators, with diverse view scales, ranging from microscope to radiology, street views, long-range, and telescope images. It contains high-resolution images of real objects, electronic documents, and composite multi-subimages. Besides, six datasets require domain-expert knowledge, while the remaining 19 belong to general domains. The diagnostic dataset is synthesized by combining the target image with the gold answer and visually similar distractors arranged in different orders to assess HRI utilization. Overall, HRScene comprises 7,068 images, with 2,008 of them being re-annotated. We conduct extensive experiments to evaluate the HRI understanding of 28 popular VLMs, including 6 proprietary VLMs: GPT [ 32 ] , Claude [ 4 ] , and Gemini [ 60 ] families, and 22 open-sourced models, including InternVL2 [ 12 ] , DeepSeek [ 72 ] , Phi [ 2 ] , Qwen [ 63 ] , MolMo [ 15 ] , LLava [ 47 ] , and Llama [ 19 ] families. As shown in Figure 1(b) , experiments on real-world tasks demonstrate that current VLMs perform modestly, with an average accuracy of around 50%, highlighting substantial challenges of HRScene. Besides, we also provide the human performance of all real-world datasets by engaging graduate-level annotators to annotate 750 image-question pairs. Our synthetic datasets provide a fine-grained understanding of VLM performance, revealing two robust issues across VLMs, including regional divergence and lost-in-the-middle, shedding light on future improvement directions. Our contributions are: (1) we propose HRScene, a unified benchmark with 25 real-world and 2 diagnostic datasets; (2) we benchmark 28 models on HRScene and show the significant performance gap; (3) we discover two salient issues of VLMs, including regional divergence and the lost-in-the-middle. Figure 2 : Distribution of resolution of each dataset. X-axis is the resolution and n k indicates the resolution is at least n^{2}*10^{6} pixels. 2 Related Work VLMs for High-resolution Image. Recent advances in vision-language models have demonstrated remarkable capabilities in understanding and reasoning about visual content [ 32 , 59 , 19 , 47 ] . However, processing high-resolution images remains a significant challenge due to computational constraints and the need to capture both fine-grained details and global context [ 8 ] . Two main categories of approaches have emerged to address

Read the complete text