π–Ήπ—ˆπ—ˆπ—†π–Ύπ—‹ \sf\small{Zoomer} : Adaptive Image Focus Optimization for Black-box MLLM

Jiaxu Qian, Chendong Wang, Yifan Yang, Chaoyun Zhang, Huiqiang Jiang, Xufang Luo, Yu Kang Β· 2025

Read π–Ήπ—ˆπ—ˆπ—†π–Ύπ—‹ \sf\small{Zoomer} : Adaptive Image Focus Optimization for Black-box MLLM by Jiaxu Qian, Chendong Wang, Yifan Yang, Chaoyun Zhang, Huiqiang Jiang, Xufang Luo, Yu Kang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-00742.txt.

Introduction

Jiaxu Qian and a large team help vision models look at the right part of a picture. Their paper, from twenty twenty-five, is called Zoomer. Big models that read text and images often invent details, above all when objects are small. Fixed shrinking of images throws away fine detail.

Zoomer spotlights zones that match the question being asked. It keeps the layout of objects intact while doing so. It also splits its budget of image tokens between the whole scene and fine zones. On nine test sets and three paid models, accuracy rose by up to twenty-seven percent. Token use fell at the same time, which cuts cost.

Catalog record

CREATORJiaxu Qian, Chendong Wang, Yifan Yang, Chaoyun Zhang, Huiqiang Jiang, Xufang Luo, Yu Kang
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE39 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-00742

Browse the full library Β· About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

π–Ήπ—ˆπ—ˆπ—†π–Ύπ—‹ \sf\small{Zoomer} : Adaptive Image Focus Optimization for Black-box MLLM Jiaxu Qian, Chendong Wang, Yifan Yang, Chaoyun Zhang, Huiqiang Jiang, Xufang Luo, Yu Kang Abstract Recent advancements in multimodal large language models (MLLMs) have broadened the scope of vision-language tasks, excelling in applications like image captioning and interactive question-answering. However, these models struggle with accurately processing visual data, particularly in tasks requiring precise object recognition and fine visual details. Stringent token limits often result in the omission of critical information, hampering performance. To address these limitations, we introduce π–Ήπ—ˆπ—ˆπ—†π–Ύπ—‹ \sf\small{Zoomer} , a novel visual prompting mechanism designed to enhance MLLM performance while preserving essential visual details within token limits. π–Ήπ—ˆπ—ˆπ—†π–Ύπ—‹ \sf\small{Zoomer} features three key innovations: a prompt-aware strategy that dynamically highlights relevant image regions, a spatial-preserving orchestration schema that maintains object integrity, and a budget-aware prompting method that balances global context with crucial visual details. Comprehensive evaluations across multiple datasets demonstrate that π–Ήπ—ˆπ—ˆπ—†π–Ύπ—‹ \sf\small{Zoomer} consistently outperforms baseline methods, achieving up to a 26.9 % 26.9\% improvement in accuracy while significantly reducing token consumption. 1 Introduction Vision-language understanding has witnessed remarkable progress with multimodal large language models (MLLMs) such as GPT-4o, Gemini Pro, and Claude 3.5 [ 13 , 8 ] . While these black-box MLLMs have demonstrated impressive capabilities in controlled settingsβ€”even being touted as having PhD-level skillsβ€”our large-scale investigation reveals a critical hallucination problem: their inherent small object blindness and limited visibility lead to catastrophic failures in real-world visual reasoning tasks. Through systematic analysis, we identify two fundamental limitations that create this problem. First, despite their sophisticated architectures, current black-box MLLMs process images with uniform attention, lacking the human-like ability to focus on relevant regions while maintaining spatial awareness. As shown in Figure 1 , this limitation leads to surprising failures in entry-level tasks like object countingβ€”a problem that persists across different models and prompting strategies. Our investigation reveals that this is not merely an implementation issue but a fundamental limitation of how these models process visual information. Second, we identify a critical tension between token budgets and visual fidelity in black-box MLLMs. While commodity black-box MLLM has treated token limitations, typically fixed token counts, as a simple engineering constraint to ensure computational efficiency and user fairness [ 4 ] , our analysis reveals it as a fundamental resource allocation problem. As demonstrated in Figure 2 , the standard approach of uniform downsampling leads to catastrophic loss of fine detailsβ€”a problem that cannot be solved by better downsampling algorithms or prompt engineering alone. Figure 1 : Illustration of a black-box MLLM’s approach to counting cacti in an image. The model identifies two small cacti on the left side and overlooks the single cactus on the right side of the image, arriving at a total of three cacti. The processed prompt highlights specific regions of interest to facilitate the correct object count. Figure 2 : Illustration of information loss during image processing in black-box MLLMs. The original high-resolution image (4240x2832) is downscaled to meet token limits (1536x1026), leading to the loss of critical details. Cropping to focus on a region of interest (146x246) allows the model to correctly identify the book title as β€œWebster’s II”. This investigation leads us to formulate a novel problem in visual prompting: How can we achieve region-aware visual processing within the strict token constraints of black-box MLLMs? This question challenges the conventional view that visual prompting is merely a simple input formatting task. Instead, we show that effective visual prompting requires solving three interconnected challenges: Region Selection : How to identify and prioritize task-relevant regions without access to model internals; Spatial Preservation : How to maintain structural relationships while maximizing detail preservation; Budget Optimization : How to allocate limited tokens between global context and local details. Based on these insights, we propose \sf\small{Zoomer} , a visual prompting framework that addresses these challenges through three technical innovations: β€’ A prompt-aware visual emphasis mechanism that enables selective attention without modifying model architecture β€’ A spatial-preserving orchestration schema that maintains global context while preserving local details β€’ A budget-aware region selection strategy that optimally allocates tokens across image regions In a comprehensive evaluation across datasets such as Vstar [ 28 ] , CVBench [ 24 ] , and RealworldQA [ 29 ] , \sf\small{Zoomer} consistently outperformed baseline methods. Notably, in the Vstar dataset, \sf\small{Zoomer} -Patches achieved a 26.9% accuracy improvement over the baseline, while in RealWorldQA, \sf\small{Zoomer} -Adaptive outperformed the baseline by 12.1%. Alongside these accuracy gains, \sf\small{Zoomer} significantly reduced token usage. For example, on the TerraIncognita dataset, \sf\small{Zoomer} achieved 6.4% higher accuracy with a 67% reduction in token consumption compared to the baseline. These results confirm that \sf\small{Zoomer} not only addresses the limitations of visual processing in black-box MLLMs but also enhances efficiency. Moreover, across APIs like GPT-4o 1 1 1 https://platform.openai.com/ , Gemini-1.5Pro 2 2 2 https://gemini.google.com/ , and Claude-3.5-Sonnet 3 3 3 https://anthropic.com/ , \sf\small{Zoomer} demonstrated consistent improvements in both accuracy and token efficiency, solidifying its potential to optimize MLLM performance in real-world, high-resolution applications. The contribution of this work can be summarized as follows: (1) We conduct a detailed investigation of GPT-4o’s image prompting strategy, exposing key limitations in its handling of visual inputs. (2) We introduce \sf\small{Zoomer} , a novel mechanism that addresses the challenge of preserving visual detail in black-box MLLMs while adhering to token constraints. (3) We present extensive experimental results across multiple datasets, demonstrating that \sf\small{Zoomer} achieves significant improvements in both accuracy and token efficiency, offering valuable insights into enhancing multimodal processing in constrained environments. 2 Pilot Experiments One of the primary challenges faced by black-box MLLMs is their inability to process visual inputs efficiently, leading to diminished accuracy in fine-grained visual tasks. Models like GPT-4o often struggle with recognizing detailed or occluded objects, particularly when dealing with complex images. These limitations are compounded by token constraints, which restrict the amount of image data that can be processed in a single prompt. Furthermore, according to the Vision pricing calculator 4 4 4 https://openai.com/api/pricing/ , GPT handle images by resizing and splitting them into basic units of 512Γ—512 pixels. Each of these units corresponds to 170 tokens. This method of processing not only imposes a strict limit on the image resolution but also increases the computational overhead due to the additional tokens generated from splitting. As a result, vital visual details may be lost when images are downsampled or resized to fit within the token limits, leading to poor performance on tasks that require precise visual grounding. Method Accuracy Prompt Tokens Unaltered Input 57% 955 Image Crop 58% 270 Zoomed Crop 64% 270 Table 1 : Performance of different methods on Image from Vstar. To evaluate this issue, we conducted a series of pilot experiments using GPT-4o-0513 on the Vstar-Bench dataset. This dataset challenges MLLMs to accurately identify detailed objects within high-resolution images, making it an ideal test for the model’s capacity to handle fine-grained visual information. The experiments compared three different image processing strategies: (1) an unprocessed prompt ( Unaltered Input ), where the image is fed to the model in its original form; (2) a prompt where the image is cropped to focus on the target object ( Image Crop ); and (3) a prompt where the cropped image

Read the complete text