πΉππππΎπ \sf\small{Zoomer} : Adaptive Image Focus Optimization for Black-box MLLM
Jiaxu Qian, Chendong Wang, Yifan Yang, Chaoyun Zhang, Huiqiang Jiang, Xufang Luo, Yu Kang Β· 2025
Read πΉππππΎπ \sf\small{Zoomer} : Adaptive Image Focus Optimization for Black-box MLLM by Jiaxu Qian, Chendong Wang, Yifan Yang, Chaoyun Zhang, Huiqiang Jiang, Xufang Luo, Yu Kang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-00742.txt.
Introduction
Jiaxu Qian and a large team help vision models look at the right part of a picture. Their paper, from twenty twenty-five, is called Zoomer. Big models that read text and images often invent details, above all when objects are small. Fixed shrinking of images throws away fine detail.
Zoomer spotlights zones that match the question being asked. It keeps the layout of objects intact while doing so. It also splits its budget of image tokens between the whole scene and fine zones. On nine test sets and three paid models, accuracy rose by up to twenty-seven percent. Token use fell at the same time, which cuts cost.
Catalog record
| CREATOR | Jiaxu Qian, Chendong Wang, Yifan Yang, Chaoyun Zhang, Huiqiang Jiang, Xufang Luo, Yu Kang |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 39 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-00742
Browse the full library Β· About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
πΉππππΎπ \sf\small{Zoomer} : Adaptive Image Focus Optimization for Black-box MLLM Jiaxu Qian, Chendong Wang, Yifan Yang, Chaoyun Zhang, Huiqiang Jiang, Xufang Luo, Yu Kang Abstract Recent advancements in multimodal large language models (MLLMs) have broadened the scope of vision-language tasks, excelling in applications like image captioning and interactive question-answering. However, these models struggle with accurately processing visual data, particularly in tasks requiring precise object recognition and fine visual details. Stringent token limits often result in the omission of critical information, hampering performance. To address these limitations, we introduce πΉππππΎπ \sf\small{Zoomer} , a novel visual prompting mechanism designed to enhance MLLM performance while preserving essential visual details within token limits. πΉππππΎπ \sf\small{Zoomer} features three key innovations: a prompt-aware strategy that dynamically highlights relevant image regions, a spatial-preserving orchestration schema that maintains object integrity, and a budget-aware prompting method that balances global context with crucial visual details. Comprehensive evaluations across multiple datasets demonstrate that πΉππππΎπ \sf\small{Zoomer} consistently outperforms baseline methods, achieving up to a 26.9 % 26.9\% improvement in accuracy while significantly reducing token consumption. 1 Introduction Vision-language understanding has witnessed remarkable progress with multimodal large language models (MLLMs) such as GPT-4o, Gemini Pro, and Claude 3.5 [ 13 , 8 ] . While these black-box MLLMs have demonstrated impressive capabilities in controlled settingsβeven being touted as having PhD-level skillsβour large-scale investigation reveals a critical hallucination problem: their inherent small object blindness and limited visibility lead to catastrophic failures in real-world visual reasoning tasks. Through systematic analysis, we identify two fundamental limitations that create this problem. First, despite their sophisticated architectures, current black-box MLLMs process images with uniform attention, lacking the human-like ability to focus on relevant regions while maintaining spatial awareness. As shown in Figure 1 , this limitation leads to surprising failures in entry-level tasks like object countingβa problem that persists across different models and prompting strategies. Our investigation reveals that this is not merely an implementation issue but a fundamental limitation of how these models process visual information. Second, we identify a critical tension between token budgets and visual fidelity in black-box MLLMs. While commodity black-box MLLM has treated token limitations, typically fixed token counts, as a simple engineering constraint to ensure computational efficiency and user fairness [ 4 ] , our analysis reveals it as a fundamental resource allocation problem. As demonstrated in Figure 2 , the standard approach of uniform downsampling leads to catastrophic loss of fine detailsβa problem that cannot be solved by better downsampling algorithms or prompt engineering alone. Figure 1 : Illustration of a black-box MLLMβs approach to counting cacti in an image. The model identifies two small cacti on the left side and overlooks the single cactus on the right side of the image, arriving at a total of three cacti. The processed prompt highlights specific regions of interest to facilitate the correct object count. Figure 2 : Illustration of information loss during image processing in black-box MLLMs. The original high-resolution image (4240x2832) is downscaled to meet token limits (1536x1026), leading to the loss of critical details. Cropping to focus on a region of interest (146x246) allows the model to correctly identify the book title as βWebsterβs IIβ. This investigation leads us to formulate a novel problem in visual prompting: How can we achieve region-aware visual processing within the strict token constraints of black-box MLLMs? This question challenges the conventional view that visual prompting is merely a simple input formatting task. Instead, we show that effective visual prompting requires solving three interconnected challenges: Region Selection : How to identify and prioritize task-relevant regions without access to model internals; Spatial Preservation : How to maintain structural relationships while maximizing detail preservation; Budget Optimization : How to allocate limited tokens between global context and local details. Based on these insights, we propose \sf\small{Zoomer} , a visual prompting framework that addresses these challenges through three technical innovations: β’ A prompt-aware visual emphasis mechanism that enables selective attention without modifying model architecture β’ A spatial-preserving orchestration schema that maintains global context while preserving local details β’ A budget-aware region selection strategy that optimally allocates tokens across image regions In a comprehensive evaluation across datasets such as Vstar [ 28 ] , CVBench [ 24 ] , and RealworldQA [ 29 ] , \sf\small{Zoomer} consistently outperformed baseline methods. Notably, in the Vstar dataset, \sf\small{Zoomer} -Patches achieved a 26.9% accuracy improvement over the baseline, while in RealWorldQA, \sf\small{Zoomer} -Adaptive outperformed the baseline by 12.1%. Alongside these accuracy gains, \sf\small{Zoomer} significantly reduced token usage. For example, on the TerraIncognita dataset, \sf\small{Zoomer} achieved 6.4% higher accuracy with a 67% reduction in token consumption compared to the baseline. These results confirm that \sf\small{Zoomer} not only addresses the limitations of visual processing in black-box MLLMs but also enhances efficiency. Moreover, across APIs like GPT-4o 1 1 1 https://platform.openai.com/ , Gemini-1.5Pro 2 2 2 https://gemini.google.com/ , and Claude-3.5-Sonnet 3 3 3 https://anthropic.com/ , \sf\small{Zoomer} demonstrated consistent improvements in both accuracy and token efficiency, solidifying its potential to optimize MLLM performance in real-world, high-resolution applications. The contribution of this work can be summarized as follows: (1) We conduct a detailed investigation of GPT-4oβs image prompting strategy, exposing key limitations in its handling of visual inputs. (2) We introduce \sf\small{Zoomer} , a novel mechanism that addresses the challenge of preserving visual detail in black-box MLLMs while adhering to token constraints. (3) We present extensive experimental results across multiple datasets, demonstrating that \sf\small{Zoomer} achieves significant improvements in both accuracy and token efficiency, offering valuable insights into enhancing multimodal processing in constrained environments. 2 Pilot Experiments One of the primary challenges faced by black-box MLLMs is their inability to process visual inputs efficiently, leading to diminished accuracy in fine-grained visual tasks. Models like GPT-4o often struggle with recognizing detailed or occluded objects, particularly when dealing with complex images. These limitations are compounded by token constraints, which restrict the amount of image data that can be processed in a single prompt. Furthermore, according to the Vision pricing calculator 4 4 4 https://openai.com/api/pricing/ , GPT handle images by resizing and splitting them into basic units of 512Γ512 pixels. Each of these units corresponds to 170 tokens. This method of processing not only imposes a strict limit on the image resolution but also increases the computational overhead due to the additional tokens generated from splitting. As a result, vital visual details may be lost when images are downsampled or resized to fit within the token limits, leading to poor performance on tasks that require precise visual grounding. Method Accuracy Prompt Tokens Unaltered Input 57% 955 Image Crop 58% 270 Zoomed Crop 64% 270 Table 1 : Performance of different methods on Image from Vstar. To evaluate this issue, we conducted a series of pilot experiments using GPT-4o-0513 on the Vstar-Bench dataset. This dataset challenges MLLMs to accurately identify detailed objects within high-resolution images, making it an ideal test for the modelβs capacity to handle fine-grained visual information. The experiments compared three different image processing strategies: (1) an unprocessed prompt ( Unaltered Input ), where the image is fed to the model in its original form; (2) a prompt where the image is cropped to focus on the target object ( Image Crop ); and (3) a prompt where the cropped image