Look Where It Matters: High-Resolution Crops Retrieval for Efficient VLMs
Nimrod Shabtay, Moshe Kimhi, Artem Spector, Sivan Haray, Ehud Rivlin, Chaim Baskin, Raja Giryes, Eli Schwartz · 2026
Read Look Where It Matters: High-Resolution Crops Retrieval for Efficient VLMs by Nimrod Shabtay, Moshe Kimhi, Artem Spector, Sivan Haray, Ehud Rivlin, Chaim Baskin, Raja Giryes, Eli Schwartz (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-16932.txt.
Introduction
Shabtay and co-authors present AwaRes, a framework that lets vision language models fetch fine image detail only when needed. Full high resolution input is costly, while low resolution input can miss small text. AwaRes starts from a low resolution global view. A tool call then retrieves high resolution crops for the parts a query needs. Training data are built with a judge that labels when cropping helps, plus an oracle that locates the evidence. Training mixes supervised fine tuning with GRPO and a reward that charges for crops.
Catalog record
| CREATOR | Nimrod Shabtay, Moshe Kimhi, Artem Spector, Sivan Haray, Ehud Rivlin, Chaim Baskin, Raja Giryes, Eli Schwartz |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 67636 |
| LICENSE | http://creativecommons.org/licenses/by-sa/4.0/ |
| RIGHTS | CC BY SA |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.16932
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Look Where It Matters: High-Resolution Crops Retrieval for Efficient VLMs Nimrod Shabtay, Moshe Kimhi, Artem Spector, Sivan Haray, Ehud Rivlin, Chaim Baskin, Raja Giryes, Eli Schwartz Abstract Vision-language models (VLMs) typically process images at a native high-resolution, forcing a trade-off between accuracy and computational efficiency: high-resolution inputs capture fine details but incur significant computational costs, while low-resolution inputs advocate for efficiency, they potentially miss critical visual information, like small text. We present AwaRes, a spatial-on-demand framework that resolves this accuracy–efficiency trade-off by operating on a low-resolution global view and using tool-calling to retrieve only high-resolution segments needed for a given query. We construct supervised data automatically: a judge compares low- vs. high-resolution answers to label whether cropping is needed, and an oracle grounding model localizes the evidence for the correct answer, which we map to a discrete crop set to form multi-turn tool-use trajectories. We train our framework with cold-start SFT followed by multi-turn GRPO with a composite reward that combines semantic answer correctness with explicit crop-cost penalties. Project Page: https://nimrodshabtay.github.io/AwaRes/ Figure 1 : AwaRes overview. Left: Given a low-resolution image, AwaRes uses tool-calling to request only the high-resolution crops needed to answer the query. Right: Accuracy vs. retained visual tokens across six benchmarks. AwaRes performs similarly to native high-resolution (80.3%) while using only 36% of the visual tokens. 1 Introduction Vision–language models (VLMs) increasingly rely on high-resolution visual inputs to solve detail-sensitive tasks such as document question answering, chart understanding, and understanding semantics and text in dense natural images. However, high resolution is expensive: the number of visual tokens grows rapidly with image resolution, making high-resolution inference a major bottleneck in practice. Existing approaches to reduce this cost largely fall into two camps. First, token pruning methods selectively discard visual tokens to reduce computation [ 4 , 26 , 27 , 30 , 32 ] . While effective in principle, they often introduce irregular token patterns and dynamic sequence lengths that can be difficult to translate into end-to-end serving speedups in common inference stacks, such as vLLM [ 13 ] , where efficiency is tied to predictable sequence length. Second, resolution escalation methods [ 19 , 12 ] learn when to request a higher-resolution view, but typically treat the decision as binary: if more details are needed, the entire high-resolution image is retrieved, wasting computation on regions irrelevant to the question. A key observation is that the demand for high fidelity is usually spatially sparse , as can be seen in Fig. 3 . Many questions require fine detail in only a small portion of the image: a single value on a chart axis, a specific cell in a table, or a tiny object in the corner of an image. In cases where low resolution image do not poses the fine-grained information, retrieving the full image at native high-resolution is unnecessarily expensive. We advocate that answering the question of where to look matters as much as whether to look. We propose VLM that is spatially aware to resolution (abbreviated AwaRes), a framework that exploits this spatial sparsity via a simple tool-calling interface that targets high-resolution crop acquisition . AwaRes processes a low-resolution global view by default, and when additional detail is required, it invokes a tool-call that requests only specific high-resolution sub-regions, and then answers conditioned on both. This multi-turn structure is naturally compatible with KV-caching: computation from the initial low-resolution turn is reused and extended in the crop turn without architectural changes, making AwaRes practical for deployment. We train AwaRes to learn a single coupled-decision policy (CDP) that jointly decides (i) whether additional resolution is needed and (ii) where to acquire it by selecting a subset of crops. Crucially, these decisions are fused into the model’s first-turn action: either answer directly, or emit a structured crop request that simultaneously signals escalation and specifies the target regions. For the cold-start phase, we construct the supervision automatically, without manual spatial annotations, by (i) identifying examples where low resolution is insufficient using an LLM as a Judge (LaaJ) that compares low- vs. high-resolution model outputs, and (ii) localizing the evidence for the correct answer using an oracle grounding model to produce target crops. We evaluate AwaRes on six benchmarks spanning document understanding and general visual QA. Across these tasks, AwaRes almost matches full high-resolution performance on average (80.3% vs. 80.46%) while using only 36% of the pixels/tokens, substantially reducing inference cost. On ChartQA, DocVQA and OCRBench, AwaRes even slightly improves over full-resolution baselines while remaining significantly more efficient. Our Contributions are listed as follows: • We introduce a spatial-on-demand inference framework for VLMs that requests only targeted high-resolution crops through tool-calling, enabling system-friendly multi-turn KV-cache reuse. • We propose an automatic data curation pipeline that produces multi-turn tool-use trajectories without manual spatial annotations. • We refine crop usage with multi-turn GRPO using an explicit accuracy–efficiency objective that penalizes unnecessary crop acquisition while discouraging missed crop requests when detail is required. 2 Related Work Several strategies have emerged to prune, compress, or dynamically reduce the number of visual tokens in Vision Language Models. One line of research focuses on dynamic token pruning. Methods such as FastV [ 4 ] , HoloV [ 32 ] , PyramidDrop [ 26 ] , FitPrune [ 28 ] , TopV [ 8 ] , SparseVILA [ 11 ] , IVTP [ 7 ] , LLaVolta [ 3 ] , and SAINT [ 9 ] discard uninformative tokens within the LLM layers based on attention scores or learned criteria. Alternatively, VisionZip [ 27 ] , FastVLM [ 22 ] , and SparseVLM [ 30 ] prune tokens directly after the vision encoder. While effective, pruning-based approaches must commit to a fixed retention ratio before inference, applying the same token budget regardless of sample complexity. In contrast, our method is fully adaptive: it dynamically determines both whether additional detail is needed and which spatial regions to acquire, allowing simple images to be processed at minimal cost while allocating more resources only when the query demands fine-grained perception. A second line of work explores resolution selection. CARES [ 12 ] uses an external lightweight model to predict the optimal input resolution before the VLM processes the image, while CROP [ 6 ] identifies contextual regions of interest via an auxiliary module. These methods rely on external components to make resolution decisions, whereas our approach enables the VLM itself to determine when and where additional detail is needed through its native capabilities, requiring no auxiliary models. Recent frameworks like ZoomEye [ 20 ] and DeepEyes [ 31 ] enhance VLM performance through dynamic zooming and high-resolution cropping. However, these methods prioritize accuracy over efficiency: ZoomEye performs multiple inference passes through a hierarchical image tree, while DeepEyes appends zoomed crops to the context, progressively increasing the token count. In contrast, our work employs cropping specifically for efficiency—requesting only the minimal high-resolution regions needed while maintaining a compact token budget. VisionThink [ 19 ] introduced a reinforcement learning approach where the model processes a low-resolution image and emits a tool call to request a high-resolution version when needed. While effective at determining resolution sufficiency, VisionThink retrieves the entire high-resolution image globally when escalation is triggered. Our method goes further by identifying the