Q-Zoom: Query-Aware Adaptive Perception for Efficient Multimodal Large Language Models

Yuheng Shi, Xiaohuan Pei, Linfeng Wen, Minjing Dong, Chang Xu · 2026

Read Q-Zoom: Query-Aware Adaptive Perception for Efficient Multimodal Large Language Models by Yuheng Shi, Xiaohuan Pei, Linfeng Wen, Minjing Dong, Chang Xu (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-06912.txt.

Introduction

Shi and four colleagues build Q-Zoom, a query-aware perception scheme for multimodal models. High-resolution inputs flood self-attention with redundant tokens, choking speed and ignoring query intent. Q-Zoom works coarse-to-fine, with a light gating network skipping high-resolution work when coarse features suffice, and a region proposal network pinpointing regions. It speeds inference about two and a half times on document and OCR tests and about four and a half times on high-resolution tasks at matched accuracy. Fidelity-first beats the baseline.

Catalog record

CREATORYuheng Shi, Xiaohuan Pei, Linfeng Wen, Minjing Dong, Chang Xu
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE70277
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.06912

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Q-Zoom: Query-Aware Adaptive Perception for Efficient Multimodal Large Language Models Yuheng Shi, Xiaohuan Pei, Linfeng Wen, Minjing Dong, Chang Xu Abstract Multimodal Large Language Models (MLLMs) require high-resolution visual inputs for fine-grained tasks like document understanding and dense scene perception. However, current global resolution scaling paradigms indiscriminately flood the quadratic self-attention mechanism with visually redundant tokens, severely bottlenecking inference throughput while ignoring spatial sparsity and query intent. To overcome this, we propose Q-Zoom , a query-aware adaptive high-resolution perception framework that operates in an efficient coarse-to-fine manner. First, a lightweight Dynamic Gating Network safely bypasses high-resolution processing when coarse global features suffice. Second, for queries demanding fine-grained perception, a Self-Distilled Region Proposal Network (SD-RPN) precisely localizes the task-relevant Region-of-Interest (RoI) directly from intermediate feature spaces. To optimize these modules efficiently, the gating network uses a consistency-aware generation strategy to derive deterministic routing labels, while the SD-RPN employs a fully self-supervised distillation paradigm. A continuous spatio-temporal alignment scheme and targeted fine-tuning then seamlessly fuse the dense local RoI with the coarse global layout. Extensive experiments demonstrate that Q-Zoom establishes a dominant Pareto frontier. Using Qwen2.5-VL-7B as a primary testbed, Q-Zoom accelerates inference by 2.52 × 2.52\times on Document & OCR benchmarks and 4.39 × 4.39\times in High-Resolution scenarios while matching the baseline’s peak accuracy. Furthermore, when configured for maximum perceptual fidelity, Q-Zoom surpasses the baseline’s peak performance by 1.1% and 8.1% on these respective benchmarks. These robust improvements transfer seamlessly to Qwen3-VL, LLaVA, and emerging RL-based thinking-with-image models, setting a new state-of-the-art for efficient, fine-grained visual perception. Project page is available at https://yuhengsss.github.io/Q-Zoom/ . I Introduction Multimodal Large Language Models (MLLMs) [ 34 , 3 , 79 ] have rapidly emerged as the cornerstone of artificial general intelligence, demonstrating unprecedented capabilities in visual reasoning [ 77 , 73 , 25 ] , document understanding [ 30 , 17 , 63 ] , and vision-language-action (VLA) modeling [ 13 , 22 , 4 ] . The bedrock of these sophisticated reasoning capabilities lies in the model’s foundational visual perception. Early pioneer architectures [ 9 , 34 , 32 , 33 ] , such as the original LLaVA series [ 34 , 32 ] , relied on frozen Vision Transformers [ 12 , 58 , 49 ] operating at a low and fixed resolution. While effective for coarse-grained image captioning, this rigid paradigm severely compressed and blurred critical local details. To overcome this perceptual bottleneck, subsequent works [ 75 , 33 , 60 , 8 ] have significantly advanced high-resolution adaptation. Sophisticated architectures, such as the AnyRes strategy [ 8 , 32 ] or native dynamic resolution encoding [ 60 , 11 ] , allow models to ingest varying and significantly higher input resolutions. These methods have achieved remarkable leaps in fine-grained perception and have become foundational mechanisms in recent state-of-the-art (SOTA) MLLMs [ 79 , 3 ] . Despite these advancements, efficient visual perception remains challenging. Current dynamic resolution solutions default to a brute-force scaling paradigm, producing visual tokens based solely on raw input resolution. Because visual representations serve to answer specific user queries, this exhaustive approach is computationally prohibitive and redundant. First, it ignores query-level intent by assuming maximum visual fidelity is universally required, wasting resources on simple questions answerable with coarse features. Second, it ignores spatial sparsity . Globally scaling the entire image floods the LLM’s quadratic self-attention mechanism with thousands of visually useless background tokens. Fig. 1 : Comparison of adaptive high-resolution perception paradigms. Training-free methods rely on handcrafted contrastive rules, requiring multiple redundant prefilling passes. RL-based methods use the LLM to auto-regressively generate code or coordinates to find the RoI. Our Q-Zoom framework operates directly on the intermediate feature space during a single prefilling pass, yielding superior efficiency. As illustrated in Figure 1 , recent literature attempts to tackle these perceptual inefficiencies through two primary paradigms. The first mitigates spatial sparsity via heuristic-driven, training-free pipelines [ 71 , 78 , 51 ] . These methods exploit the MLLM’s internal cross-attention to identify Regions-of-Interest (RoIs) on the fly, which are then cropped and re-encoded. However, because extracting these attention maps requires redundant prefilling passes or expensive auto-regressive decoding, they severely bottleneck inference efficiency and struggle to generalize due to rigid rules. A second branch reformulates adaptive perception through an Reinforcement Learning (RL) based Think-with-Image paradigm [ 48 , 73 , 77 , 25 ] . These models identify RoIs via explicit auto-regressive reasoning or sandbox code execution. While effective at reducing visual token usage, they inadvertently shift the computational burden to the language model. Relying on lengthy Chain-of-Thought (CoT) decoding drastically extends inference latency. Furthermore, optimizing these models via RL is prohibitively expensive, data-hungry, and highly unstable. To overcome these limitations, we propose Q-Zoom, a fully integrated, two-stage adaptive framework that addresses both query-level and spatial redundancy directly within the intermediate feature space. Inspired by findings that MLLM middle layers harbor robust visual grounding [ 54 , 21 ] , we attach two lightweight sub-networks to the frozen backbone. To eliminate query-level redundancy, the first stage introduces a Dynamic Gating Network to assess whether coarse features are sufficient. This router is optimized via a novel Consistency-Aware Sample Generation strategy, which derives deterministic routing labels by evaluating responses across a resolution trajectory, bypassing human annotations. For queries requiring refinement, the second stage activates the Self-Distilled Region Proposal Network (SD-RPN). Operating on intermediate tokens, the SD-RPN predicts a dense heatmap to crop and re-encode only task-relevant RoIs. It is trained via a self-distillation paradigm that mines internal cross-attention maps, filters sink tokens, and applies a selective tri-state label assignment to generate pseudo-labels. Crucially, Q-Zoom acquires these signals in a single prefilling pass. As shown in Figure 1 , this circumvents the redundant prefilling and sluggish auto-regressive decoding of prior methods, drastically accelerating throughput. Finally, to resolve the spatial misalignment caused by processing the coarse global image and fine-grained local RoI, we introduce a continuous spatio-temporal positional encoding scheme. Coupled with a targeted Post-Supervised Fine-Tuning (Post-SFT) on explicitly mined hard failure cases, this teaches the LLM to seamlessly fuse local details with the global layout, restoring robust spatial reasoning. We validate Q-Zoom across diverse base models [ 32 , 3 ] on demanding Document & OCR [ 46 , 45 , 44 , 47 ] and High-Resolution tasks [ 65 , 62 , 74 ] . Our empirical results demonstrate that Q-Zoom establishes a new Pareto frontier in the trade-off between perceptual accuracy and computational efficiency. For example, when integrated into the Qwen2.5-VL-7B backbone, Q-Zoom not only surpasses existing training-free heuristics but also outperforms concurrent cutting-edge models. Crucially, these perceptual enhancements do not bottleneck inference latency. Through the intelligent routing of simple queries and the targeted extraction of RoIs for detail-demanding tasks, Q-Zoom exceeds the peak performance of a brute-force baseline scaled to 4,096 visual tokens, all while strictly constraining its own maximum budget to just 1,024 tokens. This elegant scaling translates to exceptional 53.0% and 73.2% reductions in visual token costs, alongside 2.52\times and 4.39\times accelerations in inference throughput on Document & OCR and High-Resolution tasks, respectively. Remarkably, Q-Zoom acts as a versatile plug-and-play module, providing orthogonal performance boosts even when integrated atop advanced RL-trained thinking models [ 64 ] ,

Read the complete text