AVAM: Universal Training-free Adaptive Visual Anchoring Embedded into Multimodal Large Language Model for Multi-image Question Answering
Kang Zeng Guojin Zhong Jintao Cheng Jin Yuan Zhiyong Li · 2025
Read AVAM: Universal Training-free Adaptive Visual Anchoring Embedded into Multimodal Large Language Model for Multi-image Question Answering by Kang Zeng Guojin Zhong Jintao Cheng Jin Yuan Zhiyong Li (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-17860.txt.
Introduction
Answering questions over many images at once drowns a model in pixels it does not need. Kang Zeng and four co-authors add a step in 2025 that anchors the model's gaze on the parts that count. Their method needs no fresh training and can be dropped into models that already exist.
The anchor step squeezes the visual input in a way that bends to the question, so the count of kept pieces is not fixed in advance. It also keeps the kept pieces whole, rather than chopped into shards that break the scene's sense. A twin read then balances the full view with the squeezed one before the model answers. The team reports gains in both accuracy and speed. The design aims at any model, not one brand alone.
Catalog record
| CREATOR | Kang Zeng Guojin Zhong Jintao Cheng Jin Yuan Zhiyong Li |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 47 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2508-17860
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
AVAM: Universal Training-free Adaptive Visual Anchoring Embedded into Multimodal Large Language Model for Multi-image Question Answering Kang Zeng Guojin Zhong Jintao Cheng Jin Yuan Zhiyong Li Abstract The advancement of Multimodal Large Language Models (MLLMs) has driven significant progress in Visual Question Answering (VQA), evolving from Single to Multi Image VQA (MVQA). However, the increased number of images in MVQA inevitably introduces substantial visual redundancy that is irrelevant to question answering, negatively impacting both accuracy and efficiency. To address this issue, existing methods lack flexibility in controlling the number of compressed visual tokens and tend to produce discrete visual fragments, which hinder MLLMs’ ability to comprehend images holistically. In this paper, we propose a straightforward yet universal Adaptive Visual Anchoring strategy, which can be seamlessly integrated into existing MLLMs, offering significant accuracy improvements through adaptive compression. Meanwhile, to balance the results derived from both global and compressed visual input, we further introduce a novel collaborative decoding mechanism, enabling optimal performance. Extensive experiments validate the effectiveness of our method, demonstrating consistent performance improvements across various MLLMs. The code will be publicly available. 1 introduction Recently, Multimodal Large Language Models (MLLMs) have demonstrated remarkable cross-modal understanding and reasoning capabilities across various vision-language tasks [ 2 , 11 , 24 , 28 , 1 ] , significantly advancing Single-Image Visual Question Answering (SVQA) [ 43 , 8 , 9 ] . In this task, an MLLM processes a single image alongside natural language questions to generate accurate and contextually relevant answers. While the integration of visual information enhances the model’s capacity to predict answers by providing richer multimodal context, it also introduces computational inefficiencies in QA process. Additionally, redundant visual information may interfere with reasoning, potentially reducing the accuracy of the generated answers. To address this challenge, existing SVQA approaches primarily adopt attention-based strategies [ 45 , 10 , 51 ] to highlight salient visual tokens or query-based strategies [ 4 , 2 , 22 ] to retrieve relevant visual tokens using a learnable query. These methods effectively reduce redundant visual tokens processed by MLLMs, enhancing the efficiency of the QA process while achieving answer accuracy comparable to or even higher than that of the original MLLMs. Figure 1 : An example to illustrate the differences in visual pruning between our approach and other methods. While existing methods rely on a fixed number of visual tokens, leading to fragmented visual representations, our method adaptively prunes continuous key regions across multiple images. Research attention toward Multi-Image Visual Question Answering (MVQA) has increased significantly due to its practical relevance. Unlike SVQA, MVQA involves injecting multiple images into MLLMs for question answering, thereby furnishing abundant semantics through temporal/spatial views to facilitate complex reasoning [ 40 , 30 ] . This mode, where multimodal data with interleaved image-text interactions demonstrates more pronounced practical utility, enables applications like event prediction via temporal image sequences [ 16 , 25 , 53 ] and enhanced perceptual analysis through spatial multi-image integration [ 34 , 36 , 44 ] . However, visual redundancy becomes more pronounced in multi-image paradigm, where key visual information is increasingly obscured as the number of images expands. Existing visual redundancy filtering approaches encounter three major challenges when applied to MVQA: First, learning-based approaches [ 2 , 24 , 4 , 7 , 13 ] , represented by those utilizing learnable queries to extract specific deep-level visual features akin to using a mold for forging, often require substantial training data to develop such a generalizable “feature mold”. This data dependency and inherent scalability constraints hinder optimal accuracy-efficiency trade-offs. Second, many training-free methods [ 10 , 51 , 49 , 45 ] typically mandate fixed filtering ratios as hyperparameters. Given varying redundancy levels across images, this rigidity causes critical information loss or redundant data retention. Third, another challenge prevalent in training-free methods stems from their discrete visual token selection process. This approach disrupts the spatial continuity and local semantics within an image (as illustrated in Fig. 1 (a)). Crucially, this problem is amplified in multi-image scenarios, sparse and discrete token selections from individual images can interfere with modeling cross-image correlations and undermine spatial consistency across frames. To address the aforementioned challenges and enhance MLLM’s capability in processing multi-image inputs, this paper proposes a training-free, adaptive visual anchoring strategy that extracts critical regions from global visual representations through the response map quantifying cross-modal relevance between visual features and corresponding text. Based on response hotspots within this map, we generate a series of anchor boxes to localize question-relevant visual regions, subsequently introducing a response density metric to preserve the most critical region and filter other redundant visual tokens. Furthermore, we propose a novel collaborative decoding mechanism that dynamically fuses decoding distributions under global visual context according to varying redundancy levels, harmonizes responses from both global and compressed local visual cues and achieves improved performance. The main contributions of this work can be summarized: • We propose an adaptive visual anchoring strategy tailored for MVQA, which serves as a training-free pipeline that can be seamlessly integrated into polular MLLMs. To the best of our knowledge, this is the first work to systematically explore the issue of visual redundancy in MVQA. • We propose a novel collaborative decoding mechanism that dynamically balances probability distributions between global visual context and local cues according to varying visual redundancy ratios. • Extensive experiments demonstrate that our method can be effectively extended to most popular MLLMs, enhancing their performance on MVQA. 2 Related Work 2.1 Multimodel Large Language Models Driven by the remarkable breakthroughs of LLMs in natural language understanding and generation [ 14 , 52 , 17 , 1 , 27 ] , MLLMs extended from LLMs with visual capabilities have demonstrated impressive performance in multimodal understanding [ 42 , 38 , 1 , 46 , 29 ] . Existing MLLMs [ 29 , 33 , 20 ] typically convert images into visual token sequences via the patch embedding process of pre-trained vision encoders. For instance, LLaVA-v1.5 [ 28 ] transforms a 336\times 336 resolution image into 576 visual tokens. In multi-image scenarios, the number of visual tokens could exceed text tokens by hundreds of times. However, most current MLLMs store visual semantics through fixed quantities of visual tokens per image, failing to dynamically extract critical visual features for downstream tasks. This leads to substantial retention of task-irrelevant redundant visual information when fed into LLM, consequently impairing the performance. 2.2 Visual Token Compression for VQA Compared with text tokens, visual tokens exhibit strong sparsity in semantic representation, revealing significant compression potential. MLLMs based on learnable visual token compression strategies [ 4 , 22 , 2 ] employ learnable queries to abstract visual features, substantially reducing visual token counts and achieve a strong performance on VQA tasks. For example, Qwen-VL [ 4 ] compresses visual tokens per image from 1024 to 256 via a cross-attention adapter. Meanwhile, training-free methods have attracted considerable research interest due to their minimal computational cost and flexibility. Recent works [ 10 ] first investigated inefficient visual attention patterns in popular MLLMs and pruned visual tokens based on attention scores. Follow-up studies [ 49 , 50 , 45 ] eliminate redundancy by selecting dominant visual tokens through internal correlations. Advanced approaches [