Your Large Vision-Language Model Only Needs A Few Attention Heads For Visual Grounding
Seil Kang, Jinyeong Kim, Junhyeok Kim, Seong Jae Hwang · 2025
Read Your Large Vision-Language Model Only Needs A Few Attention Heads For Visual Grounding by Seil Kang, Jinyeong Kim, Junhyeok Kim, Seong Jae Hwang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-06287.txt.
Introduction
Seil Kang, Jinyeong Kim, Junhyeok Kim, and Seong Jae Hwang show that big vision-language models can find objects in images with almost no extra training, in a 2025 paper. The task, called visual grounding, asks a model to locate the part of an image that a text phrase describes. Most past methods needed fine-tuning or added parts that draw boxes.
The team found that a few of the model's attention heads already focus on the right spots for a phrase. They call these localization heads and use their attention maps to pick out the target object. Just three such heads match the scores of tuned systems. The result suggests these models grasp the link between words and image regions on their own.
Catalog record
| CREATOR | Seil Kang, Jinyeong Kim, Junhyeok Kim, Seong Jae Hwang |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 57 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-06287
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Your Large Vision-Language Model Only Needs A Few Attention Heads For Visual Grounding Seil Kang, Jinyeong Kim, Junhyeok Kim, Seong Jae Hwang Abstract Visual grounding seeks to localize the image region corresponding to a free-form text description. Recently, the strong multimodal capabilities of Large Vision-Language Models (LVLMs) have driven substantial improvements in visual grounding, though they inevitably require fine-tuning and additional model components to explicitly generate bounding boxes or segmentation masks. However, we discover that a few attention heads in frozen LVLMs demonstrate strong visual grounding capabilities. We refer to these heads, which consistently capture object locations related to text semantics, as localization heads. Using localization heads, we introduce a straightforward and effective training-free visual grounding framework that utilizes text-to-image attention maps from localization heads to identify the target objects. Surprisingly, only three out of thousands of attention heads are sufficient to achieve competitive localization performance compared to existing LVLM-based visual grounding methods that require fine-tuning. Our findings suggest that LVLMs can innately ground objects based on a deep comprehension of the text-image relationship, as they implicitly focus on relevant image regions to generate informative text outputs. All the source codes will be made available to the public. 1 Introduction Visual grounding is a task that, given textual descriptions, identifies and localizes relevant objects within an image, producing outputs such as bounding boxes [ 39 , 70 ] or segmentation masks [ 18 ] . Recently, this vision-language task, which inherently requires a deep understanding of the relationship between images and text, has seen significant advancements with the emergence of powerful Large Vision-Language Models (LVLMs) [ 36 , 35 , 28 , 56 ] . However, since LVLMs are primarily designed to generate text outputs, directly leveraging them as a vision-language tool to identify and localize objects within an image ( i.e . , visual grounding) presents technical challenges. Inevitably, current LVLM-based visual grounding methods require explicit fine-tuning of LVLMs with additional visual grounding datasets and modifications to model components to enable the generation of bounding boxes [ 6 , 68 , 61 ] or segmentation masks [ 26 , 45 , 74 , 65 ] . Figure 1 : Visualization of the text-to-image attention maps from LLaVA-1.5-7B [ 35 ] . While the average attention map initially seems uninformative for localization, a closer examination reveals that LVLM possesses built-in localization heads that consistently capture key areas of an image corresponding to the referring text, regardless of sample variations. L14 H24 refers to the 24th attention head in the 14th layer of the LVLM. Despite the interesting integration of LVLMs in previous visual grounding works, a fundamental question remains: since LVLMs generate text outputs that imply an understanding of specific image regions, is it possible to explicitly observe this mechanism in action? In other words, we ask whether we can extract how the LVLMs “focus” on specific image regions corresponding to given text descriptions for visual grounding. A natural first approach to addressing this question might be to examine the text-to-image attention maps, which reveal how a text description attends to different image patches. To explore this, we visualize the average attention maps of LVLMs across various layers and heads—a common method in ViTs [ 73 , 11 ] and diffusion models (DMs) [ 54 , 17 , 4 ] —anticipating that they would capture the regions associated with the referring text. However, unlike the interpretable attention patterns observed in ViTs and DMs, the text-to-image attention maps in LVLMs appear sparse and contain significant noise, as illustrated in the second column of Fig. 1 . This suggests that the current use of LVLM attention maps may struggle to accurately pinpoint relevant objects for visual grounding. However, interestingly, our work reveals that not the average of the attention maps, but some small subset of attention heads are capable of providing tangible and precise text-image attention maps. In particular, we find that a few attention heads in LVLMs consistently capture regions in images corresponding to the referred text, regardless of the samples. We refer to these heads as localization heads . For example, as presented in the third and fourth columns of Fig. 1 , the attention maps of the 24th head of the 14th layer (L14 H24) and the 13th head of the 14th layer (L14 H13) in LLaVA-1.5-7B [ 35 ] consistently highlight the regions of interest based on the referred text. In this work, we introduce how we systematically identify such localization heads based on two explicit criteria. (1) We measure how much each attention head focuses on the image by calculating the attention sum and only select the heads that dominantly attend to the image. (2) Among these heads, the ones that specifically pay attention to a certain region of the image, which is measured by spatial entropy [ 2 ] , are considered to effectively localize the referred object. We validate that the selected localization heads consistently capture objects closely associated with the text. With our localization heads, we introduce a simple yet effective training-free visual grounding framework. The attention maps from the localization heads are assembled to predict the bounding box or mask of the referred object. Notably, only three localization heads are enough to localize the referred object within the image, suggesting that they are highly specialized to attend to relevant image regions. As shown in Fig. 2 , in contrast to existing fine-tuning based methods, our framework is training-free, eliminating the need for additional fine-tuning LVLMs for visual grounding tasks. Figure 2 : Comparison of LVLM frameworks for visual grounding. (a) Existing methods generally fine-tune a LVLM to leverage specialized tokens ( e.g . , \mathtt{[SEG]} ) or language descriptions for visual grounding. (b) Our framework utilizes the attention maps of only a few localization heads from frozen LVLMs. We validate our approach across ten different LVLMs with varying parameter counts, architectures, and training datasets, demonstrating its broad applicability. Our framework outperforms the existing training-free methods by significant margins. Furthermore, our method performs comparably to specially fine-tuned LVLMs for visual grounding tasks ( e.g . , LISA [ 26 ] ). The results indicate that LVLMs can serve as effective text-referring localizers, intrinsically identifying regions that are relevant and coherent with the text expression. To the best of our knowledge, we are the first to identify the localization properties of specific attention heads in LVLMs. In summary, our contributions are as follows: • We discover that the specific attention heads in LVLMs have the capability for visual grounding, which we refer to as localization heads . • We propose a simple yet effective framework for LVLM-based training-free visual grounding with localization heads. The attention maps from a few localization heads are utilized to predict the bounding box or mask of the referred object. • We evaluate our approach across various LVLMs. Our framework demonstrates superior performance by a large margin compared to other training-free methods and even performs comparably to fine-tuned methods. 2 Related Works Visual Grounding. Visual grounding aims to identify the region in the image based on a free-form natural language expression [ 5 ] , which has expanded the scope of detection and segmentation tasks to a more realistic scenario [