Learning to See What You Need: Gaze Attention for Multimodal Large Language Models

Junha Song, Byeongho Heo, Geonmo Gu, Jaegul Choo, Dongyoon Han, Sangdoo Yun · 2026

Read Learning to See What You Need: Gaze Attention for Multimodal Large Language Models by Junha Song, Byeongho Heo, Geonmo Gu, Jaegul Choo, Dongyoon Han, Sangdoo Yun (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-13080.txt.

Introduction

Song and five co-authors introduce Gaze Attention, which lets multimodal language models look where it matters instead of attending to every visual token. Visual tokens are grouped into spatial regions and only regions relevant to the current prediction are selected, cutting attention computation. Learnable context tokens summarise images or video frames, preserving global context under selective attention. On 13 image and 6 video benchmarks it matches or beats dense-attention baselines while using up to 90% fewer visual KV entries.

Catalog record

CREATORJunha Song, Byeongho Heo, Geonmo Gu, Jaegul Choo, Dongyoon Han, Sangdoo Yun
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE39939
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.13080

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Learning to See What You Need: Gaze Attention for Multimodal Large Language Models Junha Song, Byeongho Heo, Geonmo Gu, Jaegul Choo, Dongyoon Han, Sangdoo Yun Abstract When humans describe a visual scene, they do not process the entire image uniformly; instead, they selectively fixate on regions relevant to their intended description. In contrast, current multimodal large language models (MLLMs) attend to all visual tokens at each generation step, leading to diluted focus and unnecessary computational overhead. In this work, we introduce Gaze Attention, a novel mechanism that enables MLLMs to selectively attend to task-relevant visual regions during generation. Specifically, we spatially group visual embeddings—stored as key-value caches—into compact gaze regions, each represented by a lightweight descriptor. At each decoding step, the model dynamically selects the most relevant regions and restricts attention to them, reducing redundant computation while enhancing focus. To mitigate the loss of global context caused by localized attention, we further propose learnable context tokens appended to each image or frame, allowing the model to maintain holistic visual awareness. Extensive experiments on image and video understanding benchmarks demonstrate that Gaze Attention matches or surpasses dense-attention baselines, while using up to 90% fewer visual KV entries in the attention computation. † † affiliation: KAIST † † affiliation: NAVER AI Lab † † contribution: Corresponding authors † † date: August 10, 2026 1 Introduction MLLMs encode images or videos into sequences of visual embeddings, which are concatenated with textual inputs and processed by an LLM ( 39 ; 30 ; 63 ; 9 ) . During generation, these models attend to all visual tokens, including regions that may be irrelevant to the current prediction. In contrast, human visual perception is inherently selective: rather than processing an entire scene, humans dynamically shift their gaze to fixate on regions relevant to their intended description ( 75 ; 21 ) . This discrepancy suggests that current MLLMs lack selective visual attention, motivating the development of such mechanisms. Existing efforts have explored reducing the visual information attended to by the LLM, primarily for efficiency rather than selective gaze. They are illustrated in figure 1 . Some methods compress visual tokens before they are passed to the LLM ( 64 ; 33 ; 35 ) , while others evict visual KV cache entries using importance scores computed from the question ( 15 ; 62 ; 80 ; 74 ; 7 ) . Both directions determine the usable visual information in advance, before the model knows which region will matter at each generation step. As a result, they can either discard fine-grained details or fail to adapt when generation requires information from multiple distinct regions. Neither direction enables the model to select visual regions dynamically at each generation step. In this work, we propose Gaze Attention, a mechanism that allows the model to focus on the visual regions needed for each generation step. Specifically, the cached key vectors of visual tokens are spatially grouped into gaze regions, each represented by a lightweight descriptor. At each decoding step, the model routes attention to a small subset of regions based on these descriptors and attends exclusively to them. This enables the model to perform more focused attention on task-relevant regions, improving generation quality. At the same time, because only the selected regions participate in attention computation, cost scales with the attended fraction rather than the full visual token count. Figure 1: Overview of visual attention approaches. We aim to enable MLLMs to selectively fixate on task-relevant regions during generation, rather than attending to all visual tokens. Several methods pursue this direction through token compression ( 33 ; 35 ) or KV-cache eviction ( 80 ; 68 ; 15 ) , but rely on a one-time relevance estimate from the question. In contrast, gaze attention dynamically selects query-relevant regions at each generation step. Selective attention alone, however, can limit access to global scene information. To address this, we introduce learnable context tokens that capture a compact summary of the visual content. These tokens are appended to each image or video frame and are processed within the LLM’s native attention layers. During generation, every text query has access to the KV caches of these tokens, allowing the model to retain global visual awareness while attending selectively to local regions, much as humans take in a scene as a whole before focusing on specific details. We evaluate Gaze Attention on both image and video understanding benchmarks. While attending to up to 90% fewer visual KV entries, it matches or outperforms dense-attention baselines across 13 image and 6 video understanding benchmarks. Visualization analyses further confirm that Gaze Attention yields concentrated, interpretable attention maps at each generation step, in contrast to the diffuse patterns of dense attention. These results suggest that enabling MLLMs to selectively attend to relevant visual regions is a promising alternative to dense attention. 2 Related Works Multimodal Large Language Models. Recent MLLMs adopt a similar pipeline that encodes images or videos into visual features, and processes them with an LLM backbone ( 38 ; 31 ; 58 ; 4 ; 12 ) . As images and video frames are information-dense, they typically yield far more visual tokens than text. This issue becomes more pronounced for high-resolution images ( 30 ; 70 ; 8 ) and videos ( 73 ; 72 ; 68 ) , which require an enormous number of tokens to faithfully represent their content. Despite the growing demand on LLM backbones to process large amounts of visual information, there remains limited exploration into how such information should be handled effectively within the LLM. VideoLLM with Token Reductions. To reduce the cost of long visual contexts, prior work has explored compressing visual tokens before they are processed by the LLM. Several strategies include token resampling ( 79 ; 57 ; 34 ; 64 ) and pooling ( 48 ; 44 ; 35 ) , which reduce token count by compressing or aggregating visual information. Other works adopt importance-based pruning to remove less relevant tokens ( 5 ; 33 ) . While these methods improve efficiency, they often blur or discard fine-grained details that may be needed later during generation. VideoLLM with KV-Cache Retrievals. Since attention is the dominant computational cost in LLMs, several works aim to improve efficiency, particularly in streaming scenarios, by reducing the amount of visual KV cache ( 15 ) . Some works focus on removing low-importance entries ( 62 ; 80 ; 52 ) , while others consolidate redundant entries within a fixed budget ( 15 ; 28 ; 74 ; 7 ) . In most cases, importance scores between question and visual tokens are computed once at the first generation step, and only high-scoring entries are memorized. This limits adaptivity during decoding, especially when later queries require different visual details. Our method instead routes each query to relevant visual chunks, maintaining dynamic access throughout generation. Visual Saliency & Gaze in Vision Models. Computational modeling of visual attention has a long history in computer vision. Classical saliency models ( 23 ) predict where humans fixate using low-level features, and deep learning approaches such as DeepGaze II ( 29 ) and its successors ( 37 ) substantially improved fixation prediction by leveraging features from recognition networks.

Read the complete text