Large Vision–Language Models Get Lost in Attention
Gongli Xi, Ye Tian, Mengyu Yang, Huahui Yi, Liang Lin, Xiaoshuai Hao, Kun Wang, Wendong Wang · 2026
Read Large Vision–Language Models Get Lost in Attention by Gongli Xi, Ye Tian, Mengyu Yang, Huahui Yi, Liang Lin, Xiaoshuai Hao, Kun Wang, Wendong Wang (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-05668.txt.
Introduction
Vision-language models are residual transformers at heart, whatever their training recipe. Gongli Xi and seven co-authors ask what the attention and feed-forward parts each actually do inside. Earlier attribution studies lacked a shared theoretical basis, so the team builds one from information theory and geometry to measure residual updates.
The framework reveals a clean division of labor. Attention acts as a subspace-preserving operator, moving information without rotating its geometry, while other components reshape representations. The authors argue these models can get lost in attention when that preservation goes wrong, and the diagnostics point to architectural fixes.
Catalog record
| CREATOR | Gongli Xi, Ye Tian, Mengyu Yang, Huahui Yi, Liang Lin, Xiaoshuai Hao, Kun Wang, Wendong Wang |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 76 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.05668
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Large Vision–Language Models Get Lost in Attention Gongli Xi, Ye Tian, Mengyu Yang, Huahui Yi, Liang Lin, Xiaoshuai Hao, Kun Wang, Wendong Wang Abstract Despite the rapid evolution of training paradigms, the decoder backbone of large vision–language models (LVLMs) remains fundamentally rooted in the residual-connection Transformer architecture. Therefore, deciphering the distinct roles of internal modules is critical for understanding model mechanics and guiding architectural optimization. While prior statistical approaches have provided valuable attribution-based insights, they often lack a unified theoretical basis. To bridge this gap, we propose a unified framework grounded in information theory and geometry to quantify the geometric and entropic nature of residual updates. Applying this unified framework reveals a fundamental functional decoupling: Attention acts as a subspace-preserving operator focused on reconfiguration, whereas FFNs serve as subspace-expanding operators driving semantic innovation. Strikingly, further experiments demonstrate that replacing learned attention weights with predefined values (e.g., Gaussian noise) yields comparable or even superior performance across a majority of datasets relative to vanilla models. These results expose severe misallocation and redundancy in current mechanisms, suggesting that state-of-the-art LVLMs effectively “get lost in attention” rather than efficiently leveraging visual context. Our code is publicly available at this link . 1 Introduction Large vision–language models (LVLMs) have rapidly evolved from large language models (LLMs) by extending Transformer-based sequence modeling to jointly process natural language and visual signals ( 89 ) . Early vision–language representation learning (e.g., contrastive pretraining) established strong image–text alignment that later LVLMs could leverage as a visual grounding interface ( 77 ) . Subsequent LVLMs increasingly unify pretrained vision encoders with LLM backbones, enabling few-shot multimodal generalization and instruction-following behavior at scale ( 5 ; 50 ; 57 ; 34 ) . In parallel, reasoning-oriented paradigms have further endowed these models with improved deliberation and problem-solving behaviors ( 93 ; 39 ; 32 ; 104 ; 85 ) . Despite the fast pace of architectural and training innovations, the dominant LVLM family remains fundamentally grounded in the Transformer architecture ( 89 ) . From an interpretability standpoint, the standard Transformer layer is composed of two core submodules, namely multi-head self-attention and feed-forward network (FFN), and each submodule is wrapped by residual connections, so that every submodule produces an additive update that is written back into a shared residual stream representation ( 89 ; 23 ; 83 ) . A common working hypothesis is that attention blocks are the primary substrate for in-context reasoning , implementing context-dependent algorithms such as induction/copy-based mechanisms ( 74 ) . In contrast, FFNs are often characterized as storing and retrieving distributional associations , behaving like key–value memories whose activated patterns can induce next-token distributions that resemble shallow n-gram continuations ( 27 ; 22 ) . To probe this modularity hypothesis , attention interpretability work has largely taken a statistical perspective that treats attention related signals as measurable proxies and attributes function via empirical distributions ( 106 ; 41 ) , correlations ( 40 ; 1 ) , and controlled interventions ( 81 ; 72 ) . More recently, this statistical toolkit has been extended to visual attention in LVLM decoders, where attention links text to visual tokens. Empirical analyses reveal systematic phenomena such as visual attention sink ( 42 ) and visual attention drift ( 54 ; 31 ) , which together indicate that models often under allocate attention to truly informative visual evidence. Given these advances, LVLM module-level interpretability still lacks a unifying information theoretic and geometric framework that can characterize, and explicitly contrast, how different submodules contribute to representation structure in multimodal settings. In contrast, the representation analysis literature for LLMs already uses such lenses to evaluate representation quality across depth ( 78 ; 94 ) and to study joint dynamics ( 83 ; 87 ) . This gap motivates bringing these principled lenses into LVLM analysis to address the missing perspective and enable module specific and modality grounded comparisons. To bridge this theoretical gap, we present a unified framework grounded in information theory and differential geometry to quantify and contrast module-level functional contributions in LVLM residual-stream computation. By adopting the manifold hypothesis ( 13 ) for representation space, we introduce two complementary metrics: Representation Information Discrepancy (RID) and Mixing Information Gain (MixIG) . These metrics decompose the contribution of residual updates into two distinct geometric effects: innovation , which quantifies external information injection that expands the semantic subspace or alters spectral complexity, and reconfiguration , which measures the entropic redistribution of information within the existing support. We conduct extensive experiments across 15 state-of-the-art LVLMs spanning three dominant architectures on a broad suite of multimodal benchmarks. Our analysis reveals two profound insights: first, we quantitatively validate a sharp functional decoupling in Transformer residual stream computation: attention primarily performs entropic reconfiguration that preserves the existing representation support, whereas FFNs dominate innovation by introducing new semantic directions. Building on this division of labor, we further diagnose a systemic pathology in current LVLMs: decoder visual attention often fails to perform meaningful mixing over question-relevant visual evidence, and instead exhibits substantial redundancy, frequently getting lost in interaction patterns with limited contribution to informative updates. Our main contributions are summarized as follows: • Theoretical Framework: We propose a rigorous formalism based on the manifold hypothesis to define representational information. We introduce RID and MixIG as dual metrics to quantify the geometric and entropic impact of residual updates, offering a generalized tool for probing representation dynamics. • Module-level Interpretability: We provide a quantitative explanation of the distinct roles within Transformer blocks. We demonstrate that Attention and FFNs operate in orthogonal regimes— reconfiguration versus innovation —thereby substantiating the modularity hypothesis with geometric evidence. • Empirical Diagnostics: We uncover critical inefficiencies in LVLM designs. Our results highlight that despite architectural scaling, current models suffer from severe informational redundancy in visual processing, suggesting that the integration of visual tokens is often computationally expensive yet informationally sparse. 2 Related work Interpretability of LLMs. A large body of work studies what information is encoded in LLM representations and where it appears in the network ( 11 ) . Early work uses lightweight linear probes on intermediate hidden states ( 16 ; 37 ; 12 ) . Subsequent decoding based efforts, such as the tuned lens, map hidden states to vocabulary distributions ( 12 ) . Alongside probing and decoding, sparse feature learning approaches, including transcoders ( 20 ) and sparse autoencoders ( 17 ) , map representations into a sparse and more discrete feature space ( 7 ) . Figure 1 : Overview of Our Interpretability Framework: (a) the LVLM residual stream; (b) representation information in \mathbf{X} , where SVD yields Spectrum \mathcal{S}_{\mathbf{X}} and semantic support \mathcal{D}_{\mathbf{X}} ; (c) update-level effects of \Delta\mathbf{X} , quantified by RID for innovation and MixIG for reconfiguration; (d) layer-wise functional decomposition, revealing an orthogonal division of labor where attention behaves as a subspace-preserving operator and FFNs act as subspace-expanding operators. Module Interpretability. Module interpretability asks whether internal Transformer components provide meaningful explanations of model behavior. For attention, foundational studies show that raw attention weights can be an unreliable attribution signal ( 40 ; 81 ; 96 ) . To better capture how attention-mediated influence accumulates, attention rollout and attention flow estimate propagation across layers ( 1 ; 44