Predictive Regularization Against Visual Representation Degradation in Multimodal Large Language Models
Enguang Wang, Qiang Wang, Yuanchen Wu, Ke Yan, Xinbin Yuan, Shouhong Ding, Xialei Liu, Ming-Ming Cheng · 2026
Read Predictive Regularization Against Visual Representation Degradation in Multimodal Large Language Models by Enguang Wang, Qiang Wang, Yuanchen Wu, Ke Yan, Xinbin Yuan, Shouhong Ding, Xialei Liu, Ming-Ming Cheng (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-20808.txt.
Introduction
Wang and co-authors find that multimodal language models lose visual skill in their middle layers. Language driven training, they argue, trades visual fidelity for text output. Compared with the input features, mid layer features lose global and patch level structure. Their fix is predictive regularization, called PRe. It forces mid layer features to predict the initial visual features. This keeps core visual traits alive inside the model. Broad tests showed vision language scores rose when this decay was curbed.
Catalog record
| CREATOR | Enguang Wang, Qiang Wang, Yuanchen Wu, Ke Yan, Xinbin Yuan, Shouhong Ding, Xialei Liu, Ming-Ming Cheng |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 61014 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.20808
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Predictive Regularization Against Visual Representation Degradation in Multimodal Large Language Models Enguang Wang, Qiang Wang, Yuanchen Wu, Ke Yan, Xinbin Yuan, Shouhong Ding, Xialei Liu, Ming-Ming Cheng Abstract While Multimodal Large Language Models (MLLMs) excel at vision-language tasks, the cost of their language-driven training on internal visual foundational competence remains unclear. In this paper, we conduct a detailed diagnostic analysis to unveil a pervasive issue: visual representation degradation in MLLMs. Specifically, we find that compared to the initial visual features, the visual representation in the middle layers of LLM exhibits both a degradation in global function and patch structure. We attribute this phenomenon to a visual sacrifice driven by the singular text-generation objective, where the model compromises its visual fidelity to optimize for answer generation. We argue that a robust MLLM requires both strong cross-modal reasoning and core visual competence, and propose Predictive Regularization (PRe) to force degraded intermediate features to predict initial visual features, thereby maintaining the inherent visual attributes of the MLLM’s internal representations. Extensive experiments confirm that mitigating this visual degradation effectively boosts vision-language performance, underscoring the critical importance of fostering robust internal visual representations within MLLMs for comprehensive multimodal understanding. 1 Introduction Recent years have witnessed a monumental leap in artificial intelligence, a paradigm shift largely driven by the unprecedented success of Large Language Models (LLMs) Devlin et al. 2019 ; Radford et al. 2018 ; Radford et al. 2019 ; Touvron et al. 2023 ; Bai et al. 2023a ; Team 2023 ; Liu et al. 2024a . These models, typically built upon the Transformer architecture Vaswani et al. 2017 , are pre-trained on vast text corpora to acquire a remarkable ability to comprehend, reason, and generate human-like text. Building upon this powerful linguistic foundation, researchers have begun integrating visual information into LLMs, bridging the semantic gap between vision and language to create Multimodal Large Language Models (MLLMs) Liu et al. 2024b ; Liu et al. 2023a ; Chen et al. 2024c ; Wang et al. 2025c ; Lu et al. 2024 ; Bai et al. 2023b ; Bai et al. 2025 . By conditioning the language model’s generative process on visual inputs, MLLMs are capable of performing a diverse array of vision-language tasks Hudson and Manning 2019 ; Li et al. 2023 ; Guan et al. 2024 ; Yue et al. 2024 ; Tong et al. 2024 ; Hiippala et al. 2021 ; xAI 2024 ; Singh et al. 2019 ; Ning et al. 2025 ; A et al. 2026 , establishing them as a critical step towards AGI. Figure 1 : The Proposed PRe Framework. PRe enforces the degraded representations to predict their initial, clean anchor representations via a lightweight prediction head. The dominant architectural paradigm for these MLLMs involves a straightforward yet effective design: a powerful, pre-trained vision encoder Jiang et al. 2023 ; Tschannen et al. 2025 is aligned with an LLM via a lightweight projector that maps visual features into the language embedding space. The entire system is then fine-tuned almost exclusively through a language-centric objective, i.e . , next-token prediction on text data. While this approach has proven successful in answering visual questions, it establishes a fundamental asymmetry in the optimization process. The visual representations are not optimized for their own fidelity; rather, they are implicitly and progressively transformed with the singular goal of better serving the final text generation task. Several prior works Chen et al. 2024b ; Zhang et al. 2025 ; Jiang et al. 2025 ; Kang et al. 2025 ; Neo et al. 2024 ; Wang et al. 2025b ; Kaduri et al. 2025 have provided valuable insights into this internal process, often utilizing attention analysis or probing techniques to trace how visual information flows and contributes to the model’s textual output. However, these existing analyses primarily focus on the cross-modal functionality of visual features, i.e . , how well they serve the language task. Their inherent capacity to support fundamental visual tasks independent of language remains largely unexplored. This raises a critical, yet largely overlooked question: What is the cost of this purely language-driven training to the visual representations themselves? Answering this question is critical as robust and reliable MLLMs should be both eloquent communicators (good at answering visual questions) and sharp-eyed observers (have a robust visual foundation). To this end, we conduct a comprehensive diagnostic analysis to systematically investigate this cost. Our investigation begins by directly measuring the functional validity of visual representations at each layer of various MLLMs on standard visual classification tasks. As shown in Fig. 2 , our analysis uncovers a striking and robust pattern of global functional degradation . Compared to the initial visual features fed into the LLM, the representations in the intermediate layers exhibit a significant degradation in their classification capability. Recent studies Jiang et al. 2025 ; Zhang et al. 2025 have highlighted the critical importance of intermediate-layer visual representations in MLLMs, suggesting they encode rich, abstract semantics crucial for the model’s answering. This raises a problem, how can these layers be simultaneously so crucial for final answers, yet so functionally degraded for visual tasks? To unravel this paradox, we shift our analysis to the microscopic patch level. As shown in Fig. 3 and Fig. 4 , compared to the initial visual features, the representations processed by the LLM exhibit a pronounced blurring of semantic boundaries between different objects. For example, the semantic contrast ratio consistently declines through the intermediate layers, and visualizations confirm that similarity from a single patch spills over to unrelated objects. This patch structure degradation is the direct microscopic cause of the macroscopic functional degradation. Considering that the model progressively aligns visual features with text, we argue that the visual degradation is sacrificed for achieving advanced language capabilities . The model fuses local semantics to build a globally coherent, abstract representation suitable for complex linguistic description, thereby sacrificing the fine-grained, visually-discriminative structure of the initial representation to a certain extent. The statistical properties Fig. 5 and the dynamics of the model’s visual and linguistic capabilities during pre-training Fig. 6 further substantiate this hypothesis. Considering that visual ability is the cornerstone of MLLMs, we argue that a robust MLLM should achieve high-level semantic abstraction while simultaneously preserving core visual fidelity. Inspired by the principles of predictive coding Rao and Ballard 1999 ; Friston 2005 ; Assran et al. 2023 , which posit that an efficient neural system constantly predicts its own lower-level signals to maintain a coherent world model, we propose PRe (Predictive Regularization) to regularize the degradation. As shown in Fig. 1 , PRe compels the degraded visual representations of the intermediate layers to predict the initial anchor features. This predictive objective, optimized concurrently with the standard text generation loss, encourages the model to enhance its linguistic capabilities without sacrificing its foundational visual fidelity. Extensive experiments across vision-language tasks confirm that mitigating this visual degradation effectively boosts performance, underscoring the critical importance of fostering robust internal visual representations within MLLMs for comprehensive multimodal understanding. Our main contributions are threefold: • We are the first to systematically diagnose the visual degradation phenomenon, providing a multi-level explanation for this degradation, linking the global functional degradation to a microscopic patch degradation. • We identify the