Cross-Layer Vision Smoothing: Enhancing Visual Understanding via Sustained Focus on Key Objects in Large Vision-Language Models
Jianfei Zhao, Feng Zhang, Xin Sun, Lingxing Kong, Zhixing Tan, Chong Feng · 2025
Read Cross-Layer Vision Smoothing: Enhancing Visual Understanding via Sustained Focus on Key Objects in Large Vision-Language Models by Jianfei Zhao, Feng Zhang, Xin Sun, Lingxing Kong, Zhixing Tan, Chong Feng (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-12897.txt.
Introduction
Jianfei Zhao, Feng Zhang and three co-authors add Cross-Layer Vision Smoothing in 2025 to help large vision-language models keep focus on key objects. Such models can find the right object, but their focus fades fast in deeper layers.
Their fix keeps a vision memory that smooths focus across layers. It starts from an even first-layer focus, then each layer blends past memory with new focus while the memory updates. A doubt signal stops the smoothing once seeing is done, since most seeing happens early. Tests on four benchmarks across three models show top overall scores and strong caption results.
Catalog record
| CREATOR | Jianfei Zhao, Feng Zhang, Xin Sun, Lingxing Kong, Zhixing Tan, Chong Feng |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 36 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2509-12897
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Cross-Layer Vision Smoothing: Enhancing Visual Understanding via Sustained Focus on Key Objects in Large Vision-Language Models Jianfei Zhao, Feng Zhang, Xin Sun, Lingxing Kong, Zhixing Tan, Chong Feng Abstract Large Vision-Language Models (LVLMs) can accurately locate key objects in images, yet their attention to these objects tends to be very brief. Motivated by the hypothesis that sustained focus on key objects can improve LVLMs’ visual capabilities, we propose C ross- L ayer V ision S moothing (CLVS). The core idea of CLVS is to incorporate a vision memory that smooths the attention distribution across layers. Specifically, we initialize this vision memory with position-unbiased visual attention in the first layer. In subsequent layers, the model’s visual attention jointly considers the vision memory from previous layers, while the memory is updated iteratively, thereby maintaining smooth attention on key objects. Given that visual understanding primarily occurs in the early and middle layers of the model, we use uncertainty as an indicator of completed visual understanding and terminate the smoothing process accordingly. Experiments on four benchmarks across three LVLMs confirm the effectiveness and generalizability of our method. CLVS achieves state-of-the-art performance on a variety of visual understanding tasks, with particularly significant improvements in relation and attribute understanding. 1 1 1 Our code is available at https://github.com/beta-nlp/CLVS . 1 Introduction Large Vision-Language Models (LVLMs) ( Liu et al. 2023 ; Bai et al. 2025 ) acquire the capability to comprehend visual information through training on large-scale multimodal data and exhibit strong performance across a wide range of visual understanding tasks ( Li et al. 2023 ; Wang et al. 2023 ; Fu et al. 2024 ) . Despite these successes, LVLMs still suffer from issues such as hallucinations ( Leng et al. 2024 ) , which limit their real-world applications. Recent studies ( Zhang et al. 2025 ; Chen et al. 2025 ) have revealed a strong correlation between the visual understanding capability of LVLMs and their visual attention. Visual attention is learned through end-to-end training without direct supervision. While this approach simplifies LVLM training, it also makes it difficult to avoid biases in visual attention ( Liu, Zheng, and Chen 2024 ; Kang et al. 2025 ; Zhu et al. 2025b ) , such as position bias ( Zhu et al. 2025b ; Li et al. 2025 ) , visual attention sink ( Huang et al. 2024 ; Kang et al. 2025 ) , and uncoordinated attention heads ( He et al. 2025 ; Yang et al. 2025 ) . Consequently, a series of recent works have sought to enhance the visual understanding capabilities of LVLMs by improving visual attention. Liu, Zheng, and Chen 2024 ; Zhu et al. 2025a ; Yin, Si, and Wang 2025a propose directly increasing the attention weights on visual tokens. Yang et al. 2025 ; He et al. 2025 analyze the effect of the multi-head attention mechanism on visual understanding and remove attention heads that negatively impact performance. Kang et al. 2025 ; Xie et al. 2025 ; Chen et al. 2025 adjust the visual attention distribution to achieve finer control over the model’s visual understanding process. These successes confirm the crucial role of visual attention in the visual understanding of LVLMs. Figure 1: A running example of visual attention in LLaVA-1.5-7B. In vanilla LVLMs, first-layer attention is biased toward the bottom, and focus on the key object (woman) decays after layer 15. Figure 2: Comparison of visual attention to key objects across layers between LVLMs with and without CLVS. While previous studies have demonstrated effectiveness in improving the visual attention of LVLMs, they primarily focus on enhancing attention within individual layers without considering the cross-layer evolution of visual attention. We observe that visual attention can change dramatically between consecutive layers. For example, in Figure 1 , when asked “Is the woman wearing a black shirt?”, the model attends to the “woman” and “shirt” ( i.e. , key objects) at layer 14 but ceases to focus on them afterward. Although the model accurately locates key objects, its attention to these objects is very brief. This phenomenon, commonly observed across different LVLMs and images, is referred to as advantageous attention decay . We hypothesize that such brief attention is insufficient for effective visual information fusion in LVLMs. Therefore, sustaining an appropriate focus on key objects is beneficial to enhance the model’s visual capabilities. To tackle advantageous attention decay, we propose Cross-Layer Vision Smoothing (CLVS) to enhance LVLMs’ visual comprehension by maintaining sustained focus on key objects in the image. To this end, we introduce a vision memory for smoothing visual attention across layers. Specifically, following Li et al. 2025 , we unify the visual attention in the first layer for unbiased perception and initialize the vision memory with the visual attention distribution. In subsequent layers, we progressively smooth visual attention by interpolating it with the vision memory and updating the memory iteratively. Moreover, since visual understanding primarily occurs in the early and middle layers of LVLMs, we propose an uncertainty-based criterion to determine whether the visual understanding process has been completed and, if so, terminate visual attention smoothing. With CLVS, the model can sustain focus on key objects, thereby improving the visual capabilities of LVLMs. The comparison of visual attention to key objects across layers between LVLMs with and without CLVS is shown in Figure 2 . We conduct extensive experiments on four visual understanding benchmarks and three LVLMs. The results demonstrate that CLVS effectively improves the visual capabilities of LVLMs, reducing hallucinations, particularly those related to object attributes and relations. In summary, our contributions are: • We analyze the cross-layer evolution of visual attention and identify advantageous attention decay, a phenomenon where the model’s focus on key objects is short-lived. • We propose Cross-Layer Vision Smoothing (CLVS), which sustains the model’s focus on key objects to enhance visual understanding. • We conduct extensive experiments and in-depth analyses to show that CLVS effectively alleviates advantageous attention decay and reduces hallucinations, particularly those related to object attributes and relations. 2 Related Work The cross-modal understanding ability of LVLMs is built upon the attention mechanism. However, end-to-end training lacks supervision over the model’s internal processes, leading to shortcomings in the spontaneously formed visual attention, such as position bias and visual attention sink. These issues can result in hallucinations ( Liu et al. 2024 ; Zhao et al. 2025b ) in the model’s output. Numerous works have explored improving visual attention in LVLMs. Yang et al. 2025 ; Zhu et al. 2025b incorporate supervisory signals into the attention mechanism, enabling fine-grained control during training. To avoid high training costs and improve generalizability, an increasing number of studies focus on training-free approaches to optimize visual attention. Given the critical role of visual information in LVLMs, Liu, Zheng, and Chen 2024 ; Zhu et al. 2025a ; Yin, Si, and Wang 2025a directly increase the model’s attention weights on visual tokens. While effective, these approaches overlook the internal dynamics of visual attention. Leveraging the diversity of multi-head attention, Zhang et al. 2024 ; Zhao et al. 2025a ; Yang et al. 2025 identify and utilize high-quality attention heads to optimize visual attention. Recent studies ( Kang et al. 2025 ; Xie et al. 2025 )