Mitigating Attention Localization in Small Scale: Self-Attention Refinement via One-step Belief Propagation
Nakyung Lee, Yeongoon Kim, Minhae Oh, Suhwan Kim, Jin Woo Koo, Hyewon Jo, Jungwoo Lee · 2025
Read Mitigating Attention Localization in Small Scale: Self-Attention Refinement via One-step Belief Propagation by Nakyung Lee, Yeongoon Kim, Minhae Oh, Suhwan Kim, Jin Woo Koo, Hyewon Jo, Jungwoo Lee (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-07324.txt.
Introduction
Transformer models sometimes focus their attention on too few words, missing links across a long text. Nakyung Lee and 6 co-authors propose a fix called SAOBP in a 2025 research paper. Their method runs one step of belief propagation, a technique that passes messages between connected items, through the attention graph. The goal is to spread multi-hop relationships into each attention decision.
The team also defines Global Token Dependency, a measure of how much multi-hop links contribute inside attention. Experiments show the method prevents entropy collapse, where attention becomes too narrow in deep layers. It keeps the dependency measure near task-appropriate levels. Gains were clearest in small models. That makes the approach promising for capable language models that must run on limited hardware.
Catalog record
| CREATOR | Nakyung Lee, Yeongoon Kim, Minhae Oh, Suhwan Kim, Jin Woo Koo, Hyewon Jo, Jungwoo Lee |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 44 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2509-07324
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Mitigating Attention Localization in Small Scale: Self-Attention Refinement via One-step Belief Propagation Nakyung Lee, Yeongoon Kim, Minhae Oh, Suhwan Kim, Jin Woo Koo, Hyewon Jo, Jungwoo Lee Abstract Transformer-based self-attention mechanism serves as the core of modern language models, yet it often suffers from localization , where attentions collapse onto a limited subset of tokens and fail to capture long-range dependencies. To address this issue, we propose Self-Attention One-step Belief Propagation (SAOBP) , a refinement framework that injects multi-hop relationships through a belief propagation process 1 1 1 The code is released at nakyungLee20/SAOBP . To interpret and quantify these interactions, we introduce Global Token Dependency (GTD) that captures the relative contribution of multi-hop connections within the attention graph. Empirical results indicate that SAOBP helps prevent entropy collapse in deeper layers and adaptively maintains GTD at task-appropriate levels, thereby supporting improvements in model performance. Importantly, we observe competitive gains in small-scale models, highlighting its potential for improving inference quality in resource-constrained scenarios. 1 Introduction Transformer architecture Vaswani et al. 2017 forms the backbone of modern large language models (LLMs), demonstrating remarkable performance across diverse natural language processing tasks. At the core of these models lies the self-attention mechanism, which enables dynamic modeling of contextual relationships between tokens. Despite its empirical success, several studies have identified limitations associated with attention localization , a phenomenon in which attention distributions collapse onto a limited subset of tokens. Such localized attention tends to have low entropy Dong et al. 2024 , low-rank sparsity Bao et al. 2024 and sparsed attention (Fig. 6 in the appendix), which negatively affect the representational power, training stability, and downstream model performance Zhai et al. 2023 . This problem is particularly pronounced in small-scale Transformer variants. Their limited depth and width inherently restrict the ability to propagate information across layers, making them more susceptible to localized attention than larger models. In standard self-attention mechanisms, each query token primarily attends to a few highly salient keys Shi et al. 2021 , thus modeling predominantly one-hop 2 2 2 In this paper, we consider ”local” as one-hop token dependencies, and ”global” as multi-hop token dependencies. dependencies. To address this limitation, we hypothesize that explicitly incorporating global context—defined here as multi-hop information flow—could mitigate the localization issue, particularly in compact architectures. To test this hypothesis, we propose Self-Attention One-step Belief Propagation (SAOBP) , a method that integrates global interactions by leveraging belief propagation (BP) principles combined with a repulsive Potts prior (Fig. 1 ). Our empirical evaluations show that SAOBP effectively alleviates attention collapse and consistently enhances performance across downstream tasks. To systematically investigate how SAOBP mitigates localization through multi-hop information flow and interpret its impact on model performance, we introduce Global Token Dependency (GTD) . GTD quantifies the relative attention mass contributed by intermediate, multi-hop transitions ( \geq 2 hops) within the stochastic attention graph. Our empirical analysis illustrated in Fig. 3 and Fig. 4 demonstrates that GTD serves as a principled diagnostic tool, capable of detecting specific layers and heads where attention collapses into overly localized patterns. Figure 1: Comparison between original self-attention (left) and proposed SAOBP algorithm (right): The standard self-attention formulation relies on direct (one-hop) token interactions, which often yield sparse attention distributions and entropy collapse, as illustrated in the heatmaps. In contrast, SAOBP utilizes a structured message-passing step that propagates multi-hop dependency signals via belief updates governed by a repulsive Potts prior. This enhances global context modeling and redistributes attention mass more broadly, mitigating localization and improving representational diversity in deeper layers. Contributions • We propose SAOBP , a self-attention regularization framework that suppresses entropy collapse and promotes diverse, globally-aware attention distributions. • We introduce Global Token Dependency (GTD) , a novel diagnostic concept that quantifies the extent of multi-hop information flow in attention graphs. This value offers a principled measure of intermediate context modeling and attention localization. • We empirically demonstrate that SAOBP improves model performance across a variety of downstream tasks. The benefits are more pronounced in small-scale settings, providing practical guidance for the design of efficient and expressive compact language models. 2 Related works Small-scale models in NLP Some recent studies have emphasized the practical advantages of small-scale language models, including cost-efficiency, rapid customization, and deployment feasibility in resource-constrained environments Wang et al. 2024 . Liu et al. 2024 and Thawakar et al. 2024 propose sub-billion parameter models optimized for mobile and edge applications. Similarly, Jiao et al. 2020 and Turc et al. 2019 have demonstrated that small transformer models can achieve competitive performance through effective knowledge distillation and optimized architectures. Nonetheless, compact models typically underperform compared to larger-scale alternatives, and bridging this performance gap through targeted architectural enhancements remains an important open challenge. In this work, we introduce a framework that improves the expressiveness of compact models (with \leq 50 M parameters), enhancing their utility in constrained settings. Localization of Self-Attention Self-attention mechanisms often exhibit localization, where attention mass concentrates on a few tokens, leading to entropy collapse (where attention entropy skewed to zero) and representational degradation Zhang et al. 2024 ; Qi et al. 2025 . Prior works attribute this to low-rank eigen-spectra of query-key matrices Bao et al. 2024 and the exponential nature of the softmax function Dong et al. 2024 . While several regularization strategies have been proposed Jha and Reagen 2025 ; Zhai et al. 2023 , these approaches do not explicitly quantify the extent of attention localization, making their effectiveness difficult to interpret. To overcome this limitation, we introduce GTD, a metric enabling interpretable, layer-wise analysis of attention spread. Belief Propagation for Marginalization Belief propagation (BP) is a classical inference algorithm that enables efficient marginalization in structured graphical models Pearl 1988 ; Yedidia et al. 2001 ; Koller and Friedman 2009 . Recent studies have explored integrating BP into neural architectures for structured prediction Kim et al. 2017 ; Kuck et al. 2020 ; Dai et al. 2016 . However, few studies have directly leveraged BP to refine attention patterns. Since each row of the attention matrix can be interpreted as a categorical probability distribution over tokens, BP can offer a theoretically grounded mechanism to reshape probability distributions under higher-order structural constraints. Our proposed SAOBP represents a distinct application of BP, utilizing its message-passing mechanism to inject multi-hop dependencies into attention and thereby mitigate localization effects. 3 Preliminaries Self-Attention and Attention Entropy Self-attention is mathematically defined by Eq. ( 1 ), where Q , K , and V represent learnable weight matrices, and B , H , and L denote the batch size, number of attention heads, and sequence length: A=\text{softmax}\left(\frac{QK^{T}}{\sqrt{d}}\right)\quad\in\mathbb{R}^{B\times H\times L\times L}. (1) The internal dynamics of self-attention can be analyzed using the concept of attention entropy Zhang et al. 2024 , defined formally in Eq. ( 2 ). Attention entropy quantifies the uncertainty in attention distributions, offering insight into how information is structured within models. We leverage this metric to investigate attention entropy collapse. \begin{split}\mathcal{H}(A_{i})&=-\sum_{j=1}^{L}A_{ij}\,\log A_{ij},\\ \mathcal{H}(A)&=\frac{1}{L}\sum_{i=1}^{L}\mathcal{H}(A_{i})\end{split} (2) Constructing Belief Propagation A belief propagation algorithm can be constructed with the following four components Mézard and Montanari 2009 . Let i and j be variable nodes connected by a factor node f_{ij} . 1. Messages : Messages represent probabilistic information passed between variable and factor nodes