Gated Graph Attention Networks with Learnable Temperature
Zhongtian Ma, Yexin Zhang, Qiaosheng Zhang, Zhen Wang, [0.5em] School of Cybersecurity, Northwestern Polytechnical University, Xi’an 710072, China · 2026
Read Gated Graph Attention Networks with Learnable Temperature by Zhongtian Ma, Yexin Zhang, Qiaosheng Zhang, Zhen Wang, [0.5em] School of Cybersecurity, Northwestern Polytechnical University, Xi’an 710072, China (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-29803.txt.
Introduction
Zhongtian Ma and four colleagues add two controls to graph attention networks: gating and a learnable temperature. Standard layers weigh neighbours by data-driven scores but leave shaky dimensions unchecked and fix attention sharpness. Gating filters feature and message responses to mute unreliable dimensions, while temperature tunes attention sharpness. Tests on uniform and mixed-type benchmarks show the variants beat their base models, and noise trials confirm gains under feature corruption. Theory backs the picture: gating helps when only some coordinates prove trustworthy, temperature when noise blurs node differences.
Catalog record
| CREATOR | Zhongtian Ma, Yexin Zhang, Qiaosheng Zhang, Zhen Wang, [0.5em] School of Cybersecurity, Northwestern Polytechnical University, Xi’an 710072, China |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 56230 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.29803
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Gated Graph Attention Networks with Learnable Temperature Zhongtian Ma, Yexin Zhang, Qiaosheng Zhang, Zhen Wang, [0.5em] School of Cybersecurity, Northwestern Polytechnical University, Xi’an 710072, China Abstract Graph attention networks learn neighbor importance through data-dependent coefficients, but standard layers lack explicit control over unreliable feature dimensions and use fixed sharpness of attention coefficient distributions. This paper proposes gated graph attention and learnable temperature for common graph attention mechanisms. Gated graph attention filters feature or message responses to reduce the influence of unreliable dimensions, while learnable temperature dynamically adjusts the sharpness of the attention coefficient distribution. Experiments on homogeneous and heterophilic heterogeneous benchmarks show that the proposed variants consistently improve the corresponding graph attention backbones, and controlled noise studies further verify their behavior under feature perturbations. Theoretical analysis explains these results by showing that gating improves robustness when only part of the feature coordinates are reliable, while temperature is beneficial when global noise weakens the discriminability of node features. 1 Introduction Graph neural networks (GNNs) have become a widely used framework for learning representations on graph-structured data, with applications in citation networks [ 30 ] , social networks [ 6 ] , biology [ 9 ] , computer vision [ 20 ] and recommendation systems [ 29 ] . Early graph convolutional networks (GCNs) typically aggregate neighborhood information with fixed structure-dependent weights, which makes them efficient and stable but can be limiting when neighboring nodes contribute unequally to the target node [ 16 , 11 , 23 ] . Later graph attention networks (GATs) address this limitation by computing data-dependent coefficients over neighbors, so that more informative neighbors can receive larger coefficients during message passing [ 26 , 28 ] . Most of these models can be viewed under a neighborhood message-passing paradigm, where each node updates its representation by collecting and transforming information from its neighbors [ 7 ] . Despite this flexibility, standard graph attention mainly adapts the relative importance of neighbors, while leaving two aspects of the attention layer less directly controlled. First, it maps unnormalized neighbor scores, or attention logits, to attention coefficients using a fixed softmax function. As a result, the sharpness of the resulting attention distribution is not explicitly adaptive. In mixed or noisy neighborhoods, this may cause the model to assign overly concentrated coefficients to unreliable neighbors, or to use an unsuitable level of smoothing across different layers and datasets [ 21 , 35 ] . Second, standard graph attention offers limited control over which feature dimensions are propagated after aggregation. In high-dimensional, sparse, or partially corrupted node features, different coordinates can have very different reliability. Without an explicit feature-level filtering mechanism, noisy coordinates may still affect the attention logits or be carried into the node update [ 8 ] . To address these two limitations, we introduce two lightweight modifications to graph attention. The first is learnable temperature, which rescales attention logits before softmax normalization and allows each layer to adapt the sharpness of its attention distribution. The second is gated graph attention, which modulates either the aggregated node update or the projected messages used in attention aggregation, enabling the model to suppress less reliable feature responses. These two mechanisms act on different parts of the attention layer and can be applied independently or jointly to different GAT backbones. This yields a unified family of attention variants that controls not only where information is aggregated from, but also how sharply and how selectively it is propagated. We further provide a supporting theoretical analysis to explain why these mechanisms are useful. The analysis is conducted under a two-class contextual stochastic block model (CSBM) [ 5 ] , which offers a tractable setting for studying noisy node features and attention aggregation. Specifically, we consider two common forms of feature noise, namely global perturbations that reduce the separability of node features and coordinate-wise missing or corrupted entries that leave only part of the feature vector reliable. Under global perturbations, learnable temperature can improve the node-level signal-to-noise ratio by mitigating noise-driven concentration in attention coefficients. Under coordinate-wise corruption, gated graph attention can suppress unreliable feature dimensions and improve the first-order node-level signal-to-noise ratio. These results indicate that the two mechanisms target complementary failure modes in graph attention. We further conduct extensive experiments on homogeneous and heterophilic heterogeneous benchmarks, together with controlled noise experiments, to verify the effectiveness and behavior of the proposed mechanisms. Our main contributions are summarized as follows: • We propose two lightweight and plug-in modifications to graph attention: learnable temperature for adapting attention sharpness, and gated graph attention for modulating feature-level or message-level contributions. These mechanisms can be applied to both GAT [ 26 ] and GATv2 [ 2 ] , either independently or jointly. • We provide a supporting theoretical analysis under CSBM, showing when the two mechanisms are effective. For learnable temperature, Theorem 1 shows that it can improve attention behavior when attention coefficients are affected by unreliable neighborhood signals. For gated graph attention, Theorem 2 shows that it can be beneficial when only part of the feature dimensions is reliable or task-relevant. • We evaluate the proposed variants on six homogeneous graph datasets and five heterophilic heterogeneous datasets. Overall, the proposed variants show stronger empirical performance than the corresponding GAT and GATv2 backbones across the evaluated benchmarks, and the controlled noise experiments further show behavior consistent with our theoretical analysis. 2 Related Works 2.1 Adaptive Attention Modulation Standard attention mainly controls aggregation through normalized coefficients [ 25 ] , but it does not explicitly decide whether an attention output should strongly update, weakly update, or nearly skip the current representation. Gated attention addresses this limitation by adding an explicit modulation path. Bondarenko et al. [ 1 ] show that some Transformer heads simulate no-op or weak updates by producing extreme logits, causing activation outliers and poor quantization behavior; their gated attention applies a lightweight sigmoid gate to each head output so that the model can directly control update magnitude. Qiu et al. [ 22 ] further conduct a systematic study of gated softmax attention in large language models and find that head-specific sigmoid gating after scaled dot-product attention improves performance, training stability, scaling behavior, and attention-sink mitigation. Related Transformer variants also use gates to regulate attention behavior, including content-dependent forget gates in Forgetting Transformer [ 18 ] , Gated Attention Units in FLASH [ 14 ] , moving-average equipped gated attention in MEGA [ 19 ] , and routing-style conditional attention in SwitchHead [ 4 ] . These works support the view that attention benefits from mechanisms that control not only where information comes from, but also how strongly an attention branch contributes. Another line of work modulates attention by controlling the sharpness of the attention distribution through temperature or learnable scaling. In scaled dot-product attention, the factor \sqrt{d_{k}} can be interpreted as a fixed temperature-like normalization [ 25 ] , while Query-Key Normalization replaces this fixed scaling with a learnable parameter after normalizing queries and keys [ 12 ] . Adaptive sparse attention likewise investigates how learned parameters affect whether attention is more concentrated or more diffuse [ 3 ] . In graph learning, Ma et al. [ 21 ] show that graph attention is not always advantageous and may underperform