Topologic Attention Networks: Attending to Direct and Indirect Neighbors through Gaussian Belief Propagation

Marshall Rosenhoover, Huaming Zhang · 2025

Read Topologic Attention Networks: Attending to Direct and Indirect Neighbors through Gaussian Belief Propagation by Marshall Rosenhoover, Huaming Zhang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-16871.txt.

Introduction

Marshall Rosenhoover and Huaming Zhang introduce Graph Linear Transformations, a way for graph neural networks to gather information from distant nodes in one step instead of stacking message-passing layers. Standard networks only mix direct neighbors per layer, so reaching far across the graph means going deep, with the usual training headaches.

The transformation is a single linear operator derived from the graph, computed by interpreting the graph as a walk-summable Gaussian graphical model and running Gaussian belief propagation. Each node then aggregates direct and indirect neighbors without enumerating multi-hop paths. Different precision matrix constructions give interpretable propagation biases, from selective edge-level mixing to uniform smoothing. On homophilic and heterophilic benchmarks the approach matched or beat local message-passing networks and dynamic neighborhood aggregation models.

Catalog record

CREATORMarshall Rosenhoover, Huaming Zhang
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE31 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2511-16871

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Topologic Attention Networks: Attending to Direct and Indirect Neighbors through Gaussian Belief Propagation Marshall Rosenhoover, Huaming Zhang Abstract Graph Neural Networks rely on local message passing, which limits their ability to model long-range dependencies in graphs. Existing approaches extend this range through continuous-time dynamics or dense self-attention, but both suffer from high computational cost and limited scalability. We propose Topologic Attention Networks, a new framework that applies topologic attention, a probabilistic mechanism that learns how information should flow through both direct and indirect connections in a graph. Unlike conventional attention that depends on explicit pairwise interactions, topologic attention emerges from the learned information propagation of the graph, enabling unified reasoning over local and global relationships. This method achieves provides state-of-the-art performance across all measured baseline models. Our implementation is available at https://github.com/Marshall-Rosenhoover/Topologic-Attention-Networks . 1 Introduction An intrinsic component of Graph Neural Networks (GNNs) is neighborhood aggregation: combining information from a node’s neighbors to compute its representation. Many standard GNNs, such as graph convolutional networks [ 11 ] and graph attention networks [ 17 ] , rely on aggregation functions that are only determined by immediate neighbors. This local aggregation limits expressiveness and can struggle to capture long-range dependencies [ 7 , 5 ] . Determined to solve this problem, two main branches have emerged to aggregate information beyond the local neighborhood: continuous dynamic GNNs and graph transformers. Continuous dynamic GNNs, such as Graph Neural ODE [ 14 ] and Continuous Graph Neural Networks [ 18 ] , extend message passing into continuous time by parameterizing information flow as a differential equation or continuous-time dynamical system. This formulation allows, in principle, unbounded propagation depth and smooth global interactions. However, these methods are prone to vanishing or exploding gradients, and remain inherently sequential [ 14 , 18 , 20 ] . Graph transformers [ 6 , 21 ] take the opposite approach by applying self-attention over a range of node representations to capture relationships beyond immediate neighbors. By encoding structural priors into node embeddings, they can capture complex relational patterns without explicit message passing. Yet, this expressivity comes at high compute and memory costs, and the effective incorporation of graph structure remains a nontrivial challenge [ 9 ] . Figure 1 : Illustration of a single head of topologic attention via a learned Gaussian Graphical Model, where node shading encodes confidence and edge color denotes interaction polarity and strength. In this work, we propose a third branch of dynamic neighborhood aggregation: Topologic Attention Networks (TANs). Unlike continuous dynamic GNNs, which require sequential integration, or graph transformers, which depend on dense pairwise attention, TANs perform topologic attention, learning how information should flow probabilistically through the graph’s topology to capture both local and global relationships. Figure 1 illustrates a potential learned topologic attention pattern. Laying the foundation for this new branch, we present the Topologic Attention Layer, a GNN layer that implements topologic attention. We evaluate our method across diverse graph benchmarks and find that they achieve accuracy comparable to or exceeding state-of-the-art methods, highlighting the potential of topologic attention for local and global reasoning in graphs. 2 Background A linear transformation on a graph (graph linear transformation) can be understood as a global operator that mixes node features according to the connectivity of the graph. Achieving this would require each node to integrate information from all other nodes along every possible path that connects them. However, capturing these higher-order dependencies explicitly is intractable, as enumerating all such paths is known to be NP-hard [ 2 , 8 ] . Instead of constructing this transformation explicitly, we can reinterpret the graph as a Gaussian graphical model whose equilibrium yields the solution to the graph linear transformation. In this formulation, Gaussian dependencies implicitly accumulate the influence of all paths in the graph, eliminating the need to enumerate them. 2.1 Gaussian Graphical Models Gaussian Graphical Models represent a set of continuous random variables x=[x_{1},\ldots,x_{N}]^{\top} that follow a joint multivariate Gaussian distribution with covariance matrix \Sigma . Each variable is associated with a node in a graph, where edges capture conditional dependencies between variables. The precision matrix J=\Sigma^{-1} encodes these conditional dependencies with nonzero entries J_{ij} indicate edges between nodes [ 12 , 1 , 13 ] . Together with this structural information, the model includes an information vector h representing the nodes’ local evidence. The pair (J,h) fully defines the probabilistic state of the system, with the joint distribution p(x)\propto\exp\!\left(-\tfrac{1}{2}x^{\top}Jx+h^{\top}x\right), where J\in\mathbb{R}^{N\times N} , h\in\mathbb{R}^{N} , and x\in\mathbb{R}^{N} . The equilibrium of this system \mu=J^{-1}h corresponds to the solution of the graph linear transformation introduced earlier. However, directly inverting J is computationally expensive and scales poorly with graph size. 2.2 Gaussian Belief Propagation Instead of performing the matrix inversion, Gaussian Belief Propagation (GaBP) solves the system through localized message passing. In univariate GaBP, each message represents a univariate Gaussian distribution and is parameterized by two scalars: the precision \pi (inverse variance) and the information \eta (precision-weighted mean) [ 1 , 10 , 12 ] . At each iteration, node i sends a message to a neighbor j based on incoming messages from all other neighbors except j , denoted \mathcal{N}(i)\setminus j . The message updates are defined as \pi_{i\to j}=-\frac{J_{ij}^{2}}{\alpha_{i\setminus j}},\qquad\eta_{i\to j}=-\frac{J_{ij}}{\alpha_{i\setminus j}}\,\beta_{i\setminus j}, where \alpha_{i\setminus j}=J_{ii}+\sum_{k\in\mathcal{N}(i)\setminus j}\pi_{k\to i},\qquad\beta_{i\setminus j}=h_{i}+\sum_{k\in\mathcal{N}(i)\setminus j}\eta_{k\to i}. After convergence, the linear system J\mu=h is solved implicitly, and the resulting solution can be recovered by \mu_{i}=\eta_{i}/\pi_{i} . Convergence Guarantees. GaBP is guaranteed to converge on cyclic graphs only when the precision matrix is walk-summable. Walk-summability requires that the normalized precision matrix \tilde{J}=D^{-1/2}JD^{-1/2} satisfy the spectral condition \rho\bigl(|I-\tilde{J}|\bigr)<1. Under this condition, information propagation corresponds to a convergent geometric series, \sum_{k=0}^{\infty}(I-\tilde{J})^{k}, so that contributions from longer walks decay geometrically and the total influence of all walks remains finite. 3 Topologic Attention Networks Figure 2 : Architecture of the Topologic Attention Layer. As discussed in the previous section, constructing an explicit graph linear transformation requires enumerating all paths between nodes, which is computationally intractable. Instead, we model information propagation as probabilistic inference over a walk-summable graph topology J . At equilibrium, GaBP satisfies J\mu=h\quad\Rightarrow\quad\mu_{i}=\sum_{j}(J^{-1})_{ij}h_{j}, where (J^{-1})_{ij} quantifies how information from node j influences node i through all paths in the graph. In this formulation, the effective neighborhood of a node can become dynamic in two senses. First, the precision matrix J determines which nodes are able to influence one another, so a node’s neighborhood is defined by the topology encoded in J rather than by the raw graph adjacency. Second, the observation vector h determines how strongly the nodes within this topology contribute to the aggregation: nodes with weak evidence exert little influence, while nodes with strong evidence have proportionally larger impact. Thus, J defines the dynamic neighborhood, and h defines the dynamic weighting within that neighborhood. Building on this principle, we propose making the probabilistic inference learnable by parameterizing the node evidence h and, optionally, the topology J . This allows the model to jointly learn the topological neighborhoods and how strongly each node should participate in them. We refer to this as topologic attention, implemented through the Topologic Attention Layer. 3.1 Topologic Attention Layer The overall structure of the Topologic Attention Layer, illustrated in Figure 2 , consists of two subblocks: a multi-head

Read the complete text