Beyond Attention: Learning Spatio-Temporal Dynamics with Emergent Interpretable Topologies DOI: XXXXXXX.XXXXXXX ISBN: 978-1-4503-XXXX-X/2018/06 CCS: Computing methodologies Machine learning CCS: Computing methodologies Neural networks CCS: Computing methodologies Structured outputs CCS: Computing me
Sai Vamsi Alisetti, Vikas Kalagi, Sanjukta Krishnagopal · 2025
Read Beyond Attention: Learning Spatio-Temporal Dynamics with Emergent Interpretable Topologies DOI: XXXXXXX.XXXXXXX ISBN: 978-1-4503-XXXX-X/2018/06 CCS: Computing methodologies Machine learning CCS: Computing methodologies Neural networks CCS: Computing methodologies Structured outputs CCS: Computing me by Sai Vamsi Alisetti, Vikas Kalagi, Sanjukta Krishnagopal (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-00770.txt.
Introduction
Sai Vamsi Alisetti, Vikas Kalagi, and Sanjukta Krishnagopal propose, in a 2025 paper, a forecasting model that learns the map of links between places instead of being given one. Forecasting across space and time matters for traffic, energy demand, and weather. A popular tool, graph attention networks, leans on a fixed map of which places connect, plus shifting attention scores.
Their model replaces masked attention with a fully learnable, symmetric table of how nodes affect each other. That table can discover hidden spatial links on its own. A recurrent unit then handles the time side of the forecast. On taxi data from one city and highway loop data from another, their system beat the attention based version at every horizon from 15 to 60 minutes. Gains ran from about six to at least twenty one percent, and training was faster.
Catalog record
| CREATOR | Sai Vamsi Alisetti, Vikas Kalagi, Sanjukta Krishnagopal |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 49 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2506-00770
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Beyond Attention: Learning Spatio-Temporal Dynamics with Emergent Interpretable Topologies DOI: XXXXXXX.XXXXXXX ISBN: 978-1-4503-XXXX-X/2018/06 CCS: Computing methodologies Machine learning CCS: Computing methodologies Neural networks CCS: Computing methodologies Structured outputs CCS: Computing methodologies Learning latent representations CCS: Mathematics of computing Graph theory Sai Vamsi Alisetti, Vikas Kalagi, Sanjukta Krishnagopal Abstract . Spatio-temporal forecasting is critical in applications such as traffic prediction, energy demand modeling, and weather monitoring. While Graph Attention Networks (GATs) are popular for modeling spatial dependencies, they rely on predefined connectivity and dynamic attention scores, introducing inductive biases and computational overhead, often at the cost of interpretability. We propose InterGAT , a simplified alternative to GAT that replaces masked attention with a fully learnable, symmetric node interaction matrix, capturing latent spatial relationships without relying on fixed graph topologies. Our framework, InterGAT-GRU , which incorporates a GRU-based temporal decoder, outperforms the baseline GAT-GRU in forecasting accuracy, achieving at least a 21% improvement on the SZ-Taxi dataset and a 6% improvement on the Los-Loop dataset across all forecasting horizons (15 to 60 minutes). Additionally, we observe a reduction in training time by 60-70% compared to GAT-GRU baseline. Interestingly, the learned interaction matrix reveals interpretable structure: it recovers sparse, topology-aware attention patterns that align with community structure. Spectral and clustering analyses indiate that the model captures both localized and global dynamics, offering insights into the emergent topology driving its predictions. This highlights how structure learning can simultaneously enhance predictive performance, efficiency, and topological interpretability in dynamic spatio-temporal graph domains. 1. Introduction Spatio-temporal forecasting is central to many real-world systems, from intelligent transportation networks ( Zhao et al. 2019 ) , to energy grids ( Wang et al. 2024 ) , and environmental monitoring ( Xie et al. 2023 ) . It also plays a vital role in healthcare ( Wang et al. 2022 ) , climate science ( Ravuri and et al. 2021 ) , agriculture ( Shahhosseini et al. 2021 ) , urban planning ( Li et al. 2023 ) , and prediction of social events ( Wu et al. 2023 ) , making it a foundational task in domains where spatial and temporal dependencies are key. These tasks involve modeling both the spatial dependencies between entities (e.g., road sensors) and the temporal evolution of their signals over time. Graph Neural Networks (GNNs) ( Scarselli et al. 2009 ) have become the de facto tools for spatial modeling in such domains. Graph Attention Networks (GATs) ( Veličković et al. 2018 ) , in particular, stand out due to their ability to adaptively weight neighboring nodes through attention mechanisms. However, conventional GATs have two limitations: 1) they rely on predefined adjacency matrices that encode a fixed graph structure and 2) compute pairwise attention scores at every forward pass. These design choices introduce structural biases, add computational complexity, and often limit interpretability. Previous works have attempted to address this limitation by introducing adaptive or dynamic adjacency matrices ( Zhang et al. 2021 ) , ( Bai et al. 2020 ) , ( Fang et al. 2021 ) , ( Xu et al. 2023 ) , which allow connectivity patterns to evolve based on node features, tasks, or temporal contexts. Yet these approaches typically retain costly attention computations and require recomputing the graph structure at every time step. In this work, we propose an efficient and interpretable simplification of this paradigm. Instead of relying on masked attention over a fixed graph, we introduce a fully learnable symmetric node-to-node interaction matrix that captures latent structural relationships directly from data. This matrix is shared across time and trained end-to-end, allowing the model to discover meaningful topological patterns such as modular communities or long-range dependencies without access to prior structural knowledge. This design enables not only improved training efficiency but also topology-aware interpretability. By analyzing the learned interaction matrix, we can gain insights into which node relationships are most influential for forecasting, and how attention is distributed across the graph. Spectral analyses reveal that the learned interaction matrix exhibits a full smooth eigenvalue spectrum, and the most predictive modes capture sharp contrasts across key node pairs. These findings echo trends in recent works that investigate the spectral properties of attention mechanisms in graphs ( Brody and Yahav 2021 ; Kim et al. 2022 ) and their alignment with underlying topological structure ( Ying et al. 2019 ; Dwivedi et al. 2023 ) Our community alignment analyses reveal that different heads of the model specialize to focus attention on localized and global structure respectively. The learned attention often align with topology-aware community structure, bridging interpretable deep learning and classical network science. To model temporal evolution, we pair this spatial module with a GRU-based temporal decoder, yielding InterGAT-GRU—a spatio-temporal forecasting framework. This architecture offers several advantages: • Efficiency: It improves computational efficiency across time horizons. Specifically, it reduced training time by 60-70% compared to an equivalent GAT-GRU baseline for the SZ-Taxi and Los-Loop dataset in the 15 minute horizon. These comparisons are made using identical hidden layer dimensions, GRU decoders, and identical hardware settings. • Performance: The model consistently outperforms equivalent GAT-GRU baselines in prediction on two benchmark traffic datasets across all forecasting horizons. • Interpretability: It learns symmetric, sparse interaction matrices that uncover modular, interpretable structures such as communities, functional clusters, and multi-scale patterns, offering insights from network science into latent spatial dynamics. 2. Related Work Spatio-temporal graph forecasting has been extensively studied in recent years, with most methods combining graph-based spatial modeling with recurrent or attention-based temporal encoders. Graph Convolutional Networks (GCNs) ( Kipf and Welling 2017 ) and Graph Attention Networks (GATs) have become the dominant paradigms for modeling spatial dependencies among nodes ( Wu et al. 2021 ; Veličković et al. 2018 ) . These are typically integrated with temporal models such as Gated Recurrent Units (GRUs) ( Chung et al. 2014 ) or temporal attention ( Xu et al. 2020 ) to handle time-varying signals. GAT-based forecasting models, such as ASTGCN ( Guo et al. 2019 ) , STGAT ( Kong et al. 2020 ) , and STGSL ( Zhang et al. 2021 ) , use attention to dynamically weigh node importance, but still operate under the constraint of a fixed adjacency matrix. However, these models typically assume a fixed or heuristically constructed adjacency matrix and compute attention scores at each forward pass, which can limit the model’s ability to capture long-range or evolving dependencies and reduce efficiency. Recent works like STGSL ( Zhang et al. 2021 ) and Graph WaveNet ( Wu et al. 2019 ) propose learnable adjacency matrices while Neural Relational Inference (NRI) ( Kipf et al. 2018 ) models latent dynamics via variational inference. Kazi et al. ( Kazi et al. 2023 ) employs a differentiable graph module that learns soft graph structures using attention-based updates. Unlike these methods, we learn a single, persistent interaction matrix shared across time, enabling both efficiency and interpretability Temporal modeling in spatio-temporal graphs is often handled via recurrent architectures such as GRUs or LSTMs, as seen in DCRNN ( Li et al. 2018 ) and GCN-GRU hybrids ( Zhao et al. 2019 ) . These models are effective in capturing sequential trends, but may under-use spatial flexibility when the graph structure is fixed. Transformer-based models