GIST: Gauge-Invariant Spectral Transformers for Scalable Graph Neural Operators
Mattia Rigotti ∗, Nicholas Thumiger ∗, Thomas Frick ∗ · 2026
Read GIST: Gauge-Invariant Spectral Transformers for Scalable Graph Neural Operators by Mattia Rigotti ∗, Nicholas Thumiger ∗, Thomas Frick ∗ (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-16849.txt.
Introduction
Rigotti, Thumiger, and Frick present GIST, a graph neural operator built to respect gauge invariance. Spectral encodings capture mesh geometry, but exact versions cost cubic time and break gauge symmetry through solver artifacts. GIST instead uses pairwise inner products of cheap approximate spectral embeddings. The authors prove these products estimate an exactly gauge-invariant kernel, at linear cost. They also bound the mismatch when a model moves across mesh resolutions. On mesh benchmarks with up to 750,000 nodes, GIST sets new state-of-the-art results.
Catalog record
| CREATOR | Mattia Rigotti ∗, Nicholas Thumiger ∗, Thomas Frick ∗ |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 61162 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.16849
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
GIST: Gauge-Invariant Spectral Transformers for Scalable Graph Neural Operators Mattia Rigotti ∗, Nicholas Thumiger ∗, Thomas Frick ∗ Abstract Adapting transformer positional encoding to meshes and graph-structured data presents significant computational challenges: exact spectral methods require cubic-complexity eigendecomposition and can inadvertently break gauge invariance through numerical solver artifacts, while efficient approximate methods sacrifice gauge symmetry by design. Both failure modes cause catastrophic generalization in inductive learning, where models trained with one set of numerical choices fail when encountering different spectral decompositions of similar graphs or discretizations of the same mesh. We propose GIST (Gauge-Invariant Spectral Transformers), a new graph transformer architecture that resolves this challenge by achieving end-to-end 𝒪 ( N ) \mathcal{O}(N) complexity through random projections while algorithmically preserving gauge invariance via inner-product-based attention on the projected embeddings. We prove GIST achieves discretization-invariant learning with bounded mismatch error, enabling parameter transfer across arbitrary mesh resolutions for neural operator applications. Empirically, GIST matches state-of-the-art on standard graph benchmarks (e.g., achieving 99.50% micro-F1 on PPI) while uniquely scaling to mesh-based Neural Operator benchmarks with up to 750K nodes, achieving state-of-the-art aerodynamic prediction on the challenging DrivAerNet and DrivAerNet++ datasets. † † affiliationnotice: Equal contribution 1 Introduction Following their incredible success for processing sequential data in Natural Language Processing, Transformers ( 59 ) have been demonstrating a remarkable capacity for handling data of increasing structural complexity. 41 have proposed a variant of the transformer block for permutation invariant data with their Set Transformer architecture; 21 have adapted the self-attention mechanism to 2D images with the very influential Vision Transformer architecture; and 2 have extended transformers to video analysis with their Video Vision Transformer (ViViT), demonstrating how attention mechanisms capture both spatial and temporal dependencies across video frames. This progression from sequential text to increasingly structured data indicates a trajectory suggesting that Transformers are poised to tackle even more complex data structures, including irregular meshes and graphs. Indeed, recent developments in adapting Transformers to graphs have shown promising results in capturing long-range dependencies that traditional Graph Neural Networks (GNNs) struggle with due to their reliance on localized message passing ( 23 ; 69 ) . Unlike GNNs that aggregate information from nearest neighboring nodes by iterating through layers, Transformers can directly capture global relationships across the whole graph through self-attention, enabling them to reason about distant node interactions in a single layer. Two Barriers to Scalable Graph Transformers. However, adapting transformers to graphs introduces two distinct barriers that have not been simultaneously addressed. First, there is a computational barrier : exact spectral graph embeddings, while theoretically natural, require eigendecomposition of the graph Laplacian, scaling as \mathcal{O}(N^{3}) for dense graphs or \mathcal{O}(N^{2}) for sparse graphs (where N is the number of nodes), which is prohibitively expensive for large-scale graphs. Second, there is a gauge invariance barrier : both exact and approximate spectral methods may inadvertently break gauge invariance, i.e. the inherent freedom to rotate eigenvectors, flip signs, or choose among degenerate eigenvectors. Exact methods may break it through numerical solver artifacts (sign choices, eigenspace ordering, degeneracy handling, see 5 ), while approximate methods commit to a specific basis decomposition through their approximation scheme, sacrificing freedom in spectral choices. This gauge invariance breaking introduces spurious inductive biases tied to arbitrary numerical choices, causing models trained with one set of choices to fail catastrophically when evaluated with different spectral decompositions or numerical solvers, particularly in inductive learning tasks where models must generalize to unseen graph structures. The Gauge Invariance Challenge. Consider a graph whose spectral embeddings are computed via random projection matrix R . A neural network trained on these embeddings \{R\phi_{i}\}_{i} will inevitably learn features correlated with the specific choice of R . Consequently, when evaluated on different graphs or with a different random projection R^{\prime} or different numerical eigensolver choices (sign flips, eigenspace ordering, handling eigenspace degeneracy), learned features become meaningless. This gauge dependence fundamentally undermines generalization, a critical failure mode for inductive graph learning where models must transfer to unseen structures. Beyond graphs, gauge invariance is essential to generate neural operators with bounded discretization error . Physical problems (e.g., computational fluid dynamics, structural mechanics, shape analysis) are defined on continuous manifolds but discretized into computational meshes corresponding to graphs. Different mesh resolutions produce different graph Laplacians with different spectral decompositions, each involving arbitrary gauge choices (sign flips, eigenspace rotations, solver artifacts). Without gauge invariance, parameters trained on one discretization fail to transfer to another, and the attention kernels computed from spectral embeddings at different resolutions cannot be compared meaningfully. Gauge invariance ensures that the learned operator converges to the same continuum limit regardless of discretization, enabling provably bounded discretization mismatch error that vanishes as resolution increases. Existing approaches address only one barrier at a time: (1) spectral methods like SAN ( 40 ) maintain gauge invariance but require full eigendecomposition; (2) approximate spectral methods achieve linear complexity but sacrifice gauge invariance; (3) generic linear transformers reduce attention complexity but ignore graph structure. Recent spectral invariant methods achieve gauge invariance through careful attention design, but still require full eigendecomposition ( \mathcal{O}(N^{3}) complexity) or quadratic attention ( \mathcal{O}(N^{2}) ), limiting practical scalability. Moreover, they lack discretization error controls needed for neural operator applications where consistency across mesh resolutions is critical. Our Contribution: GIST. We propose Gauge-Invariant Spectral Transformers (GIST) , which overcomes both barriers through a key insight: gauge invariance can be preserved by restricting attention to inner products between spectral embeddings, which remain invariant even under approximate spectral computations. We instantiate this framework using random projections for their computational efficiency and theoretical guarantees. For random projections, inner products between projected embeddings \langle R\phi_{i},R\phi_{j}\rangle=\langle\phi_{i},(R^{T}R)\phi_{j}\rangle\approx\langle\phi_{i},\phi_{j}\rangle remain approximately invariant under gauge transformations. By restricting attention operations to only these inner products and combining with linear attention, we achieve end-to-end \mathcal{O}(N) complexity while recovering gauge invariance algorithmically. Unlike prior approaches that address these barriers separately, GIST achieves gauge invariance and linear complexity simultaneously. Our contributions in short are: 1. identifying gauge invariance breaking as a fundamental limitation shared by both exact and approximate spectral methods, and characterizing when this breaks generalization and neural operator convergence; 2. proposing GIST, combining gauge-invariant spectral attention with multi-scale linear transformer blocks, achieving theoretical guarantees on complexity and invariance preservation, with competitive results on standard graph benchmarks (e.g. Cora, PubMed, PPI); 3. establishing GIST as a neural operator with provably bounded discretization mismatch error achieving state-of-the-art on large-scale mesh problems. 2 Related Works Graph Transformers. Graphormer ( 65 ) introduces the idea of integrating structural encodings such as shortest path distances and centrality in Transformers. Similarly, 23 propose LSPE (Learnable Structural and Positional Encodings), an architecture that decouples structural and positional representations. 40 propose Spectral Attention Network (SAN), which introduces learned positional encodings from the full Laplacian spectrum. 50 develop Graph Relative Positional Encoding (GRPE), which extends relative positional encoding to graphs by considering features representing node-topology and node-edge interactions. Hierarchical Graph Transformer ( 69 ) addresses scalability to million-node graphs through graph hierarchies and coarsening techniques. SpecFormer ( 3 ) and PolyFormer ( 12 ) are recent spectral Transformers that improve over SAN by leveraging approximate spectral bases or low-rank polynomial Laplacian filters to enhance scalability and accuracy on graph tasks. Several recent architectures aim to bridge the gap between local