Resonant Sparse Geometry Networks
Hasi Hays · 2026
Read Resonant Sparse Geometry Networks by Hasi Hays (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-18064.txt.
Introduction
Hasi Hays introduces a brain-inspired network design built on sparse, shifting links. His 2026 preprint presents Resonant Sparse Geometry Networks, or RSGN. Transformer models use dense attention, whose cost grows with the square of the input size. RSGN instead places its nodes in a learned hyperbolic space. Link strength falls off with distance in that space, and the wiring organizes itself. Only nearby nodes talk much, so the network stays sparse.
Input-dependent wiring lets the network reshape itself for each input. Hyperbolic space suits tree-like data, which is common in language and biology. If the design scales, it could cut the cost of attention sharply. The paper offers a fresh route past a known bottleneck.
Catalog record
| CREATOR | Hasi Hays |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 61 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2601-18064
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Resonant Sparse Geometry Networks Hasi Hays Abstract We introduce Resonant Sparse Geometry Networks (RSGN), a brain-inspired architecture with self-organizing sparse hierarchical input-dependent connectivity. Unlike Transformer architectures that employ dense attention mechanisms with O ( n 2 ) O(n^{2}) computational complexity, RSGN embeds computational nodes in learned hyperbolic space where connection strength decays with geodesic distance, achieving dynamic sparsity that adapts to each input. The architecture operates on two distinct timescales: fast differentiable activation propagation optimized through gradient descent, and slow Hebbian-inspired structural learning for connectivity adaptation through local correlation rules. We provide rigorous mathematical analysis demonstrating that RSGN achieves O ( n ⋅ k ) O(n\cdot k) computational complexity where k ≪ n k\ll n represents the average active neighborhood size. Experimental evaluation on hierarchical classification and long-range dependency tasks demonstrates that RSGN achieves 96.5% accuracy on long-range dependency tasks while using approximately 15 × \times fewer parameters than standard Transformers. On challenging hierarchical classification with 20 classes, RSGN achieves 23.8% accuracy (compared to 5% random baseline) with only 41,672 parameters, nearly 10 × \times fewer than Transformer baselines requiring 403,348 parameters to achieve 30.1% accuracy. Our ablation studies confirm the contribution of each architectural component, with Hebbian learning providing consistent improvements. These results suggest that brain-inspired principles of sparse, geometrically-organized computation offer a promising direction toward more efficient and biologically plausible neural architectures. I Introduction The Transformer architecture Vaswani et al. 2017 has emerged as the dominant paradigm in modern deep learning, powering breakthrough systems in natural language processing Brown et al. 2020 ; OpenAI 2023 , computer vision Dosovitskiy et al. 2020 , and multimodal artificial intelligence Alayrac et al. 2022 . The core innovation of Transformers lies in the self-attention mechanism Hays 2026a ; Hays 2026b , which allows every token to attend to every other token in a sequence, thereby capturing long-range dependencies without the sequential processing constraints of recurrent architectures. This architectural flexibility has enabled unprecedented scaling, with models now reaching hundreds of billions of parameters and demonstrating emergent capabilities that appear qualitatively different from smaller systems Wei et al. 2022 . However, the computational flexibility of self-attention incurs a significant cost: quadratic complexity O(n^{2}) with respect to sequence length n . For a sequence of 1,000 tokens, attention computation requires evaluating one million pairwise relationships. For sequences of 100,000 tokens, increasingly common in document-level understanding and long-context applications, this becomes 10 billion operations per attention layer, rendering standard Transformers computationally prohibitive Tay et al. 2022 . This scaling limitation has motivated extensive research into efficient attention variants, yet fundamental questions remain about whether dense, global attention is the optimal computational paradigm for sequence processing. In stark contrast, the human brain achieves remarkable computational efficiency through fundamentally different organizational principles. Operating on approximately 20 watts of power, comparable to a dim light bulb, the brain processes complex sensory information across multiple modalities, maintains episodic and semantic memories spanning decades, generates creative thought, plans complex action sequences, and coordinates motor actions with millisecond precision. This extraordinary capability emerges from approximately 86 billion neurons connected by roughly 100 trillion synapses Azevedo et al. 2009 , yet the computational principles underlying this efficiency remain only partially understood. Several key organizational principles distinguish biological neural computation from contemporary artificial systems. First, the brain exhibits extreme sparsity in activation : at any given moment, only approximately 1-2% of cortical neurons actively fire Olshausen and Field 1996 ; Barth and Bhalla 2012 . This sparse coding principle dramatically reduces energy consumption while providing representational advantages including noise robustness, memory capacity, and compositional generalization Földiák 2003 . Second, biological neural processing employs input-dependent routing : different inputs activate fundamentally different neural pathways rather than engaging the same fixed computational graph Dehaene 2014 ; Tononi et al. 2016 . Visual, auditory, and somatosensory information flow through distinct cortical hierarchies, with cross-modal integration occurring at specific convergence zones. Third, neural connectivity emerges through self-organizing structure via Hebbian learning principles (“neurons that fire together wire together”) Hebb 1949 and activity-dependent synaptic pruning during development and throughout life Huttenlocher 1979 ; Petanjek et al. 2011 . Fourth, information flows through hierarchical organization embedded in the physical geometry of cortical tissue, with systematic transformations as signals propagate from primary sensory areas through association cortices Felleman and Van Essen 1991 ; Harris and Mrsic-Flogel 2013 . These observations motivate our development of Resonant Sparse Geometry Networks (RSGN) , an architecture that incorporates all four biological principles through a novel combination of computational mechanisms ( Figure 1 ). We embed N computational nodes in learned hyperbolic space \mathbb{H}^{d} , which naturally encodes hierarchical relationships through its exponentially expanding geometry Sarkar 2011 ; Nickel and Kiela 2017 . Unlike Euclidean space where volume grows polynomially with radius, hyperbolic space exhibits exponential volume growth, allowing tree-like hierarchical structures to be embedded with arbitrarily low distortion Sala et al. 2018 . Connection strength between nodes decays with geodesic distance in this space, enforcing locality and sparsity without explicit pruning mechanisms. We implement input-dependent ignition where input tokens create “spark points” in the embedding space, activating only nearby nodes and establishing sparse initial activation patterns. These activations then propagate through the network via iterative dynamics with soft thresholds and local inhibition, implementing a winner-take-more competition that mirrors lateral inhibition in biological neural circuits Isaacson and Scanziani 2011 ; Carandini and Heeger 2012 . The resonance metaphor reflects that stable activation patterns emerge through iterative settling, analogous to the global workspace theory of consciousness where coherent representations arise from competitive dynamics among specialized processors Baars 1988 ; Dehaene and Changeux 2011 . Crucially, RSGN operates on two distinct timescales inspired by the separation between fast neural dynamics and slow synaptic plasticity in biological systems Friston 2003 ; Lillicrap et al. 2020 . Fast learning employs standard gradient descent through differentiable relaxations of threshold operations, optimizing for task performance on the timescale of individual forward passes. Slow learning uses local Hebbian rules where co-activated nodes strengthen their connections and drift toward each other in the embedding space, while unused connections decay and eventually prune. A global reward signal modulates plasticity strength, analogous to dopaminergic modulation of synaptic plasticity in the basal ganglia and cortex Schultz et al. 1997 ; Reynolds et al. 2001 . Figure 1 : Bio-inspired principles underlying RSGN. (A) Resonant Sparse Geometry Network (RSGN) architecture illustrating the four key principles: (1) input-dependent routing where input tokens create spark points that ignite nearby nodes; (2) hierarchical sparse connectivity with distance-based connection strength; (3) two-timescale learning combining fast gradient-based activation updates with slow Hebbian structural plasticity; and (4) hyperbolic geometry embedding with soft thresholds and local inhibition. The brain illustration shows analogous biological mechanisms including attentional networks and striatal reward modulation. (B) Bio-inspired Modular Representation comparing cortical organization (left) with RSGN implementation (right). The cortical hierarchy shows sensory input propagating from primary sensory cortex through association areas to higher cognitive regions, with Hebbian plasticity (“neurons that fire together, wire together”). The corresponding RSGN diagram shows resonant signal propagation through the Poincaré ball across iterative steps, with the ignition module processing input sequences and producing classifier output through local inhibition and soft threshold operations. Our primary contributions are fourfold: 1. Mathematical Model: We