Heterogeneous Connectivity in Sparse Networks: Fan-in Profiles, Gradient Hierarchy, and Topological Equilibria
Nikodem Tomczak · 2026
Read Heterogeneous Connectivity in Sparse Networks: Fan-in Profiles, Gradient Hierarchy, and Topological Equilibria by Nikodem Tomczak (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-10560.txt.
Introduction
Tomczak tests varied fan in wiring in sparse neural nets. Profiled sparse nets give each neuron its own dense or sparse input mix by fixed curves. At 90 percent sparsity, every wiring shape lands within about half a percent of dense scores. Hub neurons still hog gradients at 2 to 5 times the flat rate, tied to wiring spread. Smart hub starts for RigL training beat standard starts, so hub choice matters more than wiring variance.
Catalog record
| CREATOR | Nikodem Tomczak |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 61174 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.10560
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Heterogeneous Connectivity in Sparse Networks: Fan-in Profiles, Gradient Hierarchy, and Topological Equilibria Nikodem Tomczak Abstract Profiled Sparse Networks (PSN) replace uniform connectivity with deterministic, heterogeneous fan-in profiles defined by continuous, nonlinear functions, creating neurons with both dense and sparse receptive fields. We benchmark PSN across four classification datasets spanning vision and tabular domains, input dimensions from 54 to 784, and network depths of 2–3 hidden layers. At 90% sparsity, all static profiles, including the uniform random baseline, achieve accuracy within 0.2–0.6% of dense baselines on every dataset, demonstrating that heterogeneous connectivity provides no accuracy advantage when hub placement is arbitrary rather than task-aligned. This result holds across sparsity levels (80–99.9%), profile shapes (eight parametric families, lognormal, and power-law), and fan-in coefficients of variation from 0 to 2.5. Internal gradient analysis reveals that structured profiles create a 2–5 × \times gradient concentration at hub neurons compared to the ∼ 1 × {\sim}1\times uniform distribution in random baselines, with the hierarchy strength predicted by fan-in coefficient of variation ( r = 0.93 r=0.93 ). When PSN fan-in distributions are used to initialise RigL dynamic sparse training, lognormal profiles matched to the equilibrium fan-in distribution consistently outperform standard ERK initialisation, with advantages growing on harder tasks, achieving +0.16% on Fashion-MNIST ( p = 0.036 p=0.036 , d = 1.07 d=1.07 ), +0.43% on EMNIST, and +0.49% on Forest Cover. RigL converges to a characteristic fan-in distribution regardless of initialisation. Starting at this equilibrium allows the optimiser to refine weights rather than rearrange topology. Which neurons become hubs matters more than the degree of connectivity variance, i.e., random hub placement provides no advantage, while optimisation-driven placement does. Key Contributions. 1. Deterministic nonlinear fan-in profiles for structured heterogeneous sparsity, enabling continuous parameterisation of connectivity distributions as an architectural variable. 2. Practical derivation and empirical validation of mean fan-in initialisation for heterogeneous sparse networks, with immediate relevance to any method that produces non-uniform connectivity. 3. An experimental framework that decouples capacity distribution (which neurons receive how many connections) from input coverage (which specific inputs connect to each neuron). 4. Across four datasets, sparsity levels from 80% to 99.9%, and fan-in CVs from 0 to 2.5, static connectivity structure does not significantly affect accuracy at matched parameter counts. 5. Fan-in CV determines gradient heterogeneity as a structural consequence of mask geometry ( r=0.93 ), independently of profile shape or task performance. 6. Lognormal initialisation matched to the RigL equilibrium fan-in distribution consistently outperforms ERK, with advantages increasing on harder tasks. 1 Introduction Deep neural networks achieve remarkable performance across diverse domains, yet the vast majority of parameters contribute minimally to final predictions. ( 11 ; 15 ) Sparse neural networks address this inefficiency by eliminating redundant connections while maintaining competitive accuracy. ( 18 ; 12 ) Early approaches to sparsity remove individual weights based on magnitude or saliency, treating each connection independently without considering the connectivity structure of the neurons they connect. ( 23 ; 15 ) This treatment ignores organizational principles observed in complex network systems, where connectivity patterns exhibit structured heterogeneity with hub nodes maintaining dense connectivity while peripheral nodes maintain sparse, selective connections. ( 2 ; 4 ) Recent empirical evidence from dynamic sparse training methods provides support for the importance of heterogeneous connectivity. At high sparsity levels gradient-based rewiring evolves toward connectivity patterns with high variance in per-neuron fan-in, despite starting from uniform random initialization. ( 21 ) These findings suggest that networks seek heterogeneous connectivity organization when given the flexibility to evolve their topology, yet standard sparse training methods ignore this preference by imposing uniform constraints from initialization. Dynamic topology search is computationally expensive, requiring gradient computation for potential connections, periodic topology updates involving global weight ranking and selection, and storage of metadata tracking connection presence across updates. If dynamic sparse training methods naturally evolve toward heterogeneous connectivity patterns, can we accelerate convergence and improve efficiency by designing beneficial structure from initialization? If the target structure is predictable, as suggested by the consistency of emergent patterns across different architectures and sparsity levels, then starting with appropriate structure might bypass costly search while preserving its benefits. We introduce Profiled Sparse Networks (PSN), an architecture that designs heterogeneous connectivity from initialization rather than waiting for it to emerge through gradient-based rewiring. PSN assigns per-neuron fan-in according to continuous nonlinear profile functions that map neuron index to connectivity density, creating deterministic spatial arrangements of hub and specialist neurons within individual layers. By parameterizing connectivity distributions through interpretable mathematical functions, PSN enables exploration of the connectivity design space as controlled experimental variables. This approach contrasts with both uniform random sparsity, which treats all neurons identically, and dynamic sparse training, which discovers structure through topology search. The research question we address is whether structured heterogeneous connectivity provides superior inductive bias compared to uniform random sparsity when parameter counts are held constant. Recent concurrent work has begun exploring designed heterogeneous topologies from complementary angles. The Dendritic Network Model (DNM) ( 5 ) generates sparse initializations through parametric distributions of dendrites, receptive fields, and degree, enabling control over modularity and degree heterogeneity. Brain-inspired topological models from the same group ( 34 ) demonstrate that scale-free and small-world initializations can outperform random baselines at extreme sparsity, with extensions to transformers and large language models. We evaluate PSN on four classification benchmarks including MNIST ( 22 ) , Fashion-MNIST ( 33 ) , EMNIST-Balanced ( 7 ) , and Forest Cover Type ( 3 ) , spanning input dimensions from 54 to 784 and network depths of 2–3 hidden layers. We derive the initialisation requirements for heterogeneous sparse networks from forward and backward variance analysis and validate them empirically. A multi-peak interpolation experiment confirms that fan-in CV alone predicts gradient concentration ( r=0.93 ), independent of profile shape. When lognormal profiles are used to initialise RigL, matching the equilibrium fan-in distribution outperforms ERK with advantages growing on harder tasks. Static connectivity structure does not affect accuracy at matched sparsity, but understanding the structural endpoints of dynamic sparse training provides a principled basis for initialisation design. 2 Methods and the PSN Framework 2.1 Datasets and Network Architectures We evaluate PSN across four datasets spanning vision and tabular domains (Table 1 ). All image datasets use flattened 28\times 28 grayscale inputs (dimension 784). Forest Cover uses 54 tabular features. Table 1: Datasets used in this study. Dataset Train Test Input dim Classes MNIST ( 22 ) 60,000 10,000 784 10 Fashion-MNIST ( 33 ) 60,000 10,000 784 10 EMNIST-Balanced ( 7 ) 112,800 18,800 784 47 Forest Cover ( 3 ) 464,809 116,203 54 7 All experiments use multi-layer perceptrons (MLPs) with LayerNorm and ReLU activations. The three 784-input datasets share a common 784 \to 1024 \to 1024 hidden architecture with 10 or 47 output neurons. Forest Cover uses a deeper 54 \to 1024 \to 1024 \to 1024 \to 7 architecture (three hidden layers), selected by grid search over widths and depths . PSN sparse masks are applied to all hidden layers; the output layer is kept dense. Output layers are small relative to hidden layers and contribute negligibly to total parameter count. Sparsifying them provides little compression benefit while degrading class separability. Table 2 summarises the training configurations. Forest Cover architecture selection.